ALMS RV50X FW update to 4.16.0 hangs modem and then not accessible

Hi there, I am seeking some advice on any configuration settings (watchdogs?) or best practices when upgrading firmware on RV50’s, 50X’s, and 55’s through ALMS. Just about every time we push updates we end up loosing a few modems which then have to be physically visited to force reboot/recover and finish FW installation. Sometimes these locations are very remote with only helicopter access which is a HUGE cost just to go reboot a modem.

Recently we update firmware on a little over 350 of these modems and out of those, four of them never completed or came back online. In the last few days we were surprised to see two of these units suddenly come back online with FW 4.16.0. I think this is a first, at least that we have documented, and we are curious as to why and how these recovered themselves after being bricked for roughly 17 days. These units have a cellular watchdog enabled, they are not connected to the same cell tower, but they are both on AT&T in Alaska. Our only guess is that the AT&T network may have went down for a moment and the watchdog rebooted? We have about 500 more to update, but hoping to better prevent losing these units in the process. I have included a breif snippet of the log from these two units that came back online on their own 17 days later.

Dec 5 23:17:58 alert ALEOS_EVENTS_gnssmgr: GPS disabled, Assisted Data Retrieval always off
Dec 5 23:17:58 notice ALEOS_EVENTS_gnssmgr: Waiting for radio reservation to complete…
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Updating ODIS information on Radio
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Customer has selected ALL BANDS
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Band preferences need to be updated - Disconnect may follow
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current band setting: ALL BANDS
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Releasing radio access (PID: 1268)…
Dec 5 23:17:58 notice ALEOS_EVENTS_gnssmgr: Done waiting for radio reservation
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CARRIER_SELECTION
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current COPS user setting: 0
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current COPS mode: 0
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Leaving COPS mode as default: automatic
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_NOTREADY_WAIT
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CONNECT
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Current time: Mon, 05 Dec 2022 23:17:58 UTC Radio Network time used to set both system time and HW clock system time to: Fri, 23 Dec 2022 22:05:45 UTC
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Seconds since boot: 44
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: IPPreference : IPv4
Dec 23 22:05:45 debug SWI0 SDK Process: psrwds_setipfamily: instance: 0 IP family: V4 Active WDS Client: V4
Dec 23 22:05:45 debug SWI0 SDK Process: psrwds_starnet:
Dec 23 22:05:45 debug ALEOS_CELL_RadioTask: iSLQSStartStopDataSession/2975: WDS start interface verbose error: 2001
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Service is not available
Dec 23 22:05:45 notice ALEOS_CELL_detectrmdetach: 9McEmRmHal: radio module = ALEOS_RADIO_MODULE_MC7455 is active
Dec 23 22:05:45 warning ALEOS_CELL_detectrmdetach: Radio Detach Detected [1]
Dec 23 22:05:45 notice ALEOS_CELL_detectrmdetach: SIM status [ READY OK ]

Dec 6 00:58:42 notice ALEOS_CELL_RadioTask: Updating ODIS information on Radio
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Customer has selected ALL BANDS
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Band preferences need to be updated - Disconnect may follow
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current band setting: ALL BANDS
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Releasing radio access (PID: 1266)…
Dec 6 00:58:43 notice ALEOS_EVENTS_gnssmgr: Done waiting for radio reservation
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CARRIER_SELECTION
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current COPS user setting: 0
Dec 6 00:58:43 alert ALEOS_EVENTS_gnssmgr: GPS disabled, Assisted Data Retrieval always off
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current COPS mode: 0
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Leaving COPS mode as default: automatic
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_NOTREADY_WAIT
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CONNECT
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Current time: Tue, 06 Dec 2022 00:58:43 UTC Radio Network time used to set both system time and HW clock system time to: Fri, 23 Dec 2022 20:44:13 UTC
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Seconds since boot: 43
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: IPPreference : IPv4
Dec 23 20:44:13 debug SWI0 SDK Process: psrwds_setipfamily: instance: 0 IP family: V4 Active WDS Client: V4
Dec 23 20:44:13 debug SWI0 SDK Process: psrwds_starnet:
Dec 23 20:44:13 debug ALEOS_CELL_RadioTask: iSLQSStartStopDataSession/2975: WDS start interface verbose error: 2001
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Service is not available
Dec 23 20:44:13 notice ALEOS_CELL_detectrmdetach: 9McEmRmHal: radio module = ALEOS_RADIO_MODULE_MC7455 is active
Dec 23 20:44:13 warning ALEOS_CELL_detectrmdetach: Radio Detach Detected [1]
Dec 23 20:44:13 notice ALEOS_CELL_detectrmdetach: SIM status [ READY OK ]

Hi @keithw ,
Welcome you to our community!
According to your description, I understand that your RV50X lost network during firmware upgrade via ALMS, then a few days later it came back online and the upgrade completed.

So to make sure this issue doesn’t happen again, please put your RV50X in a place with strong AT&T network signal.

If you want to reboot the gateway remotely, you can refer to how to reboot Gateway via SMS in Section 8: Services Configuration on page 221 in the link below:
https://source.sierrawireless.com/resources/airlink/software_reference_docs/airlink-rv-series---configuration-guide/#sthash.UpyT44Vw.dpbs

Thanks,

Hi Jerdung,

Thank you for the welcome.

This reply is not accurate; the RV50’s have a good network connection, and never lose connection with the network. The ALMS firmware update fails mid-process which then makes the units inaccessible. Because they are then not online via IP or ALMS they cannot be rebooted as per the link you have attached.

Regards

Hi @keithw ,
There could be your issue related to AT&T 3G Sunset Impacts to AirLink LTE Routers, Please refer to the document at the link below for more details:
https://source.sierrawireless.com/resources/airlink/software_reference_docs/technical-bulletin/sierra-wireless-technical-bulletin---at-and-t-3g-sunset/#sthash.Nnqo06Qc.vmdY0YcA.dpbs

After upgrading your AirLink to ALEOS 4.16.0, Does the issue still happens in this version?

Thanks,

Hi there,

All of these modems were using 4G at the time of update.

I cannot comment on if there will be issues upgrading beyond 4.16.0, we will not know until the next update is needed; likely will not update until 4.17.x unless a security issue presents itself.

Cheers