Hi there, I am seeking some advice on any configuration settings (watchdogs?) or best practices when upgrading firmware on RV50’s, 50X’s, and 55’s through ALMS. Just about every time we push updates we end up loosing a few modems which then have to be physically visited to force reboot/recover and finish FW installation. Sometimes these locations are very remote with only helicopter access which is a HUGE cost just to go reboot a modem.
Recently we update firmware on a little over 350 of these modems and out of those, four of them never completed or came back online. In the last few days we were surprised to see two of these units suddenly come back online with FW 4.16.0. I think this is a first, at least that we have documented, and we are curious as to why and how these recovered themselves after being bricked for roughly 17 days. These units have a cellular watchdog enabled, they are not connected to the same cell tower, but they are both on AT&T in Alaska. Our only guess is that the AT&T network may have went down for a moment and the watchdog rebooted? We have about 500 more to update, but hoping to better prevent losing these units in the process. I have included a breif snippet of the log from these two units that came back online on their own 17 days later.
Dec 5 23:17:58 alert ALEOS_EVENTS_gnssmgr: GPS disabled, Assisted Data Retrieval always off
Dec 5 23:17:58 notice ALEOS_EVENTS_gnssmgr: Waiting for radio reservation to complete…
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Updating ODIS information on Radio
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Customer has selected ALL BANDS
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Band preferences need to be updated - Disconnect may follow
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current band setting: ALL BANDS
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Releasing radio access (PID: 1268)…
Dec 5 23:17:58 notice ALEOS_EVENTS_gnssmgr: Done waiting for radio reservation
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CARRIER_SELECTION
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current COPS user setting: 0
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Current COPS mode: 0
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Leaving COPS mode as default: automatic
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_NOTREADY_WAIT
Dec 5 23:17:58 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CONNECT
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Current time: Mon, 05 Dec 2022 23:17:58 UTC Radio Network time used to set both system time and HW clock system time to: Fri, 23 Dec 2022 22:05:45 UTC
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Seconds since boot: 44
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: IPPreference : IPv4
Dec 23 22:05:45 debug SWI0 SDK Process: psrwds_setipfamily: instance: 0 IP family: V4 Active WDS Client: V4
Dec 23 22:05:45 debug SWI0 SDK Process: psrwds_starnet:
Dec 23 22:05:45 debug ALEOS_CELL_RadioTask: iSLQSStartStopDataSession/2975: WDS start interface verbose error: 2001
Dec 23 22:05:45 notice ALEOS_CELL_RadioTask: Service is not available
Dec 23 22:05:45 notice ALEOS_CELL_detectrmdetach: 9McEmRmHal: radio module = ALEOS_RADIO_MODULE_MC7455 is active
Dec 23 22:05:45 warning ALEOS_CELL_detectrmdetach: Radio Detach Detected [1]
Dec 23 22:05:45 notice ALEOS_CELL_detectrmdetach: SIM status [ READY OK ]
Dec 6 00:58:42 notice ALEOS_CELL_RadioTask: Updating ODIS information on Radio
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Customer has selected ALL BANDS
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Band preferences need to be updated - Disconnect may follow
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current band setting: ALL BANDS
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Releasing radio access (PID: 1266)…
Dec 6 00:58:43 notice ALEOS_EVENTS_gnssmgr: Done waiting for radio reservation
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CARRIER_SELECTION
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current COPS user setting: 0
Dec 6 00:58:43 alert ALEOS_EVENTS_gnssmgr: GPS disabled, Assisted Data Retrieval always off
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Current COPS mode: 0
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Leaving COPS mode as default: automatic
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_NOTREADY_WAIT
Dec 6 00:58:43 notice ALEOS_CELL_RadioTask: Radio_state → RADIO_STATE_CONNECT
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Current time: Tue, 06 Dec 2022 00:58:43 UTC Radio Network time used to set both system time and HW clock system time to: Fri, 23 Dec 2022 20:44:13 UTC
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Seconds since boot: 43
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: IPPreference : IPv4
Dec 23 20:44:13 debug SWI0 SDK Process: psrwds_setipfamily: instance: 0 IP family: V4 Active WDS Client: V4
Dec 23 20:44:13 debug SWI0 SDK Process: psrwds_starnet:
Dec 23 20:44:13 debug ALEOS_CELL_RadioTask: iSLQSStartStopDataSession/2975: WDS start interface verbose error: 2001
Dec 23 20:44:13 notice ALEOS_CELL_RadioTask: Service is not available
Dec 23 20:44:13 notice ALEOS_CELL_detectrmdetach: 9McEmRmHal: radio module = ALEOS_RADIO_MODULE_MC7455 is active
Dec 23 20:44:13 warning ALEOS_CELL_detectrmdetach: Radio Detach Detected [1]
Dec 23 20:44:13 notice ALEOS_CELL_detectrmdetach: SIM status [ READY OK ]