FX30S 3G stuck searching for 3G network

andris.pavenis · February 13, 2021, 3:06pm

We are using Sierra FX30S 3G for weather station data transmission. Mostly 3G network is used.

Following problem has been observed several times:

mobile network registration is lost (no additional information about reasons, too long time before somebody gets to place to get log information)
attempt to search for mobile network after that lasts forever (days and more)
when somebody gets to place and connects through USB:
– response to cm radio is:
Power: ON
Current Network Operator:
Current RAT: UMTS network (LE_MRC_RAT_UMTS)
Status: Not registered but currently searching for a new operator (LE_MRC_REG_SEARCHING)
Signal: Very strong signal strength (5)
PS: Packet Switched Not registered (LE_MRC_REG_NONE)

– attempt to turn radio of and after that on does not seem to help
cm radio off
cm radio on
– soft reboot (by entering reboot command restores normal functionality

System:

custom FW17 build (similar problems are observed also with FW14 (original binary from Sierra Wireless) but I have not have a possibility to study them): small changes like several not normally included packages added (esim. sftp-server that was discussed here earlier separately)
custom Legato 18.06.5 build (some parts unnecessary for our use removed, some our applications added, for example to keep data connection up)
data sending is done by legacy program wrapped into Legato application (to avoid locking it to specific system type)

Related information from log (unrelated parts removed from logread output) when commands cm radio off and cm radio on were entered:

Feb  7 14:53:03 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb  7 14:54:23 fx30s user.info Legato:  INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb  7 14:54:33 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb  7 14:54:33 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb  7 14:54:33 fx30s user.err Legato: =ERR= | dcsDaemon[793]/dcsDaemon T=main | dcsServer.c PacketSwitchHandler() 1881 | New PS state unknown PS state 5
Feb  7 14:54:54 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb  7 14:54:54 fx30s user.info Legato:  INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.5 and ss.-125
Feb  7 14:54:54 fx30s user.info Legato:  INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.2 and ss.-40
Feb  7 14:54:54 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb  7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb  7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb  7 14:55:48 fx30s user.info Legato:  INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb  7 14:55:49 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed

indent preformatted text by 4 spaces

In last time I installed updated our Legato build where I included (in our added application) monitoring network registration state and:

trying to turn radio off and on 5 seconds later (timer callback) if there is no registration to home network for an hour
setting device in ULPM with wakeup 10 seconds later from timer

I did not seem to help (looks like stuck the same way again) but I did not have possibility to verify added functionality in development environment as I never experienced the same problem there.

There are more than 20 installations (part still with FW14 and part with my FW17 build) and most of them are working OK. Sevaral have however experienced described problem occasionally and currently on already 4 times in 2 week time (really one week if excluding offline time).

PS: code for automatic recovery attempt was broken but I cannot update it on device without somebody visiting weather station. This does not however invalidate observations done earlier when colleague visited place and checked state using USB connection.

jyijyi · February 14, 2021, 2:02am

you can try other WP8548 FW and see if still seeing this problem.

https://source.sierrawireless.com/resources/airprime/software/wpx5xx/wpx5xx-firmware-latest-release/#sthash.2th4isfM.dpbs

Also you can shut down legato by “legato stop” and see problem is related to legato framework:

https://docs.legato.io/latest/toolsTarget_legato.html

andris.pavenis · February 14, 2021, 8:07am

About legati stop/start: I can ask to try that when one (not me) visits weather station as one needs USB connection for this

About different firmware variants - installed generic version should work OK in Europe as far as I understand (operator Telia in Finland)

Here is some additional information about firmware from FX30S 3G on different weather station (the same my firmware build of R17.0.1.007 with some additions):

[105392 Sipoo Itatoukki] ~# fwupdate query
Connecting to service …
Firmware Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:52
Bootloader Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:39
Linux Version: 3.14.29ltsi-yocto-standard #2 PREEMPT Mon Sep 28 08:55:14 UTC 2020
FX30 Version: R17.0.1.007_FMI.20200928

This device also has similar problem once recently (from time interval since 11.2019 when it taken into use) but no additional diagnostic information is available

Information from installed firmware file:

andris.pavenis@ap-ubuntu18:~/swi-linux-src/build_bin/images/public$ swicwe -P     FX30_WP8548_full_R17.0.1.007_FMI.20200928-generic.cwe
    Type    Size(Exclude Header)    Product         Compress        Version
    *SPKG    50241947                Y912            0               9999999_9904559_SWI9X15Y_07.13.07.00_00_GENERIC_001.040_000
    *BOOT    647352                  Y912            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     SBL1    118524                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     QRPM    104672                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     SBL2    422956                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:39
    *MODM    23991251                Y912            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     DSP1    1861297                 9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     DSP2    19767922                9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:52
     DSP3    2360832                 9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
    *FILE    43645                   9X15            0               9999999_9904559_SWI9X15Y_07.13.07.00_00_GENERIC_001.040_000
    *FILE    15373                   Y912            0               9999999_9999999_MCU_002.004_00_wpx5_00_00
    *APPL    25542326                Y912            0               Mon Sep 28 08:57:07 UTC 2020
     SYST    19150080                9X15            1               Mon Sep 28 08:57:07 UTC 2020
     USER    2108490                 9X15            1               18.06.5-fmi-3.0e4239ee_bf910b46164d21a5560d0ac57548a40f 2020/09/28 08:54:43
     APBL    71812                   9X15            0               1.3.0_b12d0630d0 2020/09/28 08:46:51
     APPS    4210344                 9X15            1               3.14.29ltsi-yocto-standard 2020/09/28 08:55:33`

Our changes for firmware build:

project meta-swi/
diff --git a/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc b/meta-swi-mdm9x1/recipes-core/images/mdm9x15-image.inc
index 2fae2e5..e8f6cd3 100644
--- a/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc
+++ b/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc 
@@ -23,6 +23,14 @@ IMAGE_INSTALL_append = " crda"
IMAGE_INSTALL_append = " i2cgpioctl"
IMAGE_INSTALL_append = " sierra-init-tiwifi"

+IMAGE_INSTALL_append = " socat"
+IMAGE_INSTALL_append = " minicom"
+IMAGE_INSTALL_append = " lrzsz"
+IMAGE_INSTALL_append = " dialog"
+IMAGE_INSTALL_append = " zile"
+IMAGE_INSTALL_append = " nano"
+IMAGE_INSTALL_append = " openssh-sftp-server"
+
# Install kernel modules
IMAGE_INSTALL_append = " kernel-modules"

jyijyi · February 14, 2021, 8:50am

Suggest you trying official fw first and see if there is problem.
If no problem is found, step by step adding back your changes

andris.pavenis · February 14, 2021, 3:48pm

Collegue visited weather station and tried:

legato restart - no improvement
cm radio rat GSM - (normally we use 3G unless there is reason to fall back to 2G) - no improvement

About trying official FW and official Legato version:

cannot currently do it remotely as it would cause connection to be lost and somebody would have to do additional installation of programs at site.
installing my build of FW can be done remotely and connection should return. Even in this case we avoid doing it in weather stations providing operative weather measurement data (some would have to visit site if something goes wrong)

Thinking about having additional different Arm system that could be connected to FX30S using USB and accessed through mobile network. That would allow to debug things remotely. That site would be good place for such testing as the problem occurred already 4 times in last 2 weeks.

Currently fixed monitoring problem in my build, so FX30S should now try to put itself in ULPM for short time (timer) if no home network found in 2 hours

Additionally it was left to use 2G connection for now

cchenry · February 14, 2021, 4:36pm

Hi @andris.pavenis,

Is it possible that the devices that are failing are out of the operational temperature spec?
Have you tried both “legato restart” and “cm radio off and on”?
Did you try the above on an R14 or R17 unit?
Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
If possible, could you try suppling the AT command output for AT!GSTATUS?
If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
Are your SIMs roaming?

BR,
Chris

andris.pavenis · February 15, 2021, 5:26am

Responses to part of question:

Is it possible that the devices that are failing are out of the operational temperature spec?
NO at least for device which have the problem 4 times in last weeks. Measurements are not lost if there is no connection but sent when possible. As result I see that lowest air temperature has been in this time -25 C. I do not know whether is additional heading for this station but fore ones in Lapland there is
Have you tried both “legato restart” and “cm radio off and on
Separately yes but not at the same recovery attempt
Did you try the above on an R14 or R17 unit?
R17. There are R14 units in use installed at first. It’s planned to upgrade them (really replace with different unit and upgrade replaced one later)
Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
Only once on unit different from one having problem we are discussing (First time I tried remote upgrade from FW14 to FW17 it failed and I had to recover using swiflash. Reflashing R17 did not work, but I succeeded to flash R14 and after that upgrade to R17 using swiflash. That unit is used in testing weather station and works properly)
If possible, could you try supplying the AT command output for AT!GSTATUS?
If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
I have tried to enter AT command like AT!GSTATUS? from minicom present in my build. It has not tried in recovery attempts in last weeks. I should perhaps either add supplying them from monitoring Application or write some additional application for collecting related information
Are your SIMs roaming?
I do not know.

PS. SIMS are expected to work in home network and nomaly do. I meant that I do not know whether they supports roaming

I’m generally not doing installation myself but only provide instructions and required files.