FX30S 3G stuck searching for 3G network

IoT Gateways FX30
#1

We are using Sierra FX30S 3G for weather station data transmission. Mostly 3G network is used.

Following problem has been observed several times:

  • mobile network registration is lost (no additional information about reasons, too long time before somebody gets to place to get log information)
  • attempt to search for mobile network after that lasts forever (days and more)
  • when somebody gets to place and connects through USB:
    – response to cm radio is:
    Power: ON
    Current Network Operator:
    Current RAT: UMTS network (LE_MRC_RAT_UMTS)
    Status: Not registered but currently searching for a new operator (LE_MRC_REG_SEARCHING)
    Signal: Very strong signal strength (5)
    PS: Packet Switched Not registered (LE_MRC_REG_NONE)

– attempt to turn radio of and after that on does not seem to help
cm radio off
cm radio on
– soft reboot (by entering reboot command restores normal functionality

System:

  • custom FW17 build (similar problems are observed also with FW14 (original binary from Sierra Wireless) but I have not have a possibility to study them): small changes like several not normally included packages added (esim. sftp-server that was discussed here earlier separately)
  • custom Legato 18.06.5 build (some parts unnecessary for our use removed, some our applications added, for example to keep data connection up)
  • data sending is done by legacy program wrapped into Legato application (to avoid locking it to specific system type)

Related information from log (unrelated parts removed from logread output) when commands cm radio off and cm radio on were entered:

Feb  7 14:53:03 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb  7 14:54:23 fx30s user.info Legato:  INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb  7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb  7 14:54:33 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb  7 14:54:33 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb  7 14:54:33 fx30s user.err Legato: =ERR= | dcsDaemon[793]/dcsDaemon T=main | dcsServer.c PacketSwitchHandler() 1881 | New PS state unknown PS state 5
Feb  7 14:54:54 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb  7 14:54:54 fx30s user.info Legato:  INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.5 and ss.-125
Feb  7 14:54:54 fx30s user.info Legato:  INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.2 and ss.-40
Feb  7 14:54:54 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb  7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb  7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb  7 14:55:48 fx30s user.info Legato:  INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb  7 14:55:49 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed

indent preformatted text by 4 spaces

In last time I installed updated our Legato build where I included (in our added application) monitoring network registration state and:

  • trying to turn radio off and on 5 seconds later (timer callback) if there is no registration to home network for an hour
  • setting device in ULPM with wakeup 10 seconds later from timer

I did not seem to help (looks like stuck the same way again) but I did not have possibility to verify added functionality in development environment as I never experienced the same problem there.

There are more than 20 installations (part still with FW14 and part with my FW17 build) and most of them are working OK. Sevaral have however experienced described problem occasionally and currently on already 4 times in 2 week time (really one week if excluding offline time).

PS: code for automatic recovery attempt was broken but I cannot update it on device without somebody visiting weather station. This does not however invalidate observations done earlier when colleague visited place and checked state using USB connection.

#2

you can try other WP8548 FW and see if still seeing this problem.

https://source.sierrawireless.com/resources/airprime/software/wpx5xx/wpx5xx-firmware-latest-release/#sthash.2th4isfM.dpbs

Also you can shut down legato by “legato stop” and see problem is related to legato framework:

https://docs.legato.io/latest/toolsTarget_legato.html

#3

About legati stop/start: I can ask to try that when one (not me) visits weather station as one needs USB connection for this

About different firmware variants - installed generic version should work OK in Europe as far as I understand (operator Telia in Finland)

Here is some additional information about firmware from FX30S 3G on different weather station (the same my firmware build of R17.0.1.007 with some additions):

[105392 Sipoo Itatoukki] ~# fwupdate query
Connecting to service …
Firmware Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:52
Bootloader Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:39
Linux Version: 3.14.29ltsi-yocto-standard #2 PREEMPT Mon Sep 28 08:55:14 UTC 2020
FX30 Version: R17.0.1.007_FMI.20200928

This device also has similar problem once recently (from time interval since 11.2019 when it taken into use) but no additional diagnostic information is available

Information from installed firmware file:

andris.pavenis@ap-ubuntu18:~/swi-linux-src/build_bin/images/public$ swicwe -P     FX30_WP8548_full_R17.0.1.007_FMI.20200928-generic.cwe
    Type    Size(Exclude Header)    Product         Compress        Version
    *SPKG    50241947                Y912            0               9999999_9904559_SWI9X15Y_07.13.07.00_00_GENERIC_001.040_000
    *BOOT    647352                  Y912            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     SBL1    118524                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     QRPM    104672                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     SBL2    422956                  9X15            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:39
    *MODM    23991251                Y912            0               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     DSP1    1861297                 9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
     DSP2    19767922                9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:52
     DSP3    2360832                 9X15            1               SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 21:36:02
    *FILE    43645                   9X15            0               9999999_9904559_SWI9X15Y_07.13.07.00_00_GENERIC_001.040_000
    *FILE    15373                   Y912            0               9999999_9999999_MCU_002.004_00_wpx5_00_00
    *APPL    25542326                Y912            0               Mon Sep 28 08:57:07 UTC 2020
     SYST    19150080                9X15            1               Mon Sep 28 08:57:07 UTC 2020
     USER    2108490                 9X15            1               18.06.5-fmi-3.0e4239ee_bf910b46164d21a5560d0ac57548a40f 2020/09/28 08:54:43
     APBL    71812                   9X15            0               1.3.0_b12d0630d0 2020/09/28 08:46:51
     APPS    4210344                 9X15            1               3.14.29ltsi-yocto-standard 2020/09/28 08:55:33`

Our changes for firmware build:

project meta-swi/
diff --git a/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc b/meta-swi-mdm9x1/recipes-core/images/mdm9x15-image.inc
index 2fae2e5..e8f6cd3 100644
--- a/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc
+++ b/meta-swi-mdm9x15/recipes-core/images/mdm9x15-image.inc 
@@ -23,6 +23,14 @@ IMAGE_INSTALL_append = " crda"
IMAGE_INSTALL_append = " i2cgpioctl"
IMAGE_INSTALL_append = " sierra-init-tiwifi"

+IMAGE_INSTALL_append = " socat"
+IMAGE_INSTALL_append = " minicom"
+IMAGE_INSTALL_append = " lrzsz"
+IMAGE_INSTALL_append = " dialog"
+IMAGE_INSTALL_append = " zile"
+IMAGE_INSTALL_append = " nano"
+IMAGE_INSTALL_append = " openssh-sftp-server"
+
# Install kernel modules
IMAGE_INSTALL_append = " kernel-modules"
#4

Suggest you trying official fw first and see if there is problem.
If no problem is found, step by step adding back your changes

#5

Collegue visited weather station and tried:

  1. legato restart - no improvement
  2. cm radio rat GSM - (normally we use 3G unless there is reason to fall back to 2G) - no improvement

About trying official FW and official Legato version:

  • cannot currently do it remotely as it would cause connection to be lost and somebody would have to do additional installation of programs at site.
  • installing my build of FW can be done remotely and connection should return. Even in this case we avoid doing it in weather stations providing operative weather measurement data (some would have to visit site if something goes wrong)

Thinking about having additional different Arm system that could be connected to FX30S using USB and accessed through mobile network. That would allow to debug things remotely. That site would be good place for such testing as the problem occurred already 4 times in last 2 weeks.

Currently fixed monitoring problem in my build, so FX30S should now try to put itself in ULPM for short time (timer) if no home network found in 2 hours

Additionally it was left to use 2G connection for now

#6

Hi @andris.pavenis,

  1. Is it possible that the devices that are failing are out of the operational temperature spec?
  2. Have you tried both “legato restart” and “cm radio off and on”?
  3. Did you try the above on an R14 or R17 unit?
  4. Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
  5. If possible, could you try suppling the AT command output for AT!GSTATUS?
    If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
  6. Are your SIMs roaming?

BR,
Chris

#7

Responses to part of question:

  1. Is it possible that the devices that are failing are out of the operational temperature spec?
    NO at least for device which have the problem 4 times in last weeks. Measurements are not lost if there is no connection but sent when possible. As result I see that lowest air temperature has been in this time -25 C. I do not know whether is additional heading for this station but fore ones in Lapland there is
  2. Have you tried both “legato restart” and “cm radio off and on
    Separately yes but not at the same recovery attempt
  3. Did you try the above on an R14 or R17 unit?
    R17. There are R14 units in use installed at first. It’s planned to upgrade them (really replace with different unit and upgrade replaced one later)
  4. Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
    Only once on unit different from one having problem we are discussing (First time I tried remote upgrade from FW14 to FW17 it failed and I had to recover using swiflash. Reflashing R17 did not work, but I succeeded to flash R14 and after that upgrade to R17 using swiflash. That unit is used in testing weather station and works properly)
  5. If possible, could you try supplying the AT command output for AT!GSTATUS?
    If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
    I have tried to enter AT command like AT!GSTATUS? from minicom present in my build. It has not tried in recovery attempts in last weeks. I should perhaps either add supplying them from monitoring Application or write some additional application for collecting related information
  6. Are your SIMs roaming?
    I do not know.

PS. SIMS are expected to work in home network and nomaly do. I meant that I do not know whether they supports roaming

I’m generally not doing installation myself but only provide instructions and required files.

#8

Got response from AT!GSTATUS? when FX30S 3G is stuck looking for mobile network (required some to go to the weather station and connect using USB):

!GSTATUS:
Current Time:  3929             Temperature: 10
Bootup Time:   5                Mode:        ONLINE
System mode:   GSM              PS state:    Not attached
GSM band:      GSM900
GSM channel:   14
GMM (PS) state:DEREGISTERED     PLMN SEARCH
MM (CS) state: IDLE             NO IMSI
Serving Cell:  11 (GSM 900    )
RX level (dBm):-29.6250         LAC:         1014 (4116)
GPRS State:    GPRS IDLE        Cell ID:     00002A1B (10779)
IMS Reg State: UNKNOWN          IMS Mode:    Not Support
IMS Srv State: UNKNOWN SMS,UNKNOWN VoIP
OK

Also:

[100955 Salo Kärkkä] ~# cm radio
Power:                         ON
Current Network Operator:
Current RAT:                   GSM network (LE_MRC_RAT_GSM)
Status:                        Not registered but currently searching for a new operator (LE_MRC_REG_SEARCHING)
Signal:                        Very strong signal strength (5)
PS:                            Packet Switched Not registered (LE_MRC_REG_NONE)

In this case rebooting FX30S did not seem to help. Another FX30S 3G device with same SIM card got connection

#9

Hi @andris.pavenis,
I wonder if your SIM is not detected.
Could you please post the output of:
cm sim info

And

AT!ICCID?
AT+CIMI

BR,
Chris

#10

It is not so easy to do: somebody needs to weather station (not me) and to do testing. I could ask to do that when somebody goes there. Unfortunately the problem has never occurred here in Helsinki area (or even at my home where I have device for software development). Than it would be easy to test.

Similar problems has been observed with several FX30S devices. I added to software rebooting if mobile network registration does not happen for several hours. It helped in some cases to reestablish connection.

Took a look at our (Finnish institute of Meteorology) device registry:

  • one has remark that it begin to get stuck and end did not establish connection at all
  • the second one has remark that modem is broken and cannot turn radio on
    I have not seen myself the latest two myself and I have not attempted to do any their diagnostics. As result I cannot say more about those.

One thing that I could try (used network: Telia FI)

  • specify operator explicitly: AT+COPS=1,2, “24491” instead of the default AT+COPS=0
    Do You think, could it help?
#11

Hi @andris.pavenis,
I’m not certain if specifying the operator will help, it is something you can try if it’s not too difficult.

In your previous post, you mentioned that a different device with the same SIM registered successfully. Were you able to try the original device afterwards? I’m wondering if there was an issue with the SIM being seated properly. ie perhaps there is an issue with the SIM connector on some devices.

If we consider the second device is working properly and is running the same image with the same SIM. Did you mention this device never registered on the network ever?

BR,
Chris

#12

Second device with same SIM card worked for some time and stopped to send weather data after some time. I did not mention earlier it as there where other software related issues - noticed mislabeling of device, settings did not match what was mentioned in label, so also there could be some software issues as I tried different builds to try to workaround network registration problems.

Earlier device had connection. However often the connection was not reestablished when it was lost for some reasons and required manual rebooting. This behavior has noticed with several devices at different locations. Usually reboot helped but not always.

#13

Hi Andris, Chris

Did you have any success with solving these issues? We’re in new development with FX30 cat-M in Australia and New Zealand and are seeing very similar symptoms with modems losing their registration sometimes many times a day, sometimes perfectly fine. It can be solved immediately by issuing a manual AT+COPS=1,… connect but I’d rather understand the cause and avoid this.

Just looking for answers before submitting a new ticket. Any feedback appreciated.

Thanks
Brett