We are using Sierra FX30S 3G for weather station data transmission. Mostly 3G network is used.
Following problem has been observed several times:
mobile network registration is lost (no additional information about reasons, too long time before somebody gets to place to get log information)
attempt to search for mobile network after that lasts forever (days and more)
when somebody gets to place and connects through USB:
– response to cm radio is:
Power: ON
Current Network Operator:
Current RAT: UMTS network (LE_MRC_RAT_UMTS)
Status: Not registered but currently searching for a new operator (LE_MRC_REG_SEARCHING)
Signal: Very strong signal strength (5)
PS: Packet Switched Not registered (LE_MRC_REG_NONE)
– attempt to turn radio of and after that on does not seem to help
cm radio off
cm radio on
– soft reboot (by entering reboot command restores normal functionality
System:
custom FW17 build (similar problems are observed also with FW14 (original binary from Sierra Wireless) but I have not have a possibility to study them): small changes like several not normally included packages added (esim. sftp-server that was discussed here earlier separately)
custom Legato 18.06.5 build (some parts unnecessary for our use removed, some our applications added, for example to keep data connection up)
data sending is done by legacy program wrapped into Legato application (to avoid locking it to specific system type)
Related information from log (unrelated parts removed from logread output) when commands cm radio off and cm radio on were entered:
Feb 7 14:53:03 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb 7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb 7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb 7 14:54:23 fx30s user.info Legato: INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb 7 14:54:23 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb 7 14:54:33 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
Feb 7 14:54:33 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb 7 14:54:33 fx30s user.err Legato: =ERR= | dcsDaemon[793]/dcsDaemon T=main | dcsServer.c PacketSwitchHandler() 1881 | New PS state unknown PS state 5
Feb 7 14:54:54 fx30s user.warn Legato: -WRN- | modemDaemon[835]/le_pa T=unknown | pa_mrc_qmi.c ConvertPSState() 737 | PS state unknown
Feb 7 14:54:54 fx30s user.info Legato: INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.5 and ss.-125
Feb 7 14:54:54 fx30s user.info Legato: INFO | modemDaemon[835]/modemDaemon T=main | le_mrc.c SignalStrengthIndHandlerFunc() 819 | Signal Strength Ind Handler called with RAT.2 and ss.-40
Feb 7 14:54:54 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704ed5c) has been closed
Feb 7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/swiQmi T=main | swiQmi.c swiQmi_CheckResponse() 796 | Sending QMI_NAS_GET_PLMN_NAME_REQ_MSG_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=37.[0x25]
Feb 7 14:55:48 fx30s user.err Legato: =ERR= | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c ConvertHomeNetworkName() 889 | Failed to convert home network name
Feb 7 14:55:48 fx30s user.info Legato: INFO | modemDaemon[835]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2138 | called
Feb 7 14:55:49 fx30s user.err Legato: =ERR= | modemDaemon[835]/modemDaemon T=main | le_sms.c CloseSessionEventHandler() 2311 | SessionRef (0xb704f084) has been closed
indent preformatted text by 4 spaces
In last time I installed updated our Legato build where I included (in our added application) monitoring network registration state and:
trying to turn radio off and on 5 seconds later (timer callback) if there is no registration to home network for an hour
setting device in ULPM with wakeup 10 seconds later from timer
I did not seem to help (looks like stuck the same way again) but I did not have possibility to verify added functionality in development environment as I never experienced the same problem there.
There are more than 20 installations (part still with FW14 and part with my FW17 build) and most of them are working OK. Sevaral have however experienced described problem occasionally and currently on already 4 times in 2 week time (really one week if excluding offline time).
PS: code for automatic recovery attempt was broken but I cannot update it on device without somebody visiting weather station. This does not however invalidate observations done earlier when colleague visited place and checked state using USB connection.
About legati stop/start: I can ask to try that when one (not me) visits weather station as one needs USB connection for this
About different firmware variants - installed generic version should work OK in Europe as far as I understand (operator Telia in Finland)
Here is some additional information about firmware from FX30S 3G on different weather station (the same my firmware build of R17.0.1.007 with some additions):
[105392 Sipoo Itatoukki] ~# fwupdate query
Connecting to service …
Firmware Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:52
Bootloader Version: SWI9X15Y_07.13.07.00 92cfd2c jenkins 2018/08/22 19:18:39
Linux Version: 3.14.29ltsi-yocto-standard #2 PREEMPT Mon Sep 28 08:55:14 UTC 2020
FX30 Version: R17.0.1.007_FMI.20200928
This device also has similar problem once recently (from time interval since 11.2019 when it taken into use) but no additional diagnostic information is available
cm radio rat GSM - (normally we use 3G unless there is reason to fall back to 2G) - no improvement
About trying official FW and official Legato version:
cannot currently do it remotely as it would cause connection to be lost and somebody would have to do additional installation of programs at site.
installing my build of FW can be done remotely and connection should return. Even in this case we avoid doing it in weather stations providing operative weather measurement data (some would have to visit site if something goes wrong)
Thinking about having additional different Arm system that could be connected to FX30S using USB and accessed through mobile network. That would allow to debug things remotely. That site would be good place for such testing as the problem occurred already 4 times in last 2 weeks.
Currently fixed monitoring problem in my build, so FX30S should now try to put itself in ULPM for short time (timer) if no home network found in 2 hours
Additionally it was left to use 2G connection for now
Is it possible that the devices that are failing are out of the operational temperature spec?
Have you tried both “legato restart” and “cm radio off and on”?
Did you try the above on an R14 or R17 unit?
Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
If possible, could you try suppling the AT command output for AT!GSTATUS?
If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
Is it possible that the devices that are failing are out of the operational temperature spec?
NO at least for device which have the problem 4 times in last weeks. Measurements are not lost if there is no connection but sent when possible. As result I see that lowest air temperature has been in this time -25 C. I do not know whether is additional heading for this station but fore ones in Lapland there is
Have you tried both “legato restart” and “cm radio off and on
Separately yes but not at the same recovery attempt
Did you try the above on an R14 or R17 unit?
R17. There are R14 units in use installed at first. It’s planned to upgrade them (really replace with different unit and upgrade replaced one later)
Have you downgraded any units from R17 to R14? Please do not attempt this. While the download will be successful, the unit might not operate properly.
Only once on unit different from one having problem we are discussing (First time I tried remote upgrade from FW14 to FW17 it failed and I had to recover using swiflash. Reflashing R17 did not work, but I succeeded to flash R14 and after that upgrade to R17 using swiflash. That unit is used in testing weather station and works properly)
If possible, could you try supplying the AT command output for AT!GSTATUS?
If you execute this command several times, you can see the band and channel the radio is searching. You can also turn the radio on and off using AT+CFUN=1 or 0. The benefit of using the AT commands is that they are sent directly to the modem from the serial kernel driver, instead of being interpreted through legato.
I have tried to enter AT command like AT!GSTATUS? from minicom present in my build. It has not tried in recovery attempts in last weeks. I should perhaps either add supplying them from monitoring Application or write some additional application for collecting related information
Are your SIMs roaming?
I do not know.
PS. SIMS are expected to work in home network and nomaly do. I meant that I do not know whether they supports roaming
I’m generally not doing installation myself but only provide instructions and required files.
Got response from AT!GSTATUS? when FX30S 3G is stuck looking for mobile network (required some to go to the weather station and connect using USB):
!GSTATUS:
Current Time: 3929 Temperature: 10
Bootup Time: 5 Mode: ONLINE
System mode: GSM PS state: Not attached
GSM band: GSM900
GSM channel: 14
GMM (PS) state:DEREGISTERED PLMN SEARCH
MM (CS) state: IDLE NO IMSI
Serving Cell: 11 (GSM 900 )
RX level (dBm):-29.6250 LAC: 1014 (4116)
GPRS State: GPRS IDLE Cell ID: 00002A1B (10779)
IMS Reg State: UNKNOWN IMS Mode: Not Support
IMS Srv State: UNKNOWN SMS,UNKNOWN VoIP
OK
Also:
[100955 Salo Kärkkä] ~# cm radio
Power: ON
Current Network Operator:
Current RAT: GSM network (LE_MRC_RAT_GSM)
Status: Not registered but currently searching for a new operator (LE_MRC_REG_SEARCHING)
Signal: Very strong signal strength (5)
PS: Packet Switched Not registered (LE_MRC_REG_NONE)
In this case rebooting FX30S did not seem to help. Another FX30S 3G device with same SIM card got connection
It is not so easy to do: somebody needs to weather station (not me) and to do testing. I could ask to do that when somebody goes there. Unfortunately the problem has never occurred here in Helsinki area (or even at my home where I have device for software development). Than it would be easy to test.
Similar problems has been observed with several FX30S devices. I added to software rebooting if mobile network registration does not happen for several hours. It helped in some cases to reestablish connection.
Took a look at our (Finnish institute of Meteorology) device registry:
one has remark that it begin to get stuck and end did not establish connection at all
the second one has remark that modem is broken and cannot turn radio on
I have not seen myself the latest two myself and I have not attempted to do any their diagnostics. As result I cannot say more about those.
One thing that I could try (used network: Telia FI)
specify operator explicitly: AT+COPS=1,2, “24491” instead of the default AT+COPS=0
Do You think, could it help?
Hi @andris.pavenis,
I’m not certain if specifying the operator will help, it is something you can try if it’s not too difficult.
In your previous post, you mentioned that a different device with the same SIM registered successfully. Were you able to try the original device afterwards? I’m wondering if there was an issue with the SIM being seated properly. ie perhaps there is an issue with the SIM connector on some devices.
If we consider the second device is working properly and is running the same image with the same SIM. Did you mention this device never registered on the network ever?
Second device with same SIM card worked for some time and stopped to send weather data after some time. I did not mention earlier it as there where other software related issues - noticed mislabeling of device, settings did not match what was mentioned in label, so also there could be some software issues as I tried different builds to try to workaround network registration problems.
Earlier device had connection. However often the connection was not reestablished when it was lost for some reasons and required manual rebooting. This behavior has noticed with several devices at different locations. Usually reboot helped but not always.
Did you have any success with solving these issues? We’re in new development with FX30 cat-M in Australia and New Zealand and are seeing very similar symptoms with modems losing their registration sometimes many times a day, sometimes perfectly fine. It can be solved immediately by issuing a manual AT+COPS=1,… connect but I’d rather understand the cause and avoid this.
Just looking for answers before submitting a new ticket. Any feedback appreciated.
Sierra FX30S 3G was replaced by another device (Advantech ICR-3231) on the weather station where there were described problems of FX30S 3G being stuck looking for mobile network. Other network type (4G), but it was observed that connection often jumps between mobile cells. I do not know whether similar jumping happens there for 3G connection and whether it could cause problems.