GX450 Continuously Loses Service (AT&T)

Hi all,

I have a GX450 with integrated MC7354 LTE module. I have an AT&T mobile hotspot plan active on the router, and have been experiencing a strange issue ever since I got the plan up and running.

Every half hour or so, the router will lose LTE connectivity. Signal levels remain fine, but the cellular state goes from “Connected” to “No Service.” Without any intervention on my part, after about ten minutes or so of being out-of-service, the radio re-connects and everything starts working again. This process has been ongoing the past few days.

In the logs, I see the following sequence every time the router looses service and reconnects:

Jun 22 23:32:51 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-51, RSRQ=-8, RSRP=-75, SINR=12.3
Jun 22 23:45:06 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 23:45:06 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - No Service
Jun 22 23:45:06 notice ALEOS_LINKMGMT_linkmon: New Link Status - No Service
Jun 22 23:45:09 err ALEOS_EVENTS_dpRptSend: Sending report to server failed for 10 reason Operation not permitted
Jun 22 23:45:14 alert ALEOS_CONNECTMGMT: IPV4 Disconnect Handled
Jun 22 23:45:30 warning ALEOS_WEB_ACEmanager: 192.168.100.2 - - Invalid or missing session token
Jun 22 23:47:16 err ALEOS_EVENTS_dpRptGen: GPGGA Number of Satellites field is empty
Jun 22 23:47:51 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-51, RSRQ=-8, RSRP=-75, SINR=14.6
Jun 22 23:50:06 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 5 minutes ]
Jun 22 23:52:27 notice ALEOS_SYSTEM_SM: Swapping container, new container:'SnF21.smc'...
Jun 22 23:55:06 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 10 minutes ]
Jun 22 23:55:06 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - Connected
Jun 22 23:55:06 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Link Down
Jun 22 23:55:09 err ALEOS_EVENTS_dpRptSend: Sending report to server failed for 10 reason Operation not permitted
Jun 22 23:55:15 notice ALEOS_CONNECTMGMT: *** Network Connection successful. WAN IP [Cellular] : 10.128.xxx.xxx
Jun 22 23:55:15 notice ALEOS_CONNECTMGMT: *** IP UP 10.128.xxx.xxx : wwan0
Jun 22 23:55:15 notice ALEOS_CONNECTMGMT: Setting WWAN MTU size to 1430
Jun 22 23:55:16 notice ALEOS_CONNECTMGMT: GDNS is restarting dnsmasq
Jun 22 23:55:25 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Ready
Jun 22 23:55:25 warning ALEOS_SERIAL_HMC: not in tcp mode current mode 0 
Jun 22 23:55:30 alert ALEOS_LAN: MTU Source Auto - Set DHCP option 26 MTU: 1430
Jun 22 23:55:30 alert ALEOS_LAN: MTU Source Auto - Ignore Opt 26 MTU size 1500

When the router is disconnected, a status query of the MC7354 yields the following information:

AT!GSTATUS?
!GSTATUS: 
Current Time:  3094Temperature: 39
Bootup Time:   0Mode:        ONLINE         
System mode:   LTE        PS state:    Attached     
LTE band:      B4     LTE bw:      10 MHz  
LTE Rx chan:   2000LTE Tx chan: 20000
EMM state:     Registered     Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  

RSSI (dBm):    -50Tx Power:    0
RSRP (dBm):    -77TAC:         486C (18540)
RSRQ (dB):     -11Cell ID:     0478E217 (75031063)
SINR (dB):      5.4


OK

RRC Connected but IMS state of No Srv seems to indicate something strange going on, but my lack of knowledge on the particulars of LTE leaves me with no obvious troubleshooting steps.

I’ve tried both the ATT-specific firmware for the MC7354, as well as the generic firmware. Both exhibit the same behavior.

I’m at a loss as to what the problem is. When the radio is connected, LTE speeds are good and no issues with connectivity occur. Any suggestions for additional troubleshooting or logs to collect would be appreciated.

I noticed overnight via my logging that there were no connection drops. However again today I’m getting a few drops each hour. Same exact issue with same logs as above.

If anyone has any logging suggestions, AT commands, or anything else I can try to poke this issue please let me know. I’m at a loss as to why the router decides to randomly loose service despite perfectly fine signal levels.

For reference, the LEDs are as follows when this connection issue occurs:

Power: Green with occasional yellow flash (Valid GPS fix)
Activity: Off
Signal: Green
Network: Flashing Yellow

Which as far as I can tell would be the expected LEDs for this situation. The question remains - why is the radio randomly going out of service?

Hi @patrick ,
It is very odd you just encounter the issue during the day not overnight. Did you run any specific applications on the gateway?Can you enable debug level on the device as the screenshot and share the log? Please apply and reboot the device after changing configuration

Did you try another SIM for the device? Does the issue happen? I find some customers who are using AT&T getting the same issue as the below link. Have you tried to contact AT&T service whether they have any relevant information

Thanks

Vianney,

I’m enabling those logs now. I’ll let the router run overnight again to see if the problem continues.

Previously I was running on a US Mobile SIM card (a T-mobile MVNO) and did not experience the same symptoms.

Additionally, I noticed today during my testing that my radio showed the IMS State of No Srv even when the connection is fully working, so I believe that was a red herring.

I have not yet contacted AT&T. I fear the level of support available to a small prepaid plan like mine will be rather limited but if we run out of other options I can go down that route.

Thanks for the suggestions! I’ll report back with my findings.

Okay, I have a new log capture of a loss of connection event. I disabled the event reporting/location debug print because I have GPS reporting enabled and it was spamming the logs with unnecessary info.

I’ve attached the log file as it’s above the max character limit for forum messages. The loss of connection starts at 13:41:33 in the below logs.

Hopefully some useful information can be gathered from that. Let me know if you do actually need the Event Reporting/Location debug turned back on.

Thanks!

OOS-log-debug.txt (44.8 KB)

Here’s another log capture. The router has lost connection five times already in the past 6 hours.
OOS-event-2.txt (41.3 KB)

I’ve been plotting connectivity of the router over the past few days and I’ve noticed something interesting.

Attached is a plot of ping times over 48 hours. The big spikes are where the router looses connectivity. Notice how overnight (between ~2300 and ~800) there are zero connection issues, while during the day the connection drops quite frequently.

I’m thinking this is potentially a local cell site issue, with congestion or crowding or something along those lines.

You should contact your operator to get the exactly answer. Please feel free to share more result/question if you think that it can help
Thanks