EM7565 Verizon LTE connection not stable at hight

IoT Modules MC/EM Series
1

Hi There !

I have an embedded system with two EM7565 modems talking with Verizon. I’m using RMNet plus the latest GobiSerial / GobiNet drivers. With light traffic such as ping, both modems run for days w/o problems, however when I run speed tests it seems to upset things as I lose connectivity. Then I keep trying to enable the data connection with at!scact=1,3 and eventually it comes back to life. This seems like a carrier thing, but I’m not sure how to troubleshoot it.

So what happens is that I send out the at!scact=1,3 which returns OK, and at!scact? shows it as enabled. But then about 5 seconds later it goes back to showing it as disabled.

at!scact=1,3
OK
at!scact?
!SCACT: 1,0
!SCACT: 2,0
!SCACT: 3,1
!SCACT: 4,0
!SCACT: 5,0
!SCACT: 6,0

OK
at!scact?
!SCACT: 1,0
!SCACT: 2,0
!SCACT: 3,1
!SCACT: 4,0
!SCACT: 5,0
!SCACT: 6,0

[5 sec wait]

OK
at!scact?
!SCACT: 1,0
!SCACT: 2,0
!SCACT: 3,0
!SCACT: 4,0
!SCACT: 5,0
!SCACT: 6,0

OK

I tried enabling these notifications, but did not see anything in the terminal

at+creg=1
at+cgreg=1
at+cereg=1

Is there any other way to enable more logging ? I did enable the Gobi Driver debugging, see below

May 28 04:30:02 avoca-cb user.info kernel: [ 2293.649288] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.649296] GobiNet::ReadCallback(5825) Read 35 bytes
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.649321] GobiNet::PrintHex(5825) : 01 22 00 80 01 12 04 EF 00 22 00 16 00 01 02 00 02 00 12 01 00 04 13 02 00 80 88 14 01 00 06 15 01 00 00
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.649344] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.649359] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.701281] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.701320] GobiNet::ReadCallback(5825) Read 35 bytes
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.701345] GobiNet::PrintHex(5825) : 01 22 00 80 01 12 04 F0 00 22 00 16 00 01 02 00 02 00 12 01 00 06 13 02 00 80 88 14 01 00 06 15 01 00 00
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.701380] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.701390] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.824770] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.824786] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:02 avoca-cb user.info kernel: [ 2293.824835] GobiNet::PrintHex(4261) : 45 00 00 54 32 30 40 00 40 01 DC DC 64 4C BB 44 08 08 04 04 08 00 90 55 10 A5 06 88 88 06 00 00 2A E5 AF 60 E1 30 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.109225] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.109232] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.109275] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F1 00 01 00 40 00 10 04 00 01 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 54 00 00
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.109295] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.109301] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.825806] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.825859] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:03 avoca-cb user.info kernel: [ 2294.825860] GobiNet::PrintHex(4261) : 45 00 00 54 32 31 40 00 40 01 DC DB 64 4C BB 44 08 08 04 04 08 00 6A 50 10 A5 06 89 89 06 00 00 2B E5 AF 60 05 35 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.109101] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.109108] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.109151] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F2 00 01 00 40 00 10 04 00 02 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 A8 00 00
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.109173] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.109179] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.827037] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.827054] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:04 avoca-cb user.info kernel: [ 2295.827104] GobiNet::PrintHex(4261) : 45 00 00 54 32 32 40 00 40 01 DC DA 64 4C BB 44 08 08 04 04 08 00 B0 4A 10 A5 06 8A 8A 06 00 00 2C E5 AF 60 BD 39 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.109104] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.109111] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.109154] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F3 00 01 00 40 00 10 04 00 03 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 FC 00 00
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.109174] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.109179] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.828272] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.828325] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:05 avoca-cb user.info kernel: [ 2296.828326] GobiNet::PrintHex(4261) : 45 00 00 54 32 33 40 00 40 01 DC D9 64 4C BB 44 08 08 04 04 08 00 BF 44 10 A5 06 8B 8B 06 00 00 2D E5 AF 60 AC 3E 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.109117] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.109126] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.109169] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F4 00 01 00 40 00 10 04 00 04 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 50 01 00
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.109189] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.109195] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.829498] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.829515] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:06 avoca-cb user.info kernel: [ 2297.829565] GobiNet::PrintHex(4261) : 45 00 00 54 32 34 40 00 40 01 DC D8 64 4C BB 44 08 08 04 04 08 00 0E 3F 10 A5 06 8C 8C 06 00 00 2E E5 AF 60 5B 43 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.109107] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.109114] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.109157] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F5 00 01 00 40 00 10 04 00 05 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 A4 01 00
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.109177] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.109182] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.830753] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.830806] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:07 avoca-cb user.info kernel: [ 2298.830807] GobiNet::PrintHex(4261) : 45 00 00 54 32 35 40 00 40 01 DC D7 64 4C BB 44 08 08 04 04 08 00 16 39 10 A5 06 8D 8D 06 00 00 2F E5 AF 60 51 48 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.109140] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.109148] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.109202] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F6 00 01 00 40 00 10 04 00 06 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 F8 01 00
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.109224] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.109230] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.831015] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.831037] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:08 avoca-cb user.info kernel: [ 2299.831089] GobiNet::PrintHex(4261) : 45 00 00 54 32 36 40 00 40 01 DC D6 64 4C BB 44 08 08 04 04 08 00 21 37 10 A5 06 8E 8E 06 00 00 30 E5 AF 60 44 49 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.109223] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.109230] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.109273] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F7 00 01 00 40 00 10 04 00 07 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 4C 02 00
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.109294] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.109299] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.832144] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.832196] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:09 avoca-cb user.info kernel: [ 2300.832197] GobiNet::PrintHex(4261) : 45 00 00 54 32 37 40 00 40 01 DC D5 64 4C BB 44 08 08 04 04 08 00 9C 31 10 A5 06 8F 8F 06 00 00 31 E5 AF 60 C7 4D 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.109445] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.109453] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.109497] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F8 00 01 00 40 00 10 04 00 08 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 A0 02 00
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.109524] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.109530] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.833411] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.833432] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:10 avoca-cb user.info kernel: [ 2301.833486] GobiNet::PrintHex(4261) : 45 00 00 54 32 38 40 00 40 01 DC D4 64 4C BB 44 08 08 04 04 08 00 DC 2B 10 A5 06 90 90 06 00 00 32 E5 AF 60 85 52 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.109223] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.109230] GobiNet::ReadCallback(5825) Read 77 bytes
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.109273] GobiNet::PrintHex(5825) : 01 4C 00 80 01 12 04 F9 00 01 00 40 00 10 04 00 09 00 00 00 11 04 00 00 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 F4 02 00
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.109294] GobiNet::QMIWDSCallback(5825) Net device link is connected
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.109299] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.834698] GobiNet::GobiNetDriverTxFixup(4261)
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.834751] GobiNet::GobiNetDriverTxFixup(4261) For sending to device modified:
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.834752] GobiNet::PrintHex(4261) : 45 00 00 54 32 39 40 00 40 01 DC D3 64 4C BB 44 08 08 04 04 08 00 A6 25 10 A5 06 91 91 06 00 00 33 E5 AF 60 B9 57 0D 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.945162] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.945198] GobiNet::ReadCallback(5825) Read 34 bytes
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.945228] GobiNet::PrintHex(5825) : 01 21 00 80 01 12 04 FA 00 22 00 15 00 01 02 00 01 00 10 02 00 F0 03 11 04 00 06 00 24 00 12 01 00 06
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.945266] GobiNet::QMIWDSCallback(5825) Net device link is disconnected
May 28 04:30:11 avoca-cb user.info kernel: [ 2302.945280] GobiNet::QMIWDSCallback(5825) clientID :0x1201
May 28 04:30:12 avoca-cb user.info kernel: [ 2302.957092] GobiNet::ProcessReadWorkFunction(5825) ResubmitIntURB
May 28 04:30:12 avoca-cb user.info kernel: [ 2302.957098] GobiNet::ReadCallback(5825) Read 34 bytes
May 28 04:30:12 avoca-cb user.info kernel: [ 2302.957120] GobiNet::PrintHex(5825) : 01 21 00 80 01 12 04 FB 00 22 00 15 00 01 02 00 01 00 10 02 00 F0 03 11 04 00 06 00 24 00 12 01 00 04
May 28 04:30:12 avoca-cb user.info kernel: [ 2302.957135] GobiNet::QMIWDSCallback(5825) Net device link is disconnected
May 28 04:30:12 avoca-cb user.info kernel: [ 2302.957139] GobiNet::QMIWDSCallback(5825) clientID :0x1201

Also, here are the standard troubleshooting command responses

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.14.02.00 2e210b jenkins 2020/08/19 14:18:39
IMEI: 353533100240350
IMEI SV: 12
FSN: UF923587400315
+GCAP: +CGSM

OK
ATI8
Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.14.02.00 2e210b jenkins 2020/08/19 14:18:39
IMEI: 353533100240350
IMEI SV: 12
FSN: UF923587400315
+GCAP: +CGSM

OK
AT!BAND?
Index, Name
00, All bands

OK
AT!SELRAT?
!SELRAT: 00, Automatic

OK
AT!SELCIOT?
ERROR
AT+CGDCONT?
+CGDCONT: 1,“IPV4V6”,“ims”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 2,“IPV4V6”,“vzwadmin”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 3,“IPV4V6”,“vzwinternet”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 4,“IPV4V6”,“vzwapp”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 5,“IPV4V6”,“vzw800”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 6,“IPV4V6”,“vzwclass6”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0

OK
AT+COPS=?
+COPS: (2,“Verizon Wireless”,“VZW”,“311480”,7),(1,“T-Mobile”,“T-Mobile”,“310260”,7),(1,“311 490”,“311 490”,“311490”,7),(1,“T-Mobile”,“T-Mobile”,“310260”,2),(1,“AT&T”,“AT&T”,“310410”,2),(0,1,2,3,4),(0,1,2)

OK
AT!GSTATUS?
!GSTATUS:
Current Time: 3052 Temperature: 35
Reset Counter: 1 Mode: ONLINE
System mode: LTE PS state: Attached
LTE band: B66 LTE bw: 20 MHz
LTE Rx chan: 0 LTE Tx chan: 132122
LTE SCC1 state:NOT ASSIGNED
LTE SCC2 state:NOT ASSIGNED
LTE SCC3 state:NOT ASSIGNED
LTE SCC4 state:NOT ASSIGNED
EMM state: Registered Normal Service
RRC state: RRC Idle
IMS reg state: Full Srv IMS mode: Normal

PCC RxM RSSI: -46 PCC RxM RSRP: -76
PCC RxD RSSI: -57 PCC RxD RSRP: -89
Tx Power: – TAC: 2f06 (12038)
RSRQ (dB): -9.8 Cell ID: 054b7d16 (88833302)
SINR (dB): 22.2

OK

Any thoughts on why it is dropping the connection ?

Thanks in advance

/Otto

2

Hi @otto

Before you enable the data connection with at!scact=1,3, has a data connection existed? If a data connection existed before, you can try to disconnect it and start the data connection with at!scact=1,3.

Thanks.

3

Hi Donald !

Yes, the connection was present previously. In fact I discovered that if I release the DHCP address and bring the ethernet interface (eth1) down before issuing the at!scact=1,3 command it seems to come back every time (after also restarting udhcpc).

Thanks !

/Otto

4

Hi @otto

Could you check some information:

  1. List and checked PDP context activation by command AT+CGDCONT?, AT+CGACT? If the 3rd PDP wasn’t activated that means the response has +CGACT: 3,0 then active this PDP by command AT+CGACT=1,3.
    And then check the IP address of the 3rd PDP by command AT+CGPADDR?
  2. On Linux, help to check network interface by command “ifconfig -a” (EM7565 must be mounted to Linux).

Please check it and share your result.

Thanks.

5

Hi Donald !

In the main loop I now check for the link to be up with at!scact?, and if its down I bring down the Linux network interface (eth1) and run at!scact=1,3 again. In the same loop if I detect that the linux network interface is not in the “Running” state I restart udhcpc. That seems to be able to recover from the situations I have ran into.

My system has two modems and antennas are quite close (1" away). Can that cause one of the modems to go down ? (Due to the other modem seeing a very strong incoming signal/ high noise floor) or something like that…? I’m not sure how to explain why ping test works well but when running full speed one of the modems end up going down after 5-10sec typically. If it was a power problem I would think it would reboot ?

Thanks for you help !

/Otto

6

Hi @otto

  1. Could you check again and provide some information above?
  2. Does this problem happen on another SIM or network operation?
  3. Have you tried this problem on your system has one modem? Does it happen?

Thanks.

7

Hi Donald !

I ran speedtest -I [eth1/eth2] in parallel to cause eth2 to stop working. Then I checked the status as you requested

AT+CGDCONT?
+CGDCONT: 1,“IPV4V6”,“ims”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 2,“IPV4V6”,“vzwadmin”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 3,“IPV4V6”,“vzwinternet”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 4,“IPV4V6”,“vzwapp”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 5,“IPV4V6”,“vzw800”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 6,“IPV4V6”,“vzwclass6”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0

OK
AT+CGACT?
+CGACT: 1,1
+CGACT: 2,1
+CGACT: 3,0
+CGACT: 4,0
+CGACT: 5,0
+CGACT: 6,0

OK
AT+CGACT=1,3
OK
AT+CGPADDR?
ERROR
AT+CGACT?
+CGACT: 1,1
+CGACT: 2,1
+CGACT: 3,1
+CGACT: 4,0
+CGACT: 5,0
+CGACT: 6,0

OK
AT+CGPADDR?
ERROR
AT+CGACT=0,1
ERROR
AT+CGACT=0,1
ERROR
AT+CGACT=0,2
ERROR
AT+CGACT?
+CGACT: 1,1
+CGACT: 2,1
+CGACT: 3,1
+CGACT: 4,0
+CGACT: 5,0
+CGACT: 6,0

OK
AT+CGPADDR?
ERROR

Interestingly PDP1 and 2 are enabled, although that is not something I did (first modem, which is still in
a working state had both PDP 2 & 3 enabled after the test). PDP 3 went to 0 after running the test, but even after setting it back to one the CGPADDR? command failed. This time I was not able to get the interface back up by renewing DHCP or ifconfig eth2 down.

2). Running low-speed tests like ping runs for days w/o problems. I have not tested this on any other carrier or SIM at this point, but I can try AT&T shortly.

3). I put one modem to sleep and just ran the test on eth2. Ran 4x in a row w/o any problems. Running in parallel it usually dies first time, although once I’ve seen it get to the second test run. I checked the PDPs and 1-3 are enabled

I checked the voltage with at!pcvolt? and it did not go below 3.247V while the speedtest was running

Is there any command I can use to see reason for disconnect ? Can one of the serial ports be configured for debug logging ?

Thanks for your help Donald !

/Otto

9

Hi otto.

Please share the results of the steps below:

1. Disconnect ethernet  Linux PC
2. AT!RESET
3. AT!ENTERCND="A710"
4. AT+CGDCONT?
5. AT+CGACT?
6. AT+CGPADDR
7. AT!USBCOMP?
8. AT!SCACT=1,1
9. Wait for 5s
10. Check network interface by command “ifconfig -a”
11. Ping to 8.8.8.8 address
12. AT!SCACT?
13. AT+CGPADDR
10

Hi Donald !

Sorry about the delay on this. The main issue I have is that the modems go down in the first place. Getting them back up by resetting always works.

What kind of logging can I enable or how do I troubleshoot the reason for the disconnect ? Is it normal ?

Thanks,

/Otto

See command results below. The connection was originally Up, then it went down. The below commands brought it back up.

root@avoca-cb:~# microcom -X /dev/modemAT0
at
OK
AT!ENTERCND=“A710”
OK
AT+CGDCONT?
+CGDCONT: 1,“IPV4V6”,“ims”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 2,“IPV4V6”,“vzwadmin”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 3,“IPV4V6”,“vzwinternet”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 4,“IPV4V6”,“vzwapp”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 5,“IPV4V6”,“vzw800”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 6,“IPV4V6”,“vzwclass6”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0

OK
AT+CGDCONT?
+CGDCONT: 1,“IPV4V6”,“ims”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 2,“IPV4V6”,“vzwadmin”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 3,“IPV4V6”,“vzwinternet”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 4,“IPV4V6”,“vzwapp”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 5,“IPV4V6”,“vzw800”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0
+CGDCONT: 6,“IPV4V6”,“vzwclass6”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0

OK
AT+CGACT?
+CGACT: 1,1
+CGACT: 2,0
+CGACT: 3,0
+CGACT: 4,0
+CGACT: 5,0
+CGACT: 6,0

OK
AT+CGPADDR
+CGPADDR: 1,38.0.16.18.17.30.222.135.235.80.208.85.79.43.132.30
+CGPADDR: 2,0.0.0.0,0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
+CGPADDR: 3,0.0.0.0,0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
+CGPADDR: 4,0.0.0.0,0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
+CGPADDR: 5,0.0.0.0,0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
+CGPADDR: 6,0.0.0.0,0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0

OK
AT!USBCOMP?
Config Index: 1
Config Type: 3 (Generic)
Interface bitmask: 0000010D (diag,nmea,modem,rmnet0)

OK
AT!SCACT=1,3
OK

root@avoca-cb:~# ifconfig wwan0
wwan0 Link encap:Ethernet HWaddr c6:b9:bb:b6:99:08
BROADCAST NOARP MULTICAST MTU:1500 Metric:1
RX packets:38719 errors:0 dropped:0 overruns:0 frame:0
TX packets:39739 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3407167 (3.2 MiB) TX bytes:3594986 (3.4 MiB)

root@avoca-cb:~# killall udhcpc
root@avoca-cb:~# udhcpc -i wwan0
udhcpc (v1.21.1) started
Sending discover…
Sending select for 100.81.186.96…
Lease of 100.81.186.96 obtained, lease time 7200
/etc/udhcpc.d/50default: Adding DNS 198.224.169.135
/etc/udhcpc.d/50default: Adding DNS 198.224.171.135
root@avoca-cb:~# fping -I wwan0 8.8.4.4 -l
8.8.4.4 : [0], 84 bytes, 75.6 ms (75.6 avg, 0% loss)
8.8.4.4 : [1], 84 bytes, 44.6 ms (60.1 avg, 0% loss)
8.8.4.4 : [2], 84 bytes, 63.5 ms (61.2 avg, 0% loss)
8.8.4.4 : [3], 84 bytes, 62.5 ms (61.6 avg, 0% loss)

11

Hi Donald !

This has now become a bigger problem as the modem goes down within 10 seconds of us starting the application process (Video Tx). Ping runs for days. I captured this debug log from the Gobi Driver. How can I debug this further ?

Jun 29 09:11:27 avoca-cb user.info kernel: [ 714.136660] GobiNet::QMIWDSCallback(29) Net device link is connected
Jun 29 09:11:27 avoca-cb user.info kernel: [ 714.136670] GobiNet::QMIWDSCallback(29) clientID :0x1201
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.600621] GobiNet::GobiNetDriverTxFixup(2140)
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.600638] GobiNet::GobiNetDriverTxFixup(2140) For sending to device modified:
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.600932] GobiNet::PrintHex(2140) : 45 00 02 05 37 31 40 00 40 06 1C A7 64 77 6F 68 36 D0 DA 6B A4 5B 00 50 92 D1 AB 64 38 7 4 C4 A6 80 18 00 7F 5C B0 00 00 01 01 08 0A 00 00 A2 BA 6D 3C A3 D1 50 4F 53 54 20 2F 74 70 2F
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.658616] GobiNet::GobiNetDriverRxFixup(0) RX From Device:
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.658651] GobiNet::PrintHex(0) : 45 00 00 34 62 EA 40 00 3E 06 F4 BE 36 D0 DA 6B 64 77 6F 68 00 50 A4 5B 38 74 C4 A6 92 D1 A D 35 80 10 00 E0 F1 57 00 00 01 01 08 0A 6D 3C AD A5 00 00 A2 BA
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.658694] GobiNet::PrintHex(0) : 1E BF 0A DA A0 08 00 00 00 00 00 00 08 00 45 00 00 34 62 EA 40 00 3E 06 F4 BE 36 D0 DA 6B 6 4 77 6F 68 00 50 A4 5B 38 74 C4 A6 92 D1 AD 35 80 10 00 E0 F1 57 00 00 01 01 08 0A 6D 3C AD A5
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.674911] GobiNet::GobiNetDriverTxFixup(2109)
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.674924] GobiNet::GobiNetDriverTxFixup(2109) For sending to device modified:
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.674963] GobiNet::PrintHex(2109) : 45 00 00 3C 37 63 40 00 40 06 A5 A0 0A 00 32 00 17 16 0A A3 D1 14 0F 82 D3 83 95 A1 00 0 0 00 00 A0 02 39 08 C4 C0 00 00 02 04 05 B4 04 02 08 0A 00 00 A2 C2 00 00 00 00 01 03 03 07
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.743599] GobiNet::GobiNetDriverRxFixup(1244)
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.743613] GobiNet::GobiNetDriverRxFixup(1244) RX From Device:
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.744034] GobiNet::PrintHex(1244) : 45 00 03 63 62 EB 40 00 3E 06 F1 8E 36 D0 DA 6B 64 77 6F 68 00 50 A4 5B 38 74 C4 A6 92 D 1 AD 35 80 18 00 E0 D8 36 00 00 01 01 08 0A 6D 3C AD BA 00 00 A2 BA 48 54 54 50 2F 31 2E 31 20
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.744576] GobiNet::GobiNetDriverTxFixup(1244) For sending to device modified:
Jun 29 09:11:30 avoca-cb user.info kernel: [ 716.744610] GobiNet::PrintHex(1244) : 45 00 00 34 37 32 40 00 40 06 1E 77 64 77 6F 68 36 D0 DA 6B A4 5B 00 50 92 D1 AD 35 38 7 4 C7 D5 80 10 00 8C EE 58 00 00 01 01 08 0A 00 00 A2 C9 6D 3C AD BA
Jun 29 09:11:30 avoca-cb user.info kernel: [ 717.136570] GobiNet::ProcessReadWorkFunction(29) ResubmitIntURB
Jun 29 09:11:30 avoca-cb user.info kernel: [ 717.136587] GobiNet::ReadCallback(29) Read 77 bytes
Jun 29 09:11:30 avoca-cb user.info kernel: [ 717.136636] GobiNet::PrintHex(29) : 01 4C 00 80 01 12 04 22 00 01 00 40 00 10 04 00 6C 00 00 00 11 04 00 2E 00 00 00 12 04 00 FF FF FF FF 13 04 00 FF FF FF FF 14 04 00 FF FF FF FF 15 04 00 FF FF FF FF 19 08 00 09 33 00 00

Jun 29 09:11:30 avoca-cb user.info kernel: [ 717.136663] GobiNet::QMIWDSCallback(29) Net device link is connected

Jun 29 09:11:30 avoca-cb user.info kernel: [ 717.136673] GobiNet::QMIWDSCallback(29) clientID :0x1201
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.670912] GobiNet::GobiNetDriverTxFixup(0)
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.670928] GobiNet::GobiNetDriverTxFixup(0) For sending to device modified:
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.670969] GobiNet::PrintHex(0) : 45 00 00 3C 37 64 40 00 40 06 A5 9F 0A 00 32 00 17 16 0A A3 D1 14 0F 82 D3 83 95 A1 00 00 0 0 00 A0 02 39 08 C4 5C 00 00 02 04 05 B4 04 02 08 0A 00 00 A3 26 00 00 00 00 01 03 03 07
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.808571] GobiNet::ProcessReadWorkFunction(29) ResubmitIntURB
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.808586] GobiNet::ReadCallback(29) Read 34 bytes
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.808614] GobiNet::PrintHex(29) : 01 21 00 80 01 12 04 23 00 22 00 15 00 01 02 00 01 00 10 02 00 F0 03 11 04 00 06 00 24 00 12 01 00 06

Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.808637] GobiNet::QMIWDSCallback(29) Net device link is disconnected

Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.808652] GobiNet::QMIWDSCallback(29) clientID :0x1201
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.816568] GobiNet::ProcessReadWorkFunction(29) ResubmitIntURB
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.816582] GobiNet::ReadCallback(29) Read 34 bytes
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.816608] GobiNet::PrintHex(29) : 01 21 00 80 01 12 04 24 00 22 00 15 00 01 02 00 01 00 10 02 00 F0 03 11 04 00 06 00 24 00 12 01 00 04
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.816628] GobiNet::QMIWDSCallback(29) Net device link is disconnected
Jun 29 09:11:31 avoca-cb user.info kernel: [ 717.816637] GobiNet::QMIWDSCallback(29) clientID :0x1201

Thanks,

/Otto

12

Hello All !

I have another update on this. In our application the sockets are bound to a specific interface to send data over multiple modems. The way this is done is typically using “Bind before Connect”, but that causes the modems to disconnect (not every time but quite often). Using setsockopt() with SO_BINDTODEVICE instead has worked well so far. Bind before connect is pretty standard so my suspicion is that there is something wrong in the driver, but I’m leaving that alone. Also using a fairly old Linux version (3.10) so that’s a possibility too

Hope this helps someone out there

/Otto

13

Hi @otto

I have the same issue. How to use setsockopt() with SO_BINDTODEVICE ?
Which driver are you using?

Thanks,

14

The Title of this problem should be changed to “Instability problems when binding before connect”, as it just happens the speedtest I was using does exactly that. So its not the high throughput thats a problem.

Anyway, socket options are set this way, right after creating the socket

setsockopt(sock, SOL_SOCKET, SO_BINDTODEVICE, device, sizeof(device))