EM7455 , WWAN/QMI: RX and TX stuck after a while

IoT Modules MC/EM Series
1

Hi.

We are running Linux 4.4.32 with qmi_wwan/cdc-wdm0 drivers , and Sierra Wireless EM7455 Qualcomm® Snapdragon™ X7 LTE-A LTE modem.
Everything works, until some packets are exchanged, and RX stucks. in ifconfig, only “RX errors” increasing.
After a while, kernel complains:
[ 1959.468676] NETDEV WATCHDOG: lte0 (qmi_wwan): transmit queue 0 timed out
and TX errors start increasing only.
Only solution to restore traffic is to reload system.

Seems what causes the problem is first received fragmented packet. Till then RX and TX works normally (with non-fragmented packets).

Any suggestions ?
We worked with slightly older 74XX modems without any problem.

Here is details for the modem:
[root@lanner-1510 ~]# lsusb -v -d 1199:9071

Bus 002 Device 002: ID 1199:9071 Sierra Wireless, Inc.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 9
idVendor 0x1199 Sierra Wireless, Inc.
idProduct 0x9071
bcdDevice 0.06
iManufacturer 1 Sierra Wireless, Incorporated
iProduct 2 Sierra Wireless EM7455 Qualcomm® Snapdragon™ X7 LTE-A
iSerial 3 LF81757479041022
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 226
bNumInterfaces 4
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xa0
(Bus Powered)
Remote Wakeup
MaxPower 126mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 255 Vendor Specific Subclass
bInterfaceProtocol 255 Vendor Specific Protocol
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 2
bAlternateSetting 0
bNumEndpoints 3
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
** UNRECOGNIZED: 05 24 00 10 01
** UNRECOGNIZED: 05 24 01 00 00
** UNRECOGNIZED: 04 24 02 02
** UNRECOGNIZED: 05 24 06 00 00
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x83 EP 3 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x000a 1x 10 bytes
bInterval 9
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 3
bAlternateSetting 0
bNumEndpoints 3
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
** UNRECOGNIZED: 05 24 00 10 01
** UNRECOGNIZED: 05 24 01 00 00
** UNRECOGNIZED: 04 24 02 02
** UNRECOGNIZED: 05 24 06 00 00
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x85 EP 5 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x000a 1x 10 bytes
bInterval 9
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x84 EP 4 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x03 EP 3 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 8
bAlternateSetting 0
bNumEndpoints 3
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 255 Vendor Specific Subclass
bInterfaceProtocol 255 Vendor Specific Protocol
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x87 EP 7 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 9
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x86 EP 6 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x04 EP 4 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 0
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 22
bNumDeviceCaps 2
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000000
(Missing must-be-set LPM bit!)
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000f
Device can operate at Low Speed (1Mbps)
Device can operate at Full Speed (12Mbps)
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 1
Lowest fully-functional device speed is Full Speed (12Mbps)
bU1DevExitLat 1 micro seconds
bU2DevExitLat 500 micro seconds
can’t get debug descriptor: Resource temporarily unavailable
Device Status: 0x0000
(Bus Powered)
RAW Paste Data

Thank you.
martin.

2

@mvi76,

A few questions.

  • What firmware are you running on the unit? You said older versions worked, the only difference here would have been the firmware so what were these versions (see commands below)?
  • Have you tried using the Sierra Wireless drivers rather than the open source ones?

Can you send the following commands to the unit give a bit of background?

ati
at!gstatus?
at!priid?
at+cgdcont?

Regards

Matt

3

Hi Matt,

Thanks for reply.
Please advice if you think newer firmware will overcome the problem.

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: EM7455
Revision: SWI9X30C_02.24.05.06 r7040 CARMD-EV-FRMWR2 2017/05/19 06:23:09
MEID: 35907306087374
ESN: 12816500009, 80FBC529
IMEI: 359073060873746
IMEI SV: 12
FSN: LF817574790410
+GCAP: +CGSM

AT!GSTATUS?
!GSTATUS:
Current Time: 22082 Temperature: 47
Reset Counter: 3 Mode: ONLINE
System mode: LTE PS state: Attached
LTE band: B13 LTE bw: 10 MHz
LTE Rx chan: 5230 LTE Tx chan: 23230
LTE CA state: NOT ASSIGNED
EMM state: Registered Normal Service
RRC state: RRC Idle
IMS reg state: No Srv

PCC RxM RSSI: -66 RSRP (dBm): -97
PCC RxD RSSI: -69 RSRP (dBm): -107
Tx Power: 0 TAC: 8404 (33796)
RSRQ (dB): -14.3 Cell ID: 02044B02 (33835778)
SINR (dB): 0.8

AT!PRIID?
PRI Part Number: 9907721
Revision: 001.000
Customer: Generic-M2M

Carrier PRI: 9999999_9904609_SWI9X30C_02.24.05.06_00_GENERIC_002.026_000

OK

OK
AT+CGDCONT?
+CGDCONT: 1,“IPV4V6”,“vzwinternet”,“0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0”,0,0,0,0

OK

4

This would be a good time to propose trying the GobiNet driver, despite what I have said before :slight_smile:

EM7455 is not supposed to work with qmi_wwan on Linux v4.4. This suggests that the driver is modified. It would be useful to eliminate bugs added in the process.

5

@mvi76,

I don’t think a newer firmware would resolve this, the issue would appear to be in the drivers side given tat 2.24.05.06 is a pretty solid build. I would go for the Sierra drivers as a next thing to try.

Regards

Matt

6

@dl5162
Bjørn, thanks for your reply.

Replacing standard Kernel driver with Sierra one seems heavy and risky step for us, mostly we have to perform lot of tests.
Would you please elaborate when qmi_wwan is not supposed to work on EM7455, and if the modifications you mentioned are planned to be done any time soon ?

7

Fully understandable. I only meant for a quick test to make sure the error is caused by the driver. If you can reproduce the same problem with the GobiNet driver, then we can be almost sure it’s not a driver issue.

qmi_wwan supports EM7455 but only from Linux v4.5. The driver in Linux v4.4 does not support raw-ip mode, which is required for the EM7455. And the support will not be backported to 4.4-stable since it is considered a new feature.

But backporting is straight forward and I know many have done that with success. I assume you are using a driver with backported raw-ip support? However easy, we cannot rule out the possibility that bugs were added while backporting. It’s easy to miss a required pacth or two, since a couple of them aren’t obviously related to the raw-ip feature.

8

Thanks for your prompt replies, Bjørn and Matt.

We are already using patched 4.4. driver with raw IP support.

Just noticed, that the problematic modem has not Verizon FW, and we are using Verizon carreer. On the problematic device:
AT!PRIID?
PRI Part Number: 9907721
Revision: 001.000
Customer: Generic-M2M

Carrier PRI: 9999999_9904609_SWI9X30C_02.24.05.06_00_GENERIC_002.026_000

On another device that is located in Europe, we see wider list of Fimware packages:
Carrier PRI: 9999999_9904594_SWI9X30C_02.20.03.00_00_ATT_002.020_000
Carrier PRI: 9999999_9904609_SWI9X30C_02.24.05.06_00_GENERIC_002.026_000
Carrier PRI: 9999999_9904779_SWI9X30C_02.20.03.22_00_SPRINT_002.020_000
Carrier PRI: INTERNAL_9901234_EM7455_00.00.00.00_00_OEMPRI918_001.000_000
Carrier PRI: 9999999_9907371_SWI9X30C_02.24.05.06_00_US-Cellular_000.003_000
Carrier PRI: 9999999_9904780_SWI9X30C_02.20.03.22_00_VERIZON_002.026_001

Can the problem be due to Verizon approved firmware is not installed ?

9

@mvi76,

Obviously I responded to your post this morning about this generic subject but the short answer is yes it could easily be the issue, however, it could also be the issue that since you are using Verizon and obviously a Verizon SIM card that it is allowing you to attach, get an IP address, etc but not to send/receive data in a similar way some of the EU operators do when you run out of credit on your pay as yo go SIM.

Verizon can be pretty hot on not letting unauthorized applications on their network i.e. when you start the ODI approval process, once you have gone through the initial SFN (Safe For Network) testing you are allowed something like 20 SIM’s to perform trials while the application goes through the full approval.

Regards

Matt

10

@mvi76,

Correction the stages are

Early Network Access Program (ENAP)
 If the DEVELOPER is using a VZW certified module, up to 20 activations can be
allowed for developmental purposes. Without a VZW certified module, up to 2
activations can be allowed for developmental purposes. Upon uploading of test
Device IMEIs to the ODP, VZW will load the test Device(s) MEID/ESN/IMEI in DMD.
The DEVELOPER should then work with their VZW sales representative to subscribe
and activate lines to connect the test Device(s) with VZW network. The test MDNs
must be disconnected no later than 60 day after testing is completed

Safe For Network (SFN) Testing
 If the DEVELOPER completes SFN testing successfully, up to 500 activations
per project can be allowed for further development purposes. DEVELOPERS
must submit the Device information for the ODPT’s review and the SFN testing must
be conducted at a VZW Authorized ITL.
DEVELOPERS submit Device information on the ODP and send a test Device
request with supportive reasons and test ESNs/MEIDs/IMEIs.
o RN, TECC, and Device Solution One Pager are required for SFN.
o ODPT and NDET team review and approve the request.

Regards

Matt

11

Hi,

Brief update, using GobiNet drivers seems to resolve the problem.
Thanks for the help.

I noticed that only way to set up connection is using AT-commands, is there ready tool like qmicli for the GobiNet driver, that can simplify the job?

Martin.

12

@mvi76,

So qmicli appear to just be an application that uses the open source QMI daemon (qmilib), we offer our own QMI daemon which we validate our units with, this is generically referred to as the SDK (link below), within the SDK we give a load of examples both pre compiled and with source code, one of these is called the connection manager which performs a similar function.

https://source.sierrawireless.com/resources/airprime/software/linux-qmi-sdk-software-latest/

Regards

Matt