EM7565 unexpected reset/crash

[pierre.pele]

Hello everyone,

I’am using EM7565 in production environnement,

We are experiencing issues with two identical setups, the modem gets reset after few hours of uptime (from hours to days),

In the linux logs, we observe a disconnection on the USB port :

[259892.516755] usb 1-3: USB disconnect, device number 3
[259892.523295] qcserial ttyUSB0: Qualcomm USB modem converter now disconnected from ttyUSB0
[259892.531655] qcserial 1-3:1.0: device disconnected
[259892.536980] qcserial ttyUSB1: Qualcomm USB modem converter now disconnected from ttyUSB1
[259892.545557] qcserial 1-3:1.2: device disconnected
[259892.550887] qcserial ttyUSB2: Qualcomm USB modem converter now disconnected from ttyUSB2
[259892.559198] qcserial 1-3:1.3: device disconnected
[259892.565062] cdc_mbim 1-3:1.12 wwan0: unregister 'cdc_mbim' usb-3530000.xhci-3, CDC MBIM
[259892.742839] usb usb1: usb_suspend_both: status 0
...
[259901.706858] usb 1-3: new high-speed USB device number 4 using tegra-xusb
...
[259901.764997] usb 1-3: New USB device found, idVendor=1199, idProduct=9091
[259901.771810] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[259901.779045] usb 1-3: Product: Sierra Wireless EM7565 Qualcomm\xc2\xae Snapdragon\xe2\x84\xa2 X16 LTE-A
[259901.787143] usb 1-3: Manufacturer: Sierra Wireless, Incorporated
[259901.793492] usb 1-3: SerialNumber: XXX
[259901.800395] qcserial 1-3:1.0: Qualcomm USB modem converter detected
[259901.807023] usb 1-3: Qualcomm USB modem converter now attached to ttyUSB0
[259901.814737] qcserial 1-3:1.2: Qualcomm USB modem converter detected
[259901.821325] usb 1-3: Qualcomm USB modem converter now attached to ttyUSB1
[259901.829020] qcserial 1-3:1.3: Qualcomm USB modem converter detected
[259901.835588] usb 1-3: Qualcomm USB modem converter now attached to ttyUSB2
[259901.865397] cdc_mbim 1-3:1.12: cdc-wdm0: USB WDM device
[259901.871307] cdc_mbim 1-3:1.12 wwan0: register 'cdc_mbim' at usb-3530000.xhci-3, CDC MBIM, 32:96:f6:8d:56:45

Modem uptime indicates a modem reset,

Moreover I observe the following crash report on the modem after the reset :

AT!GCDUMP

Src:  FatalError
File: rflte_dispatch.c
Line: 5397
Str:  rflte_dispatch_handle_act_deact failed 
00000000 00000000 00000000 00000000 
Prc:  MPSS
Task: AMSS
Time: 0F6171DF
 R0: 00000000  R1: C0046130  R2: EB102000  R3: 35FBC610  R4: 000000FE
 R5: 00000000  R6: 00000008  R7: 00000001  R8: A55A6BB6  R9: 00000000
R10: 0000047A R11: 00000000 R12: C0053E10 R13: 00000000 R14: C1502EB0
R15: 00000013 R16: C507AD18 R17: 00000000 R18: 00000000 R19: 00000000
R20: C004601C R21: 0000007F R22: 00000000 R23: 00000000 R24: 00000000
R25: 00000000 R26: 35FBC610 R27: C150689C R28: 00000001 SP:  C35B16A8
FP:  C35B16B0 LR:  C009F27C
PC: 156D07E2
CPSR: 00000000
Mod: Unknown
TOS
C35B16D0 156D06D6 0000005A 0000007F 00000001 00000004 C507AD18
00000007 C35B16E0 156D07E2 C471D818 00000007 C35B1750 158CE382
00000000 00000004 00000007 00000000 00000008 00000002 00000000
0000FFFF 35FBC610 C150689C 8C4E3BD2 00000481 00000000 C19DB4D0
060708CD 00000607 0000007F 8C4E3C3F
BOS
App ver: SWI9X50C_01.08.04.00
               

OK

The reset is very arbitrary, it varies from several hours to several days,

I have other identical setups working without issues for more than 4 years, only the geographical position and the telecom operator change,

Any idea how to fix this problem ?

More information on my setup bellow,

Regards

–

HW environnement : EM7565, Linux based computer on module
SW environnement : Linux kernel 4.9, modem is in mbim mode, ModemManager 1.16.8, NetworkManager 1.32.4, libqmi 1.28.6, libmbim 1.26.0

ati

Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.08.04.00 dbb5d0 jenkins 2018/08/21 21:40:11
IMEI: XXX
IMEI SV: 8
FSN: XXX
+GCAP: +CGSM

AT+GMR

SWI9X50C_01.08.04.00 dbb5d0 jenkins 2018/08/21 21:40:11

!GSTATUS

Current Time:  10403		Temperature: 23
Reset Counter: 3		Mode:        ONLINE         
System mode:   LTE        	PS state:    Attached     
LTE band:      B7     		LTE bw:      20 MHz  
LTE Rx chan:   2850		LTE Tx chan: 20850
LTE SSC1 state:NOT ASSIGNED
LTE SSC2 state:NOT ASSIGNED
LTE SSC3 state:NOT ASSIGNED
LTE SSC4 state:NOT ASSIGNED
EMM state:     Registered     	Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  		

PCC RxM RSSI:  -63		PCC RxM RSRP:  -96
PCC RxD RSSI:  -65		PCC RxD RSRP:  -99
Tx Power:      --		TAC:         XXXX (XXXX)
RSRQ (dB):     -13.8		Cell ID:     XXXX (XXXX)
SINR (dB):     12.0

[jyijyi]

Have you tried the latest FW 01.14.13.00

[pierre.pele]

Hello, no I didn’t, I’ve checked the release notes but it’s unclear for me if there are any changes that could solve the issue I’m facing,

Would you recommand to update the modem ?

As the system is in production modem update could be difficult to achieve,

In the past to update the modem we were using a separeted Windows computer,

Unless there is a possibility to update the modem though Linux, for instance using qmicli or mbimcli ?

Regards,

[jyijyi]

Does this help to use qmi-firmware-update?

Updating EM7565 on openwrt

Here’s what I’m doing. root@OpenWrt:~# qmicli -d /dev/cdc-wdm0 -p --dms-set-firmware- preference="01.09.04.00,002.019_000,GENERIC" [/dev/cdc-wdm0] Firmware preference successfully selected You may want to power-cycle the modem now, or just set it offline and reset it: $> sudo qmicli ... --dms-set-operating-mode=offline $> sudo qmicli ... --dms-set-operating-mode=reset After reset, the modem will wait in QDL mode for new firmware. Images to download: 'modem…

[pierre.pele]

The update process is now clearer to me,

Do you have any comment on the crash report ? Before updating the modems I need to be sure the process have a chance to fix the issue,

By the way we have an other identical setup on the same area facing the same issue with a very similar crash report :

AT!GCDUMP
Src:  FatalError
File: rflte_dispatch.c
Line: 5397
Str:  rflte_dispatch_handle_act_deact failed 
00000000 00000000 00000000 00000000 
Prc:  MPSS
Task: AMSS
Time: 00219006
 R0: 00000000  R1: C0046130  R2: EB102000  R3: 52ABC660  R4: 000000FE
 R5: 00000000  R6: 00000006  R7: 00000001  R8: A55A6BB6  R9: 00000000
R10: 00000369 R11: 00000000 R12: C0053E10 R13: 00000000 R14: C1502EB0
R15: 00000013 R16: C507AD18 R17: 00000000 R18: 00000000 R19: 00000000
R20: C004601C R21: 0000007F R22: 00000000 R23: 00000000 R24: 00000000
R25: 00000000 R26: 52ABC660 R27: C150689C R28: 00000001 SP:  C35B16A8
FP:  C35B16B0 LR:  C009F27C
PC: BD6C0270
CPSR: 00000000
Mod: Unknown
TOS
C35B16D0 BD6C0344 0000005A 0000007F 00000001 00000004 C507AD18
00000007 C35B16E0 BD6C0270 C4722778 00000007 C35B1750 BD8DE610
00000000 00000004 00000007 00000000 00000006 00000002 00000000
0000FFFF 52ABC660 C150689C D523F57C 00000009 00000000 C19DC770
060708CD 00000607 0000007F D523F5C7
BOS
App ver: SWI9X50C_01.08.04.00

Waiting for your reply,

Regards,

[jyijyi]

no, i don’t know if the new FW can solve the problem.
Just see if there is some luck on this

If you are using verizon, you can also try the newest FW SWI9X50C_01.14.20.00
https://source.sierrawireless.com/resources/airprime/minicard/75xx/em_mc75xx-approved-fw-packages/#sthash.0IAOmDbB.dpbs

[pierre.pele]

Hello again,

As you suggested, I’ve upgraded the modem to the version 01.14.13.00 using qmi-firmware-update,

Unfortunately, the error still occurs, with a similar crash report :

AT!GCDUMP
Src:  FatalError
File: rflte_dispatch.c
Line: 5397
Str:  rflte_dispatch_handle_act_deact failed 
00000000 00000000 00000000 00000000 
Prc:  MPSS
Task: AMSS
Time: 0053C851
 R0: 00000000  R1: C0046130  R2: EB102000  R3: 67D77590  R4: 000000FE
 R5: 00000000  R6: 00000007  R7: 00000001  R8: A55A6BB6  R9: 00000000
R10: 000003B9 R11: 00000000 R12: C0053E20 R13: 00000000 R14: C1565DC0
R15: 00000013 R16: C51FA134 R17: 00000000 R18: 00000000 R19: 00000000
R20: C004601C R21: 0000007F R22: 00000000 R23: 00000000 R24: 00000000
R25: 00000000 R26: 67D77590 R27: C15697BC R28: 00000001 SP:  C3675758
FP:  C3675760 LR:  C009F27C
PC: 0F5AF743
CPSR: 00000000
Mod: Unknown
TOS
C3675780 0F5AF677 0000005A 0000007F 00000001 00000004 C51FA134
00000007 C3675790 0F5AF743 C489D9D0 00000007 C3675800 0FBFAFAB
00000000 00000002 00000007 00000000 00000007 00000006 00000000
0000FFFF 67D77590 C15697BC 8BA19A71 00000018 00000000 C1A57E60
060708CD 00000607 0000007F 8BA19ACE
BOS
App ver: SWI9X50C_01.14.13.00

For your information the network operator is Proximus in Belgium,

Any idea of how to dolve this ?

Regards,

–

HW environnement : EM7565, Linux based computer on module
SW environnement : Linux kernel 4.9, modem is in mbim mode, ModemManager 1.16.8, NetworkManager 1.32.4, libqmi 1.28.6, libmbim 1.26.0

ati

Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.14.13.00 883709 jenkins 2022/01/05 07:08:35
IMEI: XXX
IMEI SV: 14
FSN: XXX
+GCAP: +CGSM

AT+GMR

SWI9X50C_01.14.13.00 883709 jenkins 2022/01/05 07:08:35

!GSTATUS: 

Current Time:  892		Temperature: 23
Reset Counter: 8		Mode:        ONLINE         
System mode:   LTE        	PS state:    Attached     
LTE band:      B7     		LTE bw:      20 MHz  
LTE Rx chan:   2850		LTE Tx chan: 20850
LTE SCC1 state:NOT ASSIGNED
LTE SCC2 state:NOT ASSIGNED
LTE SCC3 state:NOT ASSIGNED
LTE SCC4 state:NOT ASSIGNED
EMM state:     Registered     	Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  		

PCC RxM RSSI:  -62		PCC RxM RSRP:  -98
PCC RxD RSSI:  -64		PCC RxD RSRP:  -101
Tx Power:      --		TAC:         XXXX (XXXX)
RSRQ (dB):     -15.7		Cell ID:     XXXX (XXXX)
SINR (dB):      6.8

[jyijyi]

Can you try with newest FW SWI9X50C_01.14.20.00 with VZW PRI and see if you can register network after changing the APN in AT+CGDCONT?

[pierre.pele]

At first sight this sounds complicated to execute as I don’t have physical access to the system,

I will first try on a identical system I have in our office and will come back to you,

[pierre.pele]

I successfully updated on of my test setup to the FW SWI9X50C_01.14.20.00 with VZW PRI, after what I reconfigured the APN in order to connect the modem to the internet,

However the modem is connected to UMTS only (no LTE connection) with very poor bandwith,

Thus I’m not confident in updating my production systems with this release, unless you provide me with more informations on how this FW update could help in solving the issue,

Waiting for your answers,

Regards,

[jyijyi]

which carrier PRI are you using before?
GENERIC?
I can put here to let you have a test

As said before, I don’t know if it can fix or not, just doing trial and error with new FW.

[pierre.pele]

Yes I’m using GENERIC,

[jyijyi]

i have sent to you in mailbox

[pierre.pele]

I’ve tested the PRI you sent me on my testing setup and am curretnly updating my production setup, will come back later to you with results,

Regards,

[pierre.pele]

I’ve upgraded the modem to the version 01.14.20.00,

Unfortunately, the error still occurs, modem is getting reseted in a arbitrary manner (2 reset in the last 2 hours),

[jyijyi]

then probably you need to contact distributor for further debugging (including getting some RAM dump after crash)