EM9191 reboots - UNKOWN reason

IoT Modules MC/EM Series
1

I was using EM9191 for quite some time with DEV KIT. Did not experience problems. However, as of today device started behaving weird in sense that it reboots “itself” for no known reason. Interestingly enough I’m able to use the device normally (e.g. attach to network) till the point it reboots “itself”. EM9191 FW version: 03.09.11.00_GENERIC.

Question: Any idea on why it might be happening or what should I check in order to figure it out?

I’m attaching output of dmesg:

[95216.788200] usb 4-1: new SuperSpeed USB device number 32 using xhci_hcd
[95216.819362] usb 4-1: config 1 has an invalid interface number: 4 but max is 3
[95216.819395] usb 4-1: config 1 has no interface number 2
[95216.821739] usb 4-1: New USB device found, idVendor=1199, idProduct=90d3, bcdDevice= 0.06
[95216.821812] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[95216.821950] usb 4-1: Product: Sierra Wireless EM9191
[95216.821994] usb 4-1: Manufacturer: Sierra Wireless, Incorporated
[95216.822007] usb 4-1: SerialNumber: 8W1164011203A121
[95216.950151] cdc_mbim 4-1:1.0: setting rx_max = 16384
[95216.952404] cdc_mbim 4-1:1.0: cdc-wdm0: USB WDM device
[95216.955140] cdc_mbim 4-1:1.0 wwan0: register 'cdc_mbim' at usb-0000:03:00.0-1, CDC MBIM, ce:d2:8a:93:a2:77
[95260.296042] usb 4-1: USB disconnect, device number 32
[95260.296227] cdc_mbim 4-1:1.0 wwan0: unregister 'cdc_mbim' usb-0000:03:00.0-1, CDC MBIM
[95274.410511] usb 4-1: new SuperSpeed USB device number 33 using xhci_hcd
[95274.439598] usb 4-1: config 1 has an invalid interface number: 4 but max is 3
[95274.439611] usb 4-1: config 1 has no interface number 2
[95274.442314] usb 4-1: New USB device found, idVendor=1199, idProduct=90d3, bcdDevice= 0.06
[95274.442325] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[95274.442358] usb 4-1: Product: Sierra Wireless EM9191
[95274.442367] usb 4-1: Manufacturer: Sierra Wireless, Incorporated
[95274.442375] usb 4-1: SerialNumber: 8W1164011203A121
[95274.585864] cdc_mbim 4-1:1.0: setting rx_max = 16384
[95274.588635] cdc_mbim 4-1:1.0: cdc-wdm0: USB WDM device
[95274.591245] cdc_mbim 4-1:1.0 wwan0: register 'cdc_mbim' at usb-0000:03:00.0-1, CDC MBIM, ce:d2:8a:93:a2:77
[95318.510072] usb 4-1: USB disconnect, device number 33
[95318.510179] cdc_mbim 4-1:1.0 wwan0: unregister 'cdc_mbim' usb-0000:03:00.0-1, CDC MBIM
[95319.873863] usb 4-1: new SuperSpeed USB device number 34 using xhci_hcd
[95319.900468] usb 4-1: New USB device found, idVendor=1199, idProduct=90d2, bcdDevice= 0.00
[95319.900481] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[95319.900490] usb 4-1: Product: Sierra Wireless EM9191
[95319.900498] usb 4-1: Manufacturer: Sierra Wireless, Incorporated
[95319.900506] usb 4-1: SerialNumber: 8W1164011203A121
[95319.964326] usbcore: registered new interface driver qcserial
[95319.964361] usbserial: USB Serial support registered for Qualcomm USB modem
[95319.964432] qcserial 4-1:1.0: Qualcomm USB modem converter detected
[95319.966270] usb 4-1: Qualcomm USB modem converter now attached to ttyUSB0
2

You can check at!gcdump
Does it happen to 5G only?

3

My current ISP does not support 5G. Usually was connected to 4G.

Outpout:

at!gcdump

Src:  FatalError
File: lte_ml1_common_dump.c
Line: 473
Str:  CPHY_START Deadlock - resp_mod:0xD0411BC8; upr_lyr:0x0; wait_mask:0x
00000080 040C0101 00000080 00000000 
Prc:  MPSS
Task: NONE
Time: 0000C08C
 R0: 00000000  R1: 00000000  R2: 00000000  R3: 00000000  R4: 00000000
 R5: 00000000  R6: 00000000  R7: 00000000  R8: 00000000  R9: 00000000
R10: 00000000 R11: 00000000 R12: 00000000 R13: 00000000 R14: 00000000
R15: 00000000 R16: 00000000 R17: 00000000 R18: 00000000 R19: 00000000
R20: 00000000 R21: 00000000 R22: 00000000 R23: 00000000 R24: 00000000
R25: 00000000 R26: 00000000 R27: 00000000 R28: 00000000 SP:  C083B4D8
FP:  C083B528 LR:  D8E30A9C
PC: C31D84E2
CPSR: 00000000
Mod: Unknown
Ctr: ARM, IRQ dis,FIQ dis

TOS
C083B540 DB773C26 00000018 CB283310 CB283300 00000000 C083B848
C31D84E2 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5
A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5
A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5
A5A5A5A5 A5A5A5A5 A5A5A5A5 A5A5A5A5
BOS
App ver: SWIX55C_03.09.11.00

Src:  FatalError
Str:  Internal error:
00000000 00000000 00000000 00000000 
Prc:  APSS
Task: 
Time: 00000000
 R0: 00000000  R1: 00000000  R2: 00000000  R3: 00000000  R4: 00000000
 R5: 00000000  R6: 00000000  R7: 00000000  R8: 00000000  R9: 00000000
R10: 00000000 R11: 00000000 R12: 00000000 R13: 00000000 R14: 00000000
PC: 00000000
CPSR: 00000000
Mod: Unknown
Ctr: ARM, IRQ dis,FIQ dis

TOS
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
BOS
k depth: 4836 bytes left
<5>[   18.005528] audit: type=1325 audit(315964801.579:2): table=mangle family=2 entries=6
<5>[   18.049418] audit: type=1325 audit(315964801.619:3): table=mangle family=2 entries=8
<5>[   18.071685] audit: type=1325 audit(315964801.649:4): table=mangle family=2 entries=9
<5>[   18.088630] audit: type=1325 audit(315964801.659:5): table=mangle family=2 entries=11
<3>[   18.341396] ipa-wan ipa3_wwan_ioctl:2045 dev(rmnet_data0) register to IPA
<12>[   19.954099] syslog: Starting syslogd/klogd: 
<12>[   20.191754] power_config: Starting powerconfig for SDX55: 
<6>[   20.295832] zram0: detected capacity change from 0 to 104857600
<6>[   20.594480] Adding 102396k swap on /dev/zram0.  Priority:-2 extents:1 across:102396k SS
<12>[   20.595151] ++++ /etc/initscripts/power_config -> ENABLE-FTRACE START
<6>[   20.683620] coresight-tmc 6048000.tmc: TMC-ETR enabled
<6>[   20.683801] coresight-dynamic-funnel 6b08000.funnel: FUNNEL inport 7 enabled
<6>[   20.683825] coresight-dynamic-funnel 6045000.funnel: FUNNEL inport 0 enabled
<6>[   20.683848] coresight-dynamic-funnel 6041000.funnel: FUNNEL inport 7 enabled
<6>[   20.683928] coresight-stm 6002000.stm: STM tracing enabled
<12>[   20.686320] ++++ /etc/initscripts/power_config -> ENABLE-FTRACE END
<12>[   20.686420] ++++ /etc/initscripts/power_config -> ENABLE-DCC START
<3>[   20.688011] msm-dcc 10a2000.dcc_v2: DCC list passed 2
<6>[   20.688048] msm-dcc 10a2000.dcc_v2: All values written to enable
<12>[   20.688149] ++++ /etc/initscripts/power_config -> ENABLE-DCC END
<12>[   20.688242] power_config: done
<12>[   20.737667] syslog: done
<4>[   22.065658] swi_netlink_data_ready: receive user pid:870, msg_cached:0
<6>[   24.102240] android_work: sent uevent USB_STATE=CONNECTED
<6>[   24.376703] android_work: sent uevent USB_STATE=DISCONNECTED
<6>[   24.384413] android_work: sent uevent USB_STATE=CONNECTED
<6>[   24.550567] configfs-gadget gadget: super-speed config #1: c
<6>[   24.550750] diag: USB channel diag: Received Connect event
<6>[   24.551990] android_work: sent uevent USB_STATE=CONFIGURED
<3>[   24.595353] id3:ipa_usb_notify_cb: Set net_ready_trigger
<3>[   46.910332] Fatal error on modem!
<3>[   46.914887] modem subsystem failure reason: lte_ml1_common_dump.c:473:CPHY_START Deadlock - resp_mod:0x80; upr_lyr:0x40C0101; wait_mask:0x80.
<6>[   46.918110] subsys-restart: subsystem_restart_dev(): Restart sequence requested for modem, restart_level = SYSTEM.
<3>[   46.928748] Ramdump(ramdump_microdump_modem): No consumers. Aborting..
<6>[   46.935895] microdump_modem_notifier_nb: do_ramdump() failed
<0>[   47.040818] Kernel panic - not syncing: subsys-restart: Resetting the SoC - modem crashed.
<4>[   47.040954] CPU: 0 PID: 14 Comm: kworker/0:1 Tainted: G        W       4.14.206 #1
<4>[   47.048056] Hardware name: Qualcomm Technologies, Inc. SDXPRAIRIE (Flattened Device Tree)
<4>[   47.055655] Workqueue: events device_restart_work_hdlr
<4>[   47.063888] [<c0110edc>] (unwind_backtrace) from [<c010ce58>] (show_stack+0x1c/0x20)
<4>[   47.068920] [<c010ce58>] (show_stack) from [<c0c0bae8>] (dump_stack+0x20/0x24)
<4>[   47.076818] [<c0c0bae8>] (dump_stack) from [<c0126318>] (panic+0x18c/0x3cc)
<4>[   47.083847] [<c0126318>] (panic) from [<c04c7a60>] (subsys_remove_restart_order+0x0/0x88)
<4>[   47.090716] [<c04c7a60>] (subsys_remove_restart_order) from [<c014331c>] (process_one_work+0x1a8/0x47c)
<4>[   47.099050] [<c014331c>] (process_one_work) from [<c01439d8>] (worker_thread+0x384/0x4f0)
<4>[   47.108241] [<c01439d8>] (worker_thread) from [<c0148554>] (kthread+0x158/0x160)
<4>[   47.116573] [<c0148554>] (kthread) from [<c0108654>] (ret_from_fork+0x14/0x20)
<3>[   47.132635] ipa ipa3_active_clients_panic_notifier:300 
<3>[   47.132635] ---- Active Clients Table ----
<3>[   47.132635] FREEZE_VOTE                              1   SPECIAL
<3>[   47.132635] USB                                      1   SPECIAL
<3>[   47.132635] TAG_PROCESS                              -7  SPECIAL
<3>[   47.132635] 
<3>[   47.132635] Total active clients count: 2
<3>[   47.132635] 

OK
4

I think you need to contact distributor on this issue if you want to know the root cause.

BTW, if you disable LTE CA on AT!RATCA, would it have improvement?

It is strange that suddenly it has problem, have you tried another SIM card on another operator?

5

The rebooting/restarting of EM9191 stopped. What I did (but don’t know if it is related) placed the SIM card of 2nd ISP (however the 2nd ISP uses network infrastructure from 1st ISP as it is sister company) and it worked just fine. And at!gcdump gives: No crash data available. Afterwards I switched back to 1st SIM card and it also worked just fine.

I don’t know what may be cause of problem . It may be related to doing attach/detach too often. If anyone can confirm it or how to check this I would preform activity on my side so we can keep track of cause. Any help most welcome.

And thank you @jyijyi for support!