EM9191 firmware crashed upon 4G/5G registration

adrien.pompee · April 18, 2024, 10:13am

Hello,
I have an issue with an EM9191 modem through PCIe interface. The communication with the modem works fine via QMI, MBIM or AT commands if the SIM PIN is not entered or the antennas not connected. Upon network registration, the firmware crashes with messages in dmesg like :

mhi-pci-generic 0000:01:00.0: firmware crashed (7)
mhi mhi0: Did not enter M3 state, MHI state: M0, PM state: SYS ERROR Detect
mhi-pci-generic 0000:01:00.0: failed to suspend device: -5
mhi-pci-generic 0000:01:00.0: firmware crashed (6)

I upgraded the modem firmware to SWIX55C_03.14.10.01. I tried different linux kernel versions. And I still have the same error messages.

I just looked at the modem dump error message, here is the report :

AT!GCDUMP

Src:  FatalError
File: lte_LL1_resource_cfg.c
Line: 988
Str:  Device FUSE doesn't support more than -801039208 layers in FW
0000000A 47FFAE8A 0000000A 00000000 
Prc:  MPSS
Task: NONE
Time: 0004BB79
 R0: 00000000  R1: 00000000  R2: 00000000  R3: 00000000  R4: 00000000
 R5: 00000000  R6: 00000000  R7: 00000000  R8: 00000000  R9: 00000000
R10: 00000000 R11: 00000000 R12: 00000000 R13: 00000000 R14: 00000000
R15: 00000000 R16: 00000000 R17: 00000000 R18: 00000000 R19: 00000000
R20: 00000000 R21: 00000000 R22: 00000000 R23: 00000000 R24: 00000000
R25: 00000000 R26: 00000000 R27: 00000000 R28: 00000000 SP:  C9AF4370
FP:  C9AF4380 LR:  D9B9D170
PC: 56A8740B
CPSR: 00000000
Mod: Unknown
Ctr: ARM, IRQ dis,FIQ dis

TOS
C9AF4390 4EC1D33F 00FE8F1F 00003800 C9AF43D0 56A8740B C8F84444
C8F84466 00000000 00000000 00000000 CF800000 62E18BBC C8F84438
62E18CC9 C9AEA3E8 00003800 0000000E C8F8454C EB1A8050 C9AF43E8
4EEB40B3 C08227C0 00000000 C9AEA3BC C9AEA3B8 C9AF4480 56F4A983
C9B15350 C9AE8D40 C9AE8D44 00000000
BOS
App ver: SWIX55C_03.14.10.01

Src:  FatalError
Str:  Internal error:
00000000 00000000 00000000 00000000 
Prc:  APSS
Task: 
Time: 00000000
 R0: 00000000  R1: 00000000  R2: 00000000  R3: 00000000  R4: 00000000
 R5: 00000000  R6: 00000000  R7: 00000000  R8: 00000000  R9: 00000000
R10: 00000000 R11: 00000000 R12: 00000000 R13: 00000000 R14: 00000000
PC: 00000000
CPSR: 00000000
Mod: Unknown
Ctr: ARM, IRQ dis,FIQ dis

TOS
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
BOS
  262.807842] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  262.814421] ipa3_dma_disable: 11 callbacks suppressed
<3>[  262.814437] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  262.835894] ipa3_dma_disable: 1 callbacks suppressed
<3>[  262.835902] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  268.991727] ipa3_dma_enable: 11 callbacks suppressed
<3>[  268.991752] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  268.998232] ipa3_dma_disable: 11 callbacks suppressed
<3>[  268.998250] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  269.020775] ipa3_dma_disable: 1 callbacks suppressed
<3>[  269.020782] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  274.762944] ipa3_dma_enable: 11 callbacks suppressed
<3>[  274.762966] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  274.769423] ipa3_dma_disable: 11 callbacks suppressed
<3>[  274.769437] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  274.792587] ipa3_dma_disable: 1 callbacks suppressed
<3>[  274.792594] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  280.852678] ipa3_dma_enable: 11 callbacks suppressed
<3>[  280.852703] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  280.859191] ipa3_dma_disable: 11 callbacks suppressed
<3>[  280.859207] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  280.881186] ipa3_dma_disable: 1 callbacks suppressed
<3>[  280.881193] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  287.227182] ipa3_dma_enable: 11 callbacks suppressed
<3>[  287.227204] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  287.233754] ipa3_dma_disable: 11 callbacks suppressed
<3>[  287.233768] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  287.254302] ipa3_dma_disable: 1 callbacks suppressed
<3>[  287.254310] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  293.447219] ipa3_dma_enable: 11 callbacks suppressed
<3>[  293.447244] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  293.453885] ipa3_dma_disable: 11 callbacks suppressed
<3>[  293.453901] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  293.474364] ipa3_dma_disable: 1 callbacks suppressed
<3>[  293.474372] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<4>[  299.563057] ipa3_dma_enable: 11 callbacks suppressed
<3>[  299.563081] ipa ipa3_dma_enable:426 Already enabled refcnt=1
<4>[  299.569561] ipa3_dma_disable: 11 callbacks suppressed
<3>[  299.569577] ipa ipa3_dma_disable:485 Multiple enablement done. refcnt=2
<4>[  299.591051] ipa3_dma_disable: 1 callbacks suppressed
<3>[  299.591058] ipa ipa3_dma_disable:497 There is pending work, can't disable.
<3>[  306.897367] Fatal error on modem!
<3>[  306.897503] modem subsystem failure reason: lte_LL1_resource_cfg.c:988:Device FUSE doesn't support more than 10 layers in FW.
<6>[  306.899762] subsys-restart: subsystem_restart_dev(): Restart sequence requested for modem, restart_level = SYSTEM.
<3>[  306.914518] Ramdump(ramdump_microdump_modem): No consumers. Aborting..
<6>[  306.922591] microdump_modem_notifier_nb: do_ramdump() failed
<0>[  307.030530] Kernel panic - not syncing: subsys-restart: Resetting the SoC - modem crashed.
<4>[  307.030633] CPU: 0 PID: 1271 Comm: kworker/0:1 Not tainted 4.14.206-perf #1
<4>[  307.037733] Hardware name: Qualcomm Technologies, Inc. SDXPRAIRIE (Flattened Device Tree)
<4>[  307.044634] Workqueue: events device_restart_work_hdlr
<4>[  307.052957] [<c010e658>] (unwind_backtrace) from [<c010b3e4>] (show_stack+0x10/0x14)
<4>[  307.057986] [<c010b3e4>] (show_stack) from [<c0121498>] (panic+0x180/0x3b4)
<4>[  307.065884] [<c0121498>] (panic) from [<c0454df4>] (subsys_remove_restart_order+0x0/0x80)
<4>[  307.072571] [<c0454df4>] (subsys_remove_restart_order) from [<ca4b8800>] (0xca4b8800)
<3>[  307.087822] ipa ipa3_active_clients_panic_notifier:300 
<3>[  307.087822] ---- Active Clients Table ----
<3>[  307.087822] 
<3>[  307.087822] Total active clients count: 3
<3>[  307.087822]

Should I downgrade the modem firmware version?

jyijyi · April 18, 2024, 3:06pm

what is the current value for AT!RFCID? and AT!HWID?

adrien.pompee · April 18, 2024, 3:18pm

AT!RFCID?
!RFCID: 
CMW_HWID: 1003
CMW_BID : 0
MMW_HWID: 0
MMW_BID : 0

OK
AT!HWID?
Revision: A


OK

jyijyi · April 18, 2024, 3:39pm

is there any improvement if you set to

AT!ENTERCND="A710"
at!rfcid=1005,0
at!RESET

adrien.pompee · April 18, 2024, 4:06pm

The firmware doesn’t crash but I have no signal at all.

AT+CSQ
+CSQ: 99,99

OK

jyijyi · April 18, 2024, 4:08pm

How many antennas have you connected?
what is the return of the following?

AT+cfun?
AT!gstatus?
AT+cpin?
at!SELRAT?
AT!BAND?
at!PCINFO?

adrien.pompee · April 18, 2024, 4:10pm

4 antennas connected

AT+CFUN?
+CFUN: 7

OK
AT!GSTATUS?
!GSTATUS: 
Current Time:  435              Temperature: 41
Thermal Mitigation Level: 0
Reset Counter: 2                Mode:        OFFLINE        
System mode:   CDMA             PS state:    Not attached 
Unknown System Mode

OK
AT+CPIN?
+CPIN: READY

OK
AT!SELRAT?
!SELRAT: 06, LTE Only


OK
AT!BAND?
Unknown band mask. Use AT!BAND to set band. 
0 - GW:    0000000000000000
1 - LTE:   0000000000000000 0000000000000000 0000000000000000 0000000000000000
3 - NRNSA: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000
4 - NRSA:  0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000

OK
AT!PCINFO?
State: Offline
LPM voters - Temp:0, Volt:0, User:0, W_DISABLE:0, IMSWITCH:0, BIOS:0, LWM2M:0, OMADM:0, FOTA:0, NVCRIT:0, RFCAL:0, MMWCAL:0, RFC_INIT:1
LPM persistence - None

OK

jyijyi · April 18, 2024, 4:16pm

is it the same for RFCID 1005 /1006/1007/1008?

Actually your module is quite old which seems to be engineering sample
Do you have a new module to test?

adrien.pompee · April 18, 2024, 4:38pm

1005 and 1006 have the same behavior.
1003 and 1007 have the same behavior.
1008 seems to work for now in LTE, I will try to make it work in 5GNR.

Yes, the module is old. We purchased 2 new modules that have yet to be delivered.

Topic		Replies	Views
MC7710 Fatal Error MC/EM Series	1	604	August 13, 2015
MC7455 crash when LTE channel conditions are bad. MC/EM Series	2	1818	January 26, 2018
MC7455 no registration after firmware update MC/EM Series	2	743	October 27, 2021
EM9191 in NSA mode MC/EM Series	25	1169	December 27, 2022
MC7455 - Works properly a few minutes and deregistered IoT Modules	0	451	June 26, 2019

EM9191 firmware crashed upon 4G/5G registration

Related Topics