EM7421 failed in NETDEV WATCHDOG: wwan0 (cdc_mbim): transmit queue 0 timed out

Noticed crash in driver in EM7421 module.
It works fine for few days before noticing crash in dmesg logs. watchdog timeout with the transmit queue timed out, and the modem stops responding to AT channel too. Only recovery is router reboot.

[425494.194583] ------------[ cut here ]------------
[425494.194590] NETDEV WATCHDOG: wwan0 (cdc_mbim): transmit queue 0 timed out
[425494.194630] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x27f/0x290
[425494.194631] Modules linked in: esp4 sch_sfq sch_hfsc xt_comment xt_MASQUERADE bridge stp llc ipt_REJECT nf_reject_ipv4 xt_policy iptable_nat xt_mark iptable_mangle xt_hashlimit xt_tcpudp xt_CT iptable_raw xt_multiport xt_conntrack nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nf_nat_tftp nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_amanda iptable_filter ip_tables x_tables xfrm_user nfnetlink qat_c3xxx(O) intel_qat(O) 8021q ppp_generic slhc vport_vxlan(O) openvswitch(O) nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 dummy fuse x86_pkg_temp_thermal efivars cdc_mbim cdc_ncm usbnet i40e(O) ixgbe(O) igb(O)
[425494.194687] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 5.4.27-yocto-standard #1
[425494.194689] Hardware name: <>, BIOS 0035.25LS 05/13/2021
[425494.194693] RIP: 0010:dev_watchdog+0x27f/0x290
[425494.194697] Code: 27 09 57 ff eb 88 4c 89 ff c6 05 e5 0e cd 00 01 e8 66 2d fb ff 44 89 e9 4c 89 fe 48 c7 c7 c8 a9 6f a2 48 89 c2 e8 1c d6 62 ff <0f> 0b e9 66 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[425494.194699] RSP: 0018:ffffaf7280178e38 EFLAGS: 00010282
[425494.194701] RAX: 0000000000000000 RBX: ffffa3f8aefa3a00 RCX: 0000000000000338
[425494.194703] RDX: 0000000000000001 RSI: 0000000000000096 RDI: ffffffffa2b6c40c
[425494.194705] RBP: ffffaf7280178e68 R08: 0000000000000001 R09: 0000000000000338
[425494.194707] R10: 00000000000102b8 R11: 0000000000000004 R12: ffffa3f8ae2eb440
[425494.194709] R13: 0000000000000000 R14: ffffa3f8aefa3a80 R15: ffffa3f8ae2eb000
[425494.194711] FS: 0000000000000000(0000) GS:ffffa3f8bb880000(0000) knlGS:0000000000000000
[425494.194713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[425494.194715] CR2: 00007fb41fad8810 CR3: 000000012900a000 CR4: 00000000003406e0
[425494.194716] Call Trace:
[425494.194720]
[425494.194727] ? qdisc_put_unlocked+0x40/0x40
[425494.194732] call_timer_fn+0x37/0x170
[425494.194736] run_timer_softirq+0x1a5/0x510
[425494.194739] ? qdisc_put_unlocked+0x40/0x40
[425494.194743] ? tick_sched_handle+0x38/0x50
[425494.194746] ? tick_sched_timer+0x3d/0x80
[425494.194751] ? recalibrate_cpu_khz+0x10/0x10
[425494.194754] ? ktime_get+0x3e/0xa0
[425494.194760] __do_softirq+0xe3/0x314
[425494.194764] ? tick_program_event+0x41/0x80
[425494.194766] ? __hrtimer_get_next_event+0x4d/0x80
[425494.194771] irq_exit+0xa8/0xb0
[425494.194775] smp_apic_timer_interrupt+0x7d/0x180
[425494.194779] apic_timer_interrupt+0xf/0x20
[425494.194781]
[425494.194785] RIP: 0010:cpuidle_enter_state+0xc4/0x460
[425494.194788] Code: 85 c0 0f 8f e0 02 00 00 31 ff e8 27 c0 74 ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 74 03 00 00 31 ff e8 f0 68 79 ff fb 45 85 e4 <0f> 88 72 01 00 00 4c 2b 75 d0 49 63 cc 48 8d 04 49 48 c1 e0 05 8b
[425494.194789] RSP: 0018:ffffaf72800a7e38 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[425494.194792] RAX: ffffa3f8bb8a8b00 RBX: ffffffffa28edde0 RCX: 000000000000001f
[425494.194794] RDX: 0000000000000000 RSI: 0000000062762762 RDI: 0000000000000000
[425494.194796] RBP: ffffaf72800a7e78 R08: 000182fc16523b4e R09: 0000000000000001
[425494.194798] R10: ffffa3f8bb8a7ae0 R11: ffffa3f8bb8a7ac0 R12: 0000000000000003
[425494.194799] R13: ffffa3f8bb8b1ea0 R14: 000182fc16523b4e R15: 0000000000000000
[425494.194805] ? cpuidle_enter_state+0xa9/0x460
[425494.194808] cpuidle_enter+0x2e/0x40
[425494.194812] call_cpuidle+0x23/0x40
[425494.194815] do_idle+0x1c3/0x230
[425494.194818] cpu_startup_entry+0x20/0x30
[425494.194822] start_secondary+0x14e/0x180
[425494.194826] secondary_startup_64+0xa4/0xb0

Looks like issue is with driver/firmware.
any pointers to debug this issue?

would you consider to use RMNET (set in AT!USBCOMP) instead of MBIM?

if so, you can see here: