EM7590 AT command ports occasionally doesn't respond at all

IoT Modules MC/EM Series
1

Hi,

We’re facing some weird behaviour with our EM7590, where occasionally, from a cold boot, the AT ports don’t seem to respond to any AT commands. We verified that no other processes are acquiring the port, and using picocom, we cannot send any commands to the AT port.

Note, we are using balena OS built using Yocto.

Attached is the mmcli output for the detected device.

  -----------------------------------
  General  |                    path: /org/freedesktop/ModemManager1/Modem/0
           |               device id: <>
  -----------------------------------
  Hardware |            manufacturer: Sierra Wireless, Incorporated
           |                   model: Sierra Wireless EM7590 Mobile Broadband Adapter
           |       firmware revision: SWIX12C_02.02.03.00
           |          carrier config: ROW_Generic_3GPP
           | carrier config revision: 08010811
           |            h/w revision: EM7590
           |               supported: gsm-umts, lte
           |                 current: gsm-umts, lte
           |            equipment id: <>
  -----------------------------------
  System   |                  device: /sys/devices/platform/bus@0/3610000.usb/usb2/2-1
           |                 physdev: /sys/devices/platform/bus@0/3610000.usb/usb2/2-1
           |                 drivers: cdc_mbim, qcserial
           |                  plugin: sierra
           |            primary port: cdc-wdm0
           |                   ports: cdc-wdm0 (mbim), wwan0 (net)
  -----------------------------------
  Status   |                   state: failed
           |           failed reason: sim-missing
           |             power state: on
  -----------------------------------
  Modes    |               supported: allowed: 3g; preferred: none
           |                          allowed: 4g; preferred: none
           |                          allowed: 3g, 4g; preferred: 4g
           |                          allowed: 3g, 4g; preferred: 3g
           |                 current: allowed: 3g, 4g; preferred: 4g
  -----------------------------------
  Bands    |               supported: utran-1, utran-4, utran-6, utran-5, utran-8, utran-9,
           |                          utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, eutran-7,
           |                          eutran-8, eutran-12, eutran-13, eutran-14, eutran-18, eutran-19,
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-29, eutran-32,
           |                          eutran-38, eutran-39, eutran-40, eutran-41, eutran-42, eutran-43,
           |                          eutran-48, eutran-66, eutran-71, utran-19
           |                 current: utran-1, utran-4, utran-6, utran-5, utran-8, utran-9,
           |                          utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, eutran-7,
           |                          eutran-8, eutran-12, eutran-13, eutran-14, eutran-18, eutran-19,
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-29, eutran-32,
           |                          eutran-38, eutran-39, eutran-40, eutran-41, eutran-42, eutran-43,
           |                          eutran-48, eutran-66, eutran-71, utran-19
  -----------------------------------
  IP       |               supported: ipv4, ipv6, ipv4v6
  -----------------------------------
  3GPP     |                    imei: <>
  -----------------------------------
  SIM      |          sim slot paths: slot 1: none
           |                          slot 2: none (active)

Verified that the device is in the correct MBIM mode

Bus 002 Device 002: ID 1199:c081 Sierra Wireless, Incorporated Sierra Wireless EM7590 Mobile Broadband Adapter

Any idea what it could be?

2

have you tried in Windows or other Linux PC to see if this is hardware issue of your environment?

3

Hey @jyijyi,

We haven’t because the module comes with the hardware we bought. It was weird that it only occurred randomly, and from dmesg, we did not notice any issues.

What we found out was when the AT command doesn’t respond from that cold boot, doing a reboot of the OS then recovers the AT command port.

4

is it only the AT port is affected?
How about MBIM interface?
Is it 100% working?

5

Yes, it does seems that only AT port is affected, because Modem Manager detects the modem every time and was able to create the wwan0 interface, which is done via MBIM interface.

mmcli always shows it detect the mbim interface as well

6

maybe you can try to use the MBPL USB driver:

7

we are using the qcserial driver, is there potentially an incompatibility between the EM7590 and the qcserial driver from Linux?

8

i saw the MBPL is also using qcserial
maybe you can backup the original qcserial binary

9

some more info,

it seems like this behaviour is only happening a subset of devices. On some device the AT ports always works. All of them have the same OS and firmware version on the modem.

10

then it could be hardware issue due to instability …

11

instability of power or the bring up of the module?

12

no idea, as said before, you need to check with other environment like windows or linux PC

13

Let me do more test on different environment and report back. Thanks!

14

@jyijyi so after testing with different OS (Balena OS and Nvidia Jetpack 6.2 based on Ubuntu 22.04) on the same hardware, we are getting similar behaviour where occasionally the AT command ports would fail to respond to anything, even though the modem is working and communicating via MBIM.

We found that by unbinding the /dev/ttyUSBX ports with qcserial via echo "2-1:1.3" | sudo tee /sys/bus/usb/drivers/qcserial/unbind , then doing a reset via mmcli -m 0 -r, that causes the modem to reset, rebinds the /dev/ttyUSB port with qcserial, then the modem responds with properly on the AT ports.

If we just unbind and bind the ports from qcserial, that doesn’t resolve it. We also need to reset the modem. This seems to imply that the modem itself is not responding on the AT ports, rather than something with the OS.

We have a batch of 200+ devices, and we are noticing this issue when doing mass testing on 10+ devices. Could this be a hardware/ firmware issue on the modem side?

15

Instead of using same hardware, is it possible to route the USB cable to Windows PC or Linux PC?

maybe you can try to use the MBPL USB driver:

16

I am following this. Yesterday, I noticed a new version of EM7590 firmware is available. I do not know if this was fixed. New suggestion - try the new module firmware version on one module and see if it helps.