EM7565 Firmware Update Troubleshooting (FW Mismatch and and Low Power Mode)

IoT Modules MC/EM Series
1

Hello,

My organization has many EM7565 modules deployed already. We have encountered issues where the EM7565 gets stuck in the bootloader, which we have found is recoverable by reflashing the firmware, so we modified our software to perform this procedure for reliable operation of the EM7565. However, we have had issues flashing some of our modems. Please advise how we can achieve this, we’ve tried many avenues detailed below.

We have more recently found that some of the (more recent?) EM7565s refuse to flash with the firmware version we use (01.08.04) due to a “Package not compatible” error, seemingly due to a difference in the flash IC.

Our testing shows:

  • EM7565 with internal flash IC KSLCMBL2VA2M2C flash as expected
  • EM7565 with internal flash IC H27S4G8F2EKPB4 refuse to flash with a “Package not compatible” error

In order to work around the issue with the H27S4G8F2EKPB4 and 01.08.04, we tested the newer firmware versions and found the following issues:

SWI9X50C_01.14.22.00

This firmware fails to flash using the latest fwdwl-lite version (1.0.2307.1) due to apparent SDK errors.

The following is sent to syslog (not the logfile) before the process exits with an error:

user.debug fwdwl-lite: fmsExtractParameter: File Name: /my-fw-path/SWI9X50C_01.14.22.00.cwe
user.debug fwdwl-lite: fmsExtractParameter: File Name: /my-fw-path/SWI9X50C_01.14.22.00.cwe
user.debug fwdwl-lite: fmsExtractParameter: File Name: /my-fw-path/SWI9X50C_01.14.22.00_GENERIC_002.059_001.nvu
user.debug fwdwl-lite: swi_ossdk_mmapro: File mapping failed
user.debug fwdwl-lite: fmsExtractParameter: File Name: /my-fw-path/SWI9X50C_01.14.22.00.cwe
user.debug fwdwl-lite: fmsExtractParameter: File Name: /my-fw-path/SWI9X50C_01.14.22.00_GENERIC_002.059_001.nvu
user.debug fwdwl-lite: swi_ossdk_readdir

SWI9X50C_01.14.13.00 + SWI9X50C_01.14.02.00

The modem accepts the firmware update, then gets stuck in low power mode. The EM7565 showed via AT!IMPREF? the preferred version was the previous fw version with generic configuration. Attemping to run AT!IMPREF="GENERIC" returned an error and caused the AT command port to hang for ~30 seconds. It appears from the below that the device is very confused about it’s loaded FW and PRIs, stuck using the new FW and old PRI.

I ran AT!IMPREF="AUTOSIM" and how have the following state:

AT!IMAGE?
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
FW   1    EMPTY  0   0 0
FW   2    EMPTY  0   0 0
FW   3    EMPTY  0   0 0
FW   4    EMPTY  0   0 0
Max FW images: 4
Active FW image is at slot 255

TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
Max PRI images: 50


OK

AT!IMPREF?
!IMPREF:
 preferred fw version:    00.00.00.00
 preferred carrier name:  AUTO-SIM
 preferred config name:   AUTO-SIM_000.000_000
 preferred subpri index:  000
 current fw version:      01.14.02.00
 current carrier name:    GENERIC
 current config name:     GENERIC_002.011_002
 current subpri index:    000

 fw version mismatch

OK

AT!HWID?
Revision: 1.0


OK

AT!PRIID?
PRI Part Number: 9908685
Revision: 002.006
Customer: Generic-M2M

Carrier PRI: None

OK

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.14.02.00 2e210b jenkins 2020/08/19 14:18:39
IMEI: <redacted>
IMEI SV: 12
FSN: <REDACTED>
+GCAP: +CGSM


OK

AT!PCINFO?
State: Low Power Mode
LPM voters - Temp:0, Volt:0, User:0, W_DISABLE:0, IMSWITCH:1, BIOS:0, LWM2M:0, OMADM:0, FOTA:0, RFCAL:0
LPM persistence - None

OK

AT!ERR
QDSP6                     ARM9 (not saved)
00   01 gpstask  04142
01   01 sys_eplm 00591

OK

Subsequently running

AT!RMARESET=1

Got me back to

AT!IMAGE?
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
FW   1    EMPTY  0   0 0
FW   2    EMPTY  0   0 0
FW   3    EMPTY  0   0 0
FW   4    EMPTY  0   0 0
Max FW images: 4
Active FW image is at slot 255

TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
PRI  FF   GOOD   0   0 0      002.008_012 01.07.02.00_ATT
PRI  FF   GOOD   0   0 0      002.015_002 01.09.04.00_DOCOMO
PRI  FF   GOOD   0   0 0      002.011_002 01.08.04.00_GENERIC
PRI  FF   GOOD   0   0 0      002.001_002 01.08.04.00_SIERRA
PRI  FF   GOOD   0   0 0      002.017_002 01.09.04.00_SOFTBANK
PRI  FF   GOOD   0   0 0      000.001_003 01.08.04.00_SPRINT
PRI  FF   GOOD   0   0 0      001.000_002 01.07.02.00_TELUS
PRI  FF   GOOD   0   0 0      002.015_006 01.08.04.00_VERIZON
Max PRI images: 50

AT!IMPREF?
!IMPREF:
 preferred fw version:    01.08.04.00
 preferred carrier name:  GENERIC
 preferred config name:   GENERIC_002.011_002
 preferred subpri index:  000
 current fw version:      01.14.02.00
 current carrier name:    GENERIC
 current config name:     GENERIC_002.011_002
 current subpri index:    000

 fw version mismatch

OK

AT!PRIID?
PRI Part Number: 9908685
Revision: 002.006
Customer: Generic-M2M

Carrier PRI: 9999999_9907259_SWI9X50C_01.08.04.00_00_GENERIC_002.011_002

Again it appears to be using the new FW with the old 01.08 PRI.

Any advice is appreciated.

2

have you tried the fwdwl tool here?
https://source.sierrawireless.com/resources/airprime/software/mbpl/mbpl-software-latest/#sthash.ZXRwiiKx.dpbs

3

I just the updated to use the firmware downloader recommended. I was able to flash successfully once, but I cannot replicate it on another module (I have about 10 I can try on). It’s possible I ran some AT command but it’s really unclear to me what’s happening.

I found that after flashing this second unit, it was similarly stuck in low power with a config firmware mismatch. I tried switching to another image (“Sierra”), which was successful and exited low power mode, but I am unable to switch back to GENERIC with the upgraded firmware, I simply get “ERROR” when I try to switch back. It appears that the firmware is not listed in any slot.

My current state is:

AT!IMAGE?
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
FW   1    GOOD   2   0 0      ?_?         01.08.04.00_?
FW   2    GOOD   1   0 0      ?_?         01.07.02.00_?
FW   3    GOOD   1   0 0      ?_?         01.09.04.00_?
FW   4    EMPTY  0   0 0
Max FW images: 4
Active FW image is at slot 1

TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
PRI  FF   GOOD   0   0 0      002.008_012 01.07.02.00_ATT
PRI  FF   GOOD   0   0 0      002.015_002 01.09.04.00_DOCOMO
PRI  FF   GOOD   0   0 0      002.035_003 01.14.02.00_GENERIC
PRI  FF   GOOD   0   0 0      002.001_002 01.08.04.00_SIERRA
PRI  FF   GOOD   0   0 0      002.017_002 01.09.04.00_SOFTBANK
PRI  FF   GOOD   0   0 0      000.001_003 01.08.04.00_SPRINT
PRI  FF   GOOD   0   0 0      001.000_002 01.07.02.00_TELUS
PRI  FF   GOOD   0   0 0      002.015_006 01.08.04.00_VERIZON
Max PRI images: 50


OK

AT!IMPREF?
!IMPREF:
 preferred fw version:    01.08.04.00
 preferred carrier name:  SIERRA
 preferred config name:   SIERRA_002.001_002
 current fw version:      01.08.04.00
 current carrier name:    SIERRA
 current config name:     SIERRA_002.001_002

OK

AT!IMPREF="GENERIC"
ERROR

AT!PCINFO?
State: Online
LPM voters - Temp:0, Volt:0, User:0, W_DISABLE:0, IMSWITCH:0, BIOS:0, LWM2M:0, OMADM:0, FOTA:0, RFCAL:0
LPM persistence - None
4

You only download the nvu file, not the cwe file 01.14.02.00 for firmware

How did you use the fwdwl tool?

5

as @jyijyi say,
the key point is AT!IMAGE? result.
You only store the 01.14.02.00_GENERIC PRI (nvu) file on the advice.
when you try to switch to AT!IMPREF=“GENERIC” you got an error due lack of the 01.14.02.00 FW in the FW slots.
The best practice is to reflash the GENERIC 01.14.02.00 from the source. Then, it will automatically stay in Generic FW, and you will be able to switch between your FWs.

6

Thank you for the time @jyijyi and @elkanam.

As you’ll find in the log below, I had already succesfully flashed the modem with both images using an archive provided from the above link, however only the PRI is listed and the firmware is not. Am I doing something wrong? I reproduced this error with a different modem. Note that ATI shows the new firmware and AT?IMAGE does not.

My flashing command was (using fwdwl-lite as the name of the newer fw-download-tool)

fwdwl-lite -P 1-1 -l /var/log/fwdwl -t 1 -c QMI -m 4 -f /my-fw-path/

firmware download tool logs (edited for brevity, full logs attached)

[20:57:46.103][6]Images list:
[20:57:46.103][6]Image[0]: /my-fw-path/SWI9X50C_01.14.02.00.cwe
[20:57:46.103][6]Image[1]: /my-fw-path/SWI9X50C_01.14.02.00_GENERIC_002.035_003.nvu
[20:57:46.115][6]litefw_download: Adjust download_type from 1 to REG_DWLD as device is already in B&H mode.
[20:57:53.380][7]File: "/my-fw-path/SWI9X50C_01.14.02.00.cwe", size in bytes: 80240356, size in sectors: 19590
<program PAGES_PER_BLOCK="64" SECTOR_SIZE_IN_BYTES="4096" filename="spkg.cwe" num_partition_sectors="19590" physical_partition_number="0" start_sector="-1"  />
<log value="SWI program command for CWE image spkg.cwe"/>
[20:57:53.382][7]Write: /my-fw-path/SWI9X50C_01.14.02.00.cwe
[20:59:55.367][7]Flash done for file: "/my-fw-path/SWI9X50C_01.14.02.00.cwe"
[20:59:55.584][7]File: "/my-fw-path/SWI9X50C_01.14.02.00_GENERIC_002.035_003.nvu", size in bytes: 10159, size in sectors: 3
<program PAGES_PER_BLOCK="64" SECTOR_SIZE_IN_BYTES="4096" filename="spkg.cwe" num_partition_sectors="3" physical_partition_number="0" start_sector="-1"  />
<log value="SWI program command for CWE image spkg.cwe"/>
[20:59:55.585][7]Write: /my-fw-path/SWI9X50C_01.14.02.00_GENERIC_002.035_003.nvu
[20:59:56.602][7]Flash done for file: "/my-fw-path/SWI9X50C_01.14.02.00_GENERIC_002.035_003.nvu"
[21:00:27.092][6]connect_to_device: Device is successfully connected.
[21:00:27.156][6]PrintFwInfo: Getting firmware info...
[21:00:27.220][6]PrintFwInfo: Model: EM7565
[21:00:27.220][6]PrintFwInfo: Boot version: SWI9X50C_01.14.02.00
[21:00:27.220][6]PrintFwInfo: APP version: SWI9X50C_01.14.02.00
[21:00:27.220][6]PrintFwInfo: SKU: 1104207
[21:00:27.220][6]PrintFwInfo: Package ID:
[21:00:27.220][6]PrintFwInfo: Carrier: 1
[21:00:27.220][6]PrintFwInfo: PRI version: 002.006
[21:00:27.220][6]PrintFwInfo: Current carrier name: GENERIC
[21:00:27.220][6]PrintFwInfo: Current carrier revision: 002.011_002
[21:00:27.284][6]disconnect_to_device: Device disconnected.
[21:00:27.284][7]Firmware download process completed successfully.

fwdwl.txt (54.8 KB)

AT!IMAGE?
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
FW   1    GOOD   2   0 0      ?_?         01.08.04.00_?
FW   2    GOOD   1   0 0      ?_?         01.07.02.00_?
FW   3    GOOD   1   0 0      ?_?         01.09.04.00_?
FW   4    EMPTY  0   0 0
Max FW images: 4
Active FW image is at slot 255

TYPE SLOT STATUS LRU FAILURES UNIQUE_ID   BUILD_ID
PRI  FF   GOOD   0   0 0      002.008_012 01.07.02.00_ATT
PRI  FF   GOOD   0   0 0      002.015_002 01.09.04.00_DOCOMO
PRI  FF   GOOD   0   0 0      002.035_003 01.14.02.00_GENERIC
PRI  FF   GOOD   0   0 0      002.001_002 01.08.04.00_SIERRA
PRI  FF   GOOD   0   0 0      002.017_002 01.09.04.00_SOFTBANK
PRI  FF   GOOD   0   0 0      000.001_003 01.08.04.00_SPRINT
PRI  FF   GOOD   0   0 0      001.000_002 01.07.02.00_TELUS
PRI  FF   GOOD   0   0 0      002.015_006 01.08.04.00_VERIZON
Max PRI images: 50


OK

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: EM7565
Revision: SWI9X50C_01.14.02.00 2e210b jenkins 2020/08/19 14:18:39
IMEI: 353533102792127
IMEI SV: 12
FSN: UF151701000310
+GCAP: +CGSM


OK
7

Hi @mregan
the flashing seen good, but your image is stored in slot 255 and not in the flash storage slots.

so you can re-flash it to see it in one of the FW slots.

8

@elkanam,

Thank you. I can confirm I’ve recovered a couple of units this way. Do you know what is slot 255, or why the firmware isn’t being loaded into a proper slot upon flashing the first time? The Image Switching documentation doesn’t seem to have any information about this.

9

Going further down the rabbit hole:

lite-fw/inc/lite-fw.h from MBPL_SDK_R35_ENG4-lite.bin.tar.gz has the following:

    /*! Non image switch type of firmware download */
    eFWDWL_TYPE_REG_DWLD,           // Non image switch scenario, plain switch to B&H mode (0x003E)

My logs show:

[20:57:46.115][6]litefw_download: Adjust download_type from 1 to REG_DWLD as device is already in B&H mode.

This all says to me that if you are flashing from B&H mode then the application will not perform a “switching download”, (as is the case for me, given that I’m trying to recover bricked modems). I tried adding -F and -z but that seemed not to make a difference.

10

how about upgrading FW like this?

11

It appears I have a workable automated solution, though it requires a software change I wanted to avoid.

The solution is:

  • When the EM7565 is in the bootloader (1199:9090), flash it
  • When the EM7565 is in low power mode and AT!PCINFO? shows IMSWITCH=1, flash it

I’m using

  • 01.14.13.00_GENERIC FW+PRI
  • fw-download-tool from R40_ENG6
  • MBPL Drivers R35_ENG4

This procedure appears to work regardless of the modem’s internal flash version reported by AT!BSINFO? (KSLCMBL2VA2M2C or H27S4G8F2EKPB4).