EM7565 Stuck in Boot and Hold Mode After Update – Permanent Issue

1

Hello everyone,

We have been using an EM7565 module, which was functioning perfectly fine until we performed a firmware update. After the update, the module is now stuck in Boot and Hold mode permanently, and even performing a power reset on the board hasn’t resolved the issue.

Current Situation:

  • The module remains in Boot and Hold mode.
  • Power cycling the modem doesn’t seem to help.

Request for Help:

Has anyone experienced something similar? Is there a known reason why an EM7565 module would enter Boot and Hold mode and stay there permanently? We would appreciate any advice or guidance on how to resolve this issue and restore normal operation.

Thanks in advance!

2

does this help?

3

Hi,
Thank you for your reply.
I am able to recover the module using the fwdwl-lite tool . I am curious to know why it went into this mode and stayed in the mode permanently.

4

Maybe due to unexpected power cycle during firmware upgrade

5

Hi,
Does EM7565 keep any log you can read using AT command.

at!err
QDSP6                     ARM9 (not saved)
00   01 gpstask  04142
01   01 sys_eplm 00591

OK

How do you read the file “sys_eplm” ?

6

Is that 100% reproducible in your device?

7

Hi,
No. I cannot set the device into permanent boot and hold situation. This happened once in field.

8

Could it be some hardware issue in the field?

9

Hi,

That’s what we are trying to find out how em7565 module can go into permanent boot and hold mode. Module which went into this mode had no firmware updates applied.
Does EM7565 keep any diagnostics logs?

10

You can check at!flog? After recovery

11

Hi,

We have six EM7565 modules deployed in the field that are all permanently stuck in Boot and Hold mode. We’ve tried various recovery methods without success.

Would it be helpful if we send one of the modules back to Sierra Wireless for further diagnostics and analysis?

12

did you contact distributor to discuss this?

13

I got the at!flog output from one of the modules which was stuck in Boot and hold mode.

at!flog?
!FLOG: 


OK
Class long:
00000,1980/01/06 00:00:00.000,C1060500,COMMENT=Log created,LOG_VERSION=1.0
00000,1980/01/06 00:00:00.000,C1060500,BOOTFATAL=boot_elf_loader.c,line=1443,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.06.05.00_?_?_?_?,TYPE=SPKG,STATUS=PASS
00001,1980/01/06 00:00:00.000,C1060500,NV_REBUILD=START,COMMENT=FW ver: SWI9X50C_01.06.05.00 2df3a4 jenkins 2018/05/08 19:40:17,FILE=DEFAULTS
00001,1980/01/06 00:00:00.000,C1060500,NV_DEFAULT=FW ver: SWI9X50C_01.06.05.00 2df3a4 jenkins 2018/05/08 19:40:17
00001,1980/01/06 00:00:00.000,C1060500,NV_REBUILD=COMPLETE,COMMENT=defaulted 333, restored 0, skipped 0, failed 0,STATUS=PASS
00008,1980/01/06 00:01:08.082,C1060500,IMAGE_PREF=00.00.00.00, NONE, NONE_000.000_000
00008,1980/01/06 00:01:08.097,C1060500,NV_BACKUP=START,COMMENT=index 1
00008,1980/01/06 00:01:11.012,C1060500,NV_BACKUP=COMPLETE,COMMENT=index 1,STATUS=PASS
00008,1980/01/06 00:01:21.626,C1060500,IMAGE_PREF=01.08.04.00, VERIZON, VERIZON_002.015_001
00008,1980/01/06 00:00:00.000,C1060500,BOOT_HOLD=3,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.08.04.00_?_?_?_?,TYPE=SPKG,SLOT_WRITE=1,SLOT_WRITE_BOOT=1,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.07.02.00_?_?_?_?,TYPE=SPKG,SLOT_WRITE=2,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.09.04.00_?_?_?_?,TYPE=SPKG,SLOT_WRITE=3,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9907257_SWI9X50C_01.08.04.00_00_VERIZON_002.015_001,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9907259_SWI9X50C_01.07.02.00_00_GENERIC_002.004_001,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9907258_SWI9X50C_01.07.02.00_00_ATT_002.008_004,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9908789_SWI9X50C_01.08.04.00_00_SIERRA_002.001_000,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9908609_SWI9X50C_01.09.04.00_00_DOCOMO_002.015_000,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9908607_SWI9X50C_01.09.04.00_00_SOFTBANK_002.017_000,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9909446_SWI9X50C_01.08.04.00_00_SPRINT_000.001_001,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=9999999_9909335_SWI9X50C_01.07.02.00_00_TELUS_001.000_000,TYPE=FILE,STATUS=PASS
00008,1980/01/06 00:00:00.000,C1060500,DOWNLOADER=boot,VERSION=1103520_9907344_EM7565_01.08.04.00_00_Generic-M2M_002.003_000,TYPE=SPKG,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_VERIZON.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_GENERIC.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_ATT.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_SIERRA.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_DOCOMO.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_SOFTBANK.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_SPRINT.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_TELUS.010,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/swisb/hash.000,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,COPY_FROM_BACKUP=/nvup/NVUP_1103520_EM7565_Generic-M2M.020,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,NV_UPDATE=START,FILE=/nvup/NVUP_1103520_EM7565_Generic-M2M.020
00009,1980/01/06 00:00:00.000,C1080400,NV_UPDATE=COMPLETE,FILE=NVUP_1103520_EM7565_Generic-M2M.020.tmp,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,IMAGE_SW,CURR=01.08.04.00 ,PREF=01.08.04.00 VERIZON_002.015_001,CALC=01.08.04.00 VERIZON_002.015_001,COMMENT=START
00009,1980/01/06 00:00:00.000,C1080400,IMAGE_SW,CURR=01.08.04.00 ,PREF=01.08.04.00 VERIZON_002.015_001,CALC=01.08.04.00 VERIZON_002.015_001,COMMENT=No persist backup for CID 0
00009,1980/01/06 00:00:00.000,C1080400,IMAGE_SW,CURR=01.08.04.00 VERIZON_002.015_001,PREF=01.08.04.00 VERIZON_002.015_001,CALC=01.08.04.00 VERIZON_002.015_001,COMMENT=COMPLETE,STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A01E0 data:0x2422adfe, 0x28913db1, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A01E8 data:0x2f9a20ea, 0x026cbda5, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A01F0 data:0x663456a5, 0x6553fd5c, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A01F8 data:0x483c9af4, 0xec6a19b9, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0200 data:0x51000000, 0x35627f85, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0400 data:0x00000000, 0x00002000, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0130 data:0x00002000, 0x00000000, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0150 data:0x0007fc76, 0x00000000, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0150 data:0x00780000, 0x00000000, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A01D0 data:0x69000000, 0x00303030, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0400 data:0x00000000, 0x00000800, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,FUSE_WRITE,COMMENT=addr: 0x000A0130 data:0x00000800, 0x00000000, STATUS=PASS
00009,1980/01/06 00:00:00.000,C1080400,BSINIT_RESET=true,REASON=fuse request
00010,1980/01/06 00:08:31.843,C1080400,NV_BACKUP=START,COMMENT=index 2
00010,1980/01/06 00:08:34.262,C1080400,NV_BACKUP_DELETE,POINT=4
00010,1980/01/06 00:08:34.274,C1080400,NV_BACKUP=COMPLETE,COMMENT=index 2,STATUS=PASS
00018,1980/01/06 00:00:00.000,C1080400,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=97 inst=c1198018,TASK=00000097,VER=SWI9X50C_01.08.04.00
00020,1980/01/06 00:00:00.000,C1080400,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=e2 inst=c04a2008,TASK=000000E2,VER=SWI9X50C_01.08.04.00
00091,1980/01/06 00:00:00.000,C1080400,CRASH=MPSS,FILE=lte_ml1_common_dump.c,LINE=477,STR=CPHY_DEDICATED_CFG Deadlock - upr_lyr: 0xC222E8E8; wait_mask: 0x8FF01524,TASK=AMSS,VER=SWI9X50C_01.08.04.00
00101,1980/01/06 00:00:00.000,C1080400,CRASH=MPSS,FILE=ps_handle_mgr.c,LINE=373,STR=Assertion 0 failed,TASK=AMSS,VER=SWI9X50C_01.08.04.00
00112,1980/01/06 00:00:14.056,C1080400,IMAGE_PREF=01.07.02.00, GENERIC, GENERIC_002.004_001
00112,1980/01/06 00:00:00.000,C1080400,SLOT_SWITCH,SLOT=2,VERSION=INTERNAL_?_SWI9X50C_01.07.02.00_?_?_?_?,STATUS=PASS
00113,1980/01/06 00:00:00.000,C1070200,IMAGE_SW,CURR=01.07.02.00 VERIZON_002.015_001,PREF=01.07.02.00 GENERIC_002.004_001,CALC=01.07.02.00 GENERIC_002.004_001,COMMENT=START
00113,1980/01/06 00:00:00.000,C1070200,IMAGE_SW,CURR=01.07.02.00 GENERIC_002.004_001,PREF=01.07.02.00 GENERIC_002.004_001,CALC=01.07.02.00 GENERIC_002.004_001,COMMENT=COMPLETE,STATUS=PASS
00140,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=ps_handle_mgr.c,LINE=287,STR=Assertion 0 failed,TASK=AMSS,VER=SWI9X50C_01.07.02.00
00141,1980/01/06 00:00:00.000,C1070200,COMMENT=Log created,LOG_VERSION=1.0
00586,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=f5 inst=c0512408,TASK=000000F5,VER=SWI9X50C_01.07.02.00
00729,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=d0 inst=c0ecc588,TASK=000000D0,VER=SWI9X50C_01.07.02.00
00731,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=f5 inst=c0850280,TASK=000000F5,VER=SWI9X50C_01.07.02.00
00797,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=ps_handle_mgr.c,LINE=269,STR=Assertion 0 failed,TASK=AMSS,VER=SWI9X50C_01.07.02.00
00802,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=lfw_syncpt_cfg.c,LINE=472,STR=CMD_VALID not cleared for VU-1037899026,TASK=AMSS,VER=SWI9X50C_01.07.02.00
00859,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=105f inst=c0019be8,TASK=0000105F,VER=SWI9X50C_01.07.02.00
00871,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=97 inst=c14bf440,TASK=00000097,VER=SWI9X50C_01.07.02.00
01106,1980/01/06 00:00:00.000,C1070200,CRASH=MPSS,FILE=,LINE=0,STR=ExIPC: Exception recieved tid=f4 inst=c001b800,TASK=000000F4,VER=SWI9X50C_01.07.02.00
02420,1980/01/06 00:00:00.000,C1070200,BOOT_HOLD=6,SMART_RECOVERY=yes,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.14.02.00_?_?_?_?,TYPE=SPKG,STATUS=PASS
02420,1980/01/06 00:00:00.000,C1070200,DOWNLOADER=boot,VERSION=9999999_9907259_SWI9X50C_01.14.02.00_00_GENERIC_002.035_003,TYPE=FILE,STATUS=PASS
02421,1980/01/06 00:00:00.000,C1140200,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_GENERIC.010,STATUS=PASS
02421,1980/01/06 00:00:00.000,C1140205,BOOT_HOLD=3,DOWNLOADER=boot,VERSION=9999999_9907259_SWI9X50C_01.14.02.00_00_GENERIC_002.035_003,TYPE=FILE,STATUS=PASS
02422,1980/01/06 00:00:00.000,C1140200,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_GENERIC.010,STATUS=PASS
02422,1980/01/06 00:23:36.267,C1140200,IMAGE_PREF=01.14.02.00, GENERIC, GENERIC_002.035_003
02423,1980/01/06 00:00:00.000,C1140200,IMAGE_SW,CURR=01.14.02.00 GENERIC_002.004_001,PREF=01.14.02.00 GENERIC_002.035_003,CALC=01.14.02.00 GENERIC_002.035_003,COMMENT=START
02423,1980/01/06 00:00:00.000,C1140200,IMAGE_SW,CURR=01.14.02.00 GENERIC_002.035_003,PREF=01.14.02.00 GENERIC_002.035_003,CALC=01.14.02.00 GENERIC_002.035_003,COMMENT=COMPLETE,STATUS=PASS
Class med:
00000,1980/01/06 00:00:00.000,C1060500,COMMENT=Log created,LOG_VERSION=1.0
00000,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 1
00001,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 2
00002,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 3
00003,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 4
00004,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 5
00005,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 6
00006,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 7
00007,1980/01/06 00:00:00.000,C1060500,RESET=HARD,COMMENT=Count 8
02423,1980/01/06 00:00:00.000,C1140200,RESET=HARD,COMMENT=Count 2403
02423,1980/01/06 00:01:35.812,C1140200,NV_BACKUP=START,COMMENT=index 4
02423,1980/01/06 00:01:41.050,C1140200,NV_BACKUP=COMPLETE,COMMENT=index 4,STATUS=PASS
02424,1980/01/06 00:00:00.000,C1140200,RESET=HARD,COMMENT=Count 2404
02425,1980/01/06 00:00:00.000,C1140200,RESET=HARD,COMMENT=Count 2405
02426,1980/01/06 00:00:00.000,C1140200,RESET=HARD,COMMENT=Count 2406

DONE
14

I believe below is for your recovery by downloading a FW 01.14.02.00 during the 2420th times of boot up:


02420,1980/01/06 00:00:00.000,C1070200,BOOT_HOLD=6,SMART_RECOVERY=yes,DOWNLOADER=boot,VERSION=INTERNAL_?_SWI9X50C_01.14.02.00_?_?_?_?,TYPE=SPKG,STATUS=PASS
02420,1980/01/06 00:00:00.000,C1070200,DOWNLOADER=boot,VERSION=9999999_9907259_SWI9X50C_01.14.02.00_00_GENERIC_002.035_003,TYPE=FILE,STATUS=PASS
02421,1980/01/06 00:00:00.000,C1140200,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_GENERIC.010,STATUS=PASS
02421,1980/01/06 00:00:00.000,C1140205,BOOT_HOLD=3,DOWNLOADER=boot,VERSION=9999999_9907259_SWI9X50C_01.14.02.00_00_GENERIC_002.035_003,TYPE=FILE,STATUS=PASS
02422,1980/01/06 00:00:00.000,C1140200,COPY_FROM_BACKUP=/swir/nvdelta/NVUP_GENERIC.010,STATUS=PASS
02422,1980/01/06 00:23:36.267,C1140200,IMAGE_PREF=01.14.02.00, GENERIC, GENERIC_002.035_003
02423,1980/01/06 00:00:00.000,C1140200,IMAGE_SW,CURR=01.14.02.00 GENERIC_002.004_001,PREF=01.14.02.00 GENERIC_002.035_003,CALC=01.14.02.00 GENERIC_002.035_003,COMMENT=START
02423,1980/01/06 00:00:00.000,C1140200,IMAGE_SW,CURR=01.14.02.00 GENERIC_002.035_003,PREF=01.14.02.00 GENERIC_002.035_003,CALC=01.14.02.00

So above that line, I don’t see anything showing why it stuck in boot and hold mode

If you are not able to reproduce the issue, it is hard to troubleshoot