Not sure when these message happens …
Do you see this after FW upgrade for this module?
Hi,
No sure to fully understand your question.
For the second module, the product was shipped and the module failed after.
That module was updated, tested, assembled in our product and retested. So we are sure it worked well.
Do you think that a power supply problem can cause this issue? What’s happen if the module main power supply is suddenly turned off when it is working (or in shutdown phase) ?
I mean i don’t know if these messages appear in the field or not:
00:00:00.000,H0082402,CRASH=MPSS,FILE=fs_device.c,LINE=401,STR=[-2095636730] Unexpected device read failure,TASK=AMSS,VER=SWI9X07H_00.08.24.02
00000,1980/01/06 00:00:00.000,H0082402,CRASH=Unknown,FILE=,LINE=0,STR=,TASK=,VER=
BTW, do you follow the power off sequence in PTS?
Ok.
Yes, we follow the power off sequence but our product power supply ( batteries) can be disconnected by the user at any time. We cannot predict when this will occur.
i check again “RC7620_Bricked_module.txt”, I think the upgrade process is not yet completed…
00000,1980/01/06 00:00:00.000,H0082402,CRASH=MPSS,FILE=fs_device.c,LINE=401,STR=[-2095636730] Unexpected device read failure,TASK=AMSS,VER=SWI9X07H_00.08.24.02
00000,1980/01/06 00:00:00.000,H0082402,CRASH=Unknown,FILE=,LINE=0,STR=,TASK=,VER=
00000,1980/01/06 00:00:18.298,H0082402,BACKUP_SYNC,PART_TO_FILE=NVBK_FULL_Factory.001,STATUS=PASS
00000,1980/01/06 00:00:20.349,H0082402,BACKUP_SYNC,PART_TO_FILE=NVBK_FULL_Provision.002,STATUS=PASS
00000,1980/01/06 00:00:20.498,H0082402,NV_REBUILD=START,COMMENT=FW ver: SWI9X07H_00.08.24.02 f90cbd jenkins 2022/03/21 03:47:54,FILE=/swibackup/NVBK_FULL_Provision.002
00000,1980/01/06 00:00:30.862,H0082402,NV_REBUILD=COMPLETE,COMMENT=defaulted 0, restored 1495, skipped 0, failed 0,STATUS=PASS
00000,1980/01/06 00:00:33.253,H0082402,NV restore NV_SWI_EF_LOG_I index=0 from Cache
00005,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00005,1980/01/06 00:00:00.000,
00006,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00006,1980/01/06 00:00:00.000,
00006,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00006,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00007,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00007,1980/01/06 00:00:00.000,
00008,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00008,1980/01/06 00:00:00.000,
00008,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00008,1980/01/06 00:00:00.000,
00008,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00008,1980/01/06 00:00:00.000,
00008,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00008,1980/01/06 00:00:00.000,
00008,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00008,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00009,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00009,1980/01/06 00:00:00.000,
00010,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00010,1980/01/06 00:00:00.000,
00010,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00010,1980/01/06 00:00:00.000,
00010,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00010,1980/01/06 00:00:00.000,
00010,1980/01/06 00:00:00.000,H0082402,no meta data:didn't write
00010,1980/01/06 00:00:00.000,
Ok. But this cannot explain our module behavior.
Is there a generated log file when we update a module?
you can take the RC7620_working_module.txt as reference as you say it is OK module
I don’t speak about the log file generated by the module itself.
Is there a log file generated by the update exe ?
you can see here:
We are having the same problem here, RC7620 modules updated and working correctly during both production test and pre-deployment testing but randomly failing in the field. The following is output from our testing:
ATIManufacturer: Sierra Wireless, Incorporated
Model: RC7620-1
Revision: SWI9X07H_00.08.44.07 2539f0 jenkins 2023/04/26 10:17:51
IMEI: 351080657501851
IMEI SV: 28
FSN: 7T2422300616B1
+GCAP: +CGSM
OK
AT!PCINFO?;!IMAGE?;!IMPREF?State: Low Power Mode
LPM voters - Temp:0, Volt:0, User:0, W_DISABLE:0, IMSWITCH:1, BIOS:0, LWM2M:0, OMADM:0, FOTA:0, RFCAL:0, AVMS:0
LPM persistence - None
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID BUILD_ID
FW 1 EMPTY 0 0 0
FW 2 EMPTY 0 0 0
FW 3 EMPTY 0 0 0
Max FW images: 1
Active FW image is at slot 255
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID BUILD_ID
PRI FF GOOD 0 0 0 001.081_000 00.08.07.00_GENERIC
PRI FF GOOD 0 0 0 000.058_000 00.08.07.00_SIERRA
Max PRI images: 50
!IMPREF:
preferred fw version: 00.08.07.00
preferred carrier name: GENERIC
preferred config name: GENERIC_001.081_000
preferred subpri index: 000
current fw version: 00.08.44.07
current carrier name: GENERIC
current config name: GENERIC_001.081_000
current subpri index: 000
fw version mismatch
OK
AT!ENTERCND="A710"OK
AT!FLOG?!FLOG:
OK
Class long:
00201,1980/01/06 00:00:08.225,H0084407,COMMENT=Log created,LOG_VERSION=1.0
00201,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00201,1980/01/06 00:00:00.000,
00202,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00202,1980/01/06 00:00:00.000,
00203,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00203,1980/01/06 00:00:00.000,
00204,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00204,1980/01/06 00:00:00.000,
00205,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00205,1980/01/06 00:00:00.000,
00206,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00206,1980/01/06 00:00:00.000,
00207,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00207,1980/01/06 00:00:00.000,
00208,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00208,1980/01/06 00:00:00.000,
00209,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
00209,1980/01/06 00:00:00.000,
00210,1980/01/06 00:00:00.000,H0084407,no meta data:didn’t write
Please advise, this has become a critical issue for our production.
are you sure module in the field can follow the power off sequence mentioned in the PTS?
It seems the required FW has been deleted or corrupted
Hello, we have the same problem with you, did you or anybody else resolve it ?
We have pinpointed it to the power shutdown of the modem. The modem writes to the flash and if in any way the power is lost while doing this it gets bricked.
Even if you follow the recommended shutdown procedure there is no way to protect it against a power outage. Only if you use special design such as super capacitors and emergency shutdown procedures which last 0.7s and require GPIO. So in simple words all we can do is minimise the probability that it can be bricked in the field.
We feel this is bad design, there is no cold boot sequence, no fail safe image or whatever needed to avoid it getting bricked. This modem lacks the stability in power loss that the previous generations had. This never happend with any other Wavecom/Sierra modem.
I am also afraid this is related to the Qualcomm chipset the RC is using.
There are 1 million ways to die with such complex modems and this is the single worst case with it.
Did anybody find any workaround ?
I have a solution for you; once you have updated the firmware in the modem issue the following commands to it:
AT!NVBACKUP=3;!ENTERCND=“A710”;!NVBACKUP=2
This is necessary as the updater supplied by SW does not update the backup image in FLASH, therefore when you get FLASH corruption due to an unexpected power down then at the next power up the modem will attempt to utilise the stored backup configuration/data which will not match your current firmware and this results in the IMSWITCH:1 error reported by AT!PCINFO.
Our testing with more than 10,000 power cycles has shown that the modem will successfully recover from the FLASH corruption if the FLASH backup has been performed.
It is unknown whether OTA firmware updates also give rise to this problem; it may be advantageous to have your system firmware issue the NVBACKUP commands if an OTA update has occurred.
Hope this helps,
Nigel
Nigel thanks a zillion. If you ever need anything feel free to ask.
We really apreciate the answer which really explains what is going on.
We have a few thousand devices in the field which where updated using the Airvantage platfrom. From the occurence of the problem i assume the NV backup commands where never issued.
We did handle the power off procedure as requested and still there where cases of the modems getting bricked.
To Sierra/Semtech:
Can someone from Sierra/Semtech confirm that OTA upgrading the RC7620 using Airvantage the SW does not update the backup image in the FLASH.
Does the EXE single file update it ?
This is a serious glitch and it should be documented immediately.
i don’t quite think NVBACKUP can 100% solve the problem.
Improper power down sequence will have chance to damage the firmware, not just the NV backup area…
I think you can do a simple test to verify.
Here is an example to show you how to restore the NV item:
ati3
Manufacturer: Sierra Wireless, Incorporated
Model: RC7620-1
Revision: SWI9X07H_00.09.10.00 85890e jenkins 2023/09/25 03:16:56
IMEI: 353635110212435
IMEI SV: 29
FSN: 7T118701512345
+GCAP: +CGSM
OK
at+cgdcont?
+CGDCONT: 1,"IP","internet","0.0.0.0",0,0,0,0
+CGDCONT: 2,"IP","smartone","0.0.0.0",0,0,0,0
+CGDCONT: 3,"IPV4V6","sos","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0",0,0,0,1
OK
at!entercnd="A710"
OK
at!nvbackup=2
!NVBACKUP:
Items Saved: 1655
OK
at+cgdcont=1,"IP","test"
OK
at!rmareset=2
!RMARESET: DEVICE REBOOT REQUIRED
Items Restored: 1646
Items Deleted: 0
Items Defaulted: 0
Items Skipped: 9
OK
aat+Cgdcont?
+CGDCONT: 1,"IP","test","0.0.0.0",0,0,0,0
+CGDCONT: 2,"IP","smartone","0.0.0.0",0,0,0,0
+CGDCONT: 3,"IPV4V6","sos","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0",0,0,0,1
OK
+CREG: 0
+CGREG: 0
+CEREG: 0
at!reset
OK
at
OK
+WDSI: 0
+WDSI: 9,35140480
+WDSI: 2
at!entercnd="A710"
OK
+KSUP: 2
+CREG: 2
+CGREG: 2
+CEREG: 2
at+cgdcont?
+CGDCONT: 1,"IP","internet","0.0.0.0",0,0,0,0
+CGDCONT: 2,"IP","smartone","0.0.0.0",0,0,0,0
+CGDCONT: 3,"IPV4V6","sos","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0",0,0,0,1
OK
Hello,
thanks for the reply.
Please explain what do you mean by <>. How can the firmware be affected/altered ?
Corrupt memory, boot loops, config storage flash degradation, bad sectors in NAND memory etc can happen. A minimal cold boot is the basis of reliable operation.
I cannot think that it gets bricked because the APN or the last good known provider was not saved correctly. What important information are written in the flash that make the modem get bricked ?
I am expecting to speak with Semtech directly on the subject hopefully soon.
We use the RC7620 modem from the very first release date and in the beginning in the migration guide there was nothing indicating the special power handling required and i mean compared to previous same form factor compatible modems. Q6528RD, HL7648, HL7692, etc.
We follow the power down procedure to the point.
Still we had bricked modems and wondered why until we got Nigel’s reply.
We discovered there is a condition when powering by a lithium battery that when the voltage is below the allowed standards and fluctuating the modems got bricked. The voltage drop caused multiple resets to the modem.
We also handle this condition now monitoring cpu and modem voltage to halt the use of the modem to avoid such a situation.
Regardless of the above though if a user removes power/battery for any reason the modem could get bricked and there is no way to avoid it. It is only a matter of low probability and that is what i described above.
What Nigel described is the only serious explanation we have up today why the modems didnt manage to recover and function regardless of what is happening.
My question still remains can you confirm that OTA upgrading the RC7620 using Airvantage the SW does not update the backup image in the FLASH.
Does the EXE single file update it ?
Thanks.
did you contact distributor?
Yes.
We had indications of the problem since Sep/Oct 2024 but lately it became serious.
did you try the AT command sequence above to verify?
For example, you can change some NV value and then do a EXE file firmware update.
If there is NV backup command proceeded, AT!RMARESET should return to the new value
No i did not try it yet alsmost all upgrades we do are via airvanatge.