HL7800 Commands Fail and Resetting

I have two HL7800s that have been running 4.7.1.0 (rc12) and recently both stopped connecting to a network. On closer inspection they both respond badly to a number of AT commands. Some ati commands work, others throw errors and ati9 causes the device to reset due to an illegal instruction. See below. at+cimi and at+ccid also return errors.

at ati3
at ati3
AT ati3

HL7800.

OK

at ati4
at ati4
AT ati4

ERROR

at ati1
at ati1
AT ati1

ERROR

at ati2
at ati2
AT ati2

ERROR

at ati5
at ati5
AT ati5

ERROR

at ati6
at ati6
AT ati6

ERROR

at ati9
at ati9
AT ati9
Illegal instruction @0x98D2E704
BadInstr=DC62797C
03E00008 27BD0018 <90820000> 14400005 00A01825
way=00000000, TagLo=0x18A9E480, DataLo=0xAD1441A5
way=00000001, TagLo=0x18D32480, DataLo=0x8FA200E4
way=00000002, TagLo=0x18D2E480, DataLo=0x90820000
way=00000003, TagLo=0x18D2A480, DataLo=0x30D8003F

####@@@@
System error!
Got exception, will be handled in stack trace function
Core : RKAPP_02_01_02_00_129__96afe6c8fdf77327e86cfbe69d5cd6cdd4af3a89
Package: N/A

0: 00000000 at: A5A5A5A5 v0: 00000000 v1: 98E5E828
a0: 00000006 a1: 98E5CB2C a2: 00000041 a3: 0000000A
t0: B01E5520 t1: 00000025 t2: 00000001 t3: 65720D0A
t4: FFFFFFC0 t5: 00000080 t6: FFFFFFE0 t7: 000000C0
s0: 0000000A s1: 98E60000 s2: 00000006 s3: 98E3448C
s4: 9C0DC870 s5: 00000000 s6: 9C0DC7D0 s7: 00000000
t8: 00000008 t9: B01E53EC k0: 98CFD1B8 k1: 0100FD03
gp: 9C0A21D0 sp: 90288B88 fp: 0000001F ra: 98ADA0A4
hi: 00000000 lo: 00000000
epc: 98D2E704
BadVAddr: 00000006
Status: 0100FD03
Cause: 00800028
BadInstr: DC62797C
BadPInstr: 12A00200

… lots more dumped following this

I tried to reflash one HL7800 with the latest 4.7.1.0 exe and I get the following:

Is it the same error when running the exe in UART1 AT command port ?

I’m using a Pinnacle 100 module and UART1 of the HL7800 isn’t easily accessible.
Does UART1 support firmware updates?

yes, UART1 AT command port supports FW upgrade

@jyijyi Ok, I’ll try to update via UART1.
Regarding the errors and reset the ati commands are causing, any idea what could be causing that or troubleshooting steps I could try?

Do you know how to reproduce the issue?

The issue appears to occur in devices that experience quite a few resets/brownouts.
A third HL7800 is now showing the same symptoms. It was intentionally subjected to many resets over a short period of time. One of the other devices also experienced many resets and I need to confirm the same happened to the other.
All devices are running 4.7.1.0.

Suggest you to contact distributor for failure analysis

The rest of the module appears to be working. Isn’t this a Semtech issue since the problem is with the HL7800?

It could be or could not be, that is why a failure analysis on problematic module should be done to confirm, right?

Btw, how is your test on downloading firmware via uart1?

It appears that I successfully updated the firmware after connecting directly to UART1. The exe went through all the steps then exited. I’ll run it again just to be sure.
Unfortunately, after the update, the problem persists and quite a few AT commands still return ERROR, including the ATI commands that print the firmware version.

Could the HL7800s flash be corrupt? That could explain why the update didn’t fix the issue. Are there certain AT commands that interact with flash that I could compare to others that do not?

yes, it could be, need to further investigate by failure analysis of module
Did you contact distributor?

BTW, is it the same situation if downloading older firmware like FW 4.6.9.4 or FW 4.6.8.0?
https://source.sierrawireless.com/resources/airprime/software/hl7800-firmware/hl78xx-firmware-release-history-item/#sthash.6lu90PQh.K1SH1ZC7.dpbs