RC7611 completely "locks up" after a while

Hi folks,

I’m trying to, at the very least, figure out how to debug an issue i’m having with the RC7611.

After running for a little while (between 10-24 hours), the modem I am using will “lock up” and become unresponsive to serial port commands. The program I am running communicates with the modem over UART, and periodically sends chunks of data to a server using the KTCP API. From my log files, the last message received from the modem, is an echo of the last command (without the expected response), and from there the modem stops responding to any commands until it is power cycled. Ideally we would run our program without hardware flow control, but this same behavior is observed both with and without flow control enabled.

When the modem is locked up like this, it does not enumerate over the USB port (i.e. if I connect it to a windows PC over USB, neither the DM nor NMEA comports show up; normally both do show up on this computer).

An example excerpt from one of my logs follows. > indicates data transmitted over the serial port, < indicates data received, and True/False indicates whether the expected response was received. The actual raw data that we transmit has been omitted (i.e. only AT commands and responses are shown).

    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:26 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:34 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:42 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\xe0\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:50 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:57 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:05 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:13 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:21 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:28 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:36 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(False, b'< AT+KTCPSTAT=1\r')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'

The test program I am using is sending chunks of a data file, in order to simulate a field deployment. I have tested this same program using an HL7588, and do not observe this problem (the modem makes it to the end of the data file without locking up; the RC7611 will lock up before making it to the end of the file).

I am looking for guidance on how to debug this issue (especially since it takes ~12 hours each run before we encounter it), or at the very least some way of recovering from this unresponsive state without having to power cycle the modem (in our application’s deployed state, we will not have access to the modem’s reset pins).

Here’s a log from the program I use to initialize the modem (prior to running the data-transmitting program), showing the firmware revision.

    INFO:root:b'> ATI\r'
    INFO:root:(True, b'< ATI\r\r\nManufacturer: Sierra Wireless, Incorporated\r\nModel: RC7611\r\nRevision: SWI9X07H_00.03.03.00 2bb7a4 jenkins 2020/04/15 07:57:29\r\nIMEI: 352138119045351\r\nIMEI SV: 14\r\nFSN: 6M1473514413A0\r\n+GCAP: +CGSM\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CFUN=1\r'
    INFO:root:(True, b'< AT+CFUN=1\r\r\nOK\r\n')
    INFO:root:b'> AT+CMEE=2\r'
    INFO:root:(True, b'< AT+CMEE=2\r\r\nOK\r\n')
    INFO:root:b'> ATE1\r'
    INFO:root:(True, b'< ATE1\r\r\nOK\r\n')
    INFO:root:b'> AT!ENTERCND="A710"\r'
    INFO:root:(True, b'< AT!ENTERCND="A710"\r\r\nOK\r\n')
    INFO:root:b'> AT!CUSTOM="CFUNPERSISTEN",1\r'
    INFO:root:(True, b'< AT!CUSTOM="CFUNPERSISTEN",1\r\r\nOK\r\n')
    INFO:root:b'> AT+CCID\r'
    INFO:root:(True, b'< AT+CCID\r\r\n+CCID: 89332401000025016219\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KSLEEP=2\r'
    INFO:root:(True, b'< AT+KSLEEP=2\r\r\nOK\r\n')
    INFO:root:b'> AT+CGDCONT=1,"IP","internet.swir"\r'
    INFO:root:(True, b'< AT+CGDCONT=1,"IP","internet.swir"\r\r\nOK\r\n')
    INFO:root:b'> AT+CESQ\r'
    INFO:root:(True, b'< AT+CESQ\r\r\n+CESQ: 99,99,255,255,13,35\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CGREG?\r'
    INFO:root:(True, b'< AT+CGREG?\r\r\n+CGREG: 0,5\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CGREG=0\r'
    INFO:root:(True, b'< AT+CGREG=0\r\r\nOK\r\n')
    INFO:root:b'> AT+CGREG?\r'
    INFO:root:(True, b'< AT+CGREG?\r\r\n+CGREG: 0,5\r\n\r\nOK\r\n')
    INFO:root:b'> AT+COPS?\r'
    INFO:root:(True, b'< AT+COPS?\r\r\n+COPS: 0,0,"T-Mobile Sierra Wireless",7\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KCNXCFG=1,"GPRS","internet.swir"\r'
    INFO:root:(True, b'< AT+KCNXCFG=1,"GPRS","internet.swir"\r\r\nOK\r\n')
    INFO:root:b'> AT+CGPADDR=1\r'
    INFO:root:(True, b'< AT+CGPADDR=1\r\r\n+CGPADDR: 1,10.126.225.97\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KHTTPCFG?\r'
    INFO:root:(True, b'< AT+KHTTPCFG?\r\r\nOK\r\n')
    INFO:root:b'> AT+KTCPCFG?\r'
    INFO:root:(True, b'< AT+KTCPCFG?\r\r\nOK\r\n')
    INFO:root:b'> AT+KCNXCFG?\r'
    INFO:root:(True, b'< AT+KCNXCFG?\r\r\n\r\n+KCNXCFG: 1,"GPRS","internet.swir","","","IPV4","0.0.0.0","0.0.0.0","0.0.0.0",0\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KTCPCFG=1,0,"www.iomt.tech",80\r'
    INFO:root:(True, b'< AT+KTCPCFG=1,0,"www.iomt.tech",80\r\r\n\r\n+KTCPCFG: 1\r\n\r\nOK\r\n')

Hi @taylor.viti

I see the problem you’re having is similar to the one that was noticed in Sierra’s Release Note.
Currently, Sierra is working on a fix for this issue. I will notify you as soon as this issue is fixed.

Thanks,