RC7611 completely "locks up" after a while

IoT Modules RC76
1

Hi folks,

I’m trying to, at the very least, figure out how to debug an issue i’m having with the RC7611.

After running for a little while (between 10-24 hours), the modem I am using will “lock up” and become unresponsive to serial port commands. The program I am running communicates with the modem over UART, and periodically sends chunks of data to a server using the KTCP API. From my log files, the last message received from the modem, is an echo of the last command (without the expected response), and from there the modem stops responding to any commands until it is power cycled. Ideally we would run our program without hardware flow control, but this same behavior is observed both with and without flow control enabled.

When the modem is locked up like this, it does not enumerate over the USB port (i.e. if I connect it to a windows PC over USB, neither the DM nor NMEA comports show up; normally both do show up on this computer).

An example excerpt from one of my logs follows. > indicates data transmitted over the serial port, < indicates data received, and True/False indicates whether the expected response was received. The actual raw data that we transmit has been omitted (i.e. only AT commands and responses are shown).

    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:26 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:34 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:42 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\xe0\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:50 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:37:57 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:05 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:13 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:21 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:28 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(True, b'< AT+KTCPSTAT=1\r\r\n\r\n+KTCPSTAT: 3,-1,0,0\r\n\r\nOK\r\n')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(True, b'< AT+KTCPSND=1,1160\r\r\nCONNECT\r\n')
    INFO:root:b'> --EOF--Pattern--\r'
    INFO:root:(True, b'< \x00\r\nOK\r\n')
    INFO:root:(True, b'< \r\n\r\n+KTCP_DATA: 1,171\r\n')
    INFO:root:(True, b'< \r\nAT+KTCPRCV=1,171\r\r\nCONNECT\r\nHTTP/1.1 200 OK\r\nServer: nginx/1.18.0\r\nDate: Tue, 19 Apr 2022 06:38:36 GMT\r\nContent-Type: application/json\r\nContent-Length: 17\r\nConnection: keep-alive\r\n\r\n{"success":true}\n--EOF--Pattern--\r\nOK\r\n')
    INFO:root:b'> AT+KTCPSTAT=1\r'
    INFO:root:(False, b'< AT+KTCPSTAT=1\r')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'
    INFO:root:(False, b'< ')
    INFO:root:b'> AT'
    INFO:root:b'> +KTCPSND=1,1160\r'

The test program I am using is sending chunks of a data file, in order to simulate a field deployment. I have tested this same program using an HL7588, and do not observe this problem (the modem makes it to the end of the data file without locking up; the RC7611 will lock up before making it to the end of the file).

I am looking for guidance on how to debug this issue (especially since it takes ~12 hours each run before we encounter it), or at the very least some way of recovering from this unresponsive state without having to power cycle the modem (in our application’s deployed state, we will not have access to the modem’s reset pins).

Here’s a log from the program I use to initialize the modem (prior to running the data-transmitting program), showing the firmware revision.

    INFO:root:b'> ATI\r'
    INFO:root:(True, b'< ATI\r\r\nManufacturer: Sierra Wireless, Incorporated\r\nModel: RC7611\r\nRevision: SWI9X07H_00.03.03.00 2bb7a4 jenkins 2020/04/15 07:57:29\r\nIMEI: 352138119045351\r\nIMEI SV: 14\r\nFSN: 6M1473514413A0\r\n+GCAP: +CGSM\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CFUN=1\r'
    INFO:root:(True, b'< AT+CFUN=1\r\r\nOK\r\n')
    INFO:root:b'> AT+CMEE=2\r'
    INFO:root:(True, b'< AT+CMEE=2\r\r\nOK\r\n')
    INFO:root:b'> ATE1\r'
    INFO:root:(True, b'< ATE1\r\r\nOK\r\n')
    INFO:root:b'> AT!ENTERCND="A710"\r'
    INFO:root:(True, b'< AT!ENTERCND="A710"\r\r\nOK\r\n')
    INFO:root:b'> AT!CUSTOM="CFUNPERSISTEN",1\r'
    INFO:root:(True, b'< AT!CUSTOM="CFUNPERSISTEN",1\r\r\nOK\r\n')
    INFO:root:b'> AT+CCID\r'
    INFO:root:(True, b'< AT+CCID\r\r\n+CCID: 89332401000025016219\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KSLEEP=2\r'
    INFO:root:(True, b'< AT+KSLEEP=2\r\r\nOK\r\n')
    INFO:root:b'> AT+CGDCONT=1,"IP","internet.swir"\r'
    INFO:root:(True, b'< AT+CGDCONT=1,"IP","internet.swir"\r\r\nOK\r\n')
    INFO:root:b'> AT+CESQ\r'
    INFO:root:(True, b'< AT+CESQ\r\r\n+CESQ: 99,99,255,255,13,35\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CGREG?\r'
    INFO:root:(True, b'< AT+CGREG?\r\r\n+CGREG: 0,5\r\n\r\nOK\r\n')
    INFO:root:b'> AT+CGREG=0\r'
    INFO:root:(True, b'< AT+CGREG=0\r\r\nOK\r\n')
    INFO:root:b'> AT+CGREG?\r'
    INFO:root:(True, b'< AT+CGREG?\r\r\n+CGREG: 0,5\r\n\r\nOK\r\n')
    INFO:root:b'> AT+COPS?\r'
    INFO:root:(True, b'< AT+COPS?\r\r\n+COPS: 0,0,"T-Mobile Sierra Wireless",7\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KCNXCFG=1,"GPRS","internet.swir"\r'
    INFO:root:(True, b'< AT+KCNXCFG=1,"GPRS","internet.swir"\r\r\nOK\r\n')
    INFO:root:b'> AT+CGPADDR=1\r'
    INFO:root:(True, b'< AT+CGPADDR=1\r\r\n+CGPADDR: 1,10.126.225.97\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KHTTPCFG?\r'
    INFO:root:(True, b'< AT+KHTTPCFG?\r\r\nOK\r\n')
    INFO:root:b'> AT+KTCPCFG?\r'
    INFO:root:(True, b'< AT+KTCPCFG?\r\r\nOK\r\n')
    INFO:root:b'> AT+KCNXCFG?\r'
    INFO:root:(True, b'< AT+KCNXCFG?\r\r\n\r\n+KCNXCFG: 1,"GPRS","internet.swir","","","IPV4","0.0.0.0","0.0.0.0","0.0.0.0",0\r\n\r\nOK\r\n')
    INFO:root:b'> AT+KTCPCFG=1,0,"www.iomt.tech",80\r'
    INFO:root:(True, b'< AT+KTCPCFG=1,0,"www.iomt.tech",80\r\r\n\r\n+KTCPCFG: 1\r\n\r\nOK\r\n')
1 Like
2

Hi @taylor.viti

I see the problem you’re having is similar to the one that was noticed in Sierra’s Release Note.
Currently, Sierra is working on a fix for this issue. I will notify you as soon as this issue is fixed.

Thanks,

3

We have the same or similar issue with the RC7620-1.
In our case, we use the UART to communicate with the modem, enable MUX mode and open a data channel for network communication.
We periodically query the modem signal strength.
When the modem is online for a while, often after reboot this happens 120 seconds after PPP connection is established. But the time can also be longer.
We then see the modem signal strength request failing. Sometimes the modem is able to recover and wakes up. But sometimes this does not happen and a PPP protocol error occurs. Restarting the modem is then needed to get back the connection.
We have tried with different KSLEEP modes (0, 1, 2). But there is not one mode, which resolves this issue.

The modem has firmware version 8.7. Hardware Flow control is enabled.

4

Ps. Sometimes the connection also just gets lost and CEREG becomes 3 or 0. Without command timeouts on the UART interface.

5

Curious if there’s been any update on this, as it continues to be a problem for me.

Extremely curious to see how others have been able to use this modem for their applications, since from where I stand, this renders the modem unusable without implementing a workaround (I’ve been using scheduled reboots, with mixed success).

1 Like