Asynchronous Modbus requests not working

Hi!

When switching from firmware 3.3.1 to firmware 3.5.1, I found a problem that sometimes Modbus stops working in asynchronous mode. That is, at first Modbus worked, but eventually stopped. Moreover, when it stopped working, neither rebooting the gateway, nor reinstalling the blueprint, nor reinstalling the firmware helps restore work. Checked on firmware 3.5.1 and 3.5.2, Gateway FX30S .
The error is that I am making a request in the “dh://modbus/rtu/cc3/request/send” resource, and the response should be in “dh://modbus/rtu/cc3/request/value”, but instead of this, response comes in “dh://modbus/rtu/cc3/value”.

The “dh://modbus/rtu/cc3/value” resource should not receive a response, since this resource should be polled once every 100 years (3155760000 seconds). I tried to set the polling period and 0 seconds and null, the result is the same, the polling of this resource occurs on an asynchronous request.

Hello Vitali,

Could you let me know the name of your Octave account? I would like to check your configuration.

Hello David!

I hope this is what you asked for:
https://octave.sierrawireless.io/company/ekatra_test/device/d5f2973c3f65503dd7f7a25f2

Hello @v.babich

I installed your blueprint on my 3.5.1 FX30 and I am not able to reproduce that issue:

The error is that I am making a request in the “dh://modbus/rtu/cc3/request/send” resource, and the response should be in “dh://modbus/rtu/cc3/request/value”, but instead of this, response comes in “dh://modbus/rtu/cc3/value”.

I can indeed see the asynchronous response in /request/value.

To avoid periodic polling you can set the period to 0 like this

With this setting there is no polling occurring on async requests


The 17 minutes match the time where the device powered on and the Developer MLode pushed to the Cloud all the resource tree values.

Hope this helps,
Nicolas

Hello Nicolas!

Initially, I had a polling period of 0 seconds and asynchronous requests worked like you did. But then something broke and asynchronous requests stopped working. And after that I began to try to set other polling periods like null or 3155760000 seconds.

If you cannot reproduce the problem on your device, you can try any manipulations on my device to fix it. This is a test device and you can’t break anything there.
Or suggest something else.

I rolled back to firmware version 3.5.0, Modbus does not work on it either. That is, Modbus does not work on firmware versions 3.5.0, 3.5.1 and 3.5.2.
After I rolled back to firmware version 3.4.2, Modbus worked.

Hi Vitaly,

Glad you are finally able to make your solution work.

Can you give more detail about the “something broke”? Since I’m not able to reproduce the failing asynchronous requests I’d like to understand what happens.

Thanks,
Nicolas

Hello Nicolas!

I have been working on a new Blueprint version on firmware 3.5.2.
Like you, at first the Modbus worked for me on firmware 3.5.2.
But after a while Modbus stopped working. What caused this I don’t know. I just worked with the Blueprint, deleted some Edge Actions that were no longer needed, created new ones, often saved new versions of the Blueprint. But even if we assume that I messed up something in the Blueprint and therefore Modbus does not work, then this is not the case, since the same Blueprint works on firmware version 3.4.2, and on versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.

I hope that we will find the cause of the error, because I do not want to stay on the old firmware version.

Hello Vitaly,

I’m trying to reproduce with a test setup:

  • FW 3.5.2
  • modbus polling period set to 0
  • modbus async requests sent every 10 minutes from an edge action. read is done on the 2 HLD registers of my modbus asset.

After 18 hours I can’t see any issues. Async requests return every time the last data. The periodic polling is successfully disabled.

I have some questions:

  • do you think my test setup is similar to what you are doing?
  • after how much time did you have modbus stopped working?
  • can you give more details about “Modbus stopped working”? Do you mean bus timeouts?
  • can you give more details about “versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.” ? Can you explicit what did you do exactly?

Thanks
Nicolas

Hello Nicolas!

do you think my test setup is similar to what you are doing?

Yes, your setup is similar to mine. Only asynchronous modbus polling goes every 5 seconds.

after how much time did you have modbus stopped working?

Modbus stopped working after a few days.

can you give more details about “Modbus stopped working”? Do you mean bus timeouts?

Modbus has stopped working - this means that the Modbus has stopped working correctly. Modbus started sending responses to asynchronous requests to resources associated with synchronous (periodic) requests.

can you give more details about “versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.” ? Can you explicit what did you do exactly?

Through the Octave web interface, I went to the resource “modbus/rtu/cc0/request/send” and pressed the “send command” button. In the pop-up window that appeared, I set the request parameters and sent the request. The answer was expected in the resource “modbus/rtu/cc0/request/value”, and received in the resource “modbus/rtu/cc0/value” as I previously described.

I am now installing firmware 3.5.2 again and will try to catch this error again. Now I will monitor my actions and if I succeed, I will write here about the result

Hello!

I got the above error again.
To get this error again, I installed firmware version 3.5.2 on November 11, 2022 and created a new Blueprint from the existing one.
After I tried to add and remove Virtual resources, Timers, Edge actions, saved the changes to the Blueprint, but the error did not appear.
Then I had to do other work and for a while I left the FX30S gateway running without my intervention.
And today, November 17, 2022, a Modbus error appeared {“errno”: -16,“errstr”:“Device or resource busy”}. After that, the response to the Asynchronous request again began to come to the resource associated with the Synchronous request.
The frequency of Asynchronous requests is set to 1 time per 5 seconds. 4 devices are polled. It took 6 days before the error appeared.
But a few minutes after rebooting the gateway, the error disappeared. Modbus started working normally again.

I hope this information helps to understand the cause of the error.

Hello Vitaly,
Sorry about the issue you encountered.
At my side, the similar test setup is running since 9 days without any issue.
The test setup is polling one device every 30 minutes. I will increase the asynchronous requests frequency to see if it makes the issue happen.
Also, are you able to SSH the device? If yes, then I would be interested in having the traces (with logread command) when you meet the issue.
Thanks,
Nicolas

Hi Vitali,

I have seen in some instances, a slow or busy or “noisy” modbus network cause the “errno”: -16,“errstr”:“Device or resource busy” error.

Is it possible to test with a higher baud rate than 19200? Depending on the root cause, it could correct the issue or make it occur more often. This could help to isolate the issue.

Hello Nicolas!

I downloaded the log file from the fx30s gateway. The error I describe occurred on November 17 at 10:31, but for some reason there is no data for this period of time in the log, although there is data before and after this time. I’m attaching the log just in case, but the next time an error occurs, I’ll try to save the log at the time of the error.
tlds000_log_001.txt (1.4 MB)

Hi David!

I will try to find out if we have the ability to change the modbus speed of the slaves, since initially we did not expect to do this. If possible, I will test at a speed of 115200.

I found out about changing the modbus speed on slave devices. Unfortunately, we are not able to do this. We do not have this option.

Hi!

I got the Modbus error again, but on another FX30S gateway. Firmware 3.5.2.
Now I have log during the error action.
The Modbus settings are the same as I described in the first post. There are 7 Modbus servers configured with addresses from 1 to 6 and with address 128. But physically, at the moment only one Modbus device (“cc2” with address 4) is connected to the FX30S gateway. The gateway I have just configured so that it asynchronously polls via Modbus only the connected device “cc2” at address 4 every 5 seconds. A synchronous request should not be made (before this, the gateway was configured to asynchronously poll all devices with addresses from 2 to 6). This can be seen from the screenshot, where in the field “send” for the device “cc2” the last request was 1 minute ago, and for the device “cc3” 18 hours ago. Also, the screenshot shows that the “cc2” device responded without errors, but the response came not from an asynchronous request, but from a synchronous one. But there shouldn’t be a synchronous request. It can also be seen that there was no asynchronous request to the “cc3” device for 18 hours, but there was a synchronous request and a response was received, indicating that there is no device with this address.

I wrote the log “tldszzz_log_20221207_001.txt” from 15:00 on December 7th. The log shows that Modbus errors occur when polling devices with addresses 1, 2, 3, 5, 6. Address 4 is not in this list, since the device exists and responds. But there is also no address 128, which has not had a single asynchronous request before, as you can see from the screenshot.

Here I lost electricity and the log recording was interrupted.
Perhaps if I make at least one asynchronous request to address 128, then this address will be constantly polled by synchronous requests like the rest? I’ll check now.
It so happened that at that moment I lost electricity again, and after it appeared, it turned out that the gateway had not been visible in the Octave for 2 hours, although the gateway had been on battery backup all this time. But it is possible that due to the fact that the electricity was lost throughout the area, including the towers of the cell operator, this affected the freezing of the gateway.
I rebooted the gateway manually. The Modbus error did not disappear.
Now I will try to do the experiment about which I wrote above. I’ll make one asynchronous request for the “broadcast” device at address 128, and if I’m right, I’ll see the constant synchronous polling of this device begin.
It didn’t work. I tried 3 times. No synchronous requests appeared for address 128, but no response came to the asynchronous request either.

I’ll save the log for today.
Log “tldszzz_log_20221208_002.txt” I wrote from 16:00 December 8.
For some reason, the log recording was interrupted, I tried to write the log again to the file “tldszzz_log_20221208_003.txt”, but the recording was soon interrupted again.

The log shows that Modbus is synchronously polled every second, and I have a setting for asynchronous polling every 5 seconds.

I hope this helps somehow.

tldszzz_log_20221207_001.txt (2.7 MB)
tldszzz_log_20221208_002.txt (1.9 MB)
tldszzz_log_20221208_003.txt (665.1 KB)

Hi Vitali,

Thank you for the data, I am looking at the logs.

1 Like

Hi Vitali,

On my side I tried again to reproduce, mixing both async and sync readings. I never could reproduce a situation where a not wanted read has been observed.

In my different tests I also tried to plug/unplug the Modbus in order to force some timeouts, suspecting something around timeouts modifying the reading queue.

Hi all!

Yesterday I got the Modbus error again, but now on the equipment working in the field. Before that, it appeared only on test samples. The error appeared on firmware version 3.5.2. Resetting the Blueprint and filling in a new Blueprint did not help. The redeploy Firmware also did not help, and in my opinion the redeploy does not work as it should, because, passes very quickly. Apparently, instead of redeploy the firmware, it’s just a check.
The rollback of the firmware back to version 3.3.1 helped.