Asynchronous Modbus requests not working

Hi!

When switching from firmware 3.3.1 to firmware 3.5.1, I found a problem that sometimes Modbus stops working in asynchronous mode. That is, at first Modbus worked, but eventually stopped. Moreover, when it stopped working, neither rebooting the gateway, nor reinstalling the blueprint, nor reinstalling the firmware helps restore work. Checked on firmware 3.5.1 and 3.5.2, Gateway FX30S .
The error is that I am making a request in the “dh://modbus/rtu/cc3/request/send” resource, and the response should be in “dh://modbus/rtu/cc3/request/value”, but instead of this, response comes in “dh://modbus/rtu/cc3/value”.

The “dh://modbus/rtu/cc3/value” resource should not receive a response, since this resource should be polled once every 100 years (3155760000 seconds). I tried to set the polling period and 0 seconds and null, the result is the same, the polling of this resource occurs on an asynchronous request.

Hello Vitali,

Could you let me know the name of your Octave account? I would like to check your configuration.

Hello David!

I hope this is what you asked for:
https://octave.sierrawireless.io/company/ekatra_test/device/d5f2973c3f65503dd7f7a25f2

Hello @v.babich

I installed your blueprint on my 3.5.1 FX30 and I am not able to reproduce that issue:

The error is that I am making a request in the “dh://modbus/rtu/cc3/request/send” resource, and the response should be in “dh://modbus/rtu/cc3/request/value”, but instead of this, response comes in “dh://modbus/rtu/cc3/value”.

I can indeed see the asynchronous response in /request/value.

To avoid periodic polling you can set the period to 0 like this

With this setting there is no polling occurring on async requests


The 17 minutes match the time where the device powered on and the Developer MLode pushed to the Cloud all the resource tree values.

Hope this helps,
Nicolas

Hello Nicolas!

Initially, I had a polling period of 0 seconds and asynchronous requests worked like you did. But then something broke and asynchronous requests stopped working. And after that I began to try to set other polling periods like null or 3155760000 seconds.

If you cannot reproduce the problem on your device, you can try any manipulations on my device to fix it. This is a test device and you can’t break anything there.
Or suggest something else.

I rolled back to firmware version 3.5.0, Modbus does not work on it either. That is, Modbus does not work on firmware versions 3.5.0, 3.5.1 and 3.5.2.
After I rolled back to firmware version 3.4.2, Modbus worked.

Hi Vitaly,

Glad you are finally able to make your solution work.

Can you give more detail about the “something broke”? Since I’m not able to reproduce the failing asynchronous requests I’d like to understand what happens.

Thanks,
Nicolas

Hello Nicolas!

I have been working on a new Blueprint version on firmware 3.5.2.
Like you, at first the Modbus worked for me on firmware 3.5.2.
But after a while Modbus stopped working. What caused this I don’t know. I just worked with the Blueprint, deleted some Edge Actions that were no longer needed, created new ones, often saved new versions of the Blueprint. But even if we assume that I messed up something in the Blueprint and therefore Modbus does not work, then this is not the case, since the same Blueprint works on firmware version 3.4.2, and on versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.

I hope that we will find the cause of the error, because I do not want to stay on the old firmware version.

Hello Vitaly,

I’m trying to reproduce with a test setup:

  • FW 3.5.2
  • modbus polling period set to 0
  • modbus async requests sent every 10 minutes from an edge action. read is done on the 2 HLD registers of my modbus asset.

After 18 hours I can’t see any issues. Async requests return every time the last data. The periodic polling is successfully disabled.

I have some questions:

  • do you think my test setup is similar to what you are doing?
  • after how much time did you have modbus stopped working?
  • can you give more details about “Modbus stopped working”? Do you mean bus timeouts?
  • can you give more details about “versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.” ? Can you explicit what did you do exactly?

Thanks
Nicolas

Hello Nicolas!

do you think my test setup is similar to what you are doing?

Yes, your setup is similar to mine. Only asynchronous modbus polling goes every 5 seconds.

after how much time did you have modbus stopped working?

Modbus stopped working after a few days.

can you give more details about “Modbus stopped working”? Do you mean bus timeouts?

Modbus has stopped working - this means that the Modbus has stopped working correctly. Modbus started sending responses to asynchronous requests to resources associated with synchronous (periodic) requests.

can you give more details about “versions 3.5.X even manually sending the Modbus Command through the interface Octave doesn’t work.” ? Can you explicit what did you do exactly?

Through the Octave web interface, I went to the resource “modbus/rtu/cc0/request/send” and pressed the “send command” button. In the pop-up window that appeared, I set the request parameters and sent the request. The answer was expected in the resource “modbus/rtu/cc0/request/value”, and received in the resource “modbus/rtu/cc0/value” as I previously described.

I am now installing firmware 3.5.2 again and will try to catch this error again. Now I will monitor my actions and if I succeed, I will write here about the result

Hello!

I got the above error again.
To get this error again, I installed firmware version 3.5.2 on November 11, 2022 and created a new Blueprint from the existing one.
After I tried to add and remove Virtual resources, Timers, Edge actions, saved the changes to the Blueprint, but the error did not appear.
Then I had to do other work and for a while I left the FX30S gateway running without my intervention.
And today, November 17, 2022, a Modbus error appeared {“errno”: -16,“errstr”:“Device or resource busy”}. After that, the response to the Asynchronous request again began to come to the resource associated with the Synchronous request.
The frequency of Asynchronous requests is set to 1 time per 5 seconds. 4 devices are polled. It took 6 days before the error appeared.
But a few minutes after rebooting the gateway, the error disappeared. Modbus started working normally again.

I hope this information helps to understand the cause of the error.

Hello Vitaly,
Sorry about the issue you encountered.
At my side, the similar test setup is running since 9 days without any issue.
The test setup is polling one device every 30 minutes. I will increase the asynchronous requests frequency to see if it makes the issue happen.
Also, are you able to SSH the device? If yes, then I would be interested in having the traces (with logread command) when you meet the issue.
Thanks,
Nicolas

Hi Vitali,

I have seen in some instances, a slow or busy or “noisy” modbus network cause the “errno”: -16,“errstr”:“Device or resource busy” error.

Is it possible to test with a higher baud rate than 19200? Depending on the root cause, it could correct the issue or make it occur more often. This could help to isolate the issue.

Hello Nicolas!

I downloaded the log file from the fx30s gateway. The error I describe occurred on November 17 at 10:31, but for some reason there is no data for this period of time in the log, although there is data before and after this time. I’m attaching the log just in case, but the next time an error occurs, I’ll try to save the log at the time of the error.
tlds000_log_001.txt (1.4 MB)

Hi David!

I will try to find out if we have the ability to change the modbus speed of the slaves, since initially we did not expect to do this. If possible, I will test at a speed of 115200.

I found out about changing the modbus speed on slave devices. Unfortunately, we are not able to do this. We do not have this option.