Since upgrading to 7.51 started getting a restart and a backtrace. The backtrace is sent to a FTP server and decoded with the virtual port. The weird thing happening now is that instead of it saying decoding backtraces the only feedback is in the trace window:
2012/08/01;09:02:32:743;001;18;4;NOT planned decrement value …
All other backtraces seem to still work with 7.51 it is just this error which doesnt decode.
Attached is the .txt and .bin backtrace files if anyone could please give some feedback.
attached again the backtrace which gets sent to the FTP using the adl_errRetrieveNextBacktrace(…) with 7.47. Using the same method to decode using the virtual port, with the only difference being in the Firmware selected to reflect the older Firmware Package (7.47.0.201202010317), I am able to decode backtraces as normal:
Invalid memory write operation (ARM Data Abort caught at 002AAC5A, Current Task 0x1F by CP15 )
_ZN6Gendac6Sefeko8Firmware3Ftp14CommandHandlerEP21_adl_atCmdPreParser_t+3e5
adlint_atProcessCmdSubscription+27a
adl_atCmdHandler+f6
mos_headerInitTask+6a
Unknown function (13a9)
Unknown function (55555555)
If anyone is using a similar method to remotely decode backtraces, or if there is an alternative simpler method which I could try would like to hear about it. I would also just like to confirm that my way of thinking is right because the error comes in where the backtrace is retrieved which seems to be accessing the wrong area of memory.
NOT planned decrement value ...
Unknown function (42e78)
Unknown function (24bd9)
Unknown function (24c75)
Unknown function (378d7)
Unknown function (38c49)
Unknown function (60371)
Unknown function (203a0)
This one started happening almost immediately after upgrading to 7.51, sometimes get stacks of these backtraces which are not decoded by DS.
2. Backtraces analysis retrieve function bug
As mentioned before it looks like the adl_ calls, adl_errStartBacktraceAnalysis, adl_errRetrieveNextBacktrace and adl_errEraseAllBacktraces has a bug where it incorrectly retrieves the backtraces, also in 7.51.
We can reproduce the offline backtrace decoding issue: ADL backtraces retrieval service seems indeed broken from 7.51
No idea about “NOT planned decrement value” (any tip to reproduce it? could you isolate a small piece of Open AT software causing this error to happen? does it make the module to reset (sometimes FW errors are logged without reset)?)
Anyway, we’ve pushed both issues to the embedded team for investigation.
Thanks for your feedback.
Thanks for the feedback daav. I’m glad it can be reproduced on your side.
We have been trying to reproduce the error with our standard firmware without success. We have however seen that this also gets generated in 7.47 but only 2 units in the last week compared to almost all our test units on a daily base. I’m not sure if our newer firmware has solved some of the resets or if it is an actual bug which has gotten worse with 7.51. On a unit with a bad signal it seems to be triggered more but havent yet been able to prove it and only a hunch.
To be honest I’m not sure if the bug resets the unit. How our remote backtraces work is after a unit reset the first time it connects to the server it checks for a backtrace and then sets a flag which tells it never to check again. So it could happen that the “NOT planned decrement value” is triggered and stored and only when we reset the unit due to some other reasons does it get sent to the server.
Don’t know if these .record files help but the attached one is the latest. It was a unit which I took home over the weekend to check on server connection. No problems/unexpected resets were found but the unit did run out of power and shut down. This morning when plugging back into the charging power supply it detected the backtrace and sent it to the server. So it could be tower switching, bad signal, low power related but as of yet havent been able to consistently reproduce.
I will implement some timer to check every x-min for backtraces and see if it doesn’t maybe get logged without a reset and let you know. Alternatively I can try write a small Open AT program but it will take some time as we have lots of components which dependent on others and seen as I cant force the error to happen not sure where to stop before I think I have isolated the problem.
“NOT planned decrement value” is an error message generated by from Layer1 controller of the firmware and is related to GSM time update. It does not cause the module to reset.
To analyze the issue, could you please try to reproduce the issue again and collect the following traces:
HWL 29
L1C 9,10,11
L3MM 3, 4, 8, 9, 15
L3RR 3, 4,8,14
SEQ 1,6,11
It is essential that the traces be collected from power-up (RR_INIT_REQ should be seen in L3RR trace level) & with the correct work space corresponding Firmware in Developer Studio.
Thank you for the feedback sushil and daav. Had a bit of time today to implement the necessary changes to try and get a log of the error with the required traces on. The system is currently running for 5 hours without any problems but will post again if anything is found.