NOT planned decrement value ... (7.51 Backtrace)

Hi,

Since upgrading to 7.51 started getting a restart and a backtrace. The backtrace is sent to a FTP server and decoded with the virtual port. The weird thing happening now is that instead of it saying decoding backtraces the only feedback is in the trace window:

2012/08/01;09:02:32:743;001;18;4;NOT planned decrement value …

All other backtraces seem to still work with 7.51 it is just this error which doesnt decode.

Attached is the .txt and .bin backtrace files if anyone could please give some feedback.

Module type: SL6087
Used packages versions:

  • Open AT Framework package (2.51.0.201206190958)
  • Open AT OS Package (6.51.0.201206010944)
  • Firmware Package (7.51.0.201205311751)
  • Internet Library Package (5.54.0.201206011257)
  • Security Library Package (1.16.0.201206041340)

~C

Nice idea :smiley:

Can’t help with the problem though, sorry!

Thanx :smiley:, took a while to get working but really helped with some restarts when not connected to a pc.

Been testing a bit and it seems to me like something is wrong in the actual adl call to retrieve the backtraces.

Did a test using the bug.c sample program to generate a bug:

/* Data abort (forbidden write) */
u32 WmRWbase = 0;
u32 Offset = 0;
u8 *MemSlot = NULL;
u8 Value = 0;

/* Swift RAM base */
WmRWbase = 0x45;

/* Set address : WM RAM base address + 1k */
MemSlot = ( u8 * ) ( WmRWbase + Offset );

/* Crash */
*MemSlot = Value;

And using something similar to:

adl_errRetrieveNextBacktrace ( backtraceHandle, buffer, size );
DUMP(4, buffer, size);

to retrieve the backtraces (Same as before nothing has changed since it working in 7.47).

I remember a post from daav which helped me get backtraces working which said:

but in the actual dump I dont see this sequence.

The weird thing is having the module connected to my PC while I generate the error correctly decodes the backtrace:

Except RTX: task AppInit, bad memory access by 0x315638
	_ZN6Gendac6Sefeko8Firmware3Ftp14CommandHandlerEP21_adl_atCmdPreParser_t+82b
	adlint_atProcessCmdSubscription+27a
	adl_atCmdHandler+f6
	mos_headerInitTask+6a
	Unknown function (203a0)

but when I check the DUMP or the file sent to the FTP they are the same and both dont work.

Actual Dump in the trace window:

00 00 00 00 11 00 00 00 FF FF FF FF 42 61 63 6B 74 72 61 63 65 20 52 65 74 75 72 6E 20 00 00 63 70 70 3A 31 FA DE FE CA 03 00 00 00 00 00 00 00 00 00 00 00 0B 00 00 00 FF FF FF FF 46 74 70 CF 63 70 70 3A 32 38 34 00 FF FF FF FF 54 63 70 49 50 43 6C 69 FA DE FE CA 0C E4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 14 07 11 18 50 72 61 63 65 20 52 65 74 75 72 6E 20 25 64 3A 00 65 73 20 FA DE FE CA 01 00 00 00 00 00 00 00 D0 B2 11 18 8A DE FE CA 05 00 00 00 00 00 00 00 00 00 00 00 14 C1 11 18 E4 BF 11 18 00 00 00 00 06 00 00 00 20 5D 20 CA 02 10 00 00 00 00 00 00 00 00 00 00 00 2C 41 2C 33 2C 30 35 FF FF FF 7F 00 00 00 00 36 2C 31 32 2C 31 33 2C 32 39 2C 2C 2C 2C 2C 2C 9A DE FE CA 03 E4 00 00 00 00 00 00 00 00 00 00 01 00 00 00 FF FF FF FF 3A 00 34 2C 31 2C 31 33 EA DE FE CA 11 00 00 00 00 00 00 00 00 00 00 00 2C 34 32 2C 31 30 34 2C 32 30 2C 30 35 2C 36 30 2C 32 34 34 2C 31 37 2C 30 37 2C 30 33 2C 30 39 39 2C 2A 37 42 0D 0A 24 47 50 47 53 56 2C 34 2C 32 2C 31 33 2C 30 38 2C 30 33 2C 30 36 32 2C 31 39 2C 31 30 2C 34 33 2C 31 34 38 2C 32 36 2C 31 32 2C 32 31 2C 32 38 34 2C 33 33 2C 31 33 2C 31 33 2C 31 34 30 2C 31 38 2A 37 37 0D 0A 24 47 50 47 53 56 2C 34 2C 33 2C 31 33 2C 31 35 2C 30 36 2C 33 33 35 2C 2C 31 37 2C 31 33 2C 30 33 31 2C 30 38 2C 32 35 2C 31 30 2C 32 34 37 2C 31 34 2C 32 36 2C 33 38 2C 30 30 31 2C 33 30 2A 37 41 0D 0A 24 47 50 47 53 56 2C 34 2C 34 2C 31 33 2C 32 39 2C 31 31 2C 32 31 38 2C 32 32 2A 34 42 0D 0A 24 47 50 47 4C 4C 2C 32 35 34 34 2E 37 33 33 36 30 2C 53 2C 30 32 38 31 36 2E 34 33 34 36 38 2C 45 2C 30 38 31 38 33 38 2E 30 30 2C 41 2C 41 2A FA DF FE CA 02 00 00 00 00 00 00 00 00 00 00 00 03 00 00 00 FF FF FF FF 35 36 31 00

Ok can confirm that it has something to do with the 7.51 adl_errRetrieveNextBacktrace() call.

Tested by right clicking project->Properties->Open AT Application and then selecting the following packages used with 7.47

  • Open AT Framework package (2.37.0.201202280640)
  • Open AT OS Package (6.37.0.201202060950)
  • Firmware Package (7.47.0.201202010317)
  • Internet Library Package (5.43.0.201202100610)
  • Security Plug-in Package (1.5.0.201108111447)

Same code base, same way of triggering error, same memory mapping also RELEASE mode build.

Dump:

Backtrace Return 624: [Ftp.cpp:285] 
AA 48 20 1B 00 41 52 4D 20 44 61 74 61 20 41 62 6F 72 74 20 63 61 75 67 68 74 20 61 74 20 25 30 38 58 2C 20 43 75 72 72 65 6E 74 20 54 61 73 6B 20 30 78 25 30 32 58 20 62 79 20 43 50 31 35 20 00 5A AC 2A 00 1F 00 00 00 00 00 61 00 00 00 00 06 00 00 00 20 66 6F 72 02 10 00 00 AA 10 22[b]04 0F 0A 05[/b] 10 02 00 00 5A AC 2A 00 45 CF 30 00 A2 FF FF FF B7 13 00 00 66 00 00 00 00 00 00 00 26 00 00 00 58 6A 10 18 50 E7 1F 18 E0 3C 00 00 02 70 00 00 36 65 32 13 3F 00 00 00 1B 00 00 00 18 F9 10 18 05 00 00 00 66 00 00 00 A2 FF FF FF 88 CD 30 00 6F D0 30 00 00 00 00 00 01 00 00 00 B8 01 11 18 01 00 00 00 70 03 10 18 5B C0 30 00 1C F6 1E 00 00 00 00 00 01 00 00 00 1E 00 00 00 08 00 00 00 06 00 00 00 38 7C 10 18 00 00 00 00 70 E7 1F 18 00 00 00 00 94 0A 07 18 18 0A 11 18 58 FE 10 18 38 7C 10 18 94 0A 07 18 F8 73 10 18 FE 00 00 00 F8 73 10 18 94 0A 07 18 00 00 00 00 98 0A 07 18 A9 C2 30 00 00 00 00 00 00 00 00 00 88 03 10 18 BC F3 1F 18 00 00 00 0C D7 C7 30 00 23 00 30 00 8C 0A 07 18 C4 E7 1F 18 C0 67 1F 18 BC F3 1F 18 A9 13 00 00 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 12

Notice the backtrace start frame 04 0F 0A 05

attached again the backtrace which gets sent to the FTP using the adl_errRetrieveNextBacktrace(…) with 7.47. Using the same method to decode using the virtual port, with the only difference being in the Firmware selected to reflect the older Firmware Package (7.47.0.201202010317), I am able to decode backtraces as normal:

Invalid memory write operation (ARM Data Abort caught at 002AAC5A, Current Task 0x1F by CP15 )
	_ZN6Gendac6Sefeko8Firmware3Ftp14CommandHandlerEP21_adl_atCmdPreParser_t+3e5
	adlint_atProcessCmdSubscription+27a
	adl_atCmdHandler+f6
	mos_headerInitTask+6a
	Unknown function (13a9)
	Unknown function (55555555)

If anyone is using a similar method to remotely decode backtraces, or if there is an alternative simpler method which I could try would like to hear about it. I would also just like to confirm that my way of thinking is right because the error comes in where the backtrace is retrieved which seems to be accessing the wrong area of memory.

Regards,

~C

It’s staring to look like two seperate errors:

1. NOT planned decrement value …

NOT planned decrement value ... 
	Unknown function (42e78)
	Unknown function (24bd9)
	Unknown function (24c75)
	Unknown function (378d7)
	Unknown function (38c49)
	Unknown function (60371)
	Unknown function (203a0)

This one started happening almost immediately after upgrading to 7.51, sometimes get stacks of these backtraces which are not decoded by DS.

2. Backtraces analysis retrieve function bug
As mentioned before it looks like the adl_ calls, adl_errStartBacktraceAnalysis, adl_errRetrieveNextBacktrace and adl_errEraseAllBacktraces has a bug where it incorrectly retrieves the backtraces, also in 7.51.

~C

Sounds like behaviors related to embedded side.
We’ll have a look to reproduce the issue(s?), and then forward them to the embedded team if needed.

Thanks daav, let met know if I can test something on my side. We reverted back to 7.47 for now but can always upgrade again to test.

~C

We can reproduce the offline backtrace decoding issue: ADL backtraces retrieval service seems indeed broken from 7.51
No idea about “NOT planned decrement value” (any tip to reproduce it? could you isolate a small piece of Open AT software causing this error to happen? does it make the module to reset (sometimes FW errors are logged without reset)?)

Anyway, we’ve pushed both issues to the embedded team for investigation.
Thanks for your feedback.

Thanks for the feedback daav. I’m glad it can be reproduced on your side.

We have been trying to reproduce the error with our standard firmware without success. We have however seen that this also gets generated in 7.47 but only 2 units in the last week compared to almost all our test units on a daily base. I’m not sure if our newer firmware has solved some of the resets or if it is an actual bug which has gotten worse with 7.51. On a unit with a bad signal it seems to be triggered more but havent yet been able to prove it and only a hunch.

To be honest I’m not sure if the bug resets the unit. How our remote backtraces work is after a unit reset the first time it connects to the server it checks for a backtrace and then sets a flag which tells it never to check again. So it could happen that the “NOT planned decrement value” is triggered and stored and only when we reset the unit due to some other reasons does it get sent to the server.

Don’t know if these .record files help but the attached one is the latest. It was a unit which I took home over the weekend to check on server connection. No problems/unexpected resets were found but the unit did run out of power and shut down. This morning when plugging back into the charging power supply it detected the backtrace and sent it to the server. So it could be tower switching, bad signal, low power related but as of yet havent been able to consistently reproduce.

I will implement some timer to check every x-min for backtraces and see if it doesn’t maybe get logged without a reset and let you know. Alternatively I can try write a small Open AT program but it will take some time as we have lots of components which dependent on others and seen as I cant force the error to happen not sure where to stop before I think I have isolated the problem.

“NOT planned decrement value” is an error message generated by from Layer1 controller of the firmware and is related to GSM time update. It does not cause the module to reset.

To analyze the issue, could you please try to reproduce the issue again and collect the following traces:
HWL 29
L1C 9,10,11
L3MM 3, 4, 8, 9, 15
L3RR 3, 4,8,14
SEQ 1,6,11
It is essential that the traces be collected from power-up (RR_INIT_REQ should be seen in L3RR trace level) & with the correct work space corresponding Firmware in Developer Studio.

Thank you for the feedback sushil and daav. Had a bit of time today to implement the necessary changes to try and get a log of the error with the required traces on. The system is currently running for 5 hours without any problems but will post again if anything is found.