Function names in backtraces

IoT Modules Development tools
1

Hello,

I have been experimenting with viewing backtraces from the “bug” sample app.

I do get some data in the backtrace view, which is a good thing, but I don’t get function names, generally speaking. (I do get some function names for OpenAT functions, but seemingly none from my own code.)

For example: after running AT+BUG=1 (which simply writes to an invalid memory address), I get a backtrace that looks like:

Watch dog reset. Tsk 24
_rand_init+0
_rand_init+0

Another example:
AT+BUG=2 (ADL “halt” error)

Error caused by the application
_rand_init+62c68
_rand_init+33b8b
_rand_init+33c41
_rand_init+1

What am I doing wrong?

Thanks,
Nick

2

For Watchdog Resets, the stack trace can’t be reliable, since it is the stack state when the CPU resets because of the watchdog (i.e. completely random WRT. the application execution).

For the other one, you should indeed have function names correctly decoded.
You must be sure that the AXF file you provided in the Backtraces view is exactly the same that the one that was running on the target when the backtraces where stored in the target memory. If you have modified the application, it is not possible to decode these backtraces properly. You have to erase the backtraces list, and try to reproduce the issue to have a fresh backtrace stored with the correct AXF file.

3

Thanks for your reply, daav.

I ran the test with the “Bug” sample from a clean slate, but the decoding is still messed up, and I don’t get a correct backtrace.

Here’s what I did:

  1. Cleaned the project. (I even deleted the folder from my disk to be sure everything was gone.)
  2. Rebuilt and re-downloaded the app.
  3. Started the app
  4. Cleared the existing backtraces (with AT+ERRBKTRC from the Bug sample.)
  5. Verified the backtraces had been cleared (with AT+GETBKTRC=?)
  6. Ran AT+BUG=2 (to cause adl_errHalt to be called). (A valid backtrace should be generated in this circumstance.)
  7. Switched to the Backtraces view and set the AXF file path to the .axl file just generated.
  8. Hit refresh in the Backtraces view. I saw one backtrace (as expected) but it’s clearly wrong. It’s referencing _rand_init (as before) which was definitely not the running function. (So I guess function names are showing, but the decoding is screwy.)

The output again looks something like this:

Error caused by the application
	_rand_init+62c68
	_rand_init+33b8b
	_rand_init+33c41
	_rand_init+2

Any suggestions?

I’m using a Fastrack xtend (FXT 003)
-“Developer Studio”,“1.2.0.201012171243-R6026”
-“Open AT Embedded Software Suite package”,“2.34.0.201009161320”
-“Open AT OS Package”,“6.34.0.201007281407”
-“Firmware Package”,“7.44.0.201008311212”

4

Please can you tell which Build Configuration you are using?

5

I’ve tried this with the [Target]_ARM_ELF_GCC_Debug and [Target]_ARM_EABI_GCC_Debug configurations.

After more testing, it seems I can get working backtraces with the Bug sample on the FXT009 (in both those configurations) but still not with the FXT003 as initially reported.

Strangely, for the FXT003, the firmware shown in the Status view is R7.44.0.201007221223, even though I attempted to update it (and the bootloader) via the status view to 7.44.0.201008311212. (The FXT009 has no such issue.) Perhaps that could be the source of my problems?

Also, backtraces aren’t working for our C++ app, even on the FXT009 - it’s only with the bug sample at the moment that I’ve had any success.

The versions of Developer Studio, etc., are the same as before (in the prior post.)

6

Sounds indeed as a potential issue. We’re going to take a look, expecting to find an explanation on this. I’ll post here as soon as we’ve found something.

Please can you elaborate on this point? How do you make the C++ application crash, and what is not working in the decoding?

7

OK, reproduced here: backtraces decoding is indeed not working on 3G FXT modules.
We’ll take care of fixing it in the next release.
Thanks for the bug report.

8

We have an intermittent crash in a C++ program that I’m trying to debug. I don’t know where it’s crashing, so that’s why I started trying out the backtrace analysis stuff.

The problem is that after I point the backtraces view to the AXF file path, the “Retrieve backtraces” dialog pauses indefinitely at the “Loading unwinding data” step (and I can’t cancel it.) Just for fun I tried using the bug sample .axf file, and the process completes (though of course the results are invalid because the bug sample has nothing to do with my C++ app.)

This has happened on all modules I’ve tried it on, even the FXT009 on which the Bug sample backtrace decoding worked.

So, in essence there’s some difference between the bug sample and my app that prevents backtraces working, but I don’t know what that is. The fact I’m using C++ just stands out as the first thing to suspect. :wink:

Thanks for listening.

9

We’ve fixed a couple of bugs in the Backtraces view in next release (including the FXT003 one), which will be available in the coming weeks. You should give another try with this one when it will be available.

10

I’ll be hanging out for the next release, that’s for sure.

11

Same issue on Q26Extreme. Dialog “Retrieve backtraces” doesn’t end using our AFX file. Using bug sample AFX file “works”.

So we are also looking forward to the next release.
Oliver.

12

It’s possible to get the backtraces via the function adl_errRetrieveNextBacktrace(). Then the backtraces may be analyzed manually.

To get an idea on how to trace through the backtrace heuristically, see http://carbidehelp.nokia.com/help/topic/com.nokia.carbide.cpp.debug.crashdebugger/html/DebuggingInformation/CrashDebuggerCallStack.guide04.html.

Example for Q26:

  • The backtrace and the related *.map file (compiler/linker output) is needed.
  • The code section is at address 0x00260000…
  • This is one line of the backtrace: 00 00 00 D3 B9 27 00 3C DE 00 0C 02 70 00 00 4B
  • D3 B9 27 00 => 0x0027B9D3 => searching in the map file for 0027B9 => 0x0027b994 Target_Reset(Target_ResetReason_t).
  • 0x0027b994 is the start address of the function Target_Reset(). 0x0027B9D3 is somewhere within this function.
13

There do seem to be at least some cases where, if DS cannot read the backtraces, then neither can adl_errRetrieveNextBacktrace()

See: Retrieving Backtraces
And: Retrieving Backtraces

Or you could get TMT to do it: antronics.co.uk/downloads/OA … _Files.pdf

Thanks - I’ll look into it…

14

The GNU Binary Utilities (binutils) have some handy tools for this kind of thing; eg,

  • nm lists the symbols from object files
  • objdump displays information about one or more object files
  • addr2line translates addresses into file names and line numbers. Given an address in an
    executable or an offset in a section of a relocatable object, it uses the debugging information
    to figure out which file name and line number are associated with it.
  • readelf displays information about one or more ELF format object files

Search for binutils.pdf in your DS installation for further details…

15

Is it actually possible to use the TMT to decode backtraces with builds produced by DS 1.2.0?

In step 4 in OAT_Playing_Backtrace_Files.pdf, we are instructed:

This “workspace” file seems to be a .wks file, (from looking at the pics in document). There’s no .wks file generated from a build in DS 1.2.0, it seems.

Where does this file come from?

Ignoring that, I have had some success - I can hit the “play” button in the “replay frames” dialog and see some info appear in the Trace screen, but it’s no more informative over what you get with “Request EEPROM logged errors” in the TMT. (Obviously the TMT needs to lookup the addresses in the backtrace somewhere…)

So, can I use DS 1.2.0, or do I need to use a previous version of the IDE, or some other tool?

(P.S. I tried pointing the TMT to my project’s .axf file in the Preferences dialog, but that made no difference.)

16

It was produced by development tools prior to DS 1.2.0.

DS 1.2.0 doesn’t use them and, therefore, doesn’t produce them.

17

“Offline” backtraces decoding is indeed not possible neither with DS 1.2.0, nor with DS 2.0.0
It’s planned to reintroduce this feature in the next release.

Meanwhile, you can do it with TMT, even if the “workaround” is not easy. Actually, from 1.2.0 DS don’t generate anymore the debug axf file required by TMT for backtraces decoding.
Anyway, it’s still possible to do it manually.
Here is the procedure:

[]Locate the AXFShrinker tool in a former DS 1.1.2 install (or older). This tool is indeed not integrated anymore from DS 1.2.0. It should be located in the plugins/com.wavecom.openat.ide.ebs.build.core.win32_XXX/os/win32 sub-directory of your old DS install;
Please just tell if you can’t grab it, I will post an archive here containing the file.[/]
[]Run “AXFShrinker yourapp.axf backtraces_app.axf” to produce the backtraces_app.axf debug file[/]
[]Locate the Firmware package corresponding to the Firmware running on the target[/]
[]In this package, look for the backtraces64k.axf file[/]
[]You now have to concatenate these two axf files (FW package’s one, and your app one;
A python script is integrated to do this under Windows (cat.py, integrated in com.wavecom.openat.ide.ebs.build.core_xxx/scripts plug-in), or if you have a Linux shell (or even maybe just a MinGW or Cygwin shell), just use “cat” utility (“cat backtraces64k.axf backtraces_app.axf > backtraces.axf”);
Let’s call the concatenation result “backtraces.axf”[/]
[]Now back to TMT, you need to provide, in the settings dialog, two files: the backtraces.axf file you’ve just generated, and the xxx.symbols files generated by DS in the output folder of your application project.[/]

From there, you should be able to decode offline backtrace files (retrieved from the ADL API) with TMT.
But please note that:

  • backtraces.axf file has to be regenerated each time the application is rebuilt (application AXF has changed…)
  • Backtraces decoding algorithm in TMT is sometimes just unable to decode backtraces

Hope this will help…