Memory leaks


#1

Hello,
I have tricky problem with Modem crashing after about 7 days (locks up and network status light stays on).

Difficult to know what’s going on, and I thought I’d try to rule out memory leaks first. But I can’t find a way to do this. There have been similar threads in the past but they seem to be broken.

I’ve tried to use static memory allocation where I can, but I do have some memget functions.

I’ve had a little play around with memGetInfo but these figures stay the same, even if I deliberately introduce massive memory leaks.

Any help much appreciated,

John


#2

People have mentioned putting a wrapper around the memget function with your own up/down counter in it.

Also check out the adl_errSubscribe(ErrEventHandler); it can tell you if there was a watchdog reset for instance. The values read back via the TMT interface are obscure but it can sometimes give you a clue. Check for any while loops or tasks that might take along time triggering a watchdog reset.

Good luck.


#3

Thanks Ben, I like both of those ideas and will try them.

John


#4

Hi,
Memory management is broken in OpenAT. Memory get de-fragmented over time and can not be recovered.
At some stage I did a small app to prove the point and send it to Sierra Wireless. They acknowledge the problem and say:
“We confirm your observation. We can not re-use the freed memory allocated by adl_memGet()”
They suggested workaround is to implement on memory management by creating large memory pool and allocate memory from it in application. And manage it in application. No plans to fix it in the near future as far as I know

Since then , I also discovered that internally openAT is also using dynamic memory allocation. For example when one sends an AT command from software (Adl_atCmdCreate()). Result is same – eentually system runs out of memory.

We have installed error handler and, when problem is detected, we reset the modem ourselves.

Rudolf


#5

Oh shit (sorry, but nothing more to say)!
Is it really so? In any versions? Can someone confirm that?

I must check it now…


#6

I take it you mean fragmented :question:


#7

This is really alarming! I’ve been carefully trying to manage maintaining connections in various network conditions (spending alot of time, i.e. cash, trying to work through various Open AT problems), but it looks like I might as well just reset once a day as its going to crash anyway!!! :angry:


#8

Yes - absolutely shocking! :angry:

What on earth are SiWi thinking :question: :open_mouth:


#9

Hmm. I can’t reproduce memory allocation errors. If memory fragmented so no solid block of requested size - yes, error (but what we can expect? relocation mechanism? wonderful memory defragmenter?)

But whole available memory (close to) can be allocated again and again, big, small or random size blocks. 100000+ allocations tested.


#10

This, of course, is one of the key reasons why dynamic allocation is generally best avoided in embedded systems!


#11

Hi,
I can e-mail sample app to demonstrate the problem, so you can check yourself.
Algorithm is pretty simple – allocate number of 1k blocks up to the maximum you can get (in my case about 234). System eventually runs out of memory.
Then try to deallocate every fifth one. So, I allocated 234 and freed up 47 1k blocks. One would expect that about 47k can be allocated again, but, in reality, I can only allocate up to about 1k in a single chunk.

I tried the code with R74a, but, as far as I know, problem still exists in later versions.

Rudolf


#12

Why not just upload it into the Code Sharing Area?
That’s what it’s there for!

Would one?

You don’t have a contiguous block of 47K - you only 47 disjoint 1K blocks!

See above.

How would the system “join” these isolated blocks into a single 47K block?
You would need some kind of “Garbage collection” to do it.

That is actually rather different from saying that Open-AT cannot re-use freed memory at all!

As already noted, this is one of the key reasons why dynamic memory allocation is best avoided on embedded system?

But this limitation should be clearly documented.


#13

No. Just as ANY other mechanism, dynamic memory must be used properly.


#14

You should us dynamic memory allocation when you need memory for short period of time. If not, use static buffers instead. If there is garbage collector then it would consume processor time, which is obviously not good.


#15

And, even with a garbage collector, how can you “coalesce” disjoint 1K blocks into a single contiguous block??


#16

If dynamic memory exists, it should perform “garbage collection”. Even if user application has to do it manually. Same problem exists when using flash memory and there is a way to “recompact” it so that one can use large consecutive block.

I agree that dynamic memory is not the best way in embedded system, but, sometimes, there is no way around it. We are working on an embedded modem that sends data through it and this generally involve creating/destroying buffers.

OpenAT uses dynamic memory internally as well, when processing AT commands, for example.

Rudolf


#17

Why not just upload it into the Code Sharing Area?
That’s what it’s there for!

Looks like attachments are not allowed. i tried .c and .txt files

Rudolf


#18

… and how you imagine this collector? Complete abandonment of memory pointers? Complex and slow relocation mechanism?


#19

What rudolfl describes as a problem is actually the expected behavior of dynamic memory allocations and will not change. The problem can only be “workarounded” on systems that support the types of hardware exceptions, like the protected mode of the x86 and x64 cpus, what applications see as complete memory block can be scattered in parts in memory and the disk.


#20

I have just experienced this using Q64s. No error message, nothing just a reset out of the blue. I have reduced the problem by cutting back the use atcmdcreate/atCmdSend. But as a failsafe I have used some persistent variables to store important accumulating statics. (ADL_MEM_UNINIT() etc). Tested with an AT+CFUN=1 values are retained but not with AT+CPOF=1.
Can anyone please tell me for sure that these variables are retained in this sort of memory related crash?