Ram(memory) heap fragmentation.

[snahmad75]

Hi,

I am using std::string which allocate and reallocate ram on heap every time function gets called.
Is this bad on Esp32. I know general rule on embedded platform to allocate all your heap variables once.
Should I avoid it.

Another Question.

I am using ESP32 WROOM-32 board. I know I have Flash of 4MB for code and nvs, etc.
but I do not know internal RAM size?
Can some one point to data sheet which can tell me total RAM size used by stack and heap.



Thanks,
Naeem

[chegewara]

http://esp-idf.readthedocs.io/en/latest ... o8uint32_t

http://esp-idf.readthedocs.io/en/latest ... eeheapsize

[snahmad75]

Yes I am using that

heap_caps_get_free_size(MALLOC_CAP_8BIT)

as first line of code in my main. which tells me 256kb. After some allocation. it called again which shows remaining heap size which is less 256kb.

I need to know total internal ram size. RTOS must be using as well before main starts.

[kolban]

The ESP32 has 512K of RAM. Some of that is used as cache for code loaded from firmware. Some of that is used for writable variables in your application (.bss and .data). Some of it is used for internals workings of ESP-IDF (including FreeRTOS). What is left is available to you as heap. When you called:

heap_caps_get_free_size(MALLOC_CAP_8BIT)

You were told how much RAM remains on the heap for future allocation by your running applications.

The best source of knowledge of RAM layout and usage is the ESP32 Technical Reference manual. Also realize that the heap size shown at the very start of an application will diminish quickly as you start leveraging major subsystems like WiFi, TCP/IP and BLE as they will carve out what they need from the same heap.

[snahmad75]

Thanks useful tips.

[snahmad75]

Hi Kolban,

You can doing heap allocation and deallocation at runtime in your C++ class.

for example.

HttpResponse::sendFile


Does this cn cause heap fragmentation on Esp32 after running for long time.

[kolban]

The potential for memory fragmentation is a potential issue in all applications where storage is dynamically allocated. See the following:

https://stackoverflow.com/questions/377 ... gmentation

The summary appears to be that danger is introduced in a long-running application that allocates memory which is "long-lived".

We would also want to study the heap management algorithms in ESP-IDF to see if they partition their allocations from heap based on size. For example, "small" allocations from one part of the heap and "large" allocations from another part of the heap.

The alternative to dynamic allocation is, of course, static allocation. With static allocation one carves out the space one needs at the start of the program and that way we will never fragment. The problem with this is that we may need more storage over the lifespan of an application than we could pre-allocate at any one given time.

I'm sure there are treatise on memory fragmentation and how to design applications to avoid it. In my travels in ESP32 world, I have not yet come across a problem that was caused by fragmentation. Now, that doesn't mean they aren't there ... just that I haven't come across one yet. That said, it doesn't take much to engineer one. For example:

Repeat the following:

Code: Select all

while(we have memory free) {
   allocate 999 bytes;
   allocate 1 byte;
}
// Here memory used is 100%

free all the allocations of 999 bytes;
// Here the vast majority of memory is free

allocate 1000 bytes;   // Here we will likely fail even though the total amount of RAM free is likely >> 1000 bytes ... there isn't a contiguous 1000 byte area.

[ESP_Sprite]

Fyi, depending on the heap allocator that does not even have to be an issue. Some heap allocators have different heaps for 'large' and 'small' blocks, specifically to stop this from happening. I don't quite remember if the esp-idf heap allocator has this, however.

[Coopdis]

I've been trying to figure this fragmentation thing out as well. I've come across this article...

https://www.freertos.org/a00111.html

Which talks about the 5 different memory allocation schemes that are available through FreeRTOS. Both "heap_4" and "heap_5" appear to enable FreeRTOS to do active defragmentation of memory (at the cost of determinism I expect).

For my (current) app, I'm happy to rely on the auto-defragmentor... but I do need to know if it is engaged or not? Better yet, how to configure it?

Hopes this helps push the ball down the court on this question.

C

[ESP_Sprite]

What we use is effectively a custom implementation of the heap_5 algorithm, if memory serves, so it coalesces free blocks. There's not much you can configure on it: if it sees two adjacent free blocks, it simply joins them.