How do I debug out of RAM conditions (SmartMatrix + AnimatedGifs + other) on ESP32

[marcmerlin]

I'm using SmartMatrix on ESP32 along with a Matrix of 64x96. I think SmartMatrix needs to allocate 2 arrays of 64x96 in 24bpp plus some more DMA arrays to push that data to the RGBPanel.
Then I have my own code (reasonably big), which you can find at "marcmerlin/NeoMatrix-FastLED-IR" if you google it (I think if I post a link, my message will end up in moderated queue) which also needs to allocate a buffer of 64x96 24bpp for a virtual framebuffer.
Last, but not least, there is a an animated gif decoding library that can use more or less RAM depending on how you set lzwMaxBits

I understand that ESP32 has multiple kinds of memory limits, and I'm hitting at least 1, probably 2.
If I set lzwMaxBits to 11, it won't build: region `dram0_0_seg’ overflowed by 3848 bytes

If I bring it down to 10, it builds, but crashes at start

Code: Select all

Global variables use 114948 bytes (35%) of dynamic memory, leaving 212732 bytes for local variables. Maximum is 327680 bytes

06:49:29.342 -> rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
06:49:29.343 -> configsip: 0, SPIWP:0xee
06:49:29.343 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
06:49:29.344 -> mode:DIO, clock div:1
06:49:29.345 -> load:0x3fff0018,len:4
06:49:29.345 -> load:0x3fff001c,len:808
06:49:29.345 -> load:0x40078000,len:6084
06:49:29.346 -> load:0x40080000,len:6696
06:49:29.346 -> entry 0x400802e4
06:49:29.412 -> E (172) boot: Assert failed in start_cpu0_default, /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/cpu_start.c:378 (res == pdTRUE)

Lowering further does not help with this crash. Instead, I can change my matrix down from 96x64 to 86x64, and then the code starts and outputs a lot of memory info. It does work, but it's the wrong resolution for my matrix

Code: Select all

Starting SmartMatrix Mallocs
Heap Memory Available: 281636 bytes total, 113792 bytes largest free block: 
8-bit Accessible Memory Available: 194284 bytes total, 113792 bytes largest free block: 
32-bit Memory Available: 281636 bytes total, 113792 bytes largest free block: 
DMA Memory Available: 194284 bytes total, 113792 bytes largest free block: 
SmartMatrix Layers Allocated from Heap:
Heap Memory Available: 247096 bytes total, 113792 bytes largest free block: 
Starting SmartMatrix DMA Mallocs
sizeof framestruct: 0000AC00
matrixUpdateFrames[0] pointer: 3FFE4374
matrixUpdateFrames[1] pointer: 3FFEEF84
Frame Structs Allocated from Heap:
Heap Memory Available: 157936 bytes total, 87352 bytes largest free block: 
8-bit Accessible Memory Available: 70584 bytes total, 25696 bytes largest free block: 
32-bit Memory Available: 157936 bytes total, 87352 bytes largest free block: 
DMA Memory Available: 70584 bytes total, 25696 bytes largest free block: 
Allocating refresh buffer:
DMA Memory Available: 70584 bytes total, 25696 bytes largest free block: 
lsbMsbTransitionBit of 0 requires 49152 RAM, 25696 available, leaving -23456 free: 
lsbMsbTransitionBit of 1 requires 24576 RAM, 25696 available, leaving 1120 free: 
Raised lsbMsbTransitionBit to 1/7 to fit in RAM
lsbMsbTransitionBit of 1 gives 56 Hz refresh, 120 requested: 
lsbMsbTransitionBit of 2 gives 111 Hz refresh, 120 requested: 
lsbMsbTransitionBit of 3 gives 213 Hz refresh, 120 requested: 
Raised lsbMsbTransitionBit to 3/7 to meet minimum refresh rate
Descriptors for lsbMsbTransitionBit 3/7 with 16 rows require 6144 bytes of DMA RAM
SmartMatrix Mallocs Complete
Heap Memory Available: 151760 bytes total, 87352 bytes largest free block: 
8-bit Accessible Memory Available: 64408 bytes total, 25696 bytes largest free block: 
32-bit Memory Available: 151760 bytes total, 87352 bytes largest free block: 
DMA Memory Available: 64408 bytes total, 25696 bytes largest free block:

Can you help me figure out what memory limit I'm hitting and whether ESP32 is indeed too small for what I'm apparently trying to do?
Thanks.

[marcmerlin]

Sigh, this forum takes around 2 days to approve my posts. Long enough that it doesn't matter by the time they get approved.
Honestly, this sucks, can this be fixed?

In the meantime, it's a problem/bug with the ESP32 compiler where it does not deal with global static arrays properly and just creates code that crashes before setup() even has a chance to run
Fix is to stop using arrays and switch to malloc, but this requires a fair amount of code changes, which is kind of sad.

[ESP_Sprite]

The approval thing is an anti-spam measure to stop people from registering gazilions of new users and have them post crap; it should go away after you've made two approved posts. FWIW, the crash seems to be because the startup code uses malloc() (because FreeRTOS uses it for its default implementations) and you seemingly have used up every single byte of memory by using static allocations.

[marcmerlin]

ESP_Sprite, thanks for your answer.
First, I have 6 posts now, so it's taking quite a while for the anti spam to go away. Sadly, my recollection of this group is that every time I took the time to post here, it took over a day (hours/overnight would have been understandable) to get approved, and by the time my post finally made it, it was too late to matter.

Back to the question at hand:
"FWIW, the crash seems to be because the startup code uses malloc() (because FreeRTOS uses it for its default implementations) and you seemingly have used up every single byte of memory by using static allocations."
Well, there is a clearly a problem there, I agree, but
1) why wouldn't FreeRTOS check its allocations?
2) I sure didn't use all the RAM available, not even close
3) The same program works perfectly ok when I remove the arrays and use mallocs and use the same amount of RAM

There is clearly something fishy happening here. I filed this bug with more details:
https://github.com/espressif/arduino-esp32/issues/2567

[ESP_Angus]

Hi Marc!

Well, there is a clearly a problem there, I agree, but
1) why wouldn't FreeRTOS check its allocations?

It does, with an assert. When the assert fails, you get a crash with "assert failed" in the log.

2) I sure didn't use all the RAM available, not even close
3) The same program works perfectly ok when I remove the arrays and use mallocs and use the same amount of RAM

There is unfortunately a long-standing limitation in ESP-IDF where not all of the RAM is available for static allocations. This means you can run out of static memory but still find that once FreeRTOS is running, a lot more RAM is made available.

(Apparently it's also possible to get close enough to the static memory limit that the build passes but FreeRTOS fails during the early boot stage - I hadn't realised this.)

Some more details:
https://github.com/espressif/esp-idf/is ... -389087100

Unfortunately this has proven difficult to fix in ESP-IDF, so for now the best option is to use malloc() or "new" at runtime for some large data structures.

There is clearly something fishy happening here. I filed this bug with more details:
https://github.com/espressif/arduino-esp32/issues/2567

Will reply and link the issue to this post.