Memory corruption / IDF heap corruption detection vs GCC Address Sanitizer

[permal]

Hi,

A part from the Wifi memory allocation issue I'm fighting, there's another one that is concerning.

As previously stated (sorry for sounding like a broken record), I'm developing my projects mainly Linux in order to be able to use things like GCC's Address Sanitizer and that debugging is so much easier. As such, I've been testing my code with AS enabled at all time and it has been a great help in catching a few bugs in the few places where I use raw pointers.

I recently got the implementation of my HTTP(s) server to such a state that I thought it worthwhile to start testing it also on the ESP32. Naturally I've enabled Stack smashing protection - Overall and Heap corruption detection - Comprehensive. This is where I encountered a very rare "CORRUPT HEAP" crash. I've seen it three times so far, but only managed to capture data from the last one.

Based on the stack trace (below), it looks as if a a bit of memory has been overwritten with zeroes, 0xbaad5678 vs 0xba000000, so its likely that the problem isn't with the shared_ptr that is currently being destructed.

I find this crash very unexpected considering that I've been running the server for hours and hours hammering it with requests with AS active, and when I compile it for ESP, I get memory corruption issues.

To me, this looks as if there is an issue somewhere in the library code that is replaced when switching from native gcc/stdlib to xtensa-gcc/newlib.

Questions:
- How much does Espressif's heap corruption detection overlap with GCC's Address Sanitizer? i.e. am likely to be correct in thinking that the problem lies in the library code since the problem only occurs when compiling with IDF?
- How would you proceed to diagnose this issue? Since it is so rare, I have no idea what triggers it.

Thanks in advance,

Per

Code: Select all

CORRUPT HEAP: Bad tail at 0x3f810de1. Expected 0xbaad5678 got 0xba000000
assertion "head != NULL" failed: file "/home/permal/esp/esp-idf/components/heap/multi_heap_poisoning.c", line 214, function: multi_heap_free
abort() was called at PC 0x4014021b on core 1
0x4014021b: __assert_func at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdlib/assert.c:62 (discriminator 8)


ELF file SHA256: 180e39140389ac0b5135b33ba28c629377b7e92e9dd8af130e7f01032f6193f3

Backtrace: 0x40096438:0x3ffd30e0 0x40096738:0x3ffd3110 0x4014021b:0x3ffd3140 0x4009b28a:0x3ffd3170 0x40088725:0x3ffd31a0 0x40095e9a:0x3ffd31d0 0x401b0a15:0x3ffd3200 0x400e7310:0x3ffd3220 0x400e78b0:0x3ffd3250 0x400d991f:0x3ffd3280 0x40100d3d:0x3ffd32b0 0x400d9742:0x3ffd32e0 0x400dce51:0x3ffd3310 0x400d7e25:0x3ffd3390 0x400d824d:0x3ffd3400 0x401b6c89:0x3ffd3430 0x401a9764:0x3ffd3450 0x40090d36:0x3ffd3480
0x40096438: invoke_abort at /home/permal/esp/esp-idf/components/esp32/panic.c:157

0x40096738: abort at /home/permal/esp/esp-idf/components/esp32/panic.c:172

0x4014021b: __assert_func at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdlib/assert.c:62 (discriminator 8)

0x4009b28a: multi_heap_free at /home/permal/esp/esp-idf/components/heap/multi_heap_poisoning.c:214 (discriminator 1)

0x40088725: heap_caps_free at /home/permal/esp/esp-idf/components/heap/heap_caps.c:268

0x40095e9a: free at /home/permal/esp/esp-idf/components/newlib/heap.c:47

0x401b0a15: operator delete(void*) at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49

0x400e7310: smooth::application::network::http::HTTPPacket::~HTTPPacket() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/ext/new_allocator.h:125
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/alloc_traits.h:462
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:304
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:285
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:570
 (inlined by) smooth::application::network::http::HTTPPacket::~HTTPPacket() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/application/network/http/HTTPPacket.h:18

0x400e78b0: std::_Sp_counted_ptr_inplace<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5>, std::allocator<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/PacketSendBuffer.h:26
 (inlined by) ?? at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/BufferContainer.h:24
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/ext/new_allocator.h:140
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/alloc_traits.h:487
 (inlined by) std::_Sp_counted_ptr_inplace<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5>, std::allocator<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:552

0x400d991f: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:155

0x40100d3d: smooth::core::network::Socket<smooth::application::network::http::HTTPProtocol<1024, 500>, smooth::application::network::http::HTTPPacket>::has_data_to_transmit() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:706
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:1145
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr.h:103
 (inlined by) smooth::core::network::Socket<smooth::application::network::http::HTTPProtocol<1024, 500>, smooth::application::network::http::HTTPPacket>::has_data_to_transmit() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/Socket.h:116

0x400d9742: smooth::core::network::SocketDispatcher::build_sets() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/network/SocketDispatcher.cpp:138

0x400dce51: smooth::core::network::SocketDispatcher::tick() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/network/SocketDispatcher.cpp:60

0x400d7e25: smooth::core::Task::exec() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/Task.cpp:157

0x400d824d: operator() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/Task.cpp:93
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/invoke.h:60
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/invoke.h:95
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:234
 (inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:243
 (inlined by) _M_run at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:186

0x401b6c89: execute_native_thread_routine at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/src/c++11/thread.cc:80

0x401a9764: pthread_task_func at /home/permal/esp/esp-idf/components/pthread/pthread.c:209

0x40090d36: vPortTaskWrapper at /home/permal/esp/esp-idf/components/freertos/port.c:143

[ESP_Sprite]

Any chance you can try running this without PSRAM? There still is an outstanding issue in the psram workaround in the toolchain that sometimes pops up (although I never saw it crop up here) that you may be running into.

[permal]

Any chance you can try running this without PSRAM? There still is an outstanding issue in the psram workaround in the toolchain that sometimes pops up (although I never saw it crop up here) that you may be running into.

I doubt it, I just enabled PSRAM for the first time ever because mbedTLS eats up so much internal RAM. I'll try to downscale to a single socket. I could also disable TLS, but that would alter the test case quite a bit, though I suppose if the issue is with with the PSRAM driver, then it doesn't matter much. What's the status of the issue, I see no mention of it in the announcement posts?

[permal]

It is now running with a single connection and without TLS. This does change the operational timing quite a lot so I can't say if it is comparable to when the heap corruption occurs, but I will let it run during the day and report back this evening.

[permal]

Update before heading off to bed: 12h stable.

[permal]

Had it running during the night for 12h with PSRAM enabled, still only a single connection, no corruption detected. It's an elusive bug.

ESP_Sprite, can you give some information on the status and nature of the bug you refer to?

[ESP_Sprite]

There are two issues I know of. Half of it is this one, the other half has something to do with the ESP32 cache getting confused if there's 16/8-bit and 32-bit loads/stores done to the same cache line. I'm planning to fix them both with a toolchain update; as soon as that's ready you can see if that fixes the issue.

[permal]

There are two issues I know of. Half of it is this one, the other half has something to do with the ESP32 cache getting confused if there's 16/8-bit and 32-bit loads/stores done to the same cache line. I'm planning to fix them both with a toolchain update; as soon as that's ready you can see if that fixes the issue.

Thanks, Sprite. Switching to single core can be a workaround for a while, at least for the first part. Can you give a rough guesstimate on when a toolchain update might become available?

[ESP_Sprite]

I'm hoping we can release at least something for everyone to test with in the next week.

[permal]

I'm hoping we can release at least something for everyone to test with in the next week.

That is much sooner than I dared hope for. Thank you.