Memory corruption / IDF heap corruption detection vs GCC Address Sanitizer
Posted: Sat May 25, 2019 6:57 pm
Hi,
A part from the Wifi memory allocation issue I'm fighting, there's another one that is concerning.
As previously stated (sorry for sounding like a broken record), I'm developing my projects mainly Linux in order to be able to use things like GCC's Address Sanitizer and that debugging is so much easier. As such, I've been testing my code with AS enabled at all time and it has been a great help in catching a few bugs in the few places where I use raw pointers.
I recently got the implementation of my HTTP(s) server to such a state that I thought it worthwhile to start testing it also on the ESP32. Naturally I've enabled Stack smashing protection - Overall and Heap corruption detection - Comprehensive. This is where I encountered a very rare "CORRUPT HEAP" crash. I've seen it three times so far, but only managed to capture data from the last one.
Based on the stack trace (below), it looks as if a a bit of memory has been overwritten with zeroes, 0xbaad5678 vs 0xba000000, so its likely that the problem isn't with the shared_ptr that is currently being destructed.
I find this crash very unexpected considering that I've been running the server for hours and hours hammering it with requests with AS active, and when I compile it for ESP, I get memory corruption issues.
To me, this looks as if there is an issue somewhere in the library code that is replaced when switching from native gcc/stdlib to xtensa-gcc/newlib.
Questions:
- How much does Espressif's heap corruption detection overlap with GCC's Address Sanitizer? i.e. am likely to be correct in thinking that the problem lies in the library code since the problem only occurs when compiling with IDF?
- How would you proceed to diagnose this issue? Since it is so rare, I have no idea what triggers it.
Thanks in advance,
Per
A part from the Wifi memory allocation issue I'm fighting, there's another one that is concerning.
As previously stated (sorry for sounding like a broken record), I'm developing my projects mainly Linux in order to be able to use things like GCC's Address Sanitizer and that debugging is so much easier. As such, I've been testing my code with AS enabled at all time and it has been a great help in catching a few bugs in the few places where I use raw pointers.
I recently got the implementation of my HTTP(s) server to such a state that I thought it worthwhile to start testing it also on the ESP32. Naturally I've enabled Stack smashing protection - Overall and Heap corruption detection - Comprehensive. This is where I encountered a very rare "CORRUPT HEAP" crash. I've seen it three times so far, but only managed to capture data from the last one.
Based on the stack trace (below), it looks as if a a bit of memory has been overwritten with zeroes, 0xbaad5678 vs 0xba000000, so its likely that the problem isn't with the shared_ptr that is currently being destructed.
I find this crash very unexpected considering that I've been running the server for hours and hours hammering it with requests with AS active, and when I compile it for ESP, I get memory corruption issues.
To me, this looks as if there is an issue somewhere in the library code that is replaced when switching from native gcc/stdlib to xtensa-gcc/newlib.
Questions:
- How much does Espressif's heap corruption detection overlap with GCC's Address Sanitizer? i.e. am likely to be correct in thinking that the problem lies in the library code since the problem only occurs when compiling with IDF?
- How would you proceed to diagnose this issue? Since it is so rare, I have no idea what triggers it.
Thanks in advance,
Per
Code: Select all
CORRUPT HEAP: Bad tail at 0x3f810de1. Expected 0xbaad5678 got 0xba000000
assertion "head != NULL" failed: file "/home/permal/esp/esp-idf/components/heap/multi_heap_poisoning.c", line 214, function: multi_heap_free
abort() was called at PC 0x4014021b on core 1
0x4014021b: __assert_func at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdlib/assert.c:62 (discriminator 8)
ELF file SHA256: 180e39140389ac0b5135b33ba28c629377b7e92e9dd8af130e7f01032f6193f3
Backtrace: 0x40096438:0x3ffd30e0 0x40096738:0x3ffd3110 0x4014021b:0x3ffd3140 0x4009b28a:0x3ffd3170 0x40088725:0x3ffd31a0 0x40095e9a:0x3ffd31d0 0x401b0a15:0x3ffd3200 0x400e7310:0x3ffd3220 0x400e78b0:0x3ffd3250 0x400d991f:0x3ffd3280 0x40100d3d:0x3ffd32b0 0x400d9742:0x3ffd32e0 0x400dce51:0x3ffd3310 0x400d7e25:0x3ffd3390 0x400d824d:0x3ffd3400 0x401b6c89:0x3ffd3430 0x401a9764:0x3ffd3450 0x40090d36:0x3ffd3480
0x40096438: invoke_abort at /home/permal/esp/esp-idf/components/esp32/panic.c:157
0x40096738: abort at /home/permal/esp/esp-idf/components/esp32/panic.c:172
0x4014021b: __assert_func at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdlib/assert.c:62 (discriminator 8)
0x4009b28a: multi_heap_free at /home/permal/esp/esp-idf/components/heap/multi_heap_poisoning.c:214 (discriminator 1)
0x40088725: heap_caps_free at /home/permal/esp/esp-idf/components/heap/heap_caps.c:268
0x40095e9a: free at /home/permal/esp/esp-idf/components/newlib/heap.c:47
0x401b0a15: operator delete(void*) at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49
0x400e7310: smooth::application::network::http::HTTPPacket::~HTTPPacket() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/ext/new_allocator.h:125
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/alloc_traits.h:462
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:304
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:285
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/stl_vector.h:570
(inlined by) smooth::application::network::http::HTTPPacket::~HTTPPacket() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/application/network/http/HTTPPacket.h:18
0x400e78b0: std::_Sp_counted_ptr_inplace<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5>, std::allocator<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/PacketSendBuffer.h:26
(inlined by) ?? at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/BufferContainer.h:24
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/ext/new_allocator.h:140
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/alloc_traits.h:487
(inlined by) std::_Sp_counted_ptr_inplace<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5>, std::allocator<smooth::core::network::BufferContainer<smooth::application::network::http::HTTPProtocol<1024, 500>, 5> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:552
0x400d991f: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:155
0x40100d3d: smooth::core::network::Socket<smooth::application::network::http::HTTPProtocol<1024, 500>, smooth::application::network::http::HTTPPacket>::has_data_to_transmit() at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:706
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr_base.h:1145
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/shared_ptr.h:103
(inlined by) smooth::core::network::Socket<smooth::application::network::http::HTTPProtocol<1024, 500>, smooth::application::network::http::HTTPPacket>::has_data_to_transmit() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/include/smooth/core/network/Socket.h:116
0x400d9742: smooth::core::network::SocketDispatcher::build_sets() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/network/SocketDispatcher.cpp:138
0x400dce51: smooth::core::network::SocketDispatcher::tick() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/network/SocketDispatcher.cpp:60
0x400d7e25: smooth::core::Task::exec() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/Task.cpp:157
0x400d824d: operator() at /home/permal/electronics/IO-Card-G3/software/externals/smooth/lib/smooth/core/Task.cpp:93
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/invoke.h:60
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/bits/invoke.h:95
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:234
(inlined by) ?? at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:243
(inlined by) _M_run at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/8.2.0/thread:186
0x401b6c89: execute_native_thread_routine at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/src/c++11/thread.cc:80
0x401a9764: pthread_task_func at /home/permal/esp/esp-idf/components/pthread/pthread.c:209
0x40090d36: vPortTaskWrapper at /home/permal/esp/esp-idf/components/freertos/port.c:143