std::shared_ptr with ESP-IDF (updated)

[permal]

Hi,

I'm writing a C++ framework on top of FreeRTOS/ESP-IDF in order to allow application development on ESP-IDF "fully C++" and also remove a lot of the hassle/pitfalls associated with multi threading. So far I've got classes for Tasks, Queues, Mutexes, Socket, Timers, Performance timers and a (memory static) Finite State Machine.

This is all working out very well and I've also begun to implement MQTT support as an example of how these things are used. Feel free to check out the project at GitHub and join if you like

Naturally I've hit a few quite a few bumps on the way but so far I've managed to solve them all. There is however one very intermittent issue causing an application crash that I'm not able to figure out. I've captured two stacktraces and although not identical they both share the same initial entrypoint (search for forward_to_taskEv in the traces) so I suspect they have the same root cause.

Can anyone of you read any useful information from these traces? All I can see is that the first one fails inside std::shared_ptr during destruction (xtensa-esp32-elf/include/c++/5.2.0/ext/new_allocator.h:124). It makes total sense that this code is executing since class ConnectionStatusEvent contains a shared_ptr), but I see no reason for it to crash at this point - it should just do what is needed to destroy the shared_ptr and continue, as it has done many many times already before it eventually fails like this.

I'm running this on a Sparkfun ESP32thing, using ESP-IDF master, commit dce7fcb90d5374c7c662323f7ab50fc28e98e2f2 and compiler xtensa-esp32-elf-linux64-1.22.0-61-gab8375a-5.2.0.tar.gz.

Update: When forcing FreeRTOS to only run on a single core I have not yet seen this issue.

Any insights will be very appreciated.

Code: Select all

V (127042) Socket: [192.168.10.247, 1883, 0x3ffcc71c]: Stopping
V (127042) SocketDispatcher: Initiating shutdown of socket 0x3ffcc71c
V (127042) Socket: [192.168.10.247, 1883, 0x3ffcc71c]: Disconnected
V (127052) SocketDispatcher: Completing shutdown of socket 0x3ffcc71c
V (127052) Socket: [192.168.10.247, 1883, 0x3ffcc768]: Created socket
V (127062) Socket: [192.168.10.247, 1883, 0x3ffcc71c]: Destructing
V (127072) Socket: [192.168.10.247, 1883, 0x3ffcc768]: Connecting
Guru Meditation Error of type LoadProhibited occurred on core  1. Exception was unhandled.
Register dump:
PC      : 0x4010222b  PS      : 0x00060730  A0      : 0x8010239c  A1      : 0x3ffcc0c0  
0x4010222b: _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE0EE10_M_releaseEv at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:309
 (inlined by) _ZNSt14__shared_countILN9__gnu_cxx12_Lock_policyE0EED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:659
 (inlined by) _ZNSt12__shared_ptrIN6smooth4core7network7ISocketELN9__gnu_cxx12_Lock_policyE0EED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:925
 (inlined by) _ZNSt10shared_ptrIN6smooth4core7network7ISocketEED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr.h:93
 (inlined by) _ZN6smooth4core7network21ConnectionStatusEventD4Ev at /home/permal/code/SmoothTest/components/Smooth/include/smooth/core/network/ConnectionStatusEvent.h:16
 (inlined by) _ZN9__gnu_cxx13new_allocatorIN6smooth4core7network21ConnectionStatusEventEE7destroyIS4_EEvPT_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/ext/new_allocator.h:124
 (inlined by) _ZNSt16allocator_traitsISaIN6smooth4core7network21ConnectionStatusEventEEE10_S_destroyIS3_EENSt9enable_ifIXsrSt6__and_IJNS5_16__destroy_helperIT_E4typeEEE5valueEvE4typeERS4_PSA_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:285
 (inlined by) _ZNSt16allocator_traitsISaIN6smooth4core7network21ConnectionStatusEventEEE7destroyIS3_EEvRS4_PT_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:414
 (inlined by) _ZNSt6vectorIN6smooth4core7network21ConnectionStatusEventESaIS3_EE8_M_eraseEN9__gnu_cxx17__normal_iteratorIPS3_S5_EE at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/vector.tcc:147

A2      : 0x3ffcc750  A3      : 0x0000000b  A4      : 0x3ffc0a04  A5      : 0x00000000  
A6      : 0x3ffc0a20  A7      : 0x3ffc0738  A8      : 0x80084452  A9      : 0x3ffcc090  
A10     : 0x3ffc0a04  A11     : 0x3ffcc0f0  A12     : 0x00000000  A13     : 0x0000ff00  
A14     : 0x00ff0000  A15     : 0xff000000  SAR     : 0x00000018  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000013  LBEG    : 0x4000c349  LEND    : 0x4000c36b  LCOUNT  : 0xffffffff  

Backtrace: 0x4010222b:0x3ffcc0c0 0x40102399:0x3ffcc0f0 0x4012f571:0x3ffcc130 0x4012f5b9:0x3ffcc170
0x4010222b: _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE0EE10_M_releaseEv at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:309
 (inlined by) _ZNSt14__shared_countILN9__gnu_cxx12_Lock_policyE0EED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:659
 (inlined by) _ZNSt12__shared_ptrIN6smooth4core7network7ISocketELN9__gnu_cxx12_Lock_policyE0EED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:925
 (inlined by) _ZNSt10shared_ptrIN6smooth4core7network7ISocketEED4Ev at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr.h:93
 (inlined by) _ZN6smooth4core7network21ConnectionStatusEventD4Ev at /home/permal/code/SmoothTest/components/Smooth/include/smooth/core/network/ConnectionStatusEvent.h:16
 (inlined by) _ZN9__gnu_cxx13new_allocatorIN6smooth4core7network21ConnectionStatusEventEE7destroyIS4_EEvPT_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/ext/new_allocator.h:124
 (inlined by) _ZNSt16allocator_traitsISaIN6smooth4core7network21ConnectionStatusEventEEE10_S_destroyIS3_EENSt9enable_ifIXsrSt6__and_IJNS5_16__destroy_helperIT_E4typeEEE5valueEvE4typeERS4_PSA_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:285
 (inlined by) _ZNSt16allocator_traitsISaIN6smooth4core7network21ConnectionStatusEventEEE7destroyIS3_EEvRS4_PT_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:414
 (inlined by) _ZNSt6vectorIN6smooth4core7network21ConnectionStatusEventESaIS3_EE8_M_eraseEN9__gnu_cxx17__normal_iteratorIPS3_S5_EE at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/vector.tcc:147

0x40102399: _ZThn4_N6smooth4core3ipc14TaskEventQueueINS0_7network21ConnectionStatusEventEE15forward_to_taskEv at ??:?

0x4012f571: _ZN6smooth4core4Task4execEv at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/char_traits.h:243

0x4012f5b9: _ZZN6smooth4core4Task5startEvENUlPvE_4_FUNES2_ at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/char_traits.h:243
 (inlined by) _FUN at /home/permal/code/SmoothTest/components/Smooth/core/Task.cpp:62


Entering gdb stub now.
$T0b#e6GNU gdb (crosstool-NG crosstool-ng-1.22.0-61-gab8375a) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_pc-linux-gnu --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/permal/code/SmoothTest/build/Smooth-test.elf...done.
Remote debugging using /dev/ttyUSB0
0x4010222b in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)0>::_M_release (this=0x3ffcc750)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:309
309	          _M_dispose();
(gdb) 


(gdb) bt
#0  0x4010222b in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)0>::_M_release (this=0x3ffcc750)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:309
#1  std::__shared_count<(__gnu_cxx::_Lock_policy)0>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:659
#2  std::__shared_ptr<smooth::core::network::ISocket, (__gnu_cxx::_Lock_policy)0>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:925
#3  std::shared_ptr<smooth::core::network::ISocket>::~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr.h:93
#4  smooth::core::network::ConnectionStatusEvent::~ConnectionStatusEvent (this=<optimized out>, __in_chrg=<optimized out>)
    at /home/permal/code/SmoothTest/components/Smooth/include/smooth/core/network/ConnectionStatusEvent.h:16
#5  __gnu_cxx::new_allocator<smooth::core::network::ConnectionStatusEvent>::destroy<smooth::core::network::ConnectionStatusEvent> (__p=<optimized out>, this=0x3ffbbd20)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/ext/new_allocator.h:124
#6  std::allocator_traits<std::allocator<smooth::core::network::ConnectionStatusEvent> >::_S_destroy<smooth::core::network::ConnectionStatusEvent> (__p=<optimized out>, __a=...)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:285
#7  std::allocator_traits<std::allocator<smooth::core::network::ConnectionStatusEvent> >::destroy<smooth::core::network::ConnectionStatusEvent> (__p=<optimized out>, __a=...)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/alloc_traits.h:414
#8  std::vector<smooth::core::network::ConnectionStatusEvent, std::allocator<smooth::core::network::ConnectionStatusEvent> >::_M_erase (this=0x3ffbbd20, __position=...)
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/vector.tcc:147
#9  0x4010239c in non-virtual thunk to smooth::core::ipc::TaskEventQueue<smooth::core::network::ConnectionStatusEvent>::forward_to_task() ()
    at /home/permal/esp/xtensa-esp32-elf/xtensa-esp32-elf/include/c++/5.2.0/bits/shared_ptr_base.h:311
#10 0x4012f574 in smooth::core::Task::exec (this=0x3ffbb9b0) at /home/permal/code/SmoothTest/components/Smooth/core/Task.cpp:93
#11 0x4012f5bc in smooth::core::Task::<lambda(void*)>::operator() (
/home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:167: internal-error: inline_frame_this_id: Assertion `frame_id_p (*this_id)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 

Code: Select all

Guru Meditation Error of type InstrFetchProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x00000000  PS      : 0x00060830  A0      : 0x801047df  A1      : 0x3ffc3540  
A2      : 0x3ffbbd48  A3      : 0x3ffc35a4  A4      : 0x00000000  A5      : 0xc0000000  
A6      : 0x00000034  A7      : 0x00000001  A8      : 0x80104734  A9      : 0x3ffbbfa0  
A10     : 0x3ffaf40c  A11     : 0x00060823  A12     : 0x00060820  A13     : 0x3ffc35a0  
A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x00000006  EXCCAUSE: 0x00000014  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  

Backtrace: 0x00000000:0x3ffc3540 0x401047dc:0x3ffc3580 0x40102d23:0x3ffc35a0 0x401346a5:0x3ffc35d0 0x401346ed:0x3ffc3610
0x401047dc: _ZThn92_N10LedControl7messageERKN6smooth4core5timer17TimerExpiredEventE at ??:?

0x40102d23: _ZThn4_N6smooth4core3ipc14TaskEventQueueINS0_5timer17TimerExpiredEventEE15forward_to_taskEv at ??:?

0x401346a5: _ZN6smooth4core4Task4execEv at /home/permal/code/SmoothTest/components/Smooth/core/Task.cpp:93

0x401346ed: _ZZN6smooth4core4Task5startEvENUlPvE_4_FUNES2_ at /home/permal/code/SmoothTest/components/Smooth/core/Task.cpp:62
 (inlined by) _FUN at /home/permal/code/SmoothTest/components/Smooth/core/Task.cpp:62


Entering gdb stub now.
$T0b#e6GNU gdb (crosstool-NG crosstool-ng-1.22.0-61-gab8375a) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_pc-linux-gnu --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/permal/code/SmoothTest/build/Smooth-test.elf...done.
Remote debugging using /dev/ttyUSB0
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0x001047df in ?? ()
#2  0x00102d26 in ?? ()
#3  0x001346a8 in ?? ()
#4  0x001346f0 in ?? ()
(gdb) 

[permal]

I've now had my tests running for six hours without any issues on a single core, longest time on dual core has been ~15 minutes.

[ESP_igrr]

I haven't gone through your library code yet, but do you think there is a chance that tasks on two CPUs are accessing/modifying shared_ptr instances pointing to a single object? Since shared_ptr is not thread-safe, that could be an issue.

[kolban]

Howdy @permal,
There has been an open source effort underway for the last 6 months attempting to provide encapsulations and function in C++ for the ESP32 environment ... see:

https://github.com/nkolban/esp32-snippe ... /cpp_utils

What I want to do is ensure you know of its existence. It is always a shame when multiple projects spring up that could have benefited through collaboration as opposed to duplicate and possibly divergent path/work.

The project above is 100% open source and all collaborators are welcome. Commits will be merged ASAP and divergent design choices are resolved by community discussions with everyone being equal.

If this interests you, feel free to post back here or email me directly at kolban1@kolban.com

Neil

[permal]

I haven't gone through your library code yet, but do you think there is a chance that tasks on two CPUs are accessing/modifying shared_ptr instances pointing to a single object? Since shared_ptr is not thread-safe, that could be an issue.

@Iggr
The standard states that std::shared_ptr IS thread-safe (but no guarantees re. the object being held by the shared_ptr). I do have two tasks accessing the same object through a shared_ptr; that object (an instance of the Socket-class) in turn uses a mutex to handle the thread-safety. My test on a single core is still running and I will let it run the entire night to see if the issue occurs.

Howdy @permal,

If this interests you, feel free to post back here or email me directly at kolban1@kolban.com

Neil

Niel, I'm very much interested! I've been poking around in your GitHub repos, but cpp-utils has gone unnoticed, and I really didn't expect to find something so valuable in a repo called "snippets" . I'm more than happy to collaborate, it makes things even more fun.

[ESP_igrr]

The standard states that std::shared_ptr IS thread-safe

Indeed it should be. However our GCC is configured without threading support, and __default_lock_policy equals _S_single, so no locking takes place (see https://github.com/jcmvbkbc/gcc-xtensa/ ... ptr_base.h and https://github.com/jcmvbkbc/gcc-xtensa/ ... currence.h).

Once https://github.com/espressif/esp-idf/issues/690 is implemented, shared_ptr should be thread-safe as the standard requires.

There may be some other way to make shared_ptr use a _S_atomic lock policy, but i haven't looked in detail.

[permal]

Oh, wow. You really should make this super-clear in the documentation - is it even documented? I can't find any note of it the doc site (https://esp-idf.readthedocs.io) and searching for "thread" yields nothing relating to this. Can you please make abundantly clear to prevent others from spending days chasing ghosts in their code

This is a real blocker for anyone wanting to use modern C++ on the ESP32. May I suggest that you do it in two steps; 1) implement the locking mechanism needed and release that (or at least make it available in the master branch) and then implement the thread management parts? As we already have the FreeRTOS task the second part is less of an issue; I assume they'll all use the same primitives in the end, right?

Do you have a time frame for this? I'd say, and I'm sure others agree, that it is a rather important feature and would benefit ESP-IDF greatly.

[ESP_igrr]

No, this is not documented. It hasn't occurred to me that shared_ptr was not in fact using atomics in our configuration until I read your post and went to check the source code.

We don't have a schedule for implementing pthreads wrapper as discussed in the issue linked above. However, now that we know that shared_ptr is also affected, we'll try to find someone to work on this.

[permal]

@ESP_Igrr That would be very much appreciated by the community And thank you for the prompt replies.