ESP32 freezes/reboots every few hours

[richardhawthorn]

Hi there everyone, I’m having a problem with my esp32 crashing randomly.

I’m using the esp-wroom-32 module using the Arduino IDE and primarily driving a set of LEDs using this driver library

https://github.com/mrfaptastic/ESP32-HU ... el-I2S-DMA

I’m also using the neopixelbus library to drive a further limited set of 6 LEDs

My code (which unfortunately I can’t paste here) polls the server every minute to get a new image to display, this is a string of values that I loop through, updating each LED in the display (about 600 LEDs in total).

The esp32 crashes randomly every few hours, and It isn’t clear why. Without the hardware WDT active core 1 just freezes (core 0 still appears to function in this time), with the WDT enabled the board panics and resets.

With debug set to verbose the serial output shows the below, and the exception decoder shows no more detail than ‘panic_abort’

task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
task_wdt: - loopTask (CPU 1)
task_wdt: Tasks currently running:
task_wdt: CPU 0: IDLE
task_wdt: CPU 1: loopTask
task_wdt: Aborting.

abort() was called at PC 0x4010bbfc on core 0
Backtrace:0x40083bc5:0x3ffbecfc |<-CORRUPTED

ELF file SHA256: 0000000000000000
Rebooting...

This appears to show that there is a problem in the loop, where it is getting stuck somewhere. I would love to see where in the loop the crash occurs, but this detail doesn’t appear to available

The crashes appear at random, and don’t seem to happen when connecting to WiFi for example. I’ve checked my code and can’t see anywhere that it could be getting stuck, but clearly there is a problem somewhere.

I think this is a software problem, but it could also be hardware related. The power supply is stable and I don’t see a dip in voltage on my scope when the crashes happen. I have a few hardware versions and all are doing the same thing.

Any pointers (despite me not being able to post the code for review) would be helpful. For example, It would be helpful to get more detail on where the code is getting stuck.

Thanks

[Craige Hales]

add print statements, before and after calls, then check the log when it crashes.
Check for arrays allocated on the stack, and look to see if they might be written beyond either end. The corrupted message probably means the return addresses on the stack have been clobbered.

You might be able to use the address (0x40083bc5:0x3ffbecfc ) with a map file. I have not done that, maybe someone has a pointer to a tutorial.

[richardhawthorn]

Hi Craige

Thanks for the tips. I've done a lot of work to investigate what is going on, looking at the integrity of the stack, looking at the stability of the power supply, and at my own code, to make sure I wasn't doing something wrong.

I found the solution, but not one I can explain.

If I add a 50ms delay at the end of the main loop the problem goes away, no freezes or reboots.

My guess is that the delay is allowing other tasks time to work, but I'm not aure that makes sense because I don't think anything else should be running on core 1 (where the main loop is running), as all the background work is on core 0.

Maybe someone else can explain why this has fixed the problem.

[remfan77]

Hello,
I read this interesting post.
I'm using
- ESPIDF4.4.6
- ESP32S3 board (240MHz)
- SPIRAM (40Mhz)
and I have a similar problem.

I use :
- wifi station (not access point mode)
- i2s
- spi triggered by a timer.

I have disabled WDT, background work (tcpip) on core 0, i2s and spi on core 1.
Try also to use a strong power supply to avoid low voltage.
I see random freezes and seems that they are rarer when the timer period that activate the spi is bigger.
To be clear I've used a timer with minimal period of 75usec...also tried 120usec...I will try to increase this value to be sure freezes do not happen.
It seems to me that sometimes (in relation to wifi connection,I think, this is the only not deterministic element) the core0 and core1 are 100% and ESP32S3 cannot run some other task.

[lbernstone]

Turn the watchdog timer back on. Removing the bulb from the check engine light does not fix the car.

[remfan77]

Thanks for your reply.

In the case there is some problem in my code I agree with you. Watchdog could help to find the problem.

Here the hypothesis (I don't know still if correct) is that a ESP32S3 must have some (I don't know how many) free cpu time to manage internal things. Without this free time the ESP32S3 freezes. With watchdog enabled I would have (maybe) a watchdog triggered but this would not solve the problem. Maybe would only say same more information. However I will try your suggestion.

I have only 1 board, so I need days to test.

[lbernstone]

It sounds like you are not providing enough time for your spi process to complete. The requests for access are likely stacking up, and eventually you either get a timeout or a memory overflow. Be realistic about what you can accomplish in the allotted time. Use mutex/semaphores with appropriate times to prevent stacking.

[remfan77]

I have also tried to disable all hardware access making a pure software firmware.
I understand there are more tasks in the system to manage TCP/IP.
What I do not understand and I don't like is that the cpu of esp32 must not be too loaded to work correctly. Unfortunately the load depends on external condition (Wifi connection). Maybe in a good envirmnment (good wifi signal) esp32 needs less idle time compared to a bad environment (bad wifi signal).
In a bad invironment I would expect slower wifi communication, but I wouldn't want to see sudden resets due to the watchdog. For this reason I initially disabled it.

[ESP_Sprite]

I have also tried to disable all hardware access making a pure software firmware.
I understand there are more tasks in the system to manage TCP/IP.
What I do not understand and I don't like is that the cpu of esp32 must not be too loaded to work correctly. Unfortunately the load depends on external condition (Wifi connection). Maybe in a good envirmnment (good wifi signal) esp32 needs less idle time compared to a bad environment (bad wifi signal).
In a bad invironment I would expect slower wifi communication, but I wouldn't want to see sudden resets due to the watchdog. For this reason I initially disabled it.

Generally, wifi traffic won't lead to the ESP32 being overloaded. It's more likely you have a bug in your software somewhere (e.g. a race condition) that leads to a hang that you can't debug because you disabled the watchdog.