ESP32 crash - Exception decoder not of any help

grescaldani
Posts: 7
Joined: Tue Mar 02, 2021 3:16 pm

ESP32 crash - Exception decoder not of any help

Postby grescaldani » Thu Dec 09, 2021 8:13 pm

Hello, I had an application running on ESP8266 and it worked fine on 5 devices for more than 3 years, it controls the 5 canvas of my home. Presently it is still running on 4 of them, while on the 5th I am testing my porting on ESP32. I have really a lot of crashes and, most of the times, it doesn't reboot and I am required to reset it manually (force EN low) to reboot.
So, I connected a serial port to investigate the crash, but the output seems not consistent. Exception Decoder just says:

Decoding 2 results
0x4008743e: _xt_user_exit at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S line 79

The output on the serial port is:
  1. Guru Meditation Error: Core  panic'ed (Interrupt wdt timeout on CPU0)
  2. Core 0 register dump:
  3. PC       0x4008743e PS       0x00060334  A0       0x8012c25  A1       0x3fbe0
  4. A2       0x3fb260  A3       0x3fbec A4       0x00000007  A5       0x000137d
  5. A6       0x00000000  A7       0x00000011  A8       0x8013ae  A9       0x3fbe0
  6. A10      0x00000000  A11      0x3fbec A12      0x3fb2c  A13      0x00000000
  7. A14      0x00000001  A15      0x00000000  SAR      0x00000010  EXCCAUSE 0x00000005
  8. EXCVADDR 0x00000000  LBEG     0x4000ce  LEND     0x4000cf  LCOUNT   0xffff
  9.  
  10. ELF file SHA256: 000000000000000
  11.  
  12. Backtrace:0x4008743e0x3fbe00x4012c22:0x3fbe0 0x401275a:0x3fbe0 0x4008a46:0x3fbe0
  13.  
  14. ore 1 register dump:
  15. PC       0x4000c71  PS       0x00060034  A0       0x8008a60  A1       0x3fbd0
  16. A2       0x3fb058  A3       0x3f15134  A4       0x00000014  A5       0x3fc0e
  17. A6       0x3fc128  A7       0x00000001  A8       0x00000001  A9       0x3f15135
  18. A10      0x000000a  A11      0x3fc43c A12      0x8008b17  A13      0x3fc0b
  19. A14      0x00000008  A15      0x00000001  SAR      0x0000001f EXCCAUSE 0x00000005
  20. EXCVADDR 0x00000000  LBEG     0x400012e  LEND     0x40001309  LCOUNT   0x800e230
  21.  
  22. ELF file SHA256: 000000000000000
  23.  
  24. Backtrace:0x4000c71:0x3fbd00x4008a5d0x3fbe00x4008c30:0x3fbe00x4008ce:0x00060320

Power is in excess of 1A and there is a huge capacitor on the 3.3V power line (4700uF) and, of course, a small 0.1uF near the ESP.
I don't tamper with core 0 - all my code is [presently] just in one task and running in core 1.

I just would like to know where I can start to investigate while all the tools I used up to date seems not effective! (on ESP8266 I had the beautiful SaveCrash functions to trap my programming errors).

Thanks!

ESP_Sprite
Posts: 9577
Joined: Thu Nov 26, 2015 4:08 am

Re: ESP32 crash - Exception decoder not of any help

Postby ESP_Sprite » Fri Dec 10, 2021 3:54 am

The raw backtrace lines you quote unfortunately seem corrupted: if any, there's semicolons missing. Agree that this makes it hard to find the error :/ The error message itself may give you a hint, though: 'Interrupt wdt timeout on CPU0' means that either an interrupt or a critical section is taking too long (from memory: >100mS) or is hung entirely. As such, a question to ask would be if you are doing anything in interrupt handlers that could take a long time?

grescaldani
Posts: 7
Joined: Tue Mar 02, 2021 3:16 pm

Re: ESP32 crash - Exception decoder not of any help

Postby grescaldani » Wed Dec 15, 2021 4:24 pm

I am very late in answer, while I redesigned my board to add some 0.1uF capacitors in strategic areas, more strong polarization resistors, etc. to be sure the problem couldn't be caused by some external spike or the like.
In every crash, the lines are always, exactly the same, including "ore 1" instead of "Core 1"
Presently I don't use interrupt handlers an I have no closed loops (that, anyway, could cause WDT in core 1, correct?)
That said, of course I don't ask what may be my present problem, but which could be the better approach to isolate such a kind of problems.
Thanks!

ESP_Sprite
Posts: 9577
Joined: Thu Nov 26, 2015 4:08 am

Re: ESP32 crash - Exception decoder not of any help

Postby ESP_Sprite » Thu Dec 16, 2021 2:03 am

Gotcha. Well, first of all, I'd look at the info I'd have... and in this case it's that the issue is an interrupt watchdog timeout, likely something funky is going on in an interrupt or critical region (an interrupt watchdog triggers when 'normal' interrupts can't be executed for too long). Secondly, but here our experiences may differ because I tend to dev in esp-idf, not in Arduino, I'd try to catch the bug. Either I'd turn on the option in ESP-IDF that saves coredumps to flash, or I'd connect a JTAG debugger and wait for the issue to happen; that way I could 'snoop around' in the program state when the issue ocurred.

Sorry, I'm not 100% sure how that maps to the Arduino environment; perhaps someone else can help you there. As a last resort, it's possible to use Arduino as a library within ESP-IDF, in which case you can still use the ESP-IDF features I mentioned.

grescaldani
Posts: 7
Joined: Tue Mar 02, 2021 3:16 pm

Re: ESP32 crash - Exception decoder not of any help

Postby grescaldani » Fri Dec 17, 2021 9:20 am

ESP_Sprite wrote:
Thu Dec 16, 2021 2:03 am
As a last resort, it's possible to use Arduino as a library within ESP-IDF, in which case you can still use the ESP-IDF features I mentioned.
This is extremely interesting! I didn't know it, and I will investigate, while this may help to solve e lot of other issues.

That said, my last night discovery is that my ESP32 sketch crashes exactly when a totally independent relay opens, even if it is idling!
The only "common" part is the power line (to both the ESP 3.3V Power Supply and that relay) and the fact that the motor is at about 1.5m distance. I am starting to think to EMF or something the like, even if I have no idea how to fight against that, while I cannot close the ESP in a lead sarcophagus!

ESP_Sprite
Posts: 9577
Joined: Thu Nov 26, 2015 4:08 am

Re: ESP32 crash - Exception decoder not of any help

Postby ESP_Sprite » Sat Dec 18, 2021 10:38 am

Sounds like EMI indeed. Make sure that there's proper decoupling near your ESP32, perhaps even add some chokes in the power line to kill spikes from the relay.

grescaldani
Posts: 7
Joined: Tue Mar 02, 2021 3:16 pm

Re: ESP32 crash - Exception decoder not of any help

Postby grescaldani » Mon Dec 20, 2021 6:46 pm

Thanks, ESP_Sprite! Yes, I had a great enhancement with two snubbers, one for each contact, but I still see some garbage on the 3.3V, even if the ESP no longer crashes. I will work on inductors on the power line and I am considering to build an EMF filter. That said, it is out of doubt the the ESP32 is way more sensible to power disturbance than ESP8266!

Who is online

Users browsing this forum: Bing [Bot] and 117 guests