HELP-ESP32-WROOVER randomly reboot with different reasons

[junglecarm]

Board: ESP32 Dev Module, node32, ttgo
Core Installation version: not sure
IDE name: Arduino IDE
Flash Frequency: 40Mhz
PSRAM enabled: not
Upload Speed: 115200
Computer OS: Windows 10

Hi,

I am working on a project, which is to use GSM to connect MQTT, and send instant data such as temperature, CO2 to MQTT server. My part functions are MQTT connecting, keep heartbeat, auto-reconnect, push message in queue and send out when connected, etc. My part works totally fine when just connect to USB Port. My partner's part functions are I/O reading, controlling, calculating, and invoking my part to send messages(BTW, my part to him has only couple of functions, config(), IoTLoop(), publishMessage(), he will invoke config() in setup(), and invoke my IoTLoop() in his loop()). When combine our parts together, it also works fine(not crashing, MQTT messages been sent) when connecting to USB port. But once we plug the board(we are using LILYGO dev board, which has ESP32-WROOVER and SIM800 in one board) to a "docker"(it connects I/O control and UART to other component, such as sensors, and a LCD), it will crash randomly. The crash reasons are different, I am posting some crash log here:

#1
Guru Meditati_on Error: Core 1 panic'ed (Unhandled debug exception)
Debug exception reason: /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:2331 (pcTaskGetTaskName)- assert failed!
abort() was called at PC 0x400893ca on core 1

Backtrace: 0x4008bca4:0x3ffccfb0 0x4008bed5:0x3ffccfd0 0x400893ca:0x3ffccff0 0x4008bfb4:0x3ffcd010 0x400847ad:0x3ffcd050

#2
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x401013bd PS : 0x00060c30 A0 : 0x800ffa83 A1 : 0x3ffb5580
A2 : 0x4c0c3d0c A3 : 0x00000000 A4 : 0x1a9f2520 A5 : 0x00080000
A6 : 0x43422c5e A7 : 0x00000000 A8 : 0x801013b8 A9 : 0x3ffb5560
A10 : 0x401004b4 A11 : 0x40003564 A12 : 0x0000000b A13 : 0x00000007
A14 : 0x00000000 A15 : 0x00000060 SAR : 0x0000001b EXCCAUSE: 0x0000001c
EXCVADDR: 0x1a9f2520 LBEG : 0x4000c46c LEND : 0x4000c477 LCOUNT : 0x00000000

Backtrace: 0x401013bd:0x3ffb5580 0x400ffa80:0x3ffb55b0 0x400ffebc:0x3ffb55d0 0x400f9a98:0x3ffb5690 0x400f9b61:0x3ffb56c0 0x40110d45:0x3ffb56f0 0x40159201:0x3ffb5720 0x4015961b:0x3ffb5740 0x4015c5f5:0x3ffb5760 0x4015c681:0x3ffb5780 0x4008f96d:0x3ffb57a0 0x40088f49:0x3ffb57e0

#3
Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1)
Core 1 register dump:
PC : 0x4000c271 PS : 0x00060034 A0 : 0x80089b37 A1 : 0x3ffd3a90
A2 : 0x3ffcbdb4 A3 : 0x3f40dbb4 A4 : 0x00000014 A5 : 0x3ffbe790
A6 : 0x3ffbe7d8 A7 : 0x00000001 A8 : 0x00000001 A9 : 0x3f40dbb5
A10 : 0x000000a5 A11 : 0x00000000 A12 : 0x8008a75b A13 : 0x3ffbe760
A14 : 0x00000008 A15 : 0x00000001 SAR : 0x0000001a EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0x00000000

Backtrace: 0x4000c271:0x3ffd3a90 0x40089b34:0x3ffd3ab0 0x4008b794:0x3ffd3ad0 0x4008b74a:0x00000000

Core 0 register dump:
PC : 0x4016126e PS : 0x00060634 A0 : 0x800fa256 A1 : 0x3ffbc140
A2 : 0x00000000 A3 : 0x00000001 A4 : 0x00000000 A5 : 0x00000001
A6 : 0x00060120 A7 : 0x00000000 A8 : 0x800f913a A9 : 0x3ffbc110
A10 : 0x00000000 A11 : 0x00000001 A12 : 0x8008915c A13 : 0x3ffbc040
A14 : 0x00000000 A15 : 0x3ffbbe40 SAR : 0x00000000 EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace: 0x4016126e:0x3ffbc140 0x400fa253:0x3ffbc160 0x4008a749:0x3ffbc180 0x40088f49:0x3ffbc1a0_

I used EspExceptionDecoder to try to see details, it looks like the Stack memory was messed because some times the assert point to a simple line code such as assigning value to a int variable, or a function returning point. I googled a lot about ESP stack memory, and tried to increase arduino loop task stack to 3x8192/4x8192, it look like relieved a little, but still crash. But if IoTLoop() is not invoked, it will not crash. (PS. The job of IoTLoop() is to connect MQTT via SIM800, maintain online, send out messages that are in queue, etc.) Sorry the codes are big(ten thousands of lines) and I don't know where the issue could happen so couldn't post here. Sorry if I couldn't express myself clearly. We are stuck here for couple of days. Kindly could anyone help to give some hint what could be the reason? Thanks a lot!

build info:
Sketch uses 425861 bytes (32%) of program storage space. Maximum is 1310720 bytes.
Global variables use 30132 bytes (9%) of dynamic memory, leaving 297548 bytes for local variables. Maximum is 327680 bytes.

[junglecarm]

a even more wield thing happened today, after couple of crash/reboot, the board suddenly ran the previous build I uploaded!!! why could this happened?

[ESP_Sprite]

That can happen because there are two regions with the OTA program: the current one, and one previous. Perhaps something flipped the 'switch' which selects which one is the 'current' one.

In general, I can't say too much about your error, but I'd suggest you also look into hardware things... low voltage spikes can easily lead to all sorts of weird and wonderful crashes, and GSM modules are notorious for using lots of power in a very spikey usage pattern... perhaps your power supply just is not up to snuff?

[junglecarm]

That can happen because there are two regions with the OTA program: the current one, and one previous. Perhaps something flipped the 'switch' which selects which one is the 'current' one.

In general, I can't say too much about your error, but I'd suggest you also look into hardware things... low voltage spikes can easily lead to all sorts of weird and wonderful crashes, and GSM modules are notorious for using lots of power in a very spikey usage pattern... perhaps your power supply just is not up to snuff?

I think you are right for the weird thing that the board rolled back to the previous load, because I did OTA test before. About the crash issue, I am trying to reduce my code and to try step by step..

[junglecarm]

Still crash, with different dumps. Sometimes doesn't crash for all day long, some times crashes several times in ten minutes, anyone can help???

crash #1
Guru Meditation Error: Core 1 panic'ed (StoreProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x400010c7 PS : 0x00060130 A0 : 0x800e0168 A1 : 0x3ffd67d0
A2 : 0x7f3d94f5 A3 : 0x3f402cc5 A4 : 0x0000000f A5 : 0x3ffc5b6c
A6 : 0xfffffce0 A7 : 0x3ffc5acc A8 : 0x800ec62d A9 : 0x3ffd67b0
A10 : 0x3f402cc5 A11 : 0x80000000 A12 : 0x00000000 A13 : 0x3ffd67f4
A14 : 0x00ff0000 A15 : 0xff000000 SAR : 0x0000000a EXCCAUSE: 0x0000001d
EXCVADDR: 0x800ec641 LBEG : 0x4000c46c LEND : 0x4000c477 LCOUNT : 0x00000000

Backtrace: 0x400010c7:0x3ffd67d0 0x400e0165:0x3ffd67f0 0x400e0505:0x3ffd68b0 0x400ecfd9:0x3ffd6900 0x40088f49:0x3ffd6920

crash #2
rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:8896
load:0x40080400,len:5816
entry 0x400806ac

Rebooting...
ets Jun 8 2016 00:22:57

[ESP_Sprite]

Again, have you looked at your power supply?

[junglecarm]

Thanks for your reply. We are also working from the power supply check direction now. Today we tried to use 12V battery as independent power source for the "docker" (the "docker" will supply 5v to ESP32), before the power of the "docker" is from a power splitter. So far works fine, hope this is the root cause. Thanks!