Crash fter running couple of days

[stevewidg]

Hi,
I do hope some kind soul can help. I have a custom board with a WROOM32 module and FTDI serial wired as per devkit C for booting. Everything runs and loads etc fine. The board is powered by a 5v supply, not USB, and is used to read various encoders and output the data normally just via BT to an Android app, or for debug etc also available via USB on request. I have had it running for several days on my bench without issue with the USB and BT connection. So time for someone else to test. They connected only the power, so just BT and everything worked fine for several hours, then switched the tablet off, leaving board on, and the next day it appearrred to be dead, and connection to a terminal revealed this (in never-ending loop):
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1216
ho 0 tail 12 room 4
load:0x6c000000,len:1073406608
1162 mmu set 00010000, pos 00010000
1162 mmu set 00020000, pos 00020000
1162 mmu set 00030000, pos 00030000
1162 mmu set 00040000, pos 00040000
1162 mmu set 00050000, pos 00050000
1162 mmu set 00060000, pos 00060000
1162 mmu set 00070000, pos 00070000
1162 mmu set 00080000, pos 00080000
1162 mmu set 00090000, pos 00090000
1162 mmu set 000a0000, pos 000a0000
1162 mmu set 000b0000, pos 000b0000
11öets Jun 8 2016 00:22:57
I'm sorry Ive spent a long time googling and reading, and cant find any clues as to what is happening. If I connect IO0 to ground then it says 'waiting for download'. But I cannot download from Arduino IDE, though it appears to go through sequence without problem. Can anybody shed any light on what this dump is saying ( and where can I get info on it!). Is it possible that writing to a disconnected BT can cause such an issue?
Any suggestions gratefully received.
Thanks
Steve

[ESP_Sprite]

It looks like something went wrong with the flash chip somehow... both the way it resets as well as the inability to download code point in that direction. Are you sure you have nothing accidentally shorting with the flash pins on the Wroom32 module (generally those at the bottom)?

[stevewidg]

Hi Sprite,
and thank you for your suggestion. I've gone over the board with a magnifier very carefully, there are only pads on 17-22, I can't see any solder whiskers. I probably programmed this particular board 20plus times during development without a hitch, as well as several weeks testing before giving it to my colleague as a cased unit. Are there any tools to test for a problem in this area?
Steve

[ESP_Sprite]

Not specifically... Can you expand a bit on the 'But I cannot download from Arduino IDE, though it appears to go through sequence without problem.' sentence? How do you conclude you cannot download?

Ah, also, have you tried erasing the entire flash chip?

[stevewidg]

Hello Sprite,
I'm fairly new to ESP32, and have only been using Arduino IDE, so no I haven't tried erasing chip, I guess that's going to involve me getting familiar with the Flash Tool, which I will certainly look at today.
What I mean by 'appeared to work' is that it established connection and went through normal download messages eg downloaded x%, and did not report any errors.
How do I get any clues as to what went wrong? I know there is an exceptions error decoder but that doesn't help if I don't have terminal connected as far as I can see.
Thanks
Steve

[ESP_Sprite]

I honestly can't tell you, sorry... something is wrong early in the boot process, possibly to do with flash, but it's not something that I have seen before. Hence me asking all those questions instead of just telling you what's going on

[stevewidg]

Well thanks for trying Sprite,
I'm really a little unsure of how to proceed. If I assume that the fault is not solder splashes corrupting the flash then is there a mechanism that the software can self destroy? An exception error as I understand it says 'you had a problem, here it is and stops' but you can restart until you hit the same problem next time. If it isn't hardware doing it , it suggests it has to be software.
Is there anything in the boot up message that gives a clue? The len statement looks very big, if that's data loaded.
Is there any way of saving more crash information, that doesn't use a serial monitor eg writing to flash? I was hoping to test in release mode, but is there any merit in testing in debug (without monitor)
I did try erasing flash, but that didn't work. Does this give any clues as to what happened?

Thanks
Steve

[ESP_Sprite]

The reason you don't get a nice backtrace is that the crash happens somewhere very early in the startup process; logic to give you a backtrace is not available there yet. (Plus, we don't even know if the crash is software-related. Hardware crashes in general give no or unusable backtraces.)

I think one of the things that may be helpful is to test if this is a hardware or a software bug. If you flash your program into a different board, do you get the same behaviour? If you flash another program (e.g. a standard 'hello world' example) into this board, do you get the same behaviour?

[stevewidg]

Hello Sprite,
I'm sorry but I dont understand. Could you please explain how you have concluded that it happened early in the process. As I said in my first post that the unit had operated successfully for several hours before being left powered up, but unconnected for a period. I also have 4 other identical boards which have all programmed and had certain levels of testing done on them for at least 8 hours each with no issue.
I guess its possible that it is a grounding/electrical noise issue, and that's another avenue I am exploring.
Thanks
Steve

[ESP_Sprite]

It's early in the boot process because your logs indicate that the device still is setting up the flash MMU tables when (or just before) it crashes; that is something that happens in the 2nd stage bootloader, before the actual esp-idf application is running.