I2C crash with release/v3.0 - what's an effective way to debug this?
Posted: Fri Jan 19, 2018 11:40 am
I'm struggling with a persistent and frequent crash scenario and I'd like to request some advice with debugging it please, which seems to happen at or around the time of calling i2c_driver_install().
I'm using the latest code on the release/v3.0 branch (e7dc749e2f953a976325964b5ac49b53c8bafc42).
What happens is that the application will happily configure and use the I2C bus for sometimes an hour or so, then suddenly crash and go into a repeating loop of crashing every time the app restarts, which is where i2c_driver_install() is called. Every crash shows pretty much the same panic output, although the PC changes each time.
I've tried a hardware JTAG debugger but it doesn't seem to catch the issue - with newer versions of the ESP-IDF it doesn't halt on a crash and as I've mentioned in another thread I have had all sorts of problems with debugging with v3.0.
Based on error message I'd say it looks like an interrupt handler is taking too long to execute, but the line number of i2c.c:1023 doesn't seem to mean much in relation to an interrupt handler - it's the end of the i2c_master_read function.
I've turned on heap corruption detection and stack smashing detection - nothing obvious shows up.
Can someone familiar with the I2C code please advise me on how to tackle this problem? Are there additional options I can configure in 'sdkconfig' to get more information? Are there any tricks or approaches anyone can recommend for narrowing this down? Any GDB "magic" that might help here? I'm inexperienced with debugging at this level on the ESP32 so any suggestions on how to proceed would be very helpful, please.
I would dearly like to solve this problem because a CPU reset doesn't clear it - once it starts crashing it just crashes over and over until intervention occurs (power cycle or a long press on the board's reset button).
Note that I didn't see this error with v2.1.1 - this only seems to happen with release/v3.0 and master. However the I2C lock-up problem in 2.1.1 may have been shielding me from this issue.
Code: Select all
Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0)
Register dump:
PC : 0x400847e7 PS : 0x00060034 A0 : 0x80084c94 A1 : 0x3ffb05b0
0x400847e7: i2c_master_cmd_begin_static at /Users/david/esp32/esp-idf-v3.0/components/driver/./i2c.c:1023
A2 : 0x00000000 A3 : 0x00000001 A4 : 0x80088f18 A5 : 0x3ffbaee0
A6 : 0x3ffb9138 A7 : 0x00000000 A8 : 0x3ffb6d88 A9 : 0x3ffb0590
A10 : 0x00000001 A11 : 0x00000000 A12 : 0x3ffb05b4 A13 : 0x00000002
A14 : 0x3ffbccec A15 : 0x00060021 SAR : 0x00000020 EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0xffffffff
Backtrace: 0x400847e7:0x3ffb05b0 0x40084c91:0x3ffb05e0 0x40082bd1:0x3ffb0610 0x4000bfed:0x00000000
0x400847e7: i2c_master_cmd_begin_static at /Users/david/esp32/esp-idf-v3.0/components/driver/./i2c.c:1023
0x40084c91: i2c_isr_handler_default at /Users/david/esp32/esp-idf-v3.0/components/driver/./i2c.c:1023
0x40082bd1: _xt_lowint1 at /Users/david/esp32/esp-idf-v3.0/components/freertos/./xtensa_vectors.S:1105
Rebooting...
What happens is that the application will happily configure and use the I2C bus for sometimes an hour or so, then suddenly crash and go into a repeating loop of crashing every time the app restarts, which is where i2c_driver_install() is called. Every crash shows pretty much the same panic output, although the PC changes each time.
I've tried a hardware JTAG debugger but it doesn't seem to catch the issue - with newer versions of the ESP-IDF it doesn't halt on a crash and as I've mentioned in another thread I have had all sorts of problems with debugging with v3.0.
Based on error message I'd say it looks like an interrupt handler is taking too long to execute, but the line number of i2c.c:1023 doesn't seem to mean much in relation to an interrupt handler - it's the end of the i2c_master_read function.
I've turned on heap corruption detection and stack smashing detection - nothing obvious shows up.
Can someone familiar with the I2C code please advise me on how to tackle this problem? Are there additional options I can configure in 'sdkconfig' to get more information? Are there any tricks or approaches anyone can recommend for narrowing this down? Any GDB "magic" that might help here? I'm inexperienced with debugging at this level on the ESP32 so any suggestions on how to proceed would be very helpful, please.
I would dearly like to solve this problem because a CPU reset doesn't clear it - once it starts crashing it just crashes over and over until intervention occurs (power cycle or a long press on the board's reset button).
Note that I didn't see this error with v2.1.1 - this only seems to happen with release/v3.0 and master. However the I2C lock-up problem in 2.1.1 may have been shielding me from this issue.