Help getting to the root of a load prohibited error ...

[this.wiederkehr]

Hello

I'm using the pycom micropython and keep getting load prohibited errors at unregular occations.
See the following thread in the pycom forums https://forum.pycom.io/topic/1361/firmw ... -7-3-b1/14.

Basically this error is triggerd by writig to the sd-card. If the only thing the esp32 does is writing to the sd-card it might take a long time till the error occures (hours). If other things are running in parallel in another task it might trigger much faster (within seconds to minutes).

Depending on the build the error triggers at different locations in code however it always triggers within freertos interrupts and for a given build at the same location:

This is the stacktrace I got when on pycom micropython 1.7.1b1:

Code: Select all

Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x40087eb9  PS      : 0x00060031  A0      : 0x80086722  A1      : 0x3ffc07b0  
A2      : 0x00000001  A3      : 0x00000001  A4      : 0x000d6f4f  A5      : 0x00000006  
A6      : 0x000002aa  A7      : 0x3ffc5fa0  A8      : 0x3ffc3fbc  A9      : 0x3ffc0790  
A10     : 0x3ffc3e40  A11     : 0x00000001  A12     : 0x00000004  A13     : 0x00000000  
A14     : 0x000000aa  A15     : 0x3ffc5f00  SAR     : 0x00000016  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000009  LBEG    : 0x400f3178  LEND    : 0x400f3180  LCOUNT  : 0x00000000  

Backtrace: 0x40087eb9:0x3ffc07b0 0x40086722:0x3ffc07d0 0x40088c1c:0x3ffc07f0 0x40081861:0x3ffc0800

0x40087eb9: xTaskIncrementTick at /components/freertos/./tasks.c:4873
0x40086722: xPortSysTickHandler at /components/freertos/./port.c:420
0x40088c1c: _frxt_timer_int at ??:?
0x40081861: _xt_lowint1 at xtensa_vectors.o:?

With the newest available pycom micropython and esp-idf sources (including some modifications):

Code: Select all

Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x400886f4  PS      : 0x00060031  A0      : 0x80087006  A1      : 0x3ffc0fb0  
A2      : 0x00000001  A3      : 0x00000001  A4      : 0x00c533dc  A5      : 0x00000006  
A6      : 0x00000005  A7      : 0x00000001  A8      : 0x3ffc5a4c  A9      : 0x3ffc0f90  
A10     : 0x3ffc58d0  A11     : 0x00000001  A12     : 0x3ffb5fe0  A13     : 0x00000000  
A14     : 0x00000000  A15     : 0x00000004  SAR     : 0x0000001a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000009  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffc  

Backtrace: 0x400886f4:0x3ffc0fb0 0x40087006:0x3ffc0fd0 0x40089424:0x3ffc0ff0 0x40081a3d:0x3ffc1000

xTaskResumeAll at /opt/pycom-esp-idf/components/freertos/./tasks.c:4940
prvCopyDataFromQueue at /opt/pycom-esp-idf/components/freertos/./queue.c:2034
_xt_context_restore at ??:?
_xt_highint4 at xtensa_vectors.o:?

Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.
Register dump:
PC      : 0x40089f76  PS      : 0x00060033  A0      : 0x80088ed5  A1      : 0x3ffc0bf0  
A2      : 0x3ffc0c40  A3      : 0x3ffc0c10  A4      : 0x00000020  A5      : 0x3ffc5844  
A6      : 0x00000005  A7      : 0x3ffc0fc0  A8      : 0x00000012  A9      : 0x00000001  
A10     : 0x3ffc0c88  A11     : 0x3ffc584c  A12     : 0x00000001  A13     : 0x3ffc0fd0  
A14     : 0x00000000  A15     : 0xffffffff  SAR     : 0x00000016  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000001  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0x00000000  

Backtrace: 0x40089f76:0x3ffc0bf0 0x40088ed5:0x3ffc0c10 0x4008a820:0x3ffc0c40 0x4008ab95:0x3ffc0e10 0x4008a500:0x3ffc0e50 0x4008a657:0x3ffc0ed0 0x400817dd:0x3ffc0ef0 0x400886f4:0x3ffc0fb0 0x400886f4:0x3ffc0fd0 0x40089424:0x3ffc0ff0 0x40081a3d:0x3ffc1000

getItemFromRingbufByteBuf at ringbuf.c:?
xTimerCreate at /opt/pycom-esp-idf/components/freertos/./timers.c:839
esp_core_dump_write at /opt/pycom-esp-idf/components/esp32/./core_dump.c:189
heap_alloc_caps_init at /opt/pycom-esp-idf/components/esp32/./heap_alloc_caps.c:203
_esp_error_check_failed at /opt/pycom-esp-idf/components/esp32/./panic.c:524
esp_core_dump_uart_write_data at /opt/pycom-esp-idf/components/esp32/./core_dump.c:488
_xt_coproc_exc at xtensa_vectors.o:?
xTaskResumeAll at /opt/pycom-esp-idf/components/freertos/./tasks.c:4940
xTaskResumeAll at /opt/pycom-esp-idf/components/freertos/./tasks.c:4940
_xt_context_restore at ??:?
_xt_highint4 at xtensa_vectors.o:?

....This repeats a few times adding another layer of faild panic handler on top.

pycom uses a modified version of esp-idf (however no tinkering within freertos):
https://github.com/pycom/pycom-esp-idf

And here the repo of there version of micropython:
https://github.com/pycom/pycom-micropython-sigfox

I have no clue where to start searching for the error which does trigger this. Could somebody help me what I could do to get to the root cause of this error? Or even better, does somebody have a hint what might be the root cause for this?

BR

This

[ESP_igrr]

Looks like a memory corruption bug, where some pointer gets overwritten due to an out-of-bounds access.

If the corruption happens in the same place every time, you could enable a debug watchpoint at the location where the corruption happens.

Your first step would be to figure out the address of the value that was overwritten. Enable GDBStub, wait for the crash to happen, inspect the value(s) which are being dereferenced in the line of the code where crash happens. Figure out which one of them is corrupted (has the value same as EXCVADDR).

Next, add a call to esp_set_watchpoint function to enable watchpoint on the particular address which gets overwritten. Once it gets written, you will get a debug exception which will trigger GDBStub.

[this.wiederkehr]

@ESP_igrr: Thanks for you detailed explanation on how to proceed. I found that a xListItem has a corrupted pointer to pvOwner (0x1) this did trigger the error in task.c in function xTaskIncrementTick.
'
pxTCB = ( TCB_t * ) listGET_OWNER_OF_HEAD_ENTRY( pxDelayedTaskList );
xItemValue = listGET_LIST_ITEM_VALUE( &( pxTCB->xGenericListItem ) );
'

Still searching for the point where it got corrupted. While on it: Is there a way to continue program execution after a watchpoint got hit and gdbstub executed?

[this.wiederkehr]

Looks like I am doing something wrong here:

The error is always the same (load prohibited), which is due to a corrupted pxDelayedTaskList:

Code: Select all

xTaskIncrementTick () at /opt/pycom-esp-idf/components/freertos/./tasks.c:2465

(gdb) p pxDelayedTaskList
$6 = (List_t * volatile) 0x3ffc53f0 <xDelayedTaskList1>
(gdb) p *pxDelayedTaskList->pxIndex
$7 = {xItemValue = 4294967295, pxNext = 0x3ffc53f8 <xDelayedTaskList1+8>,
  pxPrevious = 0x3ffc53f8 <xDelayedTaskList1+8>, pvOwner = 0x1,
  pvContainer = 0x3ffb516c}
(gdb) p &pxDelayedTaskList->pxIndex->pvOwner
$8 = (void **) 0x3ffc5404 <pxReadyTasksLists>

Notice that pvOwner is 0x1 (should be a valid address to a tcb) and that pvContainer does not point to the address of the containing List

The watchpoint does not trigger even if I have

Code: Select all

esp_set_watchpoint(0, (char*)0x3ffc5404, 1, ESP_WATCHPOINT_STORE);

right at the top of the app_main ....

Am I doing something wrong here? Shouldn't the watchpoint now trigger when the address of pvowner gets written? Does this mean, that my list already is corrupted once I start running app_main? How if the address gets written inside an isr? will the watchpoint still trigger?

[this.wiederkehr]

Looks like the above error was fixed with the following commit:

https://github.com/espressif/esp-idf/co ... 9278498965