StoreProhibited triggered by FreeRTOS

[timredfern]

I'm trying to port a firmware project to ESP32 and I'm having trouble in getting the 3 tasks correctly configured and working together.

Although the various parts all seem to work in isolation, when the whole thing comes together it triggers seemingly unrelated errors in FreeRTOS itself. I've looked at priorities, stack sizes, etc.. The underlying reason eludes me.

What does

Code: Select all

Guru Meditation Error: Core  0 panic'ed (StoreProhibited)

actually mean? Something has tried to write to an unallocated pointer?

Most of the errors follow this pattern:

Code: Select all

Guru Meditation Error: Core  0 panic'ed (StoreProhibited)
. Exception was unhandled.
Core 0 register dump:
PC      : 0x4008cf79  PS      : 0x00060033  A0      : 0x8008b9b8  A1      : 0x3ffb05a0
A2      : 0x3ffb3b48  A3      : 0x3ffcec90  A4      : 0x3ffbba7c  A5      : 0x000000ff
A6      : 0x00000001  A7      : 0x00000000  A8      : 0x3ffb7946  A9      : 0x00000000
A10     : 0x3ffcd1c0  A11     : 0x00000000  A12     : 0x00060021  A13     : 0x00000001
A14     : 0x00060021  A15     : 0x00060023  SAR     : 0x00000011  EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000004  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000
Core 0 was running in ISR context:
EPC1    : 0x4008cf79  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4008a78a

Backtrace: 0x4008cf79:0x3ffb05a0 0x4008b9b5:0x3ffb05c0 0x4008a1a3:0x3ffb05e0 0x4008c525:0x3ffb0600 0x40082582:0x3ffb0610 0x400d374b:0x00000000

And when I use EspStackTraceDecoder.jar to find the offending lines of code, it often shows an error within FreeRTOS like this:

Code: Select all

Exception Cause: Not found

0x4008cf79: vListInsertEnd at /Users/tim/ESP/esp-idf/components/freertos/list.c:131
0x4008cf79: vListInsertEnd at /Users/tim/ESP/esp-idf/components/freertos/list.c:131
0x4008a78a: xQueueGiveMutexRecursive at /Users/tim/ESP/esp-idf/components/freertos/queue.c:2037
0x4008cf79: vListInsertEnd at /Users/tim/ESP/esp-idf/components/freertos/list.c:131
0x4008b9b5: xTaskIncrementTick at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008a1a3: xPortSysTickHandler at /Users/tim/ESP/esp-idf/components/freertos/port.c:406
0x4008c525: _frxt_timer_int at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:303
0x40082582: _xt_lowint1 at /Users/tim/ESP/esp-idf/components/freertos/xtensa_vectors.S:1105
0x400d374b: esp_vApplicationIdleHook at /Users/tim/ESP/esp-idf/components/esp32/freertos_hooks.c:85

I'm uncertain why this would happen when the tasks all run happily by themselves. It seems at times that the error is triggered by accessing one global error of memory by a task, when another task is running that doesn't access this memory.

I've noticed that core panics can be caused by a task locking up a core in an infinite loop, but this doesnt seem to be the case here, and I can't pinpoint whether it's to do with resources, memory protection, priorities..

I just tried to debug this particular crash ^^^ and openocd showed a very peculiar error:

Code: Select all

esp32: target state: halted
Info : Target halted. PRO_CPU: PC=0x400D27D5 (active)    APP_CPU: PC=0x40080F6F 
Error: FreeRTOS maximum used priority is unreasonably big, not proceeding: 1061179556

All my tasks have priority 1. A task is being created where the priority is set by some random unzeroed memory? This must be within the SDK itself?

I know this is vague, but I don't see a way of making a minimal test case. Any clue would be helpful.

[ESP_Sprite]

StoreProhibited is indeed something trying to write to non-existent memory, by e.g. dereferencing a NULL pointer. The fact that you're seeing this deep within FreeRTOS usually points to some memory corruption.

What you can try is to enable the heap memory debug options, as detailed here: https://esp-idf.readthedocs.io/en/lates ... debug.html . If there's something writing past the end of a buffer, with the debugging options on you have a chance of detecting it.

The FreeRTOS task thing usually is something that happens when OpenOCD tries to enable its freertos logic when the program isn't loaded in RAM yet; it'll try to read the highest thread prio but the variable that stores that is uninitialized and it'll get random garbage. That usually causes the error; it's not necessarily a defect in your program.

[chegewara]

Im wondering about this line:

Code: Select all

0x400d374b: esp_vApplicationIdleHook at /Users/tim/ESP/esp-idf/components/esp32/freertos_hooks.c:85

Do you have vApplicationIdleHook enabled in menuconfig? And if its enabled, do you have it implemented in your code?

[ESP_Sprite]

Ah, another possible cause: do your task functions happen to return instead of calling vTaskDelete()?

[timredfern]

Thanks for the replies, I've tried a few things but it feels like playing whack-a-mole. Whatever I do, the ESP32 core panics in different ways.

- My tasks don't return, they all enter while(1) loops

- I don't have a vApplicationIdleHook enabled in menuconfig, in fact I can't find this option in menuconfig and neither can grep.

- I enabled the heap memory debugging, and the code ran to a different point, where it paniced with a brownout. I then switched to using a 5v wall-wart and the code paniced in another different way:

Code: Select all

Guru Meditation Error: Core  0 panic'ed (LoadProhibited)
. Exception was unhandled.
Core 0 register dump:
PC      : 0x4008b44e  PS      : 0x00060033  A0      : 0x4008c923  A1      : 0x3ffbef40  
A2      : 0x00050023  A3      : 0x00000000  A4      : 0x3ffb3d34  A5      : 0x3ffb3d34  
A6      : 0x00000001  A7      : 0x00000002  A8      : 0x00000005  A9      : 0x00000000  
A10     : 0x3ffb106c  A11     : 0x00000000  A12     : 0x3ffb3da0  A13     : 0x00000000  
A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x0000001a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x0000000c  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0xffffffff  

Backtrace: 0x4008b44e:0x3ffbef40 0x4008c920:0x3ffbef60 0x4008c8d6:0x00000000

CPU halted.

Exception Cause: Not found

0x4008b44e: vTaskSwitchContext at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008c923: _frxt_dispatch at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:407
0x4000c46c: ?? ??:0
0x4000c477: ?? ??:0
0x4008b44e: vTaskSwitchContext at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008c920: _frxt_dispatch at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:406
0x4008c8d6: _frxt_int_exit at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:206

This doesn't seem to have invoked the heap memory debugging, but selecting this option seems to have had some kind of side effect?

- I removed heap debugging again and got a different panic:

Code: Select all

0x4008b961: xTaskIncrementTick at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x400d35b6: esp_vApplicationIdleHook at /Users/tim/ESP/esp-idf/components/esp32/freertos_hooks.c:85
0x4008b961: xTaskIncrementTick at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008b961: xTaskIncrementTick at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008a1a3: xPortSysTickHandler at /Users/tim/ESP/esp-idf/components/freertos/port.c:406
0x4008c525: _frxt_timer_int at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:303
0x4008258e: _xt_lowint1 at /Users/tim/ESP/esp-idf/components/freertos/xtensa_vectors.S:1105
0x400d35b3: esp_vApplicationIdleHook at /Users/tim/ESP/esp-idf/components/esp32/freertos_hooks.c:85
0x4008a20f: uxPortCompareSet at /Users/tim/ESP/esp-idf/components/freertos/port.c:406
 (inlined by) vPortCPUAcquireMutexIntsDisabledInternal at /Users/tim/ESP/esp-idf/components/freertos/portmux_impl.inc.h:86
 (inlined by) vPortCPUAcquireMutexIntsDisabled at /Users/tim/ESP/esp-idf/components/freertos/portmux_impl.h:98
 (inlined by) vPortCPUAcquireMutex at /Users/tim/ESP/esp-idf/components/freertos/port.c:365
0x4008a20f: uxPortCompareSet at /Users/tim/ESP/esp-idf/components/freertos/port.c:406
 (inlined by) vPortCPUAcquireMutexIntsDisabledInternal at /Users/tim/ESP/esp-idf/components/freertos/portmux_impl.inc.h:86
 (inlined by) vPortCPUAcquireMutexIntsDisabled at /Users/tim/ESP/esp-idf/components/freertos/portmux_impl.h:98
 (inlined by) vPortCPUAcquireMutex at /Users/tim/ESP/esp-idf/components/freertos/port.c:365
0x4008afda: vTaskSwitchContext at /Users/tim/ESP/esp-idf/components/freertos/tasks.c:4571
0x4008c550: _frxt_dispatch at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:406
0x4008c506: _frxt_int_exit at /Users/tim/ESP/esp-idf/components/freertos/portasm.S:206

nb, my code image is 815184 bytes and last time I checked there was several hundred K free memory. Something just seems to have hit some limit and the ESP32 just can't run reliably, it doesn't seem to be power...

[timredfern]

I'm now trying to change / increase options in FreeRTOS component config one by one. For reference, this is exactly how I am creating the tasks: (although this example runs without a problem)

Code: Select all

#define TASK_STACK_SIZE 5000
#define TASK_PRIORITY 5

static const char *tag="Main";
#include "config.h"

#include <string.h>

#include "freertos/FreeRTOS.h"
#include "freertos/event_groups.h"
#include "esp_log.h"
#include "esp_wifi.h"

void task_1(){
	ESP_LOGI(tag, "Task 1 started");

	while (1) {
		ESP_LOGI(tag, "Task 1");
		vTaskDelay(1);
	}
}

void task_2(){
	ESP_LOGI(tag, "Task 2 started");

	while (1) {
		ESP_LOGI(tag, "Task 2");
		vTaskDelay(10);
	}
}

void task_3(){
	ESP_LOGI(tag, "Task 3 started");

	while (1) {
		ESP_LOGI(tag, "Task 3");
		vTaskDelay(100);
	}
}

void app_main()
{

	xTaskCreatePinnedToCore(task_1, "task 1", TASK_STACK_SIZE, NULL, TASK_PRIORITY, NULL,0);

	xTaskCreatePinnedToCore(task_2, "task 2", TASK_STACK_SIZE, NULL, TASK_PRIORITY, NULL,1);

	xTaskCreatePinnedToCore(task_3, "task 3", TASK_STACK_SIZE, NULL, TASK_PRIORITY, NULL,0);

}

One thing that has confused me is whether to use the function, or the address of the function, as the first argument to xTaskCreatePinnedToCore? I believe just sending the function is correct, but both appear to work, and in fact if you grep the esp-idf sdk both forms are used in different examples!?

[chegewara]

make menuconfig

Component config-> FreeRTOS-> Use FreeRTOS legacy hooks

[fly135]

Do you have ext spiram on your board? The backtrace on your first post seems to be doing something in spiram_psram.c.

John A

[ESP_Sprite]

Also, what is your hardware setup? Devboard, custom hardware, ...; cabling, LDO, power supply, ..? The issues with the brownout as well as the random-looking crashes may indicate a hardware fault instead of a software-based one.

[timredfern]

Hi there,

"use FreeRTOS legacy hooks" is disabled.

I have 2 boards, I'm seeing similar problems on both (almost, but not always identically). Neither has spiram and the spiram option is disabled (where do you see the reference to spiram_psram? I can't)

My 2 boards are:

- lolin32 LITE v1.0.0

- ESP-WROVER-KIT (v3 apparently although this isn't marked)

I'm powering the boards via USB although the brownout error does seem real as it was affected by using a 5v supply - it seems that the Wifi subsystem can draw a large current momentarily? (another panic popped up when I made the switch).

The errors are repeatable although seemingly impossible to trace. Changing something in any part of the code that runs during initial setup can make a crash pop up in the lowest level of FreeRTOS, even enabling heap protection can have this effect. The crashes appear almost identically on the 2 boards so it seems to be fundamental to the chip itself. My gut feeling was that a timing thing was causing dependency between different tasks.

It now seems to me that turning on heap corruption detection completely stops the random Store/Load/WDT errors in FreeRTOS. That doesn't seem to me to be possible if it's just a timing effect.

However.. viewtopic.php?f=13&t=5524

Could these be related? It occurs to me that this weird stuff started happening when my project was nearly completely ported, which would validate the idea of a management issue that occurs when the memory starts to fill up.