freertos mutex semaphore crashing

[italocjs]

Hello!,
I have 5 tasks accessing the multiple devices under the same i2c bus, to avoid conflict i have added semaphore on them, but i am still getting these errors (originating from different tasks):

Code: Select all

ERROR 1: assert failed: xQueueGenericSend queue.c:832 (pxQueue->pcHead != ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == xTaskGetCurrentTaskHandle())

ERROR 2: assert failed: vTaskPriorityDisinheritAfterTimeout (pxTCB->uxMutexesHeld)

I have tried to look into google but there is not much about these errors, This is how i'm creating the tasks and mutex.

Mutex creation:

Code: Select all

//Defined in a common header file
SemaphoreHandle_t xI2CSemaphore;     // Shared I2C Resource Mutex

//Running on the "setup()"
void setup_mutex() {
  if (xI2CSemaphore == NULL) {
    xI2CSemaphore = xSemaphoreCreateMutex();
    if ((xI2CSemaphore) != NULL) xSemaphoreGive((xI2CSemaphore));
  }

Task definition:

Code: Select all

#include "the_file_which_has_the_semaphore.h";
extern SemaphoreHandle_t xI2CSemaphore;
TaskHandle_t Handle_taskTMP102;
void task_temperature(void* pvParameters)
{
    ESP_LOGI("", "started");
    for (;;)
    {
        if (xSemaphoreTake(xI2CSemaphore, (TickType_t)DEFAULT_SEMAPHORE_TIMEOUT) == pdTRUE)
        {
            xLog.temperature = get_temperature(); //This is the function accessing i2c
            if (xSemaphoreGive(xI2CSemaphore) != pdTRUE)
            {
                ESP_LOGE("", "FAILED TO RETURN xI2CSemaphore");
            }
            vTaskDelay(TMP102_UPDATE_CICLE_MS / portTICK_PERIOD_MS);
        }
    }
}

Full error:

Code: Select all

[    24][I][xlog.h:94] serial_setup(): [xLOG] xLog library version 1.0.0, compiled in Mar  2 2023 12:53:16
[    36][I][nvs.h:47] setup_nvs(): [] BOOT COUNTER IS 1
[    40][I][esp32-hal-i2c.c:75] i2cInit(): Initialising I2C Master: sda=21 scl=22 freq=100000
[    43][W][Wire.cpp:301] begin(): Bus already started in Master Mode.
[   105][I][sdcard.h:211] setup_sd(): [] SD Card initialized
[   106][W][Wire.cpp:301] begin(): Bus already started in Master Mode.
[   111][I][rtc.h:54] task_sync_rtc_to_gps(): [] started
[   111][I][rtc.h:149] task_rtc(): [] started
[   411][W][Wire.cpp:301] begin(): Bus already started in Master Mode.
[   412][I][tmp102.h:94] task_temperature(): [] started
[   412][I][gps.h:85] task_update_gps(): [] started
[   426][I][accelerometer.h:79] read_calibrated_offset(): [] read offsets = x 0, y 0, z 0

assert failed: xQueueGenericSend queue.c:832 (pxQueue->pcHead != ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == ((void *)0) || pxQueue->u.xSemaphore.xMutexHolder == xTaskGetCurrentTaskHandle())


Backtrace: 0x40083c85:0x3ffd5620 0x40093ed9:0x3ffd5640 0x400997e9:0x3ffd5660 0x4009498e:0x3ffd5790 0x400d6d4f:0x3ffd57d0 0x400d6fd9:0x3ffd57f0 0x400d61c9:0x3ffd5810 0x400d6271:0x3ffd5830

  #0  0x40083c85:0x3ffd5620 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
  #1  0x40093ed9:0x3ffd5640 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
  #2  0x400997e9:0x3ffd5660 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
  #3  0x4009498e:0x3ffd5790 in xQueueGenericSend at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/freertos/queue.c:821 (discriminator 2)
  #5  0x400d6fd9:0x3ffd57f0 in TwoWire::endTransmission() at C:/Users/italo/.platformio/packages/framework-arduinoespressif32/libraries/Wire/src/Wire.cpp:642
  #6  0x400d61c9:0x3ffd5810 in get_temperature() at src/xlog/MODULES/tmp102.h:46
  #7  0x400d6271:0x3ffd5830 in task_temperature(void*) at src/xlog/MODULES/tmp102.h:99
ELF file SHA256: 7bdc8e34d23a705c
>> NOTE: THE ESP32 IS STUCK HERE, it does not reboot or do anything until the reset is pressed

Notes:
1 - i have tried DEFAULT_SEMAPHORE_TIMEOUT as 0, 10, 1000 with the same results
2 - i have tried disabling some tasks, which does help reducing the errors.

Does anyone have any idea on what could be happening?

[ESP_Sprite]

Looking at the traceback, I don't think it's your semaphore that is crashing; it looks like some mux internal to the TwoWire library. I can't tell you why it crashes, but usually these things are the result of a memory corruption bug; you may want to see if there's any of that in your code.

[italocjs]

Hello ESP_Sprite, thanks for the suggestion, i believe it was indeed corruption duo access conflict during void setup() phase, while i was setting the last modules up, the first ones already had they tasks running, causing competition to access the i2c, conflicting with the setup_functions which were not mutex protected.

I managed to "fix" it creating an bool to signal that the boot was over (bool booting), the errors stopped but i dont think this is the best solution.

Code: Select all

bool setup_rtc(){
//Run code which uses i2c to setup the module
//create task
}

void rtc_task(){
 while(booting) delay(10);
//get semaphore
//update datetime from RTC
//give semaphore
}

I've though about embedding the setup code within the task, so they would be protected by the semaphore, Would this be a better approach?

Code: Select all

void setup_rtc() {
//create rtc task
}

void rtc_task(){
//get semaphore
//setup the rtc
//give semaphore
for(;;) {
//get semaphore
//do task stuff
//give semaphore
}

[ESP_Sprite]

Why not set everything up before you create those tasks?

[italocjs]

This one is taking my sleep, lol
I still believe that the issue happens when more than one "thing" is trying to access the i2c bus, but unfortunately i cant find out who to blame! every function that access i2c are inside its own task, which is protected by the same shared semaphore. and i check if the semaphore was really taken before running any i2c functions.

I was able to live with this issue (i've designed the firmware to handle reboots and reload values from the nvs if it crashes). but the lack of reboot is causing a huge issue, sometimes the esp32 just freezes and stays frozen until powered down/up again (its installed in remote locations, usually 10+ hours of driving).

Is it possible to find out why it is not rebooting? i did not disabled the watchdog, everything is using the default parameters from the arduino port.

Could anyone give any tips on how to find out the culprit (of freezing and i2c crashes) ? jtag is not possible right now, but i'll design the next board to keep those pins free.

[noweare]

Try giving the task more stack space. Divide and conquer if you can. Just add one task at a time.

[italocjs]

Update: Found an case where the function crashed and exited without returning the semaphore first, this was causing the issue. Thanks for the tips.