vTaskDelay

davdav
Posts: 208
Joined: Thu Nov 17, 2016 2:33 pm

Re: vTaskDelay

Postby davdav » Wed Dec 12, 2018 9:58 pm

Hi ESP_Dazz,

I'm going to add the "assert(0);" to those function and start again the test.

I also report here the tasks I create on my project (with their priority, stack size and if vTaskDelay is used).

- Tasks running on device but which do not use vTaskDelay:

xTaskCreate(uart_task, "uTsk", 3500, NULL, 11, &UART_TaskHandle); -> no vTaskDelay used
xTaskCreate(GSM_uartTask, "UauxTsk", 4096, NULL, 11, &GSM_TaskHandle); -> no vTaskDelay used

-Tasks on device which do not use vTaskDelay (but they are NOT running on my test):

xTaskCreate(SOT_task, "SOTt", 3000, NULL, 8, &SOT_TaskHandle); -> no vTaskDelay used
xTaskCreate(WIEGAND_task, "WieT", 2048, NULL, 9, &WIEGAND_TaskHandle); -> no vTaskDelay used
xTaskCreate(IR_RC5_task, "ir_t", 2048, NULL, 8, &IR_TaskHandle); -> no vTaskDelay used
xTaskCreate(SD_Task, "SD_task", 3000, (uint8_t *) &val, 9, NULL); -> no vTaskDelay used
xTaskCreate(SD_boot_Task, "SD_boot", 3072, NULL, 9, NULL); -> no vTaskDelay used

-Tasks which use vTaskDelay. They are all coded as "loops" in the form of:

Code: Select all

while (1) {
..do my stuff..

vTaskDelay(xxx);
}

vTaskDelete(NULL);
xTaskCreate(COMMAND_state_machine, "cTsk", 6144, NULL, 9, &COMMAND_TaskHandle); -> vTaskDelay(20 / portTICK_PERIOD_MS); (20ms)
xTaskCreate(INPMNG_ReadTask, "I_read", 1500, NULL, 10, &INPMNG_ReadHandle); -> vTaskDelay(10 / portTICK_PERIOD_MS); (10ms)
xTaskCreate(INPMNG_task, "I_tsk", 2048, NULL, 9, &INPMNG_MngHandle); -> vTaskDelay(1000 / portTICK_PERIOD_MS); (1sec)
xTaskCreate(INPMNG_AnalogTask, "I_ana", 2048, NULL, 10, &INPMNG_AnaHandle); -> vTaskDelay(1000 / portTICK_PERIOD_MS); (1sec)
xTaskCreate(port_task, "i2c_tsk", 1024 * 2, (void* ) 0, 11, &PORT_TaskHandle); -> vTaskDelay(250 / portTICK_RATE_MS); (250ms)
xTaskCreate(RULES_State_Machine, "rTsk", 5120, NULL, 7, &RULES_TaskHandle); -> vTaskDelay(20 / portTICK_PERIOD_MS); (20ms)
xTaskCreate(SMM_task, "SmmTask", 5096, NULL, 10, &SMM_TaskHandle); -> vTaskDelay(100 / portTICK_PERIOD_MS); (100ms)

Finally I have 'main' task which I don't know what is its priority. Stack size of main task is 5120.

I'll let you know ASAP I will get something new from test.

Thank you

davdav
Posts: 208
Joined: Thu Nov 17, 2016 2:33 pm

Re: vTaskDelay

Postby davdav » Thu Dec 13, 2018 8:43 am

Hello @ESP_Dazz,
ESP_Dazz wrote:
Wed Dec 12, 2018 7:47 am
The easiest way to track down which task(s) are calling SPI flash functions would be to add an assert(0); at the first line of spi_flash_op_block_func() and spi_flash_disable_interrupts_caches_and_other_cpu(). This should cause a back trace to be printed when either functions are called.
I have inserted "assert(0);" in the two functions you mentioned (spi_flash_op_block_func() and spi_flash_disable_interrupts_caches_and_other_cpu()).

As soon as the program starts, it aborts when I call "nvs_flash_init()" in my code (one of the first stuff I do in order to read some parameter on NVS). Here is the backtrace.

Code: Select all

assertion "0" failed: file "/home/master/esp/esp-idf/components/spi_flash/cache_utils.c", line 96, function: spi_flash_disable_interrupts_caches_and_other_cpu
abort() was called at PC 0x400f7f23 on core 0

Backtrace: 0x40095663:0x3ffd1b70 0x400957bb:0x3ffd1b90 0x400f7f23:0x3ffd1bb0 0x40083362:0x3ffd1be0 0x40083bb4:0x3ffd1c00 0x40083df0:0x3ffd1c50 0x40109a03:0x3ffd1c80 0x40109b14:0x3ffd1cc0 0x401074f5:0x3ffd1ce0 0x4010751e:0x3ffd1d10 0x400e5cef:0x3ffd1d30 0x400d578b:0x3ffd1d50 0x400d384c:0x3ffd1ee0

Entering gdb stub now.
$T0b#e6

Code: Select all

(gdb) bt
#0  0x40095663 in invoke_abort ()
    at /home/master/esp/esp-idf/components/esp32/panic.c:140
#1  0x400957be in abort ()
    at /home/master/esp/esp-idf/components/esp32/panic.c:149
#2  0x400f7f26 in __assert_func (
    file=0x3f40e901 "/home/master/esp/esp-idf/components/spi_flash/cache_utils.c", line=96, func=<optimized out>, failedexpr=0x3f412                                 bfe "0")
    at ../../../.././newlib/libc/stdlib/assert.c:63
#3  0x40083365 in spi_flash_disable_interrupts_caches_and_other_cpu ()
    at /home/master/esp/esp-idf/components/spi_flash/cache_utils.c:96
#4  0x40083bb7 in spi_flash_mmap_pages (pages=0x40097b0c, page_count=1,
    memory=SPI_FLASH_MMAP_DATA, out_ptr=0x3ffd1c84, out_handle=0x3ffd1c80)
    at /home/master/esp/esp-idf/components/spi_flash/flash_mmap.c:164
#5  0x40083df3 in spi_flash_mmap (src_addr=<optimized out>,
    size=<optimized out>, memory=<optimized out>, out_ptr=0x3ffd1c84,
    out_handle=0x3ffd1c80)
    at /home/master/esp/esp-idf/components/spi_flash/flash_mmap.c:138
#6  0x40109a06 in load_partitions ()
    at /home/master/esp/esp-idf/components/spi_flash/partition.c:148
#7  esp_partition_find (type=ESP_PARTITION_TYPE_DATA,
    subtype=ESP_PARTITION_SUBTYPE_DATA_NVS, label=0x3f4082fd "nvs")
    at /home/master/esp/esp-idf/components/spi_flash/partition.c:67
#8  0x40109b17 in esp_partition_find_first (type=ESP_PARTITION_TYPE_DATA,
    subtype=ESP_PARTITION_SUBTYPE_DATA_NVS, label=0x3f4082fd "nvs")
    at /home/master/esp/esp-idf/components/spi_flash/partition.c:119
#9  0x401074f8 in nvs_flash_init_partition (part_name=0x3f4082fd "nvs")
    at /home/master/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:154
#10 0x40107521 in nvs_flash_init ()
    at /home/master/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:165
#11 0x400e5cf2 in NVS_Init ()
    at /home/master/esp/avior_esp32_wifi/main/nvs_utility.c:309
#12 0x400d578e in app_main ()
    at /home/master/esp/avior_esp32_wifi/main/avior_main.c:318
#13 0x400d384f in main_task (args=0x0)
    at /home/master/esp/esp-idf/components/esp32/cpu_start.c:476

However, the crash I had seen in my other test appears after the module is running (not at start-up). I'm going to delete the assert(0); from spi_flash_disable_interrupts_caches_and_other_cpu() and redo the test.

Thanks

ESP_Dazz
Posts: 308
Joined: Fri Jun 02, 2017 6:50 am

Re: vTaskDelay

Postby ESP_Dazz » Fri Dec 14, 2018 9:26 am

Hi davdav,
Does your application use NVS at any other point other than calling nvs_flash_init() during initialization? For example, does your application periodically read/write to NVS from a task?

davdav
Posts: 208
Joined: Thu Nov 17, 2016 2:33 pm

Re: vTaskDelay

Postby davdav » Fri Dec 14, 2018 10:21 am

ESP_Dazz wrote:
Fri Dec 14, 2018 9:26 am
Hi davdav,
Does your application use NVS at any other point other than calling nvs_flash_init() during initialization? For example, does your application periodically read/write to NVS from a task?
Yes it is, I read (and write it if necessary) NVS periodically (generally once every minute) from these tasks in my program:

main_task
INPMNG_task
port_task

I also read/write to 3 spiffs partition not periodically but asyncrously depending on external event.

Do you think the vTaskDelay issue is related to access of external flash chip?

Thanks

ESP_Dazz
Posts: 308
Joined: Fri Jun 02, 2017 6:50 am

Re: vTaskDelay

Postby ESP_Dazz » Mon Dec 17, 2018 12:30 pm

Hi davdav,
davdav wrote: Do you think the vTaskDelay issue is related to access of external flash chip?
Not exactly. My guess is that the tasks that access the external flash chip are the tasks that cause the scheduler to be suspended. Basically anything that accesses external flash (whether it be NVS, SPIFFS, FATFS) will internally call spi_flash_disable_interrupts_caches_and_other_cpu() which in turn will cause vTaskSuspendAll() to be called on both CPUs.

There could be a scheduling bug triggered by external flash access calling vTaskSuspendAll(), but I haven't been able to recreate any. Therefore, the only suggestion I can give as of now is to make sure that all of your tasks that can access SPI flash (via NVS, SPIFFS, FATFS) are pinned to a particular core, then check to see if the bug still persists. Pinning those tasks should prevent those tasks from swapping between cores hence lower the chances of a scheduling bug.

As for finding the source of the bug, I'll need to be able to recreate it first. Therefore, it would be great if you could send your application code, or a bare bones version of it that can consistently produce this bug. Thanks!

davdav
Posts: 208
Joined: Thu Nov 17, 2016 2:33 pm

Re: vTaskDelay

Postby davdav » Tue Dec 18, 2018 11:17 am

Hello @ESP_Dazz,

Thank you for reply. In effect, if I remeber correctly, I have seen the vTaskDelay assert sometimes when I wrote a file in a SPIFFS partition. I will do more test on this direction.
ESP_Dazz wrote:
Mon Dec 17, 2018 12:30 pm
Therefore, the only suggestion I can give as of now is to make sure that all of your tasks that can access SPI flash (via NVS, SPIFFS, FATFS) are pinned to a particular core, then check to see if the bug still persists. Pinning those tasks should prevent those tasks from swapping between cores hence lower the chances of a scheduling bug.
What core do you suggest to pin the task? APP cpu only? Or it is better to divide the tasks on both core (e.g. 3 task on PRO cpu and 3 tasks on APP cpu)?
Regarding "main" task: how can I pin it to a specific core?

As for finding the source of the bug, I'll need to be able to recreate it first. Therefore, it would be great if you could send your application code, or a bare bones version of it that can consistently produce this bug. Thanks!
I think it is quite difficult to create a bare bone which reproduce the issue because the assert seems "random". Maybe we could send you a sample of our product so you can test directly on it. Let's see if you suggestion to pinning task to a core helps to solve.

Thanks

ESP_Dazz
Posts: 308
Joined: Fri Jun 02, 2017 6:50 am

Re: vTaskDelay

Postby ESP_Dazz » Thu Dec 20, 2018 7:17 am

davdav wrote:Regarding "main" task: how can I pin it to a specific core?
The main task is created during cpu startup code and should be pinned to CPU0 with a priority of 1.
davdav wrote:What core do you suggest to pin the task?
I would suggest pinning the tasks to CPU0 (PRO CPU) since the main task and most background tasks (such as timer task) are also pinned to CPU0. I think an even better idea would be to try run your application in single core mode (configured in menuconfig). This should rule out any SMP related issues.

davdav
Posts: 208
Joined: Thu Nov 17, 2016 2:33 pm

Re: vTaskDelay

Postby davdav » Thu Dec 20, 2018 8:29 am

Thanks @ESP_Dazz for main task hint.
ESP_Dazz wrote:
Thu Dec 20, 2018 7:17 am
I would suggest pinning the tasks to CPU0 (PRO CPU) since the main task and most background tasks (such as timer task) are also pinned to CPU0. I think an even better idea would be to try run your application in single core mode (configured in menuconfig). This should rule out any SMP related issues.
OK I will pin the task involving access to NVS/SPIFFS to CPU0. I guess there will be a degradation in performance, but let's see how it works.

Regarding single core, I think it is a pity to use just one CPU while chip have two.

Thanks

schorsch13
Posts: 1
Joined: Fri May 01, 2020 7:54 am

Re: vTaskDelay

Postby schorsch13 » Fri May 01, 2020 9:10 am

Hello,
I came across the same problem as davdav: I am using a lot of things that are accessed by spi: WiFi uses nvs, application reads nvs every second, esp_vfs_fat_spiflash_mount() at application startup, linenoiseHistorySave() for console and fopen(), fprintf() occasionally.
I also use external spi-hardware with HSPI_HOST.
This works well until once in a while, the chip resets with:
I (112) boot: Chip Revision: 1
I (131) boot_comm: chip revision: 1, min. bootloader chip revision: 0
W (79) boot: PRO CPU has been reset by WDT.
W (80) boot: WDT reset info: PRO CPU PC=0x4008413d
W (80) boot: WDT reset info: APP CPU PC=0x4008427e
I (86) boot: ESP-IDF v4.0-dirty 2nd stage bootloader
The addresses refer to spi_flash_disable_interrupts_caches_and_other_cpu() (0x4008427e) and spi_flash_op_block_func (0x4008413d).
I do not want to go for single core, because I use also WiFi.
I can reduce the frequency of accesses to spi hardware to extend the duration between resets, but this is not a real option for a 24/7 device.
Any hints, workarounds, caveats, best practices for this kind of issue?

Who is online

Users browsing this forum: No registered users and 8 guests