Crash entering deep sleep with IDF 5.2.1

[cdollar]

I'm working on upgrading my code from IDF 5.1.2 to 5.2.1 and am seeing a crash when the code enters deep sleep. Here's the crash:

Code: Select all

I (11614) cl_main: Deep sleep for 289277 millis
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x4008206b  PS      : 0x00060b34  A0      : 0x80085a16  A1      : 0x3ffc5bf0  
0x4008206b: uart_ll_is_tx_idle at /Users/chris/esp/esp-idf/components/hal/esp32/include/hal/uart_ll.h:863
 (inlined by) esp_rom_uart_tx_wait_idle at /Users/chris/esp/esp-idf/components/esp_rom/patches/esp_rom_uart.c:22

A2      : 0x00000001  A3      : 0x00000000  A4      : 0x000023d7  A5      : 0x00006b80  
A6      : 0x00000000  A7      : 0x50001fe8  A8      : 0x3ff50000  A9      : 0x3ffc5c10  
A10     : 0x3ffb0058  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x3f404f35  A15     : 0x000003e5  SAR     : 0x00000009  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffd  
0x400014fd: strlen in ROM
0x4000150d: strlen in ROM



Backtrace: 0x40082068:0x3ffc5bf0 0x40085a13:0x3ffc5c20 0x40085ac3:0x3ffc5c40 0x40085d3d:0x3ffc5c90 0x40085d79:0x3ffc5cb0 0x400dbd2b:0x3ffc5cd0 0x400dc132:0x3ffc5cf0 0x401b8d1b:0x3ffc5d60 0x4008c15d:0x3ffc5d90
0x40082068: esp_rom_uart_tx_wait_idle at /Users/chris/esp/esp-idf/components/esp_rom/patches/esp_rom_uart.c:22 (discriminator 4)
0x40085a13: flush_uarts at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:440
0x40085ac3: esp_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:652
0x40085d3d: deep_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:982
0x40085d79: esp_deep_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:1005
0x400dbd2b: enterDeepSleep at /Users/chris/cl/firmware-esp32/main/CLMain.c:664
0x400dc132: app_main at /Users/chris/cl/firmware-esp32/main/CLMain.c:275
0x401b8d1b: main_task at /Users/chris/esp/esp-idf/components/freertos/app_startup.c:208
0x4008c15d: vPortTaskWrapper at /Users/chris/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:134

Here's my setup before entering deep sleep:

Code: Select all

esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON);
esp_sleep_enable_ulp_wakeup();
esp_sleep_enable_ext1_wakeup(1ULL << fwContext.clDevice.cfg.buttonAPin, ESP_EXT1_WAKEUP_ALL_LOW);
esp_sleep_enable_timer_wakeup(sleepMillis * 1000);
ESP_LOGI(TAG, "Deep sleep for %lu millis", sleepMillis);
esp_deep_sleep_start();

Any ideas what could be causing this? The same code has been working great on IDF v5.1.2.
Thanks!

[ESP_Sprite]

That is an interesting bug... almost makes it look like the UART power domain has been turned off already. Any chance you can post the code that has this behaviour?

[cdollar]

Hi ESP_Sprite,

I really can't post the entire project code, but I will try and create a simple reproducer for the issue.

It really is a strange bug... As far as UARTs go, my code isn't doing anything special. UART0 is just the normal serial console for downloads and logs, and I use UART1 for a short time to detect if a certain sensor is connected or not, after which I delete the driver for UART1. I do have a ULP program running that reads from this sensor using the same pins that were used with UART1, but the reads done by the ULP are bit banged, and the ULP isn't started until after the UART1 driver is deleted, so it seems unlikely that that would be contributing to the problem. And again, it all works without issue on v5.1.2. Hopefully it isn't too difficult to extract things to a simple reproducer case.

Thanks for the help!

[cdollar]

In the process of creating the reproducer test code I figured out what is causing the panic (that's how it usually goes, right?). I'm not sure if it is really a bug as much as a change in behavior, but the crux of it is this: If you enable a UART pattern detect interrupt on some UART port, AND also intend to later delete the UART driver on that UART port, then you MUST disable the pattern detect interrupt before deleting the UART driver, otherwise you'll get a panic when entering deep sleep.

The issue doesn't happen if you don't delete the UART driver.

Here's the reproducer code for the issue. If you run the code as-is then you won't see a crash, because the code calls

Code: Select all

uart_disable_pattern_det_intr()

to disable the pattern detect interrupt before it calls

Code: Select all

uart_driver_delete()

This is basically what my larger codebase was doing - using the UART driver and pattern detect interrupt to check it a sensor was connected and transmitting, before deleting the UART driver to turn things over to the ULP program. But I've not been disabling the pattern detect interrupt...

Code: Select all

#include <stdio.h>
#include "esp_log.h"
#include "esp_sleep.h"
#include "driver/uart.h"
#include "esp_check.h"

#define UART_RXBUF_SIZE (UART_HW_FIFO_LEN(UART_PORT_NUM) * 2)
#define UART_DET_QUEUE_SIZE 2
#define UART_RX_PIN 26
#define UART_TX_PIN 25

const static char *TAG = "example";

void app_main(void)
{
    ESP_LOGI(TAG, "Example starting");

    // configure and load UART1 driver
    QueueHandle_t uartQueue;
    uart_config_t uartConfig = {
        .baud_rate = 9600,
        .data_bits = UART_DATA_8_BITS,
        .parity = UART_PARITY_DISABLE,
        .stop_bits = UART_STOP_BITS_1,
        .flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
        .source_clk = UART_SCLK_APB,
    };
    ESP_ERROR_CHECK(uart_param_config(UART_NUM_1, &uartConfig));
    ESP_ERROR_CHECK(uart_set_pin(UART_NUM_1, UART_TX_PIN, UART_RX_PIN, UART_PIN_NO_CHANGE, UART_PIN_NO_CHANGE));
    ESP_ERROR_CHECK(uart_driver_install(UART_NUM_1, UART_RXBUF_SIZE, 0, UART_DET_QUEUE_SIZE, &uartQueue, 0));

    // enable UART1 pattern detect interrupt
    ESP_ERROR_CHECK(uart_enable_pattern_det_baud_intr(UART_NUM_1, '\r', 1, 9, 0, 0));
    ESP_ERROR_CHECK(uart_pattern_queue_reset(UART_NUM_1, UART_DET_QUEUE_SIZE));
    ESP_LOGI(TAG, "Loaded UART1 driver");

    // when done with UART1 it is necessary to disable this interrupt,
    // otherwise you'll get a panic when attempting to enter deep sleep
    ESP_ERROR_CHECK(uart_disable_pattern_det_intr(UART_NUM_1));

    // delete the UART1 driver
    ESP_ERROR_CHECK(uart_driver_delete(UART_NUM_1));
    ESP_LOGI(TAG, "Deleted UART1 driver");

    // configure and enter deep sleep
    esp_sleep_enable_timer_wakeup(10 * 1000 * 1000);
    ESP_LOGI(TAG, "Starting deep sleep for 10s");
    esp_deep_sleep_start();
}

If you comment out the

Code: Select all

ESP_ERROR_CHECK(uart_disable_pattern_det_intr(UART_NUM_1))

line in the above code and run it then you'll see it crash with the same panic I described in my initial post:

Code: Select all

I (307) spi_flash: detected chip: generic
I (310) spi_flash: flash io: dio
I (314) main_task: Started on CPU0
I (324) main_task: Calling app_main()
I (324) example: Example starting
I (324) uart: queue free spaces: 2
I (324) example: Loaded UART1 driver
I (334) example: Deleted UART1 driver
I (334) example: Starting deep sleep for 10s
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x40081e0a  PS      : 0x00060e34  A0      : 0x800848aa  A1      : 0x3ffb4d90  
0x40081e0a: esp_rom_uart_tx_wait_idle at /Users/chris/esp/esp-idf/components/esp_rom/patches/esp_rom_uart.c:22 (discriminator 1)

A2      : 0x00000001  A3      : 0x00000000  A4      : 0x000028f6  A5      : 0x00003380  
A6      : 0x00000000  A7      : 0x50001fe8  A8      : 0x00000000  A9      : 0x3ffb4db0  
A10     : 0x3ffb0050  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x3f40331b  A15     : 0x000003e8  SAR     : 0x00000009  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffd  
0x400014fd: strlen in ROM
0x4000150d: strlen in ROM



Backtrace: 0x40081e07:0x3ffb4d90 0x400848a7:0x3ffb4dc0 0x40084957:0x3ffb4de0 0x40084c06:0x3ffb4e30 0x40084c45:0x3ffb4e50 0x400d5817:0x3ffb4e70 0x400e5ec4:0x3ffb4eb0 0x40087ff9:0x3ffb4ee0
0x40081e07: uart_ll_is_tx_idle at /Users/chris/esp/esp-idf/components/hal/esp32/include/hal/uart_ll.h:864 (discriminator 4)
 (inlined by) esp_rom_uart_tx_wait_idle at /Users/chris/esp/esp-idf/components/esp_rom/patches/esp_rom_uart.c:22 (discriminator 4)
0x400848a7: flush_uarts at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:440
0x40084957: esp_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:652
0x40084c06: deep_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:982
0x40084c45: esp_deep_sleep_start at /Users/chris/esp/esp-idf/components/esp_hw_support/sleep_modes.c:1005
0x400d5817: app_main at /Users/chris/esp/sleep-panic-example/main/main.c:48
0x400e5ec4: main_task at /Users/chris/esp/esp-idf/components/freertos/app_startup.c:208
0x40087ff9: vPortTaskWrapper at /Users/chris/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:134

Hopefully this helps someone in the future!

[ESP_Sprite]

Good catch! I'll pass this on to the driver team; if any, crashing like you saw is not good behaviour I'd like to see in ESP-IDF. Thanks for taking the time to post what you found.