CPU cores seem to lock up on dport access

[Oromis]

This is a follow-up post on viewtopic.php?f=2&t=4246.

Summary: My ESP32 watchdog triggers after a few hours. Using the answer from the other thread, I found out the files and lines the CPUs are working on at the time of the reset.

PRO CPU: File $IDF_PATH/components/esp32/dport_panic_highint_hdl.S:181. The PRO CPU seems to be stuck in a tight loop waiting for CPU1 to finish a dport access (I might be wrong since I don't know too much about assembler):

Code: Select all

.check_dport_access_end:
    l32i    a3, a2, 0
    beqz    a3, .check_dport_access_end

APP CPU: File $IDF_PATH/components/esp32/dport_access.c:102. Meanwhile, the APP CPU seems to be busy doing the dport operation the PRO CPU is waiting for. It was executing this line:

Code: Select all

REG_READ(SPI_DATE_REG(3));

in the following function:

Code: Select all

/* stall other cpu that this cpu is pending to access dport register start */
void IRAM_ATTR esp_dport_access_stall_other_cpu_start(void)
{
#ifndef CONFIG_FREERTOS_UNICORE
    if (dport_core_state[0] == DPORT_CORE_STATE_IDLE
        || dport_core_state[1] == DPORT_CORE_STATE_IDLE) {
        return;
    }

    BaseType_t intLvl = portENTER_CRITICAL_NESTED();

    int cpu_id = xPortGetCoreID();

#ifdef DPORT_ACCESS_BENCHMARK
    ccount_start[cpu_id] = XTHAL_GET_CCOUNT();
#endif

    if (dport_access_ref[cpu_id] == 0) {
        portENTER_CRITICAL_ISR(&g_dport_mux);

        oldInterruptLevel[cpu_id]=intLvl;

        dport_access_start[cpu_id] = 0;
        dport_access_end[cpu_id] = 0;

        if (cpu_id == 0) {
            _DPORT_REG_WRITE(DPORT_CPU_INTR_FROM_CPU_3_REG, DPORT_CPU_INTR_FROM_CPU_3); //interrupt on cpu1
        } else {
            _DPORT_REG_WRITE(DPORT_CPU_INTR_FROM_CPU_2_REG, DPORT_CPU_INTR_FROM_CPU_2); //interrupt on cpu0
        }

        while (!dport_access_start[cpu_id]) {};

        // !!! APP CPU was at the following instruction during the WDT-caused reset
        REG_READ(SPI_DATE_REG(3));  //just read a APB register sure that the APB-bus is idle
    }

    dport_access_ref[cpu_id]++;

    if (dport_access_ref[cpu_id] > 1) {
        /* Interrupts are already disabled by the parent, we're nested here. */
        portEXIT_CRITICAL_NESTED(intLvl);
    }
#endif /* CONFIG_FREERTOS_UNICORE */
}

How can this happen? What causes a dport access in the first place? How can I prevent this apparent deadlock from happening?

Thank you for any suggestions. I'm not deep enough into ESP32 internals to be able to solve this on my own.

[ESP_Angus]

Hi Oromis,

There are a few places where a CPU has to make an access from DPORT memory (the most common during normal operation is hardware crypto acceleration). Due to a silicon bug we have to ensure the other CPU is not doing anything while the DPORT access takes place (hence all the fuss around "dport access"). It's possible that somewhere the DPORT protection has a bug causing it not to unlock correctly. Most other DPORT access happens at startup.

There is a feature currently in review (probably to be merged early next week) which dumps the full call stack on both CPUs when an interrupt watchdog like this happens. Probably the easiest thing to do is wait for this feature to land on github, then dump (& decode) the call stack from both CPUs to get some more context about what it's doing when it crashes.


Angus

[Oromis]

Hey Angus,

thanks for your reply and the insight you provided! My program makes use of hardware-accelerated crypto, so this might be related indeed.

I will wait for the change to be merged. Is there some pull request or branch I can watch to know when it is done?

Thanks for your help!