spinlock on esp32s3

[mikkeldamsgaard]

Hey

Generally, it looks like to me that the spinlock logic has some issue when the freertos tasks do not have a core affinity specified.

For example, the lwip task can be configured without core affinity and if I do that (CONFIG_LWIP_TCPIP_TASK_AFFINITY_NO_AFFINITY=y), then this happens after a few seconds:

assert failed: spinlock_release spinlock.h:158 (core_id == lock->owner)


Backtrace:0x4037c266:0x3fcc45f00x4038aad5:0x3fcc4610 0x4039341d:0x3fcc4630 0x4038e393:0x3fcc4750 0x4038c2e8:0x3fcc4770 0x403927d5:0x3fcc47b0 0x420c5ea7:0x3fcc47d0 0x40391d48:0x3fcc47f0 0x42035be5:0x3fcc4810 0x4003966f:0x3fcc4830 |<-CORRUPTED
0x4037c266: panic_abort at esp-idf/components/esp_system/panic.c:402

0x4038aad5: esp_system_abort at esp-idf/components/esp_system/esp_system.c:128

0x4039341d: __assert_func at esp-idf/components/newlib/assert.c:85

0x4038e393: spinlock_release at esp-idf/components/esp_hw_support/include/soc/spinlock.h:158
(inlined by) vPortExitCritical at esp-idf/components/freertos/port/xtensa/port.c:310

0x4038c2e8: xQueueSemaphoreTake at esp-idf/components/freertos/queue.c:1612

0x403927d5: sys_mutex_lock at esp-idf/components/lwip/port/esp32/freertos/sys_arch.c:82

0x420c5ea7: sys_arch_protect at esp-idf/components/lwip/port/esp32/freertos/sys_arch.c:480

0x40391d48: pbuf_free at esp-idf/components/lwip/lwip/src/core/pbuf.c:755
(inlined by) pbuf_free at esp-idf/components/lwip/lwip/src/core/pbuf.c:729

0x42035be5: esp_netif_netstack_buf_free at esp-idf/components/esp_netif/lwip/esp_netif_lwip.c:862

If I change the affinity to be a specific core, for example CONFIG_LWIP_TCPIP_TASK_AFFINITY_CPU0=y, then the system returns to being stable.

Kind regards
Mikkel

[ESP_Dazz]

Could you enable CONFIG_ESP_SYSTEM_PANIC_GDBSTUB in menuconfig to trigger GDBstub to run on panic, then use GDBstub to check the value of the spinlock's "lock->owner" field. If could be stack overflow (or some memory corruption) that is corrupting the spinlock contents.

[godzilla2]

I am also hitting this issue.

esp32s3
esp-idf 4.4.1 (Tue May 31 10:28:03)

It does appear to be corruption:
core_id: 43947
lock->owner: 3007315967

I used ESP_EARLY_LOGI to grab the values:

Code: Select all

    if (core_id != lock->owner){
        ESP_EARLY_LOGI("","core_id %u, lock->owner %u", core_id, lock->owner);
    }
    assert(core_id == lock->owner); // This is a mutex we didn't lock, or it's corrupt

[ESP_Sprite]

Those numbers are 'abab' and 'b33fffff' in hexadecimal, which very much sounds like heap allocator guard numbers. Are you sure the spinlock isn't in a memory structure that is free()'ed already?

[godzilla2]

Good catch!

Note: I do have heap & stack protection enabled with comprehensive & overall (strongest protection)

How would I know? I did not allocate this spinlock... I don't think

I've made a Github Issue: https://github.com/espressif/esp-idf/issues/9120

[godzilla2]

found these:

#define PRO_CPUID (0xcdcd)
#define APP_CPUID (0xabab)
#define SPINLOCK_FREE 0xB33FFFFF

[ESP_Sprite]

Ah. Never mind then. I guess those got changed to magic numbers in the spinlock code somewhere in the past >_<

[godzilla2]

please see my newest update in: https://github.com/espressif/esp-idf/issues/9120