StoreProhibited when starting OTA update

[tobobo]

I'm getting consistent `StoreProhibited` errors when starting an OTA update, seemingly related to `calloc` calls. I'm using esp-idf 4.4.3 on an esp32-s3. The error seems to happen immediately before any HTTPS connection takes place. Any suggestions for how to investigate this error would be appreciated!

[Codebox]
OtaManager *otaManager = this;
ESP_LOGI(TAG, "starting ota update");
xTaskCreate(
[](void *param)
{
ESP_LOGI(TAG, "creating ota update config");
OtaManager *otaManager = (OtaManager *)param;
esp_http_client_config_t config;
config.url = otaManager->url;
config.cert_pem = TUNESHINE_OTA_CERT;
config.event_handler = otaManager->eventHandler;
config.keep_alive_enable = true;
ESP_LOGI(TAG, "starting ota update");
esp_err_t ret = esp_https_ota(&config);
if (ret == ESP_OK)
{
ESP_LOGI(TAG, "Firmware upgrade successful, restarting");
esp_restart();
}
else
{
ESP_LOGE(TAG, "Firmware upgrade failed with code %i", ret);
esp_restart();
}
},
"otaManagerUpdate",
8192,
this,
5,
NULL);[/Codebox]

And here's the stack trace.

[Codebox]
15:46:09.203 > I (7303) OtaManager: starting ota update
15:46:09.203 > I (7303) OtaManager: creating ota update config
15:46:09.203 > I (7303) OtaManager: starting ota update
15:46:09.203 > Guru Meditation Error: Core 0 panic'ed (StoreProhibited). Exception was unhandled.
15:46:09.203 >
15:46:09.203 > Core 0 register dump:
15:46:09.203 > PC : 0x40384d95 PS : 0x00060933 A0 : 0x80385314 A1 : 0x3fcd6200
15:46:09.203 > A2 : 0x3fca9264 A3 : 0xb5359950 A4 : 0x00000000 A5 : 0x00060923
15:46:09.203 > A6 : 0x00060923 A7 : 0x00000000 A8 : 0x3fcd24e4 A9 : 0x3fcd24fc
15:46:09.203 > A10 : 0x00000001 A11 : 0xfffffffc A12 : 0x3fcd2504 A13 : 0x3fcd24ec
15:46:09.203 > A14 : 0x75687468 A15 : 0xfffffffc SAR : 0x0000001b EXCCAUSE: 0x0000001d
15:46:09.203 > EXCVADDR: 0xb5359950 LBEG : 0x400570e8 LEND : 0x400570f3 LCOUNT : 0x00000000
15:46:09.203 >
15:46:09.203 >
15:46:09.203 > Backtrace: 0x40384d92:0x3fcd6200 0x40385311:0x3fcd6220 0x403855ab:0x3fcd6240 0x4037769d:0x3fcd6260 0x40377750:0x3fcd6280 0x403864f8:0x3fcd62a0 0x4038651c:0x3fcd62c0 0x420b2d07:0x3fcd62e0 0x420b201f:0x3fcd6310 0x420334f1:0x3fcd6340 0x42033ac3:0x3fcd6380 0x42007919:0x3fcd63c0 0x4038275d:0x3fcd6470
15:46:09.332 >
15:46:09.332 > #0 0x40384d92:0x3fcd6200 in block_next at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf_block_functions.h:146
15:46:09.332 > (inlined by) block_link_next at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf_block_functions.h:154
15:46:09.332 > (inlined by) block_mark_as_free at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf_block_functions.h:162
15:46:09.332 > (inlined by) block_split at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf.c:299
15:46:09.332 > (inlined by) block_trim_free at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf.c:380
15:46:09.332 > (inlined by) block_prepare_used at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf.c:460
15:46:09.332 > (inlined by) tlsf_malloc at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_tlsf.c:780
15:46:09.332 > #1 0x40385311:0x3fcd6220 in multi_heap_malloc_impl at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/multi_heap.c:197
15:46:09.332 > #2 0x403855ab:0x3fcd6240 in multi_heap_malloc at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/multi_heap_poisoning.c:230
15:46:09.332 > #3 0x4037769d:0x3fcd6260 in heap_caps_malloc_base at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_caps.c:154
15:46:09.332 > #4 0x40377750:0x3fcd6280 in heap_caps_malloc_default at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/heap/heap_caps.c:207
15:46:09.332 > #5 0x403864f8:0x3fcd62a0 in _calloc_r at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/newlib/heap.c:65
15:46:09.332 > #6 0x4038651c:0x3fcd62c0 in calloc at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/newlib/heap.c:29
15:46:09.332 > #7 0x420b2d07:0x3fcd62e0 in http_header_init at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/esp_http_client/lib/http_header.c:34
15:46:09.332 > #8 0x420b201f:0x3fcd6310 in esp_http_client_init at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/esp_http_client/esp_http_client.c:589
15:46:09.332 > #9 0x420334f1:0x3fcd6340 in esp_https_ota_begin at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/esp_https_ota/src/esp_https_ota.c:216 (discriminator 4)
15:46:09.332 > #10 0x42033ac3:0x3fcd6380 in esp_https_ota at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/esp_https_ota/src/esp_https_ota.c:601
15:46:09.332 > #11 0x42007919:0x3fcd63c0 in OtaManager::update()::{lambda(void*)#1}::_FUN(void*) at src/comms/OtaManager.cpp:141
15:46:09.332 > (inlined by) _FUN at src/comms/OtaManager.cpp:152
15:46:09.332 > #12 0x4038275d:0x3fcd6470 in vPortTaskWrapper at /Users/tobiasbutler/.platformio/packages/framework-espidf@3.40403.0/components/freertos/port/xtensa/port.c:131
15:46:09.332 > [/Codebox]

Here's my esp-idf config:

sdkconfig.development-s3.zip

(10.65 KiB) Downloaded 176 times

[ESP_Sprite]

Looks like the address that is dereferenced is garbage. That's usually a memory corruption done earlier in the code somewhere.

[tobobo]

Thanks. Heap corruption detection didn’t come up with anything, but I’ll start removing code and try to diagnose the issue. How can someone tell in this situation that the address is bogus?

[ESP_Sprite]

You can look at the EXCVADDR in the stack dump. I'm going by heuristics from seeing lots of valid addresses during my years at Espressif and going like 'this ain't one', but you can check the memory map in the TRM if you want to be sure. (Generally, valid addresses tend to start with 0x40 or 0x3f.)

[tobobo]

It turned out to have something to do with how I was creating the http config struct. Now I have some HTTP "Out of buffer" errors but at least the memory corruption is fixed.

Before:


[Codebox]
esp_http_client_config_t config;
config.url = url;
config.cert_pem = TUNESHINE_OTA_CERT;
config.event_handler = eventHandler;
config.keep_alive_enable = true;
[/Codebox]

After:

[Codebox]
esp_http_client_config_t config = {
.url = url,
.cert_pem = TUNESHINE_OTA_CERT,
.event_handler = eventHandler,
.keep_alive_enable = true,
.buffer_size = 10240,
};
[/Codebox]