OTA upgrade is aborting

[Ritu21]

Hi Team,

I have applied OTA upgrade in my project and it was working properly but now it has started giving problem which I am not able to understand.
Below is the error received while trying to update over OTA:

abort() was called at PC 0x40115128 on core 0

ELF file SHA256: 4b5b2176653ec245b44f3e5e5a1672bd23decd2a834fb9c0b585c22a74fb1ff2

Backtrace: 0x40088070:0x3ffcb1c0 0x400882bd:0x3ffcb1e0 0x40115128:0x3ffcb200 0x40115899:0x3ffcb240 0x40115f8d:0x3ffcb260 0x4008b6c2:0x3ffcb2b0 0x4008def5:0x3ffcb2f0

I have backtraced as below but still not getting any idea.

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40088070: invoke_abort at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/esp32/panic.c:715


rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x400882bd: abort at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/esp32/panic.c:715


rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40115128: pm_on_beacon_rx at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40115f8d: ppRxPkt at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x4008b6c2: ppTask at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x4008def5: vPortTaskWrapper at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/freertos/port.c:403

Can you please check and tell where is the problem and how to resolve it?

Waiting for your early response.

Thanks
Ritu

[Ritesh]

Hi Team,

I have applied OTA upgrade in my project and it was working properly but now it has started giving problem which I am not able to understand.
Below is the error received while trying to update over OTA:

abort() was called at PC 0x40115128 on core 0

ELF file SHA256: 4b5b2176653ec245b44f3e5e5a1672bd23decd2a834fb9c0b585c22a74fb1ff2

Backtrace: 0x40088070:0x3ffcb1c0 0x400882bd:0x3ffcb1e0 0x40115128:0x3ffcb200 0x40115899:0x3ffcb240 0x40115f8d:0x3ffcb260 0x4008b6c2:0x3ffcb2b0 0x4008def5:0x3ffcb2f0

I have backtraced as below but still not getting any idea.

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40088070: invoke_abort at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/esp32/panic.c:715


rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x400882bd: abort at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/esp32/panic.c:715


rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40115128: pm_on_beacon_rx at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x40115f8d: ppRxPkt at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x4008b6c2: ppTask at ??:?

rajan@B06 MINGW32 /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new
$ xtensa-esp32-elf-addr2line -pfia -e /d/Ritu/ESP32/ESP32-WROMM-32/esp-idf/examples/SmartAttendance_new/build/smartAttendanceNew.elf

0x4008def5: vPortTaskWrapper at D:/Ritu/ESP32/ESP32-WROMM-32/esp-idf/components/freertos/port.c:403

Can you please check and tell where is the problem and how to resolve it?

Waiting for your early response.

Thanks
Ritu

Hi,

As you have mentioned that it was working earlier and now you are facing issue so did you add any other code into component or application side?

which ESP32 IDF you are using? If older then did you just try with newer IDF?

Would you please share code snippet for same to view like is there any issue into code perspective?

[boarchuz]

An issue that I and others have run into is that the OTA flash erase can take a long time and hogs a lot of resources, causing all kinds of panics and aborts. I think my main problem, from memory, was that CPU0 was starved so clients were dropping from the AP. I wonder if you've recently increased the size of the firmware and/or the partition?
It may not be your particular problem but I'll leave a solution here anyway in case it's useful to anyone.

The easiest solution, actually, is to increase priority of the task responsible for the flash erase, but I was still having issues with that.
What I ended up doing was altering esp_ota_ops.c to do the erase in chunks, yielding for a moment in between. See here: https://github.com/espressif/esp-idf/bl ... ops.c#L157

Original:

Code: Select all

    if ((image_size == 0) || (image_size == OTA_SIZE_UNKNOWN)) {
        ret = esp_partition_erase_range(partition, 0, partition->size);
    } else {
        ret = esp_partition_erase_range(partition, 0, (image_size / SPI_FLASH_SEC_SIZE + 1) * SPI_FLASH_SEC_SIZE);
    }
    if (ret != ESP_OK) {
        return ret;
    }

My workaround:

Code: Select all

        size_t bytesToDelete = partition->size;
        if(image_size != 0 && image_size != OTA_SIZE_UNKNOWN)
        {
            bytesToDelete = (image_size / SPI_FLASH_SEC_SIZE + 1) * SPI_FLASH_SEC_SIZE;
        }
        size_t bytesDeleted = 0;
        while(bytesDeleted < bytesToDelete)
        {   //0x10000 seems stable enough without too much overhead, you can do larger or smaller chunks
            size_t thisBytesToDelete = (bytesDeleted <= (bytesToDelete - 0x10000) ? 0x10000 : bytesToDelete - bytesDeleted);
            ret = esp_partition_erase_range(partition, bytesDeleted, thisBytesToDelete);
            if(ret != ESP_OK){
            	return ret;
            }
            bytesDeleted+=thisBytesToDelete;
            vTaskDelay( 10 / portTICK_PERIOD_MS); //Yield
        }

[Ritu21]

Hi Ritesh,

Yes I have added extra application and the size has also increased but what difference does that make in OTA???

Current version of ESP-IDF is v4.0-dev-217-g5b2247df2.

Attached is the code of OTA I am using in my application.

Earlier I was downloading image from HTTPS but now I am using HTTP for the same due to memory issue.

Thanks
Ritu

[Ritesh]

Hi Ritesh,

Yes I have added extra application and the size has also increased but what difference does that make in OTA???

Current version of ESP-IDF is v4.0-dev-217-g5b2247df2.

Attached is the code of OTA I am using in my application.

Earlier I was downloading image from HTTPS but now I am using HTTP for the same due to memory issue.

Thanks
Ritu

Hi,

Would you please check your older working code with HTTP request as well? it is working or not?

[Ritu21]

I have only added spiffs and changed the partition table and for now commented that part and checked, yet getting the same error. Tried the solution given by @boarchuz, the frequency of getting error has reduced but not gone.

[Ritesh]

I have only added spiffs and changed the partition table and for now commented that part and checked, yet getting the same error. Tried the solution given by @boarchuz, the frequency of getting error has reduced but not gone.

Hi,

Just make sure that partition is not overriding some how into OTA and SPIFFS while adding that type of support.

[Ritu21]

Hi,

How do I check this?? Address on which esp is writing ota update seems to be correct. It gets stuck after writing new image to ota app.

Below is the current partiion table:
I (53) boot: ## Label Usage Type ST Offset Length
I (61) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (68) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (76) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (83) boot: 3 factory factory app 00 00 00010000 00100000
I (90) boot: 4 storage Unknown data 01 82 00110000 0007d000
I (98) boot: 5 ota_0 OTA app 00 10 00190000 00100000
I (105) boot: 6 ota_1 OTA app 00 11 00290000 00100000
I (113) boot: End of partition table


I (44451) native_ota_example: Starting OTA...
I (44821) native_ota_example: Running partition type 0 subtype 16 (offset 0x00190000)
I (44841) native_ota_example: Writing to partition subtype 17 at offset 0x290000
I (44851) native_ota_example: New firmware version: 11.0
I (44851) native_ota_example: Running firmware version: 10.0
I (49511) native_ota_example: esp_ota_begin succeeded

After writing image, this message is printed.

I (87831) native_ota_example: Connection closed,all data received
I (87831) native_ota_example: Total Write binary data length : 876880
I (88451) esp_image: segment 0: paddr=0x00290020 vaddr=0x3f400020 size=0x239b8 (145848) map
I (89361) esp_image: segment 1: paddr=0x002b39e0 vaddr=0x3ffb0000 size=0x03274 ( 12916)
I (90181) esp_image: segment 2: paddr=0x002b6c5c vaddr=0x40080000 size=0x00400 ( 1024)
I (92071) esp_image: segment 3: paddr=0x002b7064 vaddr=0x40080400 size=0x08fac ( 36780)
I (93231) esp_image: segment 4: paddr=0x002c0018 vaddr=0x400d0018 size=0x9e794 (649108) map
I (94891) esp_image: segment 5: paddr=0x0035e7b4 vaddr=0x400893ac size=0x07978 ( 31096)
I (97861) esp_image: segment 0: paddr=0x00290020 vaddr=0x3f400020 size=0x239b8 (145848) map
I (99601) esp_image: segment 1: paddr=0x002b39e0 vaddr=0x3ffb0000 size=0x03274 ( 12916)
I (100831) esp_image: segment 2: paddr=0x002b6c5c vaddr=0x40080000 size=0x00400 ( 1024)
I (101041) esp_image: segment 3: paddr=0x002b7064 vaddr=0x40080400 size=0x08fac ( 36780)
I (102671) esp_image: segment 4: paddr=0x002c0018 vaddr=0x400d0018 size=0x9e794 (649108) map
I (104011) esp_image: segment 5: paddr=0x0035e7b4 vaddr=0x400893ac size=0x07978 ( 31096)

As per my code, it should reboot after above message, sometimes it does but most of the time it get stuck here only. When hard rebooted, device starts with the new image.

Please suggest what is happening??

Thanks
Ritu.

[Ritesh]

Hi,

How do I check this?? Address on which esp is writing ota update seems to be correct. It gets stuck after writing new image to ota app.

Below is the current partiion table:
I (53) boot: ## Label Usage Type ST Offset Length
I (61) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (68) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (76) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (83) boot: 3 factory factory app 00 00 00010000 00100000
I (90) boot: 4 storage Unknown data 01 82 00110000 0007d000
I (98) boot: 5 ota_0 OTA app 00 10 00190000 00100000
I (105) boot: 6 ota_1 OTA app 00 11 00290000 00100000
I (113) boot: End of partition table


I (44451) native_ota_example: Starting OTA...
I (44821) native_ota_example: Running partition type 0 subtype 16 (offset 0x00190000)
I (44841) native_ota_example: Writing to partition subtype 17 at offset 0x290000
I (44851) native_ota_example: New firmware version: 11.0
I (44851) native_ota_example: Running firmware version: 10.0
I (49511) native_ota_example: esp_ota_begin succeeded

After writing image, this message is printed.

I (87831) native_ota_example: Connection closed,all data received
I (87831) native_ota_example: Total Write binary data length : 876880
I (88451) esp_image: segment 0: paddr=0x00290020 vaddr=0x3f400020 size=0x239b8 (145848) map
I (89361) esp_image: segment 1: paddr=0x002b39e0 vaddr=0x3ffb0000 size=0x03274 ( 12916)
I (90181) esp_image: segment 2: paddr=0x002b6c5c vaddr=0x40080000 size=0x00400 ( 1024)
I (92071) esp_image: segment 3: paddr=0x002b7064 vaddr=0x40080400 size=0x08fac ( 36780)
I (93231) esp_image: segment 4: paddr=0x002c0018 vaddr=0x400d0018 size=0x9e794 (649108) map
I (94891) esp_image: segment 5: paddr=0x0035e7b4 vaddr=0x400893ac size=0x07978 ( 31096)
I (97861) esp_image: segment 0: paddr=0x00290020 vaddr=0x3f400020 size=0x239b8 (145848) map
I (99601) esp_image: segment 1: paddr=0x002b39e0 vaddr=0x3ffb0000 size=0x03274 ( 12916)
I (100831) esp_image: segment 2: paddr=0x002b6c5c vaddr=0x40080000 size=0x00400 ( 1024)
I (101041) esp_image: segment 3: paddr=0x002b7064 vaddr=0x40080400 size=0x08fac ( 36780)
I (102671) esp_image: segment 4: paddr=0x002c0018 vaddr=0x400d0018 size=0x9e794 (649108) map
I (104011) esp_image: segment 5: paddr=0x0035e7b4 vaddr=0x400893ac size=0x07978 ( 31096)

As per my code, it should reboot after above message, sometimes it does but most of the time it get stuck here only. When hard rebooted, device starts with the new image.

Please suggest what is happening??

Thanks
Ritu.

Hi,

I just want to make sure that before ESP32 Restart Did you put any delay or not? Because we have faced this type of issue before and we resolved it to start timer for 5 seconds and restart ESP32 Device on that 5 seconds of timer expiration.

So, Please make sure for that and also is that code is running under any ISR or not?

If possible then share your whole sample code of OTA update.

[Ritu21]

I haven't put any delay before rebooting the device. Do I need to add delay??

Actual application is server dependent, we get the time from server fro image update, then start rtos timer like this:

imageDownloadTimerHandle = xTimerCreate("imageDownloadTimer",pdMS_TO_TICKS(difference_in_time*1000),pdFALSE,(void*)1, vTimerCallbackApplyTimeExpired);
if(imageDownloadTimerHandle == NULL){
printf("Upload Timer Not Created\n");
}
else{
printf("Upload timer Started\n");
xTimerStart(imageDownloadTimerHandle, 0);
}

This is a callback function in which ota update function is called which I had already send you :
void vTimerCallbackApplyTimeExpired(xTimerHandle pxTimer1)
{
printf("Image Download Timer Expired\n");
ota_ImageUpdate();
printf("Image update finished\n");
}