ESP32: Partition Table Update through OTA

[vindi503]

Hi,

Currently I am facing issue in updating partition table through OTA.

As my new OTA file size has been exceeded more than 1MB so I want to create an intermediate OTA file which just updates partition
table to a higher size than 1MB and then do a actual OTA whose size is more than 1MB.

This requirement is because I cannot flash the unit(exists another place) through cable with updated partition table hence looking for
an option to do it via creating 'intermediate OTA' file.

I tried creating 'intermediate OTA' file with updating partitions_example.csv(attached) but after OTA while booting it is still old partition.

I (239) boot: Partition Table:
I (250) boot: ## Label Usage Type ST Offset Length
I (273) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (296) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (319) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (342) boot: 3 factory factory app 00 00 00010000 00110000
I (366) boot: 4 ota_0 OTA app 00 10 00120000 00110000
I (389) boot: 5 ota_1 OTA app 00 11 00230000 00110000
I (412) boot: 6 storage Unknown data 01 82 00340000 000b0000
I (436) boot: End of partition table


Your help will be greatly appreciated.

Regards,
Vinay

[jcsbanks]

There has been discussion on how to flash the factory app from OTA app. Maybe you could do similar to flash the partition partition. Might need to look at CONFIG_SPI_WRITING_DANGEROUS_REGIONS_* in sdkconfig too.

[vindi503]

Hi,

I went through that section(flash the factory app from OTA app), It seems it requires modification in the esp-idf bootloader code for
my case where I need updated partition table while booting after OTA.

As I am not familiar with the esp-idf bootloader code, would you tell me the filename and what's needs to be modified such that
I create intermediate OTA file where after OTA I see the updated partition table on boot up.

Your help will be appreciated.

Regards,
Vinay

[jcsbanks]

esp_https_ota and bootloader are the components I copied from esp-idf to my project's components directory. Then started editing.

[vindi503]

Hi,

Our OTA implementation is similar to esp_https_ota, The only thing current issue is once the OTA happens at the offset
ota_1(0x250000) on the next bootup I get the below error.

W (6727) ota: Configured OTA boot partition at offset 0x00250000, but running from offset 0x00130000
W (6727) ota: (This can happen if either the OTA boot data or preferred boot image become corrupted somehow.)

Could you tell me how to keep make esp_ota_set_boot_partition configuration reflect on the bootloader section.

I see bootloader_utility_load_boot_image fxn call in bootloader_start.c, Any changes need here?

[jcsbanks]

My changes were for updating factory and choosing which app to boot. I worked through the errors/barriers one at a time making changes and adding other features.

For your changes I would think you need a running app from which you can reprogram the partition table and bootloader, making sure you have a bootable app before you reset. If anything is interrupted you will have a brick in the field unless you can do something where you write everything to new locations that do not affect the ability to boot and then do a dangerous overwrite at the very end to keep the danger period as short as possible and only after checks. I guess this is why this sort of OTA change is not supported. Should not be impossible though, but will need fair bit of testing.

[luisonoff]

Hello, I just wanted to share my findings about this topic of updating partition table through OTA, for future users who fight this.

I faced this problem because we wanted to change partition table from FACTORY+OTA to OTA0+OTA1, plus we wanted to expand NVS partition size.
We got several devices already installed on clients, plus our devices' case is not easy to open, so we decided to do everything thru OTA.

I read many posts about this, but I just found the same answer: it is not recommended to change partition table from active app.
But I didn't know if it was possible or not at all.
We tried and tested and I can tell you partition table update thru OTA is doable.

We used SPI Flash driver, esp_partition and esp_ota from IDF.

These are the steps we followed in order to get it done:
1. Decide what partition table we wanted for the long term, in order to not change it again if possible.
2. Compile any app with this partition table, so we can find partition.bin in the build directory.
3. Attach this partition.bin to our app, adding this file to our project folder and using COMPONENT_EMBED_FILES in component.mk.
4. Modify our code in order to create a new app version that would be used for this update.
5. Device performs OTA firmware update of this new app.
6. This app checks after boot if factory partition exists, if it exists, then it is the old partition table.
7. If the old partition table is found, app downloads to RAM this same binary from server, and copies it to the new partition using its address (does not exist yet). This partition must not overlap with the actual running partition.
8. After copy bin to memory, app erases old partition table (address 0x8000), and copies new partition table in same place.
9. Then the location of new partitions NVS, otadata, etc is erased. Make sure they do not overlap with current partition.
10. Then we restart ESP, because new partition table is not loaded until restart.
11. After restart ESP tries to start each app partition, probably some of them will fail, but at the end it will find the correct one where we copied the bin file directly.
12. Then it starts from there, checks partition table, no factory partition exists, so partition table update is finished.
13. We do one more step for correctness, after detecting that the new partition table is used, we perform a new OTA now just using the library from IDF. This OTA works ok and we are done.
14. Last we modify our app code with the new partition table and (optional) we delete every code related to this partition table update.
15. We can continue developing with new partition table and every device is updated.

TIPS:
DO THIS AT YOUR OWN RISK
TEST EVERYTHING with developing units before trying the real ones.
Before writing to flash it is necesary to erase first.
After writing partition table it is necessary to restart ESP in order to load it and be able to detect partitions.
Before restart make sure you have a correct bin written to where the new partition table expects.
Be careful not to write or erase any memory address that overlaps your current partition.
In order for all this to work, you have to change menu config option, in Components/SpiFlashDriver select Allowed in "Writing to dangerous flash regions".

These are the functions we used, just for reference:

Code: Select all

error = spi_flash_read(address, buffer, size);
error = spi_flash_erase_range(address, sizeToErasemultipleof4096);
error = spi_flash_write(address, downloadedFile, downloadedFileSize);
esp_partition_t * partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "factory");
if (partition != NULL)
{
// partition factory found
}
esp_ota_set_boot_partition(partition); //we didn't use this one at all

[MaxVapor]

We just did something similar, but relied on the existing OTA mechanism to shift the application for us.

I posted about it here:
viewtopic.php?f=13&t=12004

[filzek]

Hello, I just wanted to share my findings about this topic of updating partition table through OTA, for future users who fight this.

I faced this problem because we wanted to change partition table from FACTORY+OTA to OTA0+OTA1, plus we wanted to expand NVS partition size.
We got several devices already installed on clients, plus our devices' case is not easy to open, so we decided to do everything thru OTA.

I read many posts about this, but I just found the same answer: it is not recommended to change partition table from active app.
But I didn't know if it was possible or not at all.
We tried and tested and I can tell you partition table update thru OTA is doable.

We used SPI Flash driver, esp_partition and esp_ota from IDF.

These are the steps we followed in order to get it done:
1. Decide what partition table we wanted for the long term, in order to not change it again if possible.
2. Compile any app with this partition table, so we can find partition.bin in the build directory.
3. Attach this partition.bin to our app, adding this file to our project folder and using COMPONENT_EMBED_FILES in component.mk.
4. Modify our code in order to create a new app version that would be used for this update.
5. Device performs OTA firmware update of this new app.
6. This app checks after boot if factory partition exists, if it exists, then it is the old partition table.
7. If the old partition table is found, app downloads to RAM this same binary from server, and copies it to the new partition using its address (does not exist yet). This partition must not overlap with the actual running partition.
8. After copy bin to memory, app erases old partition table (address 0x8000), and copies new partition table in same place.
9. Then the location of new partitions NVS, otadata, etc is erased. Make sure they do not overlap with current partition.
10. Then we restart ESP, because new partition table is not loaded until restart.
11. After restart ESP tries to start each app partition, probably some of them will fail, but at the end it will find the correct one where we copied the bin file directly.
12. Then it starts from there, checks partition table, no factory partition exists, so partition table update is finished.
13. We do one more step for correctness, after detecting that the new partition table is used, we perform a new OTA now just using the library from IDF. This OTA works ok and we are done.
14. Last we modify our app code with the new partition table and (optional) we delete every code related to this partition table update.
15. We can continue developing with new partition table and every device is updated.

TIPS:
DO THIS AT YOUR OWN RISK
TEST EVERYTHING with developing units before trying the real ones.
Before writing to flash it is necesary to erase first.
After writing partition table it is necessary to restart ESP in order to load it and be able to detect partitions.
Before restart make sure you have a correct bin written to where the new partition table expects.
Be careful not to write or erase any memory address that overlaps your current partition.
In order for all this to work, you have to change menu config option, in Components/SpiFlashDriver select Allowed in "Writing to dangerous flash regions".

These are the functions we used, just for reference:

Code: Select all

error = spi_flash_read(address, buffer, size);
error = spi_flash_erase_range(address, sizeToErasemultipleof4096);
error = spi_flash_write(address, downloadedFile, downloadedFileSize);
esp_partition_t * partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "factory");
if (partition != NULL)
{
// partition factory found
}
esp_ota_set_boot_partition(partition); //we didn't use this one at all

Hi, Seens that you can rewrite new partition table on the fly, can you send you entire working code to test?

Also, how to write the partition table and flash it would be useful.

thanks

[rafalino]

Hello,
Did anyone try it with IDF 4.3 ?
I followed the instructions and it fails booting after I have modified the Partition table and reset the ESP32.

Thanks in advance ,

Rafalino