Change Partition Subtype during runtime

[MaxVapor]

Apologies if this has been asked elsewhere, I tried to use the search function but did not seem to yield any results.

I have an application which is currently using the standard 1 Factory Partition / 2 OTA layout. I would like to push a new OTA update out to users which changes the Partition Subtype of the Factory Partition when the app is running off one of the OTA partitions.

I was not able to locate any existing function in the esp-idf which would allow this, but presumably it could be changed if I knew the location in the bootloader?

Any advice?

[WiFive]

To do this you would have to erase the existing partition table and if you fail to write a valid replacement for some reason like power loss the device will be bricked. And you also have to enable SPI_FLASH_DANGEROUS_WRITE

[MaxVapor]

Is there a reason I would need to erase the existing partition table?

Assuming SPI_FLASH_DANGEROUS_WRITE is enabled, could I not use esp_flash_write or something similar to write a single byte in the flash to change the subtype if I knew the correct offset? (and perhaps a new CRC).

[WiFive]

You can only erase a whole sector, you can't change a single byte without doing that. Technically you can change 1's to 0's but in this case the value is already 0 and the new hash/checksum will almost certainly require setting 0's to 1's.

[MaxVapor]

We managed to successfuly update our partition table on deployed devices.
The end goal was to remove the default factory partition to allow us to increase the size of the OTA partitions as our application is nearing the 1MB limit.

This was facilitated by our application and supporting infrastructure already having an OTA update mechanism in place.

The hardware as shipped uses the default Factory / 2 OTA partition table, in the latest firmware we do the following.

- Enable SPI_FLASH_DANGEROUS_WRITE
- Check for the existence of "factory" partition with subtype of 0x00.
- If factory partition exists, check the current running partition.
- If running from factory partition or ota_0, apply OTA updates until the running partition is ota_1 @ 0x210000
- When running from original ota_1 (1MB) @ 0x210000, erase and write new partition table with precomputed checksum and reboot.
- Partition table is now ota_0 (1.5MB) @ 0x10000 and original ota_1 (1MB) @ 0x210000. Factory partition is gone.
- Request OTA update, running application will now be ota_0 @ 0x10000
- Erase and Write final partition table, reboot.
- Application is now running from ota_0 (1.5MB) @ 0x10000 , with ota_1 (1.5MB) @ 0x190000

[filzek]

We managed to successfuly update our partition table on deployed devices.
The end goal was to remove the default factory partition to allow us to increase the size of the OTA partitions as our application is nearing the 1MB limit.

This was facilitated by our application and supporting infrastructure already having an OTA update mechanism in place.

The hardware as shipped uses the default Factory / 2 OTA partition table, in the latest firmware we do the following.

- Enable SPI_FLASH_DANGEROUS_WRITE
- Check for the existence of "factory" partition with subtype of 0x00.
- If factory partition exists, check the current running partition.
- If running from factory partition or ota_0, apply OTA updates until the running partition is ota_1 @ 0x210000
- When running from original ota_1 (1MB) @ 0x210000, erase and write new partition table with precomputed checksum and reboot.
- Partition table is now ota_0 (1.5MB) @ 0x10000 and original ota_1 (1MB) @ 0x210000. Factory partition is gone.
- Request OTA update, running application will now be ota_0 @ 0x10000
- Erase and Write final partition table, reboot.
- Application is now running from ota_0 (1.5MB) @ 0x10000 , with ota_1 (1.5MB) @ 0x190000

@MaxVapor could you provide your working code as reference for Parition Table change on the fly as an example to test?

[MaxVapor]

"repartition.c"

Code: Select all

#include <stdio.h>
#include "freertos/FreeRTOS.h"
#include "esp_ota_ops.h"
#include "esp_err.h"
#include "nvs_flash.h"
#include "nvs.h"
#include <esp_log.h>

#include "modes.h"

extern int current_mode;

static const char no_factory[] = {
    0xAA, 0x50, 0x01, 0x02, 0x00, 0x90, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x6E, 0x76, 0x73, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x01, 0x00, 0x00, 0xD0, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x6F, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x01, 0x01, 0x00, 0xF0, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x70, 0x68, 0x79, 0x5F, 0x69, 0x6E, 0x69, 0x74, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x00, 0x10, 0x00, 0x00, 0x01, 0x00, 0x00, 0x70, 0x17, 0x00, 0x6F, 0x74, 0x61, 0x5F, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x00, 0x11, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x10, 0x00, 0x6F, 0x74, 0x61, 0x5F, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xEB, 0xEB, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xB6, 0x35, 0x8C, 0xF0, 0xCA, 0x76, 0x39, 0xCA, 0xAB, 0x6F, 0x58, 0x44, 0xF6, 0x19, 0xD6, 0x7A,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
};

static const char final_table[] = {
    0xAA, 0x50, 0x01, 0x02, 0x00, 0x90, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x6E, 0x76, 0x73, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x01, 0x00, 0x00, 0xD0, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x6F, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x01, 0x01, 0x00, 0xF0, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x70, 0x68, 0x79, 0x5F, 0x69, 0x6E, 0x69, 0x74, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x00, 0x10, 0x00, 0x00, 0x01, 0x00, 0x00, 0x70, 0x17, 0x00, 0x6F, 0x74, 0x61, 0x5F, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xAA, 0x50, 0x00, 0x11, 0x00, 0x00, 0x19, 0x00, 0x00, 0x70, 0x17, 0x00, 0x6F, 0x74, 0x61, 0x5F, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0xEB, 0xEB, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xB8, 0x39, 0x22, 0x9C, 0xAD, 0x07, 0xCD, 0x06, 0xD9, 0x24, 0x54, 0x4A, 0x9E, 0xE8, 0x38, 0x07,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
    0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
};

void write_partition_table(const char *new_table){
    current_mode = MODE_REPARTITION;
    esp_err_t ret = spi_flash_erase_range(CONFIG_PARTITION_TABLE_OFFSET, 0x2000);
    if(ret == ESP_OK){
        spi_flash_write(CONFIG_PARTITION_TABLE_OFFSET, new_table, 0x2000);
        nvs_flash_erase();
        nvs_flash_init();
        esp_restart();
    }
}

// This function is called when a factory partition exists
// We need our active partition to be OTA1 so that we can modify the lower end
// of the partition table.
void alter_default_partitions() {
    const esp_partition_t *boot_partition = esp_ota_get_running_partition();
    if(boot_partition->subtype == 0){ // This should not happen in the wild
        ESP_LOGW("repartition", "Running from Factory partition");
        //write_partition_table(final_table);
    }
    else if(boot_partition->subtype == 16){
        ESP_LOGI("repartition", "Running from OTA0, wait until next update");
    }
    else if(boot_partition->subtype == 17){
        ESP_LOGW("repartition", "Running from OTA1, modify partition table");
        write_partition_table(no_factory);
    }
}

// This function is called if the factory partition is missing
// This means we are at least partially upgraded, so we need to check
// the size of ota_1 to see if the upgrade is completed.
void check_upgraded_partitions() {
    const esp_partition_t *ota_0 = esp_partition_find_first(ESP_PARTITION_TYPE_APP, 16, "ota_0");
    const esp_partition_t *ota_1 = esp_partition_find_first(ESP_PARTITION_TYPE_APP, 17, "ota_1");
    if(ota_0->size == 1536000 && ota_1->size == 1536000){
        ESP_LOGI("repartition", "Fully upgraded partition table");
    } 
    else if(ota_0->size == 1536000 && ota_1->size == 1048576){
        const esp_partition_t *boot_partition = esp_ota_get_running_partition();
        ESP_LOGW("repartition", "Partially upgraded table");
        if(boot_partition->subtype == 16){
            ESP_LOGW("repartition", "Running from OTA0, write final table");
            write_partition_table(final_table);
        }
        else if(boot_partition->subtype == 17){
            ESP_LOGI("repartition", "Running from OTA1, wait until next update");
        }
    }
}

void repartition_device() {
    const esp_partition_t *factory_partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, 00, "factory");
    const esp_partition_t *ota_0 = esp_partition_find_first(ESP_PARTITION_TYPE_APP, 16, "ota_0");
    const esp_partition_t *ota_1 = esp_partition_find_first(ESP_PARTITION_TYPE_APP, 17, "ota_1");
    if(factory_partition != NULL){
        ESP_LOGI("repartition", "Default partition scheme detected");
        alter_default_partitions();
    }
    if(factory_partition == NULL && ota_0 != NULL && ota_1 != NULL){
        ESP_LOGI("repartition", "Modified partition scheme detected");
        check_upgraded_partitions();
    }
}

Call repartition_device() in app_main after hardware is initialized.
extern var current_mode can be ignored, it just keeps some background tasks from launching when in the interim mode (writing to partition table).

[ryanf55]

I've implemented a similar version of this, but doesn't require you to wait for two OTA cycles for it to work.

I have it almost complete, but need to copy the current running partition's data to a different partition.
I know I need to use esp_partition_read() and esp_partition_write(), but could use some help on how to copy all the data over and get it to boot. For some reason after copying and erasing the OTA data partition to make it boot factory, it can't boot the factory image. esp_ota_write() doesn't let me write data to the factory partition.

Note: By default, the partition sizes for ota0, ota1, and factory are all the same.

Code: Select all

    esp_partition_subtype_t FACTORY = ESP_PARTITION_SUBTYPE_DATA_OTA;
    esp_partition_subtype_t OTA0 = ESP_PARTITION_SUBTYPE_APP_OTA_0;
    esp_partition_subtype_t OTA1 = ESP_PARTITION_SUBTYPE_APP_OTA_1;

    const esp_partition_t *boot_partition = esp_ota_get_running_partition();

	ESP_LOGI(TAG, "On OTA1 now.");
        const esp_partition_t *fp = esp_partition_find_first(ESP_PARTITION_TYPE_APP, FACTORY, "factory");
        if (fp==NULL){
            ESP_LOGE(TAG,"No factory partition found to erase.");
        }
        ESP_ERROR_CHECK(esp_partition_erase_range(fp, 0, fp->size));

        ESP_LOGI(TAG, "Copying ota1 (current part) over");
        const uint16_t s = 8192;
        void * buffer = malloc(s);
        for (int i = 0 ; i < fp->size ; i += s) {
            ESP_LOGI(TAG, "Copying i=%i", i);
            esp_partition_read(boot_partition, 0, buffer, s);
            esp_partition_write(fp, 0, buffer, s);
        }
        free(buffer);
        ESP_LOGE(TAG, "TODO boot to factory");
        // ESP_ERROR_CHECK(esp_ota_set_boot_partition(fp)); //this can't validate
        const esp_partition_t *find_partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_OTA, NULL);
        if (find_partition != NULL) {
            ESP_ERROR_CHECK(esp_partition_erase_range(find_partition, 0, find_partition->size));
            esp_restart();
        }
      

Here's the output after the above code runs.

Code: Select all

E (155) esp_image: invalid segment length 0xffffffff
E (156) boot: Factory app partition is not bootable
E (156) esp_image: image at 0x110000 has invalid magic byte
W (162) esp_image: image at 0x110000 has invalid SPI mode 255
W (168) esp_image: image at 0x110000 has invalid SPI size 15
E (174) boot: OTA app partition slot 0 is not bootable

Happy to share all the code with it working

[ftylitak]

Hello all,

thank you for the valuable input and the code share. It was very useful and saved me from a lot of manual work.

In the hope of providing some extra help, here is a blogpost trying to explain further our usecase of repartitioning over the air.
* https://insigh.io/blog/over-the-air-swi ... cropython/

and the source code of the updater:
* https://github.com/insighio/esp-idf-repartitioner

[jr.hernandezSio]

Hi,

It is amazng what you have manged. Thanks for sharing. I am planing to repartition my deployed devices in order to support file operations into a new SPIFFS partition.

App partitions OTA_0 and OTA_1 shoulf be reduced in size in order to fit the new SPIFFS partition at the end.
Current size of program running is less than 1000K, so I expect this is doable.

CURRENT SCHEME
-------------------------------------------------------------------------
Name, Type, SubType, Offset, Size, Flags
reserved, data, 0xfe, 0x9000, 16k
otadata, data, ota, 0xd000, 8k
phy_init, data, phy, 0xf000, 4k
ota_0, app, ota_0, 0x10000, 1920k
ota_1, app, ota_1, , 1920k
coredump, data, coredump, , 64K
nvs, data, nvs, , 128K

NEW SCHEME
-------------------------------------------------------------------------
Name, Type, SubType, Offset, Size, Flags
reserved, data, 0xfe, 0x9000, 16k
otadata, data, ota, 0xd000, 8k
phy_init, data, phy, 0xf000, 4k
ota_0, app, ota_0, 0x10000, 1780k
ota_1, app, ota_1, , 1780k
coredump, data, coredump, , 64K
nvs, data, nvs, , 128K
storage, data, spiffs, , 256K

I am trying to understand the repartitionerhttps://github.com/insighio/esp-idf-rep ... /main/main
and not sure hot to get the

Code: Select all

final_table[]

array. It is 352 long, but my partitions.bin file is 4KB.
Is it possible that

Code: Select all

final_table 

is just the first part of partitions.bin where meaningful data lies?

Is it possible to resize all partitions when running from OTA_0?

Any help will be welcome.

Regards, Ramon.