Using OTA with multi-partition application

[mbratch]

I have an application that's pretty straightforward, but uses two partitions: one for the factory app, and SPIFFS partition for web pages and javascript, etc, which is the embedded web app used to access and configure, etc. More specifically:

Code: Select all

# Name        Type       SubType      Offset      Size
nvs           data       nvs          0x9000      0x6000
phy_init      data       phy          0xf000      0x1000
factory       app        factory      0x10000     0x130000
www           data       spiffs       0x140000    0x20000

I want to add OTA capability, but I'm unsure as to how to handle the www partition. I need two partitions for each "instance" of the application, but the OTA examples seem to only use one partition per application instance (ota_1, ota_2, etc...).

So I reckon it needs maybe to look something like this, although I suppose this isn't really correct for what I'm trying to achieve (e.g., how do I do OTA for the www partitions?):

Code: Select all

# Name        Type       SubType      Offset      Size
nvs           data       nvs          0x9000      0x4000
otadata       data       ota          0xd000      0x2000
phy_init      data       phy          0xf000      0x1000
factory       app        factory      0x10000     0x130000
www           data       spiffs       0x140000    0x20000
ota_0         app        factory      0x160000    0x130000
www_0         data       spiffs       0x290000    0x20000
ota_1         app        factory      0x2B0000    0x130000
www_1         data       spiffs       0x3E0000    0x20000

How is OTA done when each instance of the app is multiple partitions? Seems like this would be a common scenario, so I'm sure I'm just missing something obvious in the docs or examples that I've overlooked. So maybe I just need to be pointed at the right thing.

[boarchuz]

If these files are so tightly coupled with the firmware, embedding them in the firmware might be more suitable than a separate SPIFFS partition.

Does the firmware function properly without the files? If not, I would suggest they must be embedded in the binary. Are they ever altered, except in the case of a firmware update? Do you gain anything by having these accessible from different firmwares?

If there are optional and/or user-uploaded files then these may be good candidates for SPIFFS (eg. images) or even NVS (eg. settings).

[leemoore1966]

@mbratch

Hi, did you find a solution to your problem ?
I have exactly the same requirement, I need to update my code AND the html/js files which are in a spiffs file system
I would be very interested if you were able to solve your issue, having great difficulty finding a solution
Thx
Lee

[mbratch]

@leemoore1966, I'm glad you ask. Yes, I found a solution.

As a little background, here is my situation, in contrast to the original response I received to my question:

• I have an application that does not require the embedded web pages (or "web application") to operate since it operates with a REST API. So in that sense, the web application is not tightly coupled to my application. However, it is functionally coupled, of course, and is always provided as a convenience to the user as a pre-built client that uses the REST API.
• The embedded web application isn't terribly large, but it's large enough that it would be very impractical to simply make it a big pile of string data in the application partition itself.

Given the constraints, here is the solution that I came up with. I have two OTA application partitions but only one SPIFFs partition, as is required. I configured my application to build a SPIFFs partition file which is a .bin product output of the build process. I inject this binary into the elf using the technique described in Embedding Binary Data. When my application boots up, it checks to see if the current SPIFFs partition matches the binary SPIFFs image embedded in the ELF file. This is done by a literal compare which is unsophisticated but effective and only takes a second. If it does not match, it copies the ELF SPIFF binary image to the SPIFFs partition. This way, the SPIFF partition will always match whatever application version is booted up. I use the `esp_partition_find_first`, `esp_partition_erase_range`, `esp_partition_read` and `esp_partition_write` functions to do this.

Here is the CMakeList.txt file at the main app level that I use:

Code: Select all

cmake_minimum_required(VERSION 3.16.0)
cmake_policy(SET CMP0076 NEW)

set(PROJECT_VER "1.0")

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include) 

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

project(MyProject)

# ..... OTHER STUFF IN MY CMakeList.txt .....

# Create the www SPIFFS partition binary and inject into elf

set(WWW_SIZE "0x30000")      # This should match the size of the SPIFFs partition
set(WWW_DIR ${CMAKE_CURRENT_SOURCE_DIR}/web_root)
set(SPIFFSGEN_DIR ${IDF_PATH}/components/spiffs)
set(WWW_FILE "www.bin")

file(GLOB_RECURSE WWW_SRC ${WWW_DIR}/*.*)

if(EXISTS ${WWW_DIR})
    add_custom_command(OUTPUT ${BUILD_DIR}/${WWW_FILE}
        COMMAND python ${SPIFFSGEN_DIR}/spiffsgen.py ${WWW_SIZE} ${WWW_DIR} ${BUILD_DIR}/${WWW_FILE} --page-size=256 --obj-name-len=64 --meta-len=4 --use-magic --use-magic-len
		DEPENDS ${WWW_SRC}
    )

    target_add_binary_data(${PROJECT_NAME}.elf ${BUILD_DIR}/${WWW_FILE} BINARY)
    add_custom_target(www_fs DEPENDS ${BUILD_DIR}/${WWW_FILE})
	add_dependencies(${PROJECT_NAME}.elf www_fs)
else()
    message(FATAL_ERROR "${WWW_DIR} not found")
endif()

This is my partition file:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
nvs,      data, nvs,     0xa000,  0x3000,
otadata,  data, ota,     0xd000,  0x2000,
phy_init, data, phy,     0xf000,  0x1000,
ota_0,    app,  ota_0,   0x10000, 0x1e0000,
ota_1,    app,  ota_1,   0x1f0000,0x1e0000,
www,      data, spiffs,  0x3D0000,0x30000

As you would expect, I need to have 0x30000 bytes of space available in both OTA partitions to contain the application code itself and the appropriate copy of the SPIFF partition data. This means I do have a redundant copy of the SPIFF image, but I have enough available memory, so it's not a problem for me and the convenience is worth it.

[leemoore1966]

Many thanks for your comprehensive response.

I was also wondering about the redundant copy of the SPIFFS in the program binary, luckily I am in a similar situation to yourself that, the convenience of this approach outweighs the redundant data
many thanks

Lee

[phatpaul]

If you want a read-only filesystem that is compiled into your firmware binary, I use EspFs (a.k.a. FrogFs) and it is fast and easy to use.

https://github.com/jkent/frogfs

[leemoore1966]

@mbratch

Many thanks for your pointers, I finally got this implemented.
Just a note for anyone reading, I made a silly mistake of using the API call of esp_partition_find_first() incorrectly
I simply called with

Code: Select all

const esp_partition_t *part = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, NULL);

However this matches the first data partition which is the nvs, so this really screwed me up for a while before I realised
I should have used the following with the label of the partition

Code: Select all

const esp_partition_t *part = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, "storage");

For completeness, here is the code that checks, then overwrites if necessary the data in the partition - hope this helps anyone attempting the same

Code: Select all

void fs_update() {
    // Check FS integrity
    esp_err_t status;
    
    const char *label = "storage";
    const esp_partition_t *part = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, label);

    // 2Mb (0x200000 / 2097152) to compare, generate a bool, and if fail erase, write
    // 1kb pages = 2048 iterations
    const int pagesize = 1024;
    const int pages    = 2048;
    const int partsize = (pagesize * pages);
    uint8_t buffer[pagesize];
    int page = 0;
    Bool match = 1;
    for(page=0; page<pages; page++) {
        size_t offset = page * pagesize;
        status = esp_partition_read(part, offset, buffer, pagesize);
        if (status != ESP_OK) {
            return;
        }
        // compare memory buffers
        uint8_t *pelf = storage_start + offset;
        int mc = memcmp(pelf, buffer, pagesize);
        if(mc) {
            match = 0;
            break;
        }
    }
    
    if (!match) {
        // erase
        status = esp_partition_erase_range(part, 0, partsize);

        // write
        status = esp_partition_write(part, 0, storage_start, partsize);
    }
}