Dynamically resizing SPIFFS partition to "largest possible"

[RandomInternetGuy]

Hi! I'm a developer of an ESP32 ESP-IDF + Arduino project. Our project ships source or binaries (for user convenience) for a wide variety of configurations, including different Espressif parts, board configurations, and supported pinouts. The current issue is boards that differ only by flash memory sizes.

We're currently shipping multiple binaries and build configurations for different partition table configurations from various partition_table.csv files. This is inefficient and difficult to maintain. We'd like to be able to ship a single binary that can be dynamically updated on first boot or upon update to add or change partitions.

We propose to read the partition table at startup, stash the contents of SPIFFS, erase the sector containing the partition table, do the math to add crash and grow/change the filesystem tables to use all the storage, ]and then write the new partition table back to flash and restore it. The SPIFFS partition is always at the end, so growing it and adding a crashdump partition seems likely to work. We'd not change the partition tables if there were not all true. We're aware that this is a risky operation, but we're confident that we can make it safe by desk-checking the code path and regenerating the checksum, though we haven't yet found info on that.

Are there any best practices for safely adding or changing partitions? Is there a Partition Table Write API? Are there any other risks that we should be aware of?

[ESP_igrr]

Hi RandomInternetGuy,

The partition table is stored in a single flash sector without redundancy. I'm afraid that overwriting it at run time may lead to the device getting bricked, if the power gets disconnected at the right (or wrong) moment. If the partition table has been erased but not written and the device reboots, it won't be able to boot anymore.

It is still technically possible to do this, you just need to enable CONFIG_SPI_FLASH_DANGEROUS_WRITE_ALLOWED option, and use esp_flash_erase and esp_flash_write APIs. No specific API is provided for writing to the partition table as it is deemed to be a dangerous thing to do.

Another option would be to not add SPIFFS to the partition table, and instead register the partition at run time by calling esp_partition_register_external function. (It is intended to be used with additional external SPI flash chips, but it does work for the main flash chip as well.)

Note that you can't register the core dump partition dynamically this way... This is because the core dump partition has to be known earlier at startup process, before the application entry point runs. So I would recommend keeping the core dump partition as the last entry in the partition table, and allocate the remainder of the space to SPIFFS at run time using esp_partition_register_external.

[lbernstone]

The chip size must be set properly because it respects the partition boundaries. Thus you can't make a universal image, but:

Code: Select all

#include "esp_partition.h"
#include "esp_spi_flash.h"
#include "esp_flash.h"
#if CONFIG_IDF_TARGET_ESP32
#include "esp32/rom/spi_flash.h"
#elif CONFIG_IDF_TARGET_ESP32S2
#include "esp32s2/rom/spi_flash.h"
#elif CONFIG_IDF_TARGET_ESP32S3
#include "esp32s3/rom/spi_flash.h"
#elif CONFIG_IDF_TARGET_ESP32C3
#include "esp32c3/rom/spi_flash.h"
#endif

#include <FFat.h>

size_t last_flash_used() {
  esp_partition_iterator_t it;
  size_t endpt = 0;
  it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, NULL); 
  for (; it != NULL; it = esp_partition_next(it)) {
    const esp_partition_t *part = esp_partition_get(it);
    endpt = part->address >= endpt ? part->address + part->size : endpt;
  }
  esp_partition_iterator_release(it);
  return endpt;
}

bool register_partition() {
  size_t endpt = last_flash_used();
  if (endpt == g_rom_flashchip.chip_size) {
    log_i("No free space on flash");
    return false;
  }
  esp_err_t err;
  err = esp_partition_register_external(esp_flash_default_chip, endpt, g_rom_flashchip.chip_size - endpt, "ffat",
              ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_FAT, NULL);
  if (err) {
    log_e("Error registering partition");
    return false;
  }
  return true;
}

void setup() {
  Serial.begin(115200);
  if (register_partition()) Serial.println("Fat partition created");
  if (!FFat.begin(true)) {
    Serial.println("FFat Mount Failed");
    return;
  }
  Serial.printf("Free space: %10u\n", FFat.freeBytes());  
}

void loop() {}

and a partitions.csv to go with it

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
coredump, data, coredump,0x290000,0x10000,

[RandomInternetGuy]

Thank you, ESP_igrr, for confirming that I wasn't overlooking something in the doc; IDF is "helpfully" trying to protect me from myself. I was aware of the danger of rewriting that on first boot, of course. That partition table, like the boot sector and everything else, has to be written on a fresh system on that first boot anyway, right? One of the strengths of this product family - unlike some competing products - is that the products are never really bricked and recovery just requires a trip to a computer. Sure, there will be that One Guy that doesn't have a computer, takes it out of the box, immediately installs it on his island surrounded by moat or something and THEN has a brownout on first boot. We know that is going to happen, but as we're going into a mostly technical market, we considered that risk manageable.

You do, however, present and extremely helpful option in esp_partition_register_external. From the name, I wasn't at all sure that allowed combinations of the partition to fall outside the core chip itself and into external SPI or if it was external SPI at all, or if it was meant to be something like an SDI card on relatively slow SPI bus, or something in between like on the WROOM or other module, but off the ESP32-S3 proper or even a W25Q-whatever on the board from the vendor, so external to the SoC and external to the module, but internal to the board. [ Whew, the combinations! ]

I think that would let us reduce this axis to only two combinations to support: small and large. The small configuration is too small to have two system partitions, so we can't enable the dual partitions for OTA. We just have to special case the smallest chips. We then have an "and everything else" partitions.csv that we ship with APP partitions A and B to allow OTA. On first boot, we try to mount spiffs and fail. We can then use the partitions API to find the end of the B parition, query the chip for its actual flash size, shave off 64K for a recovery partition, and know the size remaining is the right size for SPIFFS. Then we create those two paritions and retry to mount the user filesystem before booting as usual.

If that combination is likely to technically work, I'm comfortable coding that and think it will solve our user's needs of having appropriately sized partitions and a reasonable matrix to develop and support. That person whose device failed that first write on first boot is still going to have a bad day, but at least we boot loop and have a chance to emit some telemetry on the network or I/O instead of going back to the reset vector on a blank chip.

Once we've registered the partition(s) in this way, will ESP-IDF/Arduino remember the size and location of these secondary partitions for subsequent mounts (i.e. is there a secondary partition table stashed somewhere that it writes that's consulted on normal startup) or is this a process we'll have to recompute and re-register on every boot before attempting the spiffs start and whatever has to be done to register the crash partition?

If my understanding of your suggestion is correct, I think this should work for us. Thank you so much!

[RandomInternetGuy]

Hi, and Thanks!

This conversation is likely to be out of order as I'm still earning posting privileges here and am awaiting moderation.

That code is very close to what I was thinking for the "large" (Two app partitions; OTA allowed) case. I was fixated on having the crash partition floating at the end, but your approach of putting it at the end of the fixed ones makes a lot of sense. I wasn't familiar with g_rom_flashchip, but spi_flash_get_chip_size or its replacement? " uint32_t size_flash_chip; esp_flash_get_size(NULL, &size_flash_chip);" and "ESP.getFlashChipSize())" - which is probably Arduino's respelling of the former hopefully read the device fuses to get the available flash size. (If someone solders down chips without setting the fuses, well, that's on them, right?)

In your example, we know that flash is taken through 0x30.0000 (192k) and can see that's 4K aligned, so it's an OK start size. Take whatever size we got from esp_flash_get_size() (let's say 2M) and subtract the 192K, assert the alignment - which should only jump the rails if esp_flash_get_size() isn't 4K aligned, which would be pretty shocking in itself) and can use that as the end, right? (I'm not going to risk embarrassment of showing the math here. Too easy to make typos."

Other than my second post (which you may not be able to see) pseudo-coding that same approach with the the crash partition at the very end, after the user filesystem, why wouldn't that work? Then you only have to ship two bins - one with no OTA and one with) that are built two two different parition tables, with the second using an approach llike I discussed. No more special casing 2 vs. 4 vs. 8 vs. 16MB. (Eek! I now see 32's are available...)

We don't quite get to universal, but we go from "too small for OTA(0? 1? I'm not sure, but let's say "<2" for this discussion + 2 + 4 + 8 + 16 + 32" to " too small" (I suppose that might theoretically be more than one image, but I think we can just artificially say "no") and "bit enough for two OTA app partitions + dump + we'll use whatever's left for spiffs or littlefs or fat or whatever" and that's a pretty big win.

It's not quite universal, but what limit remains from getting it down to really only two such builds/.bins?

Thank you for talking it through!

[lbernstone]

The binary will have a chip size written into it, and the register function will not allow you to go past that. I don't know how you are having people upload the file, but if they are using esptool on the command line, there is no way to change this without some surgery.
I am able to manually fix it like this (after writing the image with 4MB flash size):

Code: Select all

~/esp32/esptool/esptool.py --baud 961200 read_flash 0 0x150000 autopart.img
~/esp32/esptool/esptool.py --baud 961200 write_flash 0 autopart.img
~/esp32/esptool/esptool.py write_mem 0x1003 0x4F

The first octet of that byte at 0x1003 is the size (2M = 1, 4M = 2, 8M = 3, 16M = 4)
If you come up with a script to get the flash_info and then set that value then you can make it universal. Maybe there is a way to force esptool to set this (putting --flash_size in the write_flash does not override it).

As per Ivan, the coredump needs to be in your partition table. So, it can't be at the end of the disk.

[RandomInternetGuy]

The binary will have a chip size written into it,

So spi_flash_get_chip_size() doen't actually get the spi chip size from looking at programmed fuses or interrogating parts and such, but instead works by reading what we told it the size was at build time?

If that's a correct understanding, them this whole idea is dead in the water. My enthusiasm for not having to provide a target + binary per flash size is dashed if there's really not a way to read the usable flash size from the hardware but it instead relies upon us telling IT at build time. I'd really hoped the human installer didn't have to know the details of the device they're using, but this instead means that a user could download any of the 2M, 4M, 8M, 16M variations and it would probably "work" until their filesystem use hit the sector range of flash memory that's actually not there.

I appreciate the honesty. I can consider this idea dead and move on. The user will have to just know how much flash they have and we'll just have to provide build targets and .bins for all the interesting cases.

This explains why even major products like https://github.com/micropython/micropyt ... orts/esp32 and every combination of memory size + (memory size + OTA" gets its own partition table and its own resulting .bin.

Thank you, everyone.

[RandomInternetGuy]

Can we instead rely on [ESP.getFlashChipRealSize](https://github.com/esp8266/Arduino/pull/2210/files) to tell me the real flash size and not just parrot back what the build system told it the image was being built for?

Without being able to clearly determine the hardware size of the flash, this idea is indeed dead.

I know that link is for ESP8266 and not the others.

Perhaps the IDF natives are
https://docs.espressif.com/projects/esp ... P8uint32_t
or
https://docs.espressif.com/projects/esp ... P8uint32_t

... with the latter looking to be the preferred interface.

SO with that in hand, can we determine real flash size, put the crash partition after OTAA and OTAB, and then plop the userdata/SPIFFS/LIttleFS parition in the 'rest' of the space by creating the secodary external partition table and letting SPIFFS.open (or init() or whatever the constructor is called) all do their thing?

[lbernstone]

I made a bad assumption that the disk size would be a const...
Finished code at https://github.com/lbernstone/autopart

[RandomInternetGuy]

That's ... not very helpful. (TBC: the function's behavior - you're amazingly helpful!)

Lying to the build system "you're a 128MB chip! Take that!" so that I could only ever shrink it just seems unwise.

So I'm clear, the end result of all this is that you code on the ESP32 part really can't ever really size its own partition "appropriately", right? As a practical matter, the build system has to be told the size of the chip you're building for, it's probably only sane for it to include a static partition table, and it's up to the user somehow to pick the appropriate one for their device.

To go back to the problem statement: this project contains 33 build targets. (Not all of them make sense in the grid I'm about to describe as some are already combinations of environment + hardware + flash) We'd like to support ESP32S3, S2, C3, and probably soon H6 and maybe even 8266. Each of those boards contain multiple flash memory targets. Some combinations are kind of nonsensical on some hardware and some combinations probably don't exist for market or technical reasons, but we wanted to avoid having different builds for every combination and flash size seemed "obvious" since it's almost literally three different entries in a table in the image.

I appreciate your help. I'd rather be definitively be handed defeat in the prototyping stage than after shipping.