Best way to store large objects on flash?

[greggm]

Hi,

I am looking for the best way to store a larger image object (around 10k) on the internal flash. I tried to store it as binary using NVS, since NVS is the best described way in the documentation (including examples). But it seems i am hitting a 4k boundary here (along with some guru meditation which i would need to follow up on).

Can i extend this boundary?

What other options should i consider (partition table apis with or without mmap, virtual filesystem, ...?).

Would be great to have an overview on this. I did not find too much on memory restrictions of the different options.
Thanks,
Gregg

[ESP_igrr]

For NVS, the maximum value size is around 2k. This limit is related to the flash sector size (4k) and can not be increased.

You may take a look at storage/wear_levelling example in ESP-IDF. It demonstrates how to use a FAT filesystem on top of wear levelling library to store files. There is no officially documented way of uploading FAT FS to the ESP32 yet; see discussion in another topic related to this.

There is also a SPIFFS component and example prepared by loboris, based on SPIFFS port from Lua-RTOS project. It comes with a method to generate filesystem containing given files and to upload the filesystem to the ESP32 (using mkspiffs utility).

[greggm]

Thanks a lot for your fast reply and your pointers - from what you are saying it seems that also other flash access library (like the partition_table api) will also hit the same boundary as NVS. Right?

So it seems that SPIFFS might be the only well-documented way of handling with larger files on flash...

Gregg

[greggm]

I have tried out the wear-levelling example at
https://github.com/espressif/esp-idf/tr ... _levelling
but i get:

Code: Select all

E (321) vfs_fat_spiflash: Failed to find FATFS partition (type='data', subtype='fat', partition_label='storage'). Check the partition table.
E (322) example: Failed to mount FATFS (0x105)

Am i missing a step that creates a FATFS partition?

If so, how would i do it? I assume "somewhere" along the lines of
https://esp-idf.readthedocs.io/en/lates ... ables.html.

Do you have any details?


Gregg

[BuddyCasino]

If you only need a single immutable file, theres also the option of letting the build embed it.
Example for a file called "what_time.raw":

Code: Select all

/* embedded file */
extern uint8_t file_start[] asm("_binary_what_time_raw_start");
extern uint8_t file_end[] asm("_binary_what_time_raw_end");

component.mk:

Code: Select all

COMPONENT_EMBED_FILES := what_time.raw

[greggm]

That is an interesting option, but in my case the file is not immutable.
I will receive the data via wifi and want to write it down to flash.

What would be a suitable option here?

[ESP_igrr]

wear_levelling example in the latest IDF comes with the correct partition table, which is in the same directory as the example, and sdkconfig.defaults has an option to use it by default. If you run 'make clean' and 'make flash' in the wear_levelling example directory, are you still getting this error?

Regarding you other question, esp_partition APIs don't have the limitation on partition size (other than the available flash space). So creating a partition and writing the file there using esp_partiton_erase_range/esp_partition_write is an option. You can then use either esp_partition_read or esp_partition_mmap to read the data.

[loboris]

That is an interesting option, but in my case the file is not immutable.
I will receive the data via wifi and want to write it down to flash.

What would be a suitable option here?

You can check my curl example (viewtopic.php?f=18&t=1833), it creates fat fs in flash, then downloads some files, exactly what you want to do. Of course, you don't need to use curl, just use the part which deals with file system.

[greggm]

Thanks for the many replies - Will go down one way - either using the partition API or the file system solution.
Are there any reason to prefer one over the other?

WRT the error i was reporting before: I have tried it out without ESP_IDF using the Arduino variant on platformio. I am now setting up ESP-IDF to make the sample work.

Gregg

[ESP_igrr]

either using the partition API or the file system solution.
Are there any reason to prefer one over the other?

Filesystem (FAT FS + wear_levelling):
+ can use C standard library and POSIX APIs for working with files
+ easy to extend if you need to store more than one file
+ easy to update the file reliably
+ can use with encrypted flash (contents of the partition will be encrypted)
- high overhead, both in terms of size and run time (FAT filesystem needs at least 524kB of flash + ~10kB of RAM)
- currently no officially documented way to generate the filesystem and upload it

Filesystem (SPIFFS):
+ can use C standard library and POSIX APIs for working with files
+ easy to extend if you need to store more than one file
+ easy to update the file reliably
+ has a documented way of generating the filesystem and uploading it
- when using encrypted flash, SPIFFS partition can not be encrypted
- high overhead, although lower than FATFS + wear_levelling

Direct access to partitions:
+ lowest overhead — partition can be as small as the size of the file
+ can map the file directly into memory (very efficient if you need to send a large file over network)
- can not use C standard library APIs, lowest code portability
- difficult to extend if you need more than one file
- difficult to update the file reliably