Thoughts on a read only file system that maps to flash ...

[kolban]

I have been working on a project where I need to have large tables of read-only text. Now imagine a black box function that I have no control over that takes as input a pointer to a NULL terminated string. Let us call this function "process(char *data)". What I have been doing up till now is using SPIFFs to read the text data into malloc'd RAM using SPIFFs file I/O and then calling "process()" on the data. All has been working well ... however, I hadn't thought the design through in its entirety and now I see that I can't afford to copy the SPIFFs file data into RAM to process it. The 3rd party library requires ALL the data to be supplied as its parameter.

So for example, if my data file is 10K, I malloc 10K, copy the file into the malloc storage and then pass the 10K pointer to process(). All is well. However, what is my data file is 100K or 500K ... now I'm sunk because I don't have enough RAM to accommodate. What solutions come to mind?

The first is to abandon SPIFFs and just write the raw data into flash using esptool.py. I am then assuming that I have a technique to map the flash address space into processor address space (what is the right phrase for that?) ... now I have a readily available pointer to my read only data without first copying to RAM. My thinking is that I can now pass a pointer to this flash housed storage to "process()" and we never touched RAM at all. Perfect!!!

Assuming I'm on the right track with this, the next puzzle is that there isn't one data file ... but many. They don't change at runtime but they will often change during my development cycles. What I am thinking about is a directory of text files and a tool (to be written). The tool will process the text files and concatenate them together (adding a NULL terminator) and remember the offset into the resulting "blob" where each "file" starts. There will also be a table at the beginning that is comprised of the pair of "filename/offset". I can then create a library that will provide a function that takes as input a file name and returns a pointer to the start of the data.

To my way of thinking, this sounds and feels generically useful ... and that's usually when I put the brakes on and ask the obvious question "Has it been done already?". So ... this is the core question ... does anyone know of a library/tool that does this today that ideally is an open source project?

[ESP_igrr]

Sprite_TM's ESPHTTPD project has an "espfs" read-only filesystems. With some work, it can be integrated into the ESP-IDF VFS system.
https://github.com/Spritetm/libesphttpd ... ster/espfs

[kolban]

Thank you sir, that sub-module does indeed look interesting. Its exposed APIs seem to be:

EspFsFile *espFsOpen(char *fileName)
int espFsRead(EspFsFile *fh, char *buff, int len)
int espFsFlags(EspFsFile *fh)
void espFsClose(EspFsFile *fh)

What I think I need would be one more API that might look like:

int espFsAccess(EspFsFile *fh, char **buf, int *len)

On invocation, it would populate the "buf" pointer with the mapped memory location of start of the file and also populate the "len" pointer with the length of the data pointed to by buf (which would be the whole file length).

I'll have a study of Sprite_TM's code base and see if that is something I can achieve by myself. Any comments from yourself or Mr Sprite_TM more than welcome.

... Later ...
Mr Sprite_TM's code was fantastic. It uses spi_flash_read() to read in data. For my purposes, I changed this to use a memory map of the flash data and added an "espFsAccess()" function that gives me a memory map to the file content as a whole. I don't consider this to be any more than a technology test on my part and certainly not an enhancement of the existing code. For those that might find use in it, one can find my efforts here:

https://github.com/nkolban/esp32-snippe ... tems/espfs

[WiFive]

How about zip with no compression?

https://en.wikipedia.org/wiki/Zip_(file ... #Structure

[ESP_Sprite]

We actually have been discussing this internally. We're already thinking of using a specific flavour of CPIO-archives for our OTA updates; it's probably trivial to implement this as a VFS as well.

[WiFive]

We actually have been discussing this internally. We're already thinking of using a specific flavour of CPIO-archives for our OTA updates; it's probably trivial to implement this as a VFS as well.

But cpio and tar have no directory/index right so you have to follow headers in a linked list style which could mean more cache misses?

[ESP_Sprite]

Yes, this is true. For a filesystem, we could implement something like a hash bucket style of cache to look up filenames quickly, though. Also, espfs from esphttpd works kind-of similar and also is somewhat braindead when it comes to filesystems with a very large amount of files.

[grooves]

Hi,

thanks for the porting of nice espfs to esp-idf system,
I have added the heatshrink again,
need it for porting the amazing httpd project from Spritetm,

compression is not supported for the new espFsAccess method,

Regards,
grooves