Thoughts on a read only file system that maps to flash ...
Posted: Mon Dec 12, 2016 5:46 am
I have been working on a project where I need to have large tables of read-only text. Now imagine a black box function that I have no control over that takes as input a pointer to a NULL terminated string. Let us call this function "process(char *data)". What I have been doing up till now is using SPIFFs to read the text data into malloc'd RAM using SPIFFs file I/O and then calling "process()" on the data. All has been working well ... however, I hadn't thought the design through in its entirety and now I see that I can't afford to copy the SPIFFs file data into RAM to process it. The 3rd party library requires ALL the data to be supplied as its parameter.
So for example, if my data file is 10K, I malloc 10K, copy the file into the malloc storage and then pass the 10K pointer to process(). All is well. However, what is my data file is 100K or 500K ... now I'm sunk because I don't have enough RAM to accommodate. What solutions come to mind?
The first is to abandon SPIFFs and just write the raw data into flash using esptool.py. I am then assuming that I have a technique to map the flash address space into processor address space (what is the right phrase for that?) ... now I have a readily available pointer to my read only data without first copying to RAM. My thinking is that I can now pass a pointer to this flash housed storage to "process()" and we never touched RAM at all. Perfect!!!
Assuming I'm on the right track with this, the next puzzle is that there isn't one data file ... but many. They don't change at runtime but they will often change during my development cycles. What I am thinking about is a directory of text files and a tool (to be written). The tool will process the text files and concatenate them together (adding a NULL terminator) and remember the offset into the resulting "blob" where each "file" starts. There will also be a table at the beginning that is comprised of the pair of "filename/offset". I can then create a library that will provide a function that takes as input a file name and returns a pointer to the start of the data.
To my way of thinking, this sounds and feels generically useful ... and that's usually when I put the brakes on and ask the obvious question "Has it been done already?". So ... this is the core question ... does anyone know of a library/tool that does this today that ideally is an open source project?
So for example, if my data file is 10K, I malloc 10K, copy the file into the malloc storage and then pass the 10K pointer to process(). All is well. However, what is my data file is 100K or 500K ... now I'm sunk because I don't have enough RAM to accommodate. What solutions come to mind?
The first is to abandon SPIFFs and just write the raw data into flash using esptool.py. I am then assuming that I have a technique to map the flash address space into processor address space (what is the right phrase for that?) ... now I have a readily available pointer to my read only data without first copying to RAM. My thinking is that I can now pass a pointer to this flash housed storage to "process()" and we never touched RAM at all. Perfect!!!
Assuming I'm on the right track with this, the next puzzle is that there isn't one data file ... but many. They don't change at runtime but they will often change during my development cycles. What I am thinking about is a directory of text files and a tool (to be written). The tool will process the text files and concatenate them together (adding a NULL terminator) and remember the offset into the resulting "blob" where each "file" starts. There will also be a table at the beginning that is comprised of the pair of "filename/offset". I can then create a library that will provide a function that takes as input a file name and returns a pointer to the start of the data.
To my way of thinking, this sounds and feels generically useful ... and that's usually when I put the brakes on and ask the obvious question "Has it been done already?". So ... this is the core question ... does anyone know of a library/tool that does this today that ideally is an open source project?