Store static file in flash

[MickJagger]

I 'd like to pass some static file to the spi memory when flashing the chip, so as to send it after client's request.
How can I add something to the project, that is not part of a code, to be able to open it with fopen?

[kolban]

I have been using the SPIFFS file systems that ports easily to run on an ESP32. SPIFFSs provides a flat file system structure that apparently can do wear leveling on SPI devices. It has its own file API which is quite POSIX compliant. I was able to map that easily to the Virtual File System (VFS) APIs so that I could then open, close, read and write SPIFFs based files using the newlib POSIX apis.

[WiFive]

I have been using the SPIFFS file systems that ports easily to run on an ESP32. SPIFFSs provides a flat file system structure that apparently can do wear leveling on SPI devices. It has its own file API which is quite POSIX compliant. I was able to map that easily to the Virtual File System (VFS) APIs so that I could then open, close, read and write SPIFFs based files using the newlib POSIX apis.

Have you developed a way to write to the spiffs using esptool.py?

[kolban]

Yes sir. The following tool from igrr can be used to build a "SPIFFS" image from a local file system:

https://github.com/igrr/mkspiffs

Once built, this can then be "flashed" using esptool.py to the flash space that we are using for SPIFFs storage. For example, I am using 1MByte of storage for my SPIFFs area located at offset 3*1024*1024 to 4*1024*1024-1. I haven't gotten around to making up a partition table definition to "reserve" that space for SPIFFs and am "just using it" but as long as my app bin doesn't tread on it, I've been ok so far.

[mjmorrison]

Hi @kolban, did you need to make any modifications to the SPIFFS code to get it to work with the ESP32?

I am looking to buffer some data when lacking wifi connectivity, and one of the devs here recommended I use the partition tool to make a partition and store my data there. Any sense on which would be more work?

Thanks much,
M

[kolban]

If you need to store data, you have two options ... RAM and flash. Obviously you lose the data when/if you lose power. Writing to flash means that the data will be present when you restart. The sizes of data differ, the amount of RAM is only 512K (but yet my tests have never shown me more than about 180K available) ... while many (most/all/some?) of the available ESP32 modules have 4MBytes of flash. Let us assume you want to store in flash.

You have a number of options for writing to flash. You can use Non Volatile Storage which defaults to about 24K (max) with a blob max size of about 4K. Alternatively you can use the flash write APIs and simply write to somewhere in flash storage you know isn't used. However, my "favorite" is to use a file system on top of the flash storage. I believe there are two popular ones ... FAT and SPIFFS. So far I have been using SPIFFS in the code base found in this project:

https://github.com/whitecatboard/Lua-RT ... nts/spiffs

And have had no issues at all. In addition, this great library also includes support for "wear leveling" which may or may not be an issue depending on how often you are writing.

Now ... the partition table provides an allocation "map" to where things can be found in flash. So rather than hard-coding flash addresses into your applications and code, my understanding is that we can create the partition table that provides a "table of contents" of how the flash space is divided up. I believe ESP-IDF uses this to find out where to put important things such as the NVS table and where to load other goodies. Ideally, we would set up a partition table entry that would then be used "automatically" by the SPIFFs libraries to indicate where in the flash address space the SPIFFS data will be found.

If you want to do it "really properly"

1) Get SPIFFS working from the project above and demonstrate to yourself that you can use SPIFFS.
2) Map to a Virtual File System so you can use POSIX file I/O. For example: https://github.com/nkolban/esp32-snippe ... vfs/spiffs
3) Study the Partition table APIs and configure SPIFFS to find an use a "SPIFFS" tagged partition.

[mjmorrison]

Ok Wow this is really comprehensive! Thank you for your time.

I want to store around 1-2MB of data, so flash is definitely the way to go. I already have a project going on the esp8266, using spiffs to buffer some sensor reads. Re-using some of my old spiffs code would be super nice if possible!

I had passed by the Lua RTOS ESP32 implementation you linked, but I thought it would be full of lua code so I didn't delve too deeply. It actually looks like there is barely any Lua in there... (0.1%) Next week I'm going to dig into this lua RTOS code you linked, and that spiffs snippet and try to get them talking properly!

I think I will push for getting it running quick-and-dirty for the time being, but I'm interested in helping getting this working the "really proper" way once this is working.

I don't mean to highjack this thread... but I'm having a little trouble seeing how lua fits into the freeRTOS / esp-idf ecosystem. Maybe it will become clear after some time with the Lua RTOS implementation. Thanks again!

[kolban]

The way I see it, at the core is the ESP32 (hardware) and above that are the Espressif provided software drivers which include LWIP for the TCP/IP stack, FreeRTOS as the task management stack, newlib for much of the "posix like libraries" and more. The thinking is primarily that applications will be written in C and run on top of these "loose collection of tested libraries". And we could stop there and say "perfect ... good job everyone". However, as one might say, this is the 21st century and despite the success and test of time, C coding is low level stuff. Many folks like to work with higher level computing paradigms such as OO, modules, functional programming, closures, loose typing/strong typing and all the other rich considerations used in choosing an design and implementation language for a project. In the "old days" when one had "kilobytes" of RAM and programs were stored in ROM/EEPROM, there weren't such choices. Even with the ESP8266, it only having about 80K of RAM put "constraints" on the shape of the world. However, with the arrival of the ESP32, we have the luxury of up to 512K of RAM.

And that opens doors....

Now we have opportunities to provide alternative (no one is arguing better ... just different) programming platforms. There are a variety of platforms ALREADY available for the ESP32 including a few JavaScripts, Python and LUA. I'm sure before all is said and done there will be many more instances of these platforms as well as additional platforms. Some folks are already chatting about .NET/C# on ESP32.

I fully expect (opinion) that these platforms will be implemented on top of ESP-IDF (or at least portions of it) and, as such, they will likely perform "poorer" than those written in native C (which already are likely to perform poorer than those hand written in assembler). However, it is all relative. The "cost" of building an App is not just how fast can it possibly run ... but how easy was it to design, how long did it take to write, how easy is it to maintain and fix bugs (etc etc). Where a higher level paradigm moves us away from "optimal" performance and efficiency, it will likely gain in terms of ease of implementation.

So ... to your question ... where does "LUA fit?". LUA is a programming language with an environment that allows you to write applications on the ESP32. It is an "option" for you (just like JavaScript, Python and others). If I had a project in front of me, I would ask "What platform should I use to implement that project?" ... considerations would include available skills, time to deliver and any obvious reasons that it could NOT be done in one platform or another.

[mjmorrison]

Hi,

I'm working with the spiffs library from Lua RTOS mentioned above, as well as @kolban's spiffs_vfs driver.

In trying to register the VFS, function is objecting to receiving a spiffs_t pointer instead of spiffs. I've tried several different ways of accessing the "spiffs" type instead of the "spiffs_t" type, but no luck.

I tried a typedef and casting as (spiffs *) Any advice?

It results in the following error

Code: Select all

/path/to/main/./main.cpp:221: undefined reference to `spiffs_registerVFS(char*, spiffs_t*)'

[kolban]

The sample function that uses VFS to map to SPIFFS uses "spiffs" data type ... see:

https://github.com/nkolban/esp32-snippe ... iffs_vfs.h

The spiffs data type is defined here:

https://github.com/whitecatboard/Lua-RT ... ffs.h#L291

Where is "spiff_t" coming from?