ESP32 Forum

Is there any log-structured file system that is supported by ESP32 (ESP-IDF > 4.0)? SPIFF is probably the closest, isn't it? If I understand correctly, FatFS and LittleFS are more generic file system, aren't they?
We are using WROVER-E with 16MB of flash - roughly half for the application (ota 1 and 2) and another half to be used as a data storage.
During development there are many printf and/or ESP_LOGx calls that we would like to redirect to a text file, store it in the flash, make available to a host application via TCP/IP sockets, https or otherwise and be analyzed by some service application. I think that one file should be enough for this as it can be parsed and if needed split later, after being extracted from the ESP board. In order to store this file in ESP32 flash, it appears that a log-structured file system would be the best fit as appending and sequential reading are the only two required operations. After all, if needed we can split the data partition into two parts with two different file systems, can't we?
So, just to repeat the question, are you aware of any log-structured file system that has been ported to ESP32, and if not, which file system you recommend for the described purpose?
Thank you.

Possibly not the answer you want, but the NVS subsystem effectively works as a log-structured 'file' system. Although personally, I'd roll my own solution using the low-level partition SPI flash api... it looks like your use case is simple enough for it.

Thank you very much, I'll look into this - both NVS and the low level SPI APIs.
You say that my use case is a simple one, and I think it is. When it comes to plain system logging, I also think that this is the usual scenario which would maybe justify having a log-structured file system in IDF, would it?

ndjurov wrote: ↑

Thu Jan 20, 2022 4:23 pm

When it comes to plain system logging, I also think that this is the usual scenario which would maybe justify having a log-structured file system in IDF, would it?

You're not wrong there. I think people prefer sending everything directly over WiFi to the server (or keeping it in a small memory buffer in case WiFi is not there) but a log file system might be a good alternative. I'll float the idea within the software team. Meanwhile, if you decide to roll something yourself, we're always open to pull requests

ESP-Sprite, thank you very much.

Would you please help me understand something about ESP32 flash.
My understanding is that in order to change data in NOR flash, all flash 'units' first must be set to FFs as it is not possible to change 0 to 1, only 1 to 0. What I don't understand is what are the 'units' - can this operation can be done on a single byte or a word, or it is only block based? In general - what is the smallest addressable unit in the flash?
Thera are two reasons I would like to know this - the first one of course being curiosity, and the second one is to be able to decide what is the best way to organize my data files in flash. Considering that all files need to be deleted at some point and that deletion may take considerable amount of time (if that involves switching all bytes to FF - again - on byte or block basis), I'm wondering is it "better" to have several smaller files or one or two big ones. I guess that having smaller files will introduce fragmentation, but I'm not sure how big an issue it is. On the other hand, it may take long time to erase a long file, does it? Another option is to have several small data partitions with one file per partition? Is this better from user point of view - are file operations any faster this way?
Of course, better understanding of NOR flash operation will help making these and similar decisions.

Thanks again.

ndjurov wrote: ↑

Fri Jan 21, 2022 7:25 pm

Would you please help me understand something about ESP32 flash.
My understanding is that in order to change data in NOR flash, all flash 'units' first must be set to FFs as it is not possible to change 0 to 1, only 1 to 0. What I don't understand is what are the 'units' - can this operation can be done on a single byte or a word, or it is only block based? In general - what is the smallest addressable unit in the flash?

That is the 'erase block size', and you can generally find it in the datasheet of the flash chip. For any NOR chips that are used as the main flash chip for ESP32 chips, the erase block size will be 4Kbytes.

I don't have any comments on what is best to do from an user PoV wrt speed etc - that really depends on the filesystem as each of them will abstract away the NOR flash behaviour in various ways.

(Also, if you ever go down the rabbit hole of talking to the raw flash yourself, keep in mind flash encryption makes the behaviour of NOR flash even weirder - erasing it will set the *encrypted* flash cells to 0xff, but that means the *decrypted* data there, as read by the ESP32, is random. With flash encryption, you can't assume that 0xFF = erased; you need different methods like CRCs to differ between unused and used flash space.)

Hi,
I posted this question in github.com/joltwallet/esp_littlefs, but I don't know how active it is, so I'll ask the same thing here.
The file system in use is littleFS.
I'm logging ESP32 debug output to a temporary log file and every time ESP32 starts, I would like to concatenate this temporary file to a "permanent" log file that can be later extracted by some host computer (no wifi connection, only BLE - sometimes). New temporary log file would be then opened.
I was hoping that doing the concatenation instead of coping and appending to a big log file would minimize flash wear as the location of the new temporary file would always be different. Does littleFS allow this and would it really help with the flash wear? Another option is to write directly to a "big" log, but in case this one ever gets corrupt, everything is lost. How big is "big"? My board has WROVER with 16MB of flash and half of this is dedicated for debug log data.
I would also have an indexing file that keeps position of each temporary log file inside the big log.
Thanks.

I was having some issues using littleFS, so I tried to use spifFS to see if I can get different results.
What I'm doing is probably a common workflow used to save debug output to a file.
After initializing SpifFS (as littleFS was), vprintf is redirected to a simple function that places all debug output into a file (occasionally calling fsync() that didn't work with littleFS, but fflush did). Writing into the file is also protected with a semaphore by waiting for up to 10 ticks and if semaphore is not acquired within that time the current string is not written into the file.
That's all.

What I got was a stack overflow:

***ERROR*** A stack overflow in task BTC_TASK has been detected.

Backtrace:0x40090dfb:0x3ffe09c0 0x40091455:0x3ffe09e0 0x40094c75:0x3ffe0a00 0x400932fd:0x3ffe0a80 0x4009154c:0x3ffe0aa0 0x400914fe:0x3ffe0454 |<-CORRUPTED
0x40090dfb: panic_abort at C:/ESP-IDF/components/esp_system/panic.c:368

0x40091455: esp_system_abort at C:/ESP-IDF/components/esp_system/system_api.c:112

0x40094c75: vApplicationStackOverflowHook at C:/ESP-IDF/components/freertos/port/xtensa/port.c:490

0x400932fd: vTaskSwitchContext at C:/ESP-IDF/components/freertos/tasks.c:3290

0x4009154c: _frxt_dispatch at C:/ESP-IDF/components/freertos/port/xtensa/portasm.S:432

BTW, the application is configured to use BLE only.

After increasing the bludroid task stack size from 3072 (default) to 4096, everything worked as expected.

Do you have any idea why was this happening with spifFS but not with littleFS?

Thank you.