SPIFFS fast enough for a one second data logger?

[rin67630]

SPIFFS or SD-Card?

I am planning a data logger with an ESP32 WROOM.
It should store one byte every second in a file. At the end of the day, a new file will be created. The file will be retrieved by FTP, then erased.

The available SPIFF memory should be enough for the job, but I am woried about the time performance.
I intend to store data in a 60 byte RAM array first and update the SPIFF file every minute.
But will I be able to finish the SPIFF update task within ~ 800mS?

If not, will a SD-Card on a separate shield be faster?

Thank you for your advice.

[Archibald]

I've just tried writing 60 bytes to flash and timed it using the micros() function (before and after). It took 212μs (including some time taken by the 'micros' function calls, but excluding opening of file). I wonder if it may sometimes take longer, such as if the filing system sometimes needs to rewrite a sector.

[ESP_igrr]

Both SPIFFS and SD card will have higher write latency as the disk fills up. For SPIFFS, see this issue: https://github.com/pellepl/spiffs/issues/202. For SD cards, from memory we've seen delays as long as 0.5 seconds for erase/write operations on some cards, after writing a few megabytes of data. So make sure you have a sufficiently large buffer.

[ESP_Angus]

To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)

[rin67630]

To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

Ouch, that is absolutely low-level fiddling!
Out of my reach, and out of reach of 99% of the people here, isn't it?

[ESP_igrr]

That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.

[rin67630]

That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.

But you want to be able to read the results as a regular file e.g. to send it via FTP. So that's not so straight forward as you pretend.

[ESP_igrr]

Ah i see, I missed the FTP requirement. In that case, yes, it's going to be a bit more work to keep track of the beginning and the end of the ring buffer.

[akbarhash]

Hi ESP_Angus and ESP_igrr.
Sorry to resurrect this thread, if possible could a simple example be shared of the correct way to initialize and do serial write to the internal flash device using the library. A simple example like writing a small string and reading it back from flash memory would do. SPIFFS and NVS have a lot of overhead and is not suitable for the project I am working on. The spi flash api reference is a bit mixed up and not clear. Thanks.

That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.

To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)