ESP32 Forum

Hello,

i'm struggling with my ESP32-C6-WROOM-N8 module to store adc measurements over some days.

I want to store 4 adc measurements every minute in for example a .txt or .csv file.
With this stored data i create a diagram on my http server read out by javascript to show it to the user.

Therefore i'm not sure which storage medium is the best for this application.
NVS seems not to be the right choice because it is a too large amount of data.
And i'm not sure if the 8 MB of flash would be the right choice with SPIFFS, because of the NOR flash lifetime limitation.

What would be the professional way here?

How much data loss is acceptable in case of loss of power?

You're right in that NVS probably is not ideal here.

If you store data as a csv, you'd use up, say, 5 bytes per record, or 20 bytes per minute. If a filesystem like spiffs doesn't do any superfluous erases (Microcontrollers question comes in here) you'd need 291 days to write to the entire 8MiB partition; in other words every sector on flash in that partition saw one erase/write cycle. ESP32 flash can handle 100.000 cycles at minimum, meaning you should be good until the year 81749. Obviously, if you write more data and add in the inefficiencies of SPIFFS, you'll wear out the flash a bit earlier, but that's about the size of magnitude of lifetime I'd expect.

MicroController wrote: ↑

Mon Nov 27, 2023 7:17 pm

How much data loss is acceptable in case of loss of power?

No data should be loss in case of loss of power.

ESP_Sprite wrote: ↑

Tue Nov 28, 2023 2:12 am

If you store data as a csv, you'd use up, say, 5 bytes per record, or 20 bytes per minute. If a filesystem like spiffs doesn't do any superfluous erases (Microcontrollers question comes in here) you'd need 291 days to write to the entire 8MiB partition; in other words every sector on flash in that partition saw one erase/write cycle. ESP32 flash can handle 100.000 cycles at minimum, meaning you should be good until the year 81749. Obviously, if you write more data and add in the inefficiencies of SPIFFS, you'll wear out the flash a bit earlier, but that's about the size of magnitude of lifetime I'd expect.

Thank's for the detailled answer!
Because there is a timestamp and the data in the following format "hh:mm,xx.xx,xxx.xx,xxx.xx,xx.xx\n" with 33 bytes, it would be no problem to use 2 or 4 MiB of the ESP32-C6 flash memory over some houndred years. Even if i consider the inefficiencies of the SPIFFS filesystem.

Marius28 wrote: ↑

Thu Nov 30, 2023 2:02 pm

No data should be loss in case of loss of power.

Ok, so buffering any data in RAM is out.

Because there is a timestamp and the data in the following format "hh:mm,xx.xx,xxx.xx,xxx.xx,xx.xx\n" with 33 bytes, it would be no problem to use 2 or 4 MiB of the ESP32-C6 flash memory over some houndred years. Even if i consider the inefficiencies of the SPIFFS filesystem.

Probably not. Though it should be pretty unnecessary to use ASCII/CSV as format for internal storage. From the looks of it, you want to store 4 values (ADC samples?) and one timestamp per record (per minute?). An easy fit into 5 uint16_t's, i.e. 10 bytes. Taking into account that 4 samples represent exactly one minute you don't even need to store the timestamp for any but the first record.
You said no data loss is acceptable; does that mean you need to write to flash 4 times per minute? Or is one write per minute/record sufficient (potentially losing up to three samples upon power loss)?

Personally, I wouldn't use any file system for this. I'd just sequentially write my data to a 'raw' partition, erasing one block of old data each time the partition is full. A persistent cyclic buffer.

MicroController wrote: ↑

Thu Nov 30, 2023 2:56 pm

Though it should be pretty unnecessary to use ASCII/CSV as format for internal storage. From the looks of it, you want to store 4 values (ADC samples?) and one timestamp per record (per minute?). An easy fit into 5 uint16_t's, i.e. 10 bytes. Taking into account that 4 samples represent exactly one minute you don't even need to store the timestamp for any but the first record.
You said no data loss is acceptable; does that mean you need to write to flash 4 times per minute? Or is one write per minute/record sufficient (potentially losing up to three samples upon power loss)?

Personally, I wouldn't use any file system for this. I'd just sequentially write my data to a 'raw' partition, erasing one block of old data each time the partition is full. A persistent cyclic buffer.

The thought to use ASCII/CSV as format was to serve this files later via webserver for download. The problem with only one timestemp at the beginning is at power loss. So i don't know how long the power loss was.

I write the 4 ADC samples every minute so one write per minute.

Currently i'm struggling with the file_server example and my webserver. I have the file_server example running on my ESP32-C6 but i need to match these uri's:
192.168.4.1
192.168.4.1/
192.168.4.1/sometext.csv
and not
192.168.4.1/sometext/sometext.csv
but /* for the uri in the example matches all behind the IP address...

Marius28 wrote: ↑

Thu Nov 30, 2023 5:58 pm

The thought to use ASCII/CSV as format was to serve this files later via webserver for download.
...
Currently i'm struggling with the file_server example and my webserver.

I got it.
Generating CSV from the stored binary data 'on the fly' only when responding to an HTTP request shouldn't be hard (may want to use chunked transfers); and this avoids any dependency on filenames or mount points the file server may impose on you...

MicroController wrote: ↑

Thu Nov 30, 2023 6:59 pm

Generating CSV from the stored binary data 'on the fly' only when responding to an HTTP request shouldn't be hard (may want to use chunked transfers); and this avoids any dependency on filenames or mount points the file server may impose on you...

What would be the advantage to store the data internaly in binary format versus in ascii format, except that i don't need any filesystem?

With the filesystem i can create day seperated files and delete the oldest one if the partition is full. So i can show the data day wise to the chart on my website.

Marius28 wrote: ↑

Fri Dec 01, 2023 10:58 am

What would be the advantage to store the data internaly in binary format versus in ascii format, except that i don't need any filesystem?

1) less memory required (and with it faster/less reads/writes, less time spent in a state vulnerable to power loss)
2) known, fixed size of every record, easy searching/skipping/jumping to records
3) possibility to easily do computations on the stored data (e.g. "min/max/average value over the last hour")

With the filesystem i can create day seperated files and delete the oldest one if the partition is full. So i can show the data day wise to the chart on my website.

Yes, you can definitely do that. Functionally, you can do the same with a bare cyclic buffer.
My thought is that a file system is a lot of overhead for this specific use case where you don't need random writes, don't need multiple files &c. The most important 'overhead' to me in this case is the requirement for a file system to manage the metadata (file size, allocated/free blocks), which implies multiple write operations every time you append even a single byte to a file; and with this, increased risk of corruption/loss of all data in case of an unexpected loss of power or crash/reset.

With that said, I do realize that just plugging in a file system may be more comfortable/convenient and quicker, especially if one has not used the IDF's partition API before or dealt with some of the peculiarities of flash memory; and that, while implementing the 'raw partition persistent cyclic buffer' approach is a good opportunity to learn some new stuff and create a rock-solid data logging solution in the process, this may not be at the top of the priority list of a project

Edit:
A plus when using a 'raw' partition is also that you can directly mmap the data into the CPU's address space, so that all read accesses can be done like memory reads (no flash/partition API needed for reading) and you won't have to deal with disabled cache during reads.

ESP_Sprite wrote: ↑

Tue Nov 28, 2023 2:12 am

You're right in that NVS probably is not ideal here.

If you store data as a csv, you'd use up, say, 5 bytes per record, or 20 bytes per minute. If a filesystem like spiffs doesn't do any superfluous erases (Microcontrollers question comes in here) you'd need 291 days to write to the entire 8MiB partition; in other words every sector on flash in that partition saw one erase/write cycle. ESP32 flash can handle 100.000 cycles at minimum, meaning you should be good until the year 81749. Obviously, if you write more data and add in the inefficiencies of SPIFFS, you'll wear out the flash a bit earlier, but that's about the size of magnitude of lifetime I'd expect.

Does SPIFFS automatically do wear leveling or is it necessary to initialize the wear leveling for SPIFFS filesystem?