ESP32 Forum

I need to save data to send periodically to the cloud.
There will be many write operations and a single read operation during each task cycle that sends the data to the cloud.
Where do you recommend storing this data: in the RTC memory, in the heap, in flash memory, or where?

Hi @yaghmr,
this depends on data update period in your application, the amount of data to be stored, etc... assuming you would like to store the data in ESP chip once per second (or slower), and send them to the cloud eventually, I recommend using standard file(s) in SPI Flash (FatFS partition). Please, take a look into https://github.com/espressif/esp-idf/tr ... _levelling

In case you need really fast processing, you probably have to keep the data in RAM (==heap). RTC memory is designed for sleep mode stubs and is not suitable for your case.

If you need further help, please let me know.

pacucha42 wrote: ↑

Mon Oct 23, 2023 7:25 am

this depends on data update period in your application, the amount of data to be stored, etc... assuming you would like to store the data in ESP chip once per second (or slower), and send them to the cloud eventually, I recommend using standard file(s) in SPI Flash (FatFS partition). Please, take a look into https://github.com/espressif/esp-idf/tr ... _levelling

In case you need really fast processing, you probably have to keep the data in RAM (==heap). RTC memory is designed for sleep mode stubs and is not suitable for your case.

Thank you for the response; the frequency of writing depends on the device, it isn't constant, initially, it's very high, then it decreases, so I need the write operation to be fast so that it doesn't interfere too much with normal operation. In summary, I need quick writing (more than very close writings), while reading has no time constraints.
What advantage is there in saving it in SPI flash compared to NVS as a blob?

NVS is sort of a file-system (key-value pairs) and is designed for keeping the system/application configuration values. It is not recommended for storing production data. Beside of that, NVS is runs over SPI Flash partition, alike the FatFS in the example provided.

Still not sure what "very high" frequency means - could you describe in milliseconds, please? Like the shortest possible period, or approximate average

pacucha42 wrote: ↑

Mon Oct 23, 2023 9:57 am

Still not sure what "very high" frequency means - could you describe in milliseconds, please? Like the shortest possible period, or approximate average

Writing operations are linked to network initialization, handler registration, network traffic, task operations, so the frequency is not easily determinable.

Ok. Please try the solution us suggested and let me know, if it doesn't work - I would try to find something more suitable. I don't expect troubles due SPI throughput, though, it's pretty fast compared to network operations

pacucha42 wrote: ↑

Mon Oct 23, 2023 2:00 pm

Ok. Please try the solution us suggested and let me know, if it doesn't work - I would try to find something more suitable. I don't expect troubles due SPI throughput, though, it's pretty fast compared to network operations

Certainly, I will first try using the SPI flash, and if I notice any significant slowdowns, I will switch to the heap. I will keep you updated. Thanks again.