How to update a value in a partition?

[rocotocloc]

Hello,

I am migrating one project from PIC to ESP32 and I have the following question regarding data storage.

At the moment I use an external memory in the PIC to store some statistical data. The memory is divided in 366 chunks, one per day of the year, where I save what the equipment consumes every day.

So let's say position 0x00 is January 1st. When the equipment needs to save data there it just first reads the current value, adds the new value and save the updated value to the same position 0x00. It just goes like this:
- New value to save is 125
- Reads what is already in 0x00
- Ads 125 to that
- Save the updated value again to 0x00

I am now trying to do this in ESP32 so I defined a data partition like the following (storage)

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
nvs,        data, nvs,      0x9000,  0x6000,
phy_init,   data, phy,      0xf000,  0x1000,
factory,    app,  factory,  0x10000, 1M,
storage,    data, ,             , 0x40000,

The problem is, as I understand from this other reply (https://esp32.com/viewtopic.php?t=16229#p61746) that esp_partition_write() makes bits go from 1 to 0 but not viceversa, the only way to go back to 1 is erasing the corresponding chunk through esp_partition_erase_range()

So, how do I update a particular value here by using esp_partition_write() if only the first will really work (when all bits are set to 1)? According to this I should erase first but this seems not to be a good practice since I am gonna wear the memory too fast.

I know in this paticular situation, where it's very simple, I could use NVS system but what if it were not only 366 positions but much larger data map? Should I still use NVS and increase its partition size if needed?

Thank you.

[nopnop2002]

How about connecting an SPI-Flash like W25Q64 and reading/writing in sectors?

W25Q64 has 2048 sectors and W25Q128 has 4096 sectors and MX25L256 has 8192 sectors.

In other words, January 1st is sector 0, January 2nd is sector 1, and December 31st is sector 364.

[ESP_Sprite]

In other words, January 1st is sector 0, January 2nd is sector 1, and December 31st is sector 364.

You would still wear out flash because you'd need to erase each sector every time you increase its counter.

One way around this would be to use the bit count in each sector. To read the number, you'd count the number of 0 bits in the sector. To increment a sector, you'd read the data in there, find the first bit that is 1 and change it into a 0, then write the changed data back. That allows you to count up to (4K*8=)32767 without having to erase anything.

[rocotocloc]

@nopnop2002, I could do what you suggest with W25 chips but in the ESP32 too, right? I mean they seem to have similar write/erase cycle limits.

@ESP_Sprite, this is very clever indeed and very useful for this simple case where there are only 366 values but this sacrifices huge amount of memory space in a real scenario where I need to handle not only 366 values but many more.

Thanks for the replies, let me ask you something else by comparing straight flash access like this and NVS.

Let's say I have to handle several thousands of different count variables that need to be updated over time in the way we were talking about. Each of these variables is 32 bits so I had 1024 variables per sector (4096 bytes per sector).

As we already discussed, If I follow this straight memory process, one particular sector always contain the same 1024 variables. And in case I want to update any of those 1024 variables, since they belong to the same sector, I would have to the following, correct me If am wrong:

- Locate the sector where the current variable is stored by using some kind of logic particular to my project
- Read the entire sector to a RAM buffer
- Now modify the 32 bits that belong to the variable I want to update
- Erase the whole sector
- Write back the buffer to flash

Ok so what if I use NVS in terms of memory wear out?

Now I change the partition type from this:

Code: Select all

storage,    data, ,             , 0x40000,

To this

Code: Select all

storage,    nvs, ,             , 0x40000,

And I use the NVS functions to get/update each of those 1024 values, each key/value pair corresponds to one count variable. Would this be better in terms of memory wear out for this particular case due to the implicit wear levelling mechanism of the NVS system? Is there any drawback here like maximum amount of key/value pairs I can store in NVS?

Thank you.

[MicroController]

Each of these variables is 32 bits so I had 128 variables per sector (4096 bytes per sector).

Actually, that'd be 1024 32-bit values per 4096-byte sector.

Would [NVS] be better in terms of memory wear out for this particular case due to the implicit wear levelling mechanism of the NVS system?

Yes.
You can also check the NVS documentation to see how it does its thing: https://docs.espressif.com/projects/esp ... #internals

The 'trick' is always to store some form of metadata to indicate which data in the partition is 'current' (or unused), and to make it so that the update of this metadata requires bits to only be changed from 1 to 0.

[MicroController]

By the way, how often do you need to update the statistics values?
Can you buffer them in RAM for some time to reduce the number of flash writes needed?

[rocotocloc]

I got confused there, of course it’s 1024 variables, thanks.

How often do I need to update the variables? Well, as long as I could be able to store them in flash if electricity goes off, I could handle them in RAM, but none of them shold be lost, they are indeed important statistical data the user needs to generate sales reports.

Regarding NVS, I guess one drawback is that it uses several registers to do its magic so less space for real storing, right?

[MicroController]

Ok, no data loss acceptable.
But are we talking about some 2 updates each day or more like 2 per second?

Yes, NVS has some overhead. And wear-levelling in general needs some over-provisioning.
365x4 bytes is only 1460 bytes, i.e. not even one full 4k sector. 8-16 sectors of NVS should get you pretty far, and that's only 32-64k of flash, or 1-2% of a 4MB flash. Shouldn't be a problem I guess.

[rocotocloc]

Ok thanks, I have the numbers now so I will choose the proper option according to final requirements.

[RandomInternetGuy]

Commodity filesystems like LittleFS and SPIFFS are good at wear-leveling. Hand them a partition that's large enough that they're not always on the hunt for fresh sectors and just write a CSV or a file per sample or some other natural case where you can reconstruct missed uploads and retire out old data and you should be fine.

Espressif has published the life cycle math. It's not IMPOSSIBLE to wear out the internal flash if you're writing thousands of samples at 100Hz to the flash around the clock or something crazy but for a reasonable amount of data captures, splattered across a reasonable number sectors that it can append and write to (maybe you just take the risk of holding 60 seconds worth of data in RAM and do them all in one write...) and you have enough sectors so it's not constantly reusing them, the real world life was into many, many hundreds of years.

Also, don't be afraid to have tiered storage. Maybe you write primarily to network and only write to flash when the connection is down. Consider multiple write schemes that combine reliabilty and redundancy and define bounds of acceptable losses. Maybe, you don't NEED to know the air pressure of a gymnasium at 1000Hz to local networking. and if you do and the primary write system fails and falls back to 100Hz to local disk, maybe that's OK. (All Examples totally made up.....)

The data is on the spec sheets. Generally don't do silly things in the flash writes, be aware of the issue, and you'll be fine.

...and if not, use a device with USB-C host most and attach a big-ole spinning disk or SSD if you can't count on writing directly to the network that's storing it onto a striped RAID configuration that you just don't have in a $3USD part.