Read/Write to NVS with DIRECT ADDRESSING

[Sunil Dullolli]

Hi All,

I have had enough information about writing to NVS using key/value pair and by using File Systems. These seem to make things much easier for some cases but for my application I would be better off writing to NVS with direct addressing with help of spi_flash_{read|write|erase} API's. Is it possible to use these API's to Read/write/Erase NVS part of the flash? Did any one go down this road?

Any help is appreciated.

[ESP_Sprite]

Why do you think you'd be better off by doing this manually?

[Sunil Dullolli]

Hi ESP_Sprite,

Well, the Master that will be sending around 4KB of data to ESP32 has historically maintained a format. The Master will have the same 4KB data stored in an EEPROM. The Read/Write instructions from Master to EEPROM are all address based and these are communicated through CAN interface. Now for the sake of writing to NVS of ESP32, I thought I would just logically map the EEPROM's addresses to my NVS partition addresses. This way I don't have to worry about having the my master to change its already existing format.

Initially it was great working around Key/Value pairs but having to update whole "value" not just the individual byte gives lot of overhead to our application. Is there any way to work around this?

Also, if I am using both direct addressing and Key/value pair to write/read to nvs, are there any chances of memory overlap? Does Direct addressing work with nvs? please share pros/cons is any.

Thank you for your suggestions.

[ESP_Sprite]

NVS isn't really meant to work with changing bits within a value, like you intend. One of the reasons for this is that flash doesn't really allow random writes: generally, if you write a region, you need to erase the page (4K) the region is in before you can re-write it. You will have this limitation, independent of if you use NVS or another technique, by the way...

Hmm, this is a somewhat hard issue. I can imagine a few options to get around that, but the most versatile one would more-or-less mean you'd have to write a specialized journaling file system. Do you know how often this 'eeprom' is written to, in general? That affects the answer.

[Sunil Dullolli]

On power up Master writes all the available data to EEPROM. I would say this is a discrete data that comes through CAN interface. Thereafter, when some configuration in the Master changes, the exact byte with its exact address location will be sent to update in the EEPROM. In worst case scenario we might be updating EEPROM 'n' times.

[ESP_Sprite]

Sure but my question is more: how often, over the lifetime of a device, is a byte in the 'EEPROM' changed? Because at a certain point, if you erase the 4K range every time a byte is written, you're going to run into those limitations. Flash just is not the same as EEPROM and you cannot treat them as exactly the same, without a fair amount of abstraction.

I'm asking because you have a few options here, but they depend on what you want to do. They are, more or less:
- Keep the current approach, writing the 4K keys directly to memory. This effectively means that every changed byte will chew up one flash erase in the NVS range (because the flash page size happens to be 4K). Will perform like crap, but given that you have a NVS partition that's big enough, it will work.
- Keep using NVS, but give every byte its own key. Yes, I know, that's pretty damn dirty and wasteful of space, but this way, you can write data pretty quickly (will only erase a page when really needed). Problem is that each NVS entry is 32 bytes, so the entire 'EEPROM' is going to use up 128K. Might work if you use a large enough NVS partition (256K or so) to do wear levelling.
- Write your own journalling log-based 'file'system. Essentially, keep a record of [byte index, byte value] bytes in flash, appending new entries as needed.
- Add an external EEPROM to the ESP32.