Fundamental question about nvs library

[Zingemneire]

Hi,

I have a fundamental question about the operation of the nvs library. I understand that it adds new key/value pairs at the end of the reserved flash area for storing data. If a value needs to be updated it will set the status of the existing key/value pair to "erased" and put the new value at the end again.

What I am wondering about is: what will it do when it reaches the end of the available space?

As an example: say that I have two key/value pairs, one that I write once and one that I update every couple of minutes. If I run the application long enough it will eventually run out of space. The first "write once" key/value pair will still be in the original location while the regularly updated value will eventually end up at the end of the allocated flash space. The previously updated key/value pairs are still in flash but they all have the "erased" status bits.
End result: there is plenty of free space ( pages ) that just need to be erased again in order to be reused for writing new and/or updated key/value pairs to.

Is there a built in mechanism in the nvs library to deal with erasing pages so that there is always enough flash memory to write to or is it something you need to watch out for yourself and deal with it?

I have not found anything yet that explains whether or not that sort of things is in the nvs library so I thought I would pose the question here, in the mean time I will look further into the documentation to see if I can find it.

Any help much appreciated.

[Zingemneire]

Ok, I have found something: the nvs library can put a page into the "erasing" state wherein it moves remaining key-value pairs to another page to empty it and finally erase the page to make it available again. What I have not been able to find yet is when and why the nvs library will do that.

[WiFive]

https://github.com/espressif/esp-idf/bl ... r.cpp#L127

[Zingemneire]

Hi WiFive,

Many thanks, that clarifies a lot. there are just a few questions that remain to really make sure I understand this correctly. Simple YES/NO answers should be more than enough.

1) Given that you have provided a direct link to the software I assume there is no description of it anywhere?
2) Assuming the above and having looked at the code you referred to I assume that all the necessary functionality is present and will keep working OK as long as there are pages available with enough "erased" key-value entries in them to make moving the ones not yet erased possible. The not erased ones are moved to a free page while the "old" one is then erased and becomes the next free page or simply a free page. Correct ?

[ESP_Angus]

Hi Zingemneire,

There's a description of the on-flash NVS implementation here:
http://esp-idf.readthedocs.io/en/latest ... flash.html

However I don't think it specifically describes the behaviour once a page is full. Your explanation is essentially correct, although all erasing happens at the page level not the key/value level. To simplify: there's always at least one empty/erased page, and when one page is full (including some stale entries), the remaining valid entries from the full page are copied to the empty page, then the full page is erased and becomes the next erased page.

The only way the NVS can become "out of space" is if enough key/value pairs are added to take up all of the available space (ie all pages except the reserved empty page are full, with no "stale" entries that can be reclaimed). If you reach this condition and the NVS API returns an out of space error, deleting a key/value pair will be enough to free up space and allow you to continue.

[Zingemneire]

Hi ESP_Angus,

Thank you very much for the confirmation, from the beginning I assumed there was a provision to handle recycling pages but now I am 100% sure .

Our application won't ever run out of space as I am planning to allocate a 128 kByte partition and we are likely to need only a few kilobyte at most for our parameter storage. If it ever reaches the 128 kByte limit there are bound to be many 4096 byte pages, with a lot of "erased" i.e. "updated" key-value pairs in them, to always make freeing a page to store more stuff possible.

[ESP_igrr]

If you are certain that the application will not ever use the whole 128k partition, it is worth reducing partition size. Since NVS builds a cache of items in RAM, that would reduce heap usage and reduce startup time.

[Zingemneire]

Thanks, that is very interesting to know.

Our application will be battery powered so anything that will reduce power consumption such as shortening the start up time will come in handy. When I have a final idea as to how much space we will need for our parameter set I will set the partition size to about double that with a minimum of about six pages or so.