What exactly is the String size which can be stored in NVS with the preferences lib?

[ullixesp]

I use an ESP32-dev with the Arduino framework (on platformio under vscode), and the NVS storage with the preferences lib for some log data. I am quite pleased how easy and reliable this lib can be used.

But I now tried to expand the storage of strings and I am running into unexpected limits.

The Espressif docu says:

String values are currently limited to 4000 bytes. This includes the null terminator.

My 4MB flash is configured for an NVS space of 0x5000 bytes. With every key at "full", the pref lib tells me I have 288 free entries.

Under key WARN I store "Strings" (actually chars) of a maximum size of 3500. When adding new records, I add them at the top and cut from the bottom until the new String fits into the 3500 limit, and putString() it back. So in my case with records of near 70 bytes I can keep the latest approx. 50 records. Good, that is how I want it.

However, I have another key DEVEL, also with a max size of 3500, which I would like to treat and behave the same way. Each one of the two keys is well under the 4000 char limit, yet DEVEL cannot hold more than 3150 bytes. Any attempts to save more results in the error message:

Code: Select all

[  3063][E][Preferences.cpp:273] putString(): nvs_set_str fail: warn NOT_ENOUGH_SPACE

So, somewhere there is an additional limit. Where does it come from?

What do I need to set in order to get the maximum string storage with my two keys WARN and DEVEL?

Here is an excerpt of the content for these two keys with some test data:

Code: Select all

# WARN: Bytes used:3450 out of 3500 (no errors)
18 14:16:07.515 51_123456789_123456789_123456789_123456789_123456789
18 14:16:07.641 52_123456789_123456789_123456789_123456789_123456789
18 14:16:07.769 53_123456789_123456789_123456789_123456789_123456789
...
18 14:16:13.386 98_123456789_123456789_123456789_123456789_123456789
18 14:16:13.515 99_123456789_123456789_123456789_123456789_123456789
18 14:16:13.641 100_123456789_123456789_123456789_123456789_123456789


# DEVEL: 3158 out of 3500 
18 14:19:30.327  1_123456789_123456789_123456789_123456789_123456789
18 14:19:30.441  2_123456789_123456789_123456789_123456789_123456789
18 14:19:30.491  3_123456789_123456789_123456789_123456789_123456789
...
18 14:19:34.312 43_123456789_123456789_123456789_123456789_123456789
18 14:19:34.441 44_123456789_123456789_123456789_123456789_123456789
18 14:19:34.587 45_123456789_123456789_123456789_123456789_123456789
# (errors 'NOT_ENOUGH_SPACE' when trying to save more)

[lbernstone]

A couple comments here:
1) Free entries is internal housekeeping for nvs. It really has no bearing on how much free space is available on the disk. There's no good mechanism for getting the free space in an nvs partition other than trying to put your key and see if it fails.
2) nvs does a safe replacement, so modifying an entry requires space to put down the new information, before the old info is deleted.
3) The size of a string is likely limited more by your stack space than the hard limit at 4000.
4) The default nvs partition has some system entries in it. If you need to keep track of the space used, I'd recommend making your own nvs partition.
5) Blobs are not limited on space like strings, and require very little processing to go back and forth from String to blob. The limit on a blob is 496K or 97% of the size of an nvs partition. They are also more flexible, and will tell how big they should be before you actually fill a buffer. I'd strongly recommend you use blobs instead of strings. See the prefs2struct example for a demonstration of how you efficiently use them for records (aka struct). You can also do something like using an entry in a struct as a pointer to another key/namespace where you store a freeform string.

[ullixesp]

Thank you. Doesn't seem to be as easy as it initially looked.

One would assume that "Free Entries" has to be at least greater than zero, but the docu isn't very forthcoming in explanation:

Arduino or c-string String types use a minimum of two key table entries with the number of entries increasing with the length of the string.
keys storing values of type Bytes use a minimum of three key table entries with the number of entries increasing with the number of bytes stored.

But whatever the expected value, I do see a minimum of 288 free entries, which let me conclude that this isn't a limit?

The stack also does not seem to limiting, as I see a High water Mark of 5840 after all NVS processing.

Heap also does not seem to be a problem:

Code: Select all

Heap:298K Free:108K Min:81K maxAloc:63476 Frg:79.2%

My NVS partition is set as:

Code: Select all

Type       : SubType Label     Address    Size[B] Size[MB]
0x01 Data  :    0x02 nvs      0x009000  0x0005000    0.020

So my NVS size is 0x5000 = 20480. My code defines 2 further keys, both Bytes, but small at 40 and 136 Bytes. When I use a maximum of 4000+4000+40+136=8176, and I am updating no more than one of the keys at a time, I'd expect that this also poses no limit (though I have seen that it does!). You say there is no way to determine the amount of NVS used by mine and other functions? I believe WiFi is one of those other functions?

If I wanted to define my own NVS space, how would I name Type, SubType and Label in my partitions.csv file? And how would I inform preferences to use this custom-NVS-space?

Large text info like my log data in WARN and DEVEL seem to suggest String use. But I'll give the Bytes a try.

[lbernstone]

partitions.csv:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
prefs1,   data, nvs,     0x290000,0x160000,
coredump, data, coredump,0x3F0000,0x10000,

Code: Select all

void setup() {
  Serial.begin(115200);
  Serial.println();
  prefs.begin("test", false, "prefs1"); 
  prefs.clear();
  
  String myStr;
  myStr.reserve(100);
  for (int x=0; x<100; x++) myStr += "-";
  prefs.putBytes("mystr", myStr.c_str(), myStr.length());
  Serial.println(myStr);
  
  myStr = "";
  size_t strsz = prefs.getBytesLength("mystr");
  char* buffer = (char*) calloc(strsz+1, sizeof(char));
  strsz = prefs.getBytes("mystr", buffer, strsz);
  myStr = buffer;
  free(buffer);
  Serial.println(myStr);
}

[ullixesp]

Looks rather straight forward, thank you, I will try!

Can you tell what the minimum size of the original nvs must be to keep all things working?

[lbernstone]

The default info is probably about 1k, and then there is the nvs overhead consumption. Call it 1 sector (4k) to be safe. This information truly does not change often. Some of it is hardware calibrations, some of it is stuff like the saved ssid & password.
I don't mean to discourage you from using the default partition generally, say if you know you have one CA and a private key to store, but it is shared space. In your case, you may get close to the limits, and it's better safe than sorry over a few sectors.

[ullixesp]

I now use a separate NVS for my data, and put the data as blobs. It does seem cleaner. Thanks for the suggestions!

I have now set the size of the original NVS at 0x2000 (so far all is ok), and my custom NVS at 0x3000. This turns out to be surprisingly small for my application.

What gave me a bit of a headache was your "+1" in the buffer size definition "calloc(strsz+1, sizeof(char));". Was this to accommodate any NULL termination at the end of the string? I think this does not make sense when you put down text as blobs, because there is no need for this ending (I flag the end with a "\n") and it actually is harmful since you are gobbling up undefined characters.

But now it works, though the pref lib does seem to use quite a bit of overhead.

I have now in my custom NVS of size 0x3000 (=12288) 4 keys as Bytes of max sizes 40+136+2000+2700=4876 when all is full and thus free space of at least 12288-4876=7412. The big keys take some kind of log data: I read the blob, add the latest line and remove the first, then I put this back. So it always stays below my defined maximum. Since you said that pref writes the new blob first before it removes the old one, I would need room for 2700 bytes. As I have 7412 bytes available, this should be easy. But no.

When I increase the biggest key to 2750 the updates fail with NOT_ENOUGH_SPACE while the High-WaterMark is at over 9000, and Free entries at 131!

Does the pref lib really need 270% spare bytes, i.e. 170% overhead on top of the 100% reserved room (7412 / 2750=2.70)? This does look like a very generous helping on the part of pref!