Are 'preferences' supposed to maintain values even over uploads of new code?

[ullixesp]

I am using preferences with 5 keys (Bytes, Strings, Short) and as expected the values are preserved over any reboot.

However, when I upload fresh code - and do NOT touch the partitions - there is a mix of results. In some rare(!) occasions the values are preserved, but most of the time from 1 to all 5 of the keys are missing (prefs.isKey("mykey") is false).

I am surprised: why does an upload (via USB cable) change the nvs section?

While I find nvs easy to use, this makes it less valuable than storing under a file system, like FFat or LittleFS. Or is there any trick I can apply to make nvs keep the values?

[ullixesp]

I have more details, and I think I got trapped by a bug in the preferences lib:
When I use the default nvs to store my pref variables, all works fine - values are maintained over reboots as well as over uploads.
But when I use a custom nvs as a 2nd nvs, my values are still maintained over reboots, but randomly lost from none to all over an upload.

Here my configuration and situation with a single, default nvs (using platformio in vscode):

Code: Select all

File 'partitions.csv' (only default is active)
#       Name,       Type,   SubType,    Offset,     Size,      Flags
# nvs default
        nvs,        data,   nvs,              ,   0x3000,
# nvs custom 
#       nvscustom,  data,   nvs,              ,   0x2000,


Code initiating nvs (default is used)
    #define AMBIOMON "ambiomon"
    #define NVSPARTITION "nvs"                  // using default nvs
    // #define NVSPARTITION "nvscustom"         // using custom nvs
    bool mynvs = prefs.begin(AMBIOMON, false, NVSPARTITION);                                    // false => allow both read and write


Flash Storage Partitions:  (Chip Size: 4 MiB)
Type       : SubT Label      Address     Bytes    MiB  Blk
SecureBoot                  0x000000 0x0001000  0.004    1
Bootloader                  0x001000 0x0007000  0.027    7
Partition Table             0x008000 0x0001000  0.004    1
0x01 Data  : 0x00 otadata   0x009000 0x0002000  0.008    2
0x01 Data  : 0x02 nvs       0x00b000 0x0003000  0.012    3
0x00 App   : 0x10 app0      0x010000 0x0160000  1.375  352
0x00 App   : 0x11 app1      0x170000 0x0160000  1.375  352
0x01 Data  : 0x03 coredump  0x2d0000 0x0010000  0.062   16
0x01 Data  : 0x82 spiffs    0x2e0000 0x0120000  1.125  288
Flash Size Total                     0x03fe000  3.992 1022
Flash Size Free                      0x0002000  0.008    2


From the code log:
Initializing NVS
Success initializing namespace: 'ambiomon'
Free Entries   : 231  
NVS key  amcfg     : Using: 136  Bytes (Type Bytes)
NVS key  crc       : Using: 40   Bytes (Type Bytes)
NVS key  syslogNo  : Using: 2    Bytes (Type Short)

So far my past and current experience with a single, default nvs.

Now I activate a 2nd nvs as custom nvs, and I store my pref values in this custom nvs:

Code: Select all

File 'partitions.csv' (both default and custom are active)
#       Name,       Type,   SubType,    Offset,     Size,      Flags
# nvs default
        nvs,        data,   nvs,              ,   0x3000,
# nvs custom 
        nvscustom,  data,   nvs,              ,   0x2000,


Code initiating nvs (custom is used)
    #define AMBIOMON "ambiomon"
    // #define NVSPARTITION "nvs"                  // using default nvs
    #define NVSPARTITION "nvscustom"         // using custom nvs
    bool mynvs = prefs.begin(AMBIOMON, false, NVSPARTITION);                                    // false => allow both read and write


Flash Storage Partitions:  (Chip Size: 4 MiB)
Type       : SubT Label      Address     Bytes    MiB  Blk
SecureBoot                  0x000000 0x0001000  0.004    1
Bootloader                  0x001000 0x0007000  0.027    7
Partition Table             0x008000 0x0001000  0.004    1
0x01 Data  : 0x00 otadata   0x009000 0x0002000  0.008    2
0x01 Data  : 0x02 nvs       0x00b000 0x0003000  0.012    3
0x01 Data  : 0x02 nvscustom 0x00e000 0x0002000  0.008    2
0x00 App   : 0x10 app0      0x010000 0x0160000  1.375  352
0x00 App   : 0x11 app1      0x170000 0x0160000  1.375  352
0x01 Data  : 0x03 coredump  0x2d0000 0x0010000  0.062   16
0x01 Data  : 0x82 spiffs    0x2e0000 0x0120000  1.125  288
Flash Size Total                     0x0400000  4.000 1024
Flash Size Free                      0x0000000  0.000    0


From the log:
Initializing NVS
Success initializing namespace: 'ambiomon'
Free Entries   : 231  
NVS key  amcfg     : MISSING  resetNVSkey: amcfg
NVS key  crc       : MISSING  resetNVSkey: crc
NVS key  syslogNo  : MISSING  - has now been created

You see that all three pref variables (2 type Bytes, 1 type Short) are missing after an upload (same when type String is used, not shown).

If this is not a bug, please help and explain what I made wrong!

[ullixesp]

I reported what I think is a severe bug in the preferences lib - or even in code behind it - and I am somewhat surprised that I do not get a response to this?

I'd like to know: is this a bug? If not, what did I do wrong, please help me!

[boarchuz]

Minimum NVS partition size is 0x3000b, so the inconsistent results you're seeing would fall under "undefined behaviour".

Is mynvs == false with the custom 0x2000 partition? If not, an issue on the esp32-arduino repo is probably the best way to have this addressed.

[ullixesp]

@boarchuz: are you saying than each and every nvs partition requires a minimum of 0x3000, i.e. 3 sectors of 4096 bytes each?

This is in stark contrast to what Ibernstone stated, https://esp32.com/viewtopic.php?f=19&t= ... 18#p118909 namely that 1000 bytes can be enough, but recommends to use 0x1000, equal to 1 sector. Well, I found out that this is NOT enough, but for the internal needs of the ESP at least 0x2000 is needed. This 0x2000 worked fine, but you say this also must be raised to 0x3000?

After adding only 176 bytes via keys from my code, I had to increase this to 0x3000, and then again all was fine.

The problem came when I put these 176 bytes in a custom nvs of 0x2000 size. Rebooting also was never a problem, but after uploading from none to all my custom keys were missing. This will all work after increasing custom-nvs partition to 0x3000 size?

[ullixesp]

I ran some more tests:

I can definitively rule out the need for a minimum 0x3000 size for a nvs partition! Both the default as well as a custom nvs do work with size 0x2000.

Both do NOT work with a size of 0x1000! While I do not know what the default nvs is used for, my custom nvs gets only 3 keys with a total of 176 bytes. So with 4096 bytes not being enough to hold 176 bytes suggests quite a significant overhead! This underlines my earlier observation for huge overheads needed for String storage, see https://esp32.com/viewtopic.php?f=19&t= ... 18#p118974

However, there are more problems, resulting from where the partition is located:

I used a default nvs, plus a custom nvs, each of size 0x3000, and later of size 0x2000, with same results. When the custom nvs came right after the default nvs, a perpetual reboot of the cpu would result.

When the custom nvs came at the very end of the partition list, all worked - be it reboot or upload, the values were maintained. This is how my partitioning (4 MB chip size) looks like:

Code: Select all

Type       : SubT Label      Address     Bytes    MiB  Blk
0x01 Data  : 0x00 otadata   0x009000 0x0002000  0.008    2
0x01 Data  : 0x02 nvs       0x00b000 0x0003000  0.012    3
0x01 Data  : 0x02 nvscustom 0x3f0000 0x0003000  0.012    3	# shown placed at the end; it works
0x00 App   : 0x10 app0      0x010000 0x0160000  1.375  352
0x00 App   : 0x11 app1      0x170000 0x0160000  1.375  352
0x01 Data  : 0x82 spiffs    0x2d0000 0x0120000  1.125  288

Something isn't kosher with that preferences lib!

[ullixesp]

Is it possible that partition "app0" MUST be placed at address 0x010000? Because I let the ESP-firmware decide where to put the partitions, and if I want nvs at size 0x3000, it always adds a gap of 0x2000 after nvs. So there is no point in "saving" nvs space!? Actually, for wear-leveling it might be better to give all 0x5000 to nvs??

Code: Select all

0x01 Data  : 0x00 otadata   0x009000 0x0002000  0.008    2
0x01 Data  : 0x02 nvs       0x00b000 0x0003000  0.012    3
0x00 App   : 0x10 app0      0x010000 0x0160000  1.375  352
0x00 App   : 0x11 app1      0x170000 0x0160000  1.375  352
0x01 Data  : 0x82 spiffs    0x2d0000 0x0130000  1.188  304