[Solved] Recipes for saving application state to flash at run-time.

[kolban]

I am working on an app that needs to save data into flash so that it would be present on a restart. Are there any suggestions for:

1) APIs to use to read/write flash memory?
2) Locations of storage within a memory map that would be recommend to use?

The back story is that I am working on porting a JavaScript run-time to ESP32 which is coming along very well. We are at the stage now where we would like to save the JavaScript and have it loaded at boot time. For the sake of discussion, let us assume that the JavaScript program is saved as text and re-parsed to execute. What we do is run our ESP32 app (called Espruino) on the ESP32. This then runs the JavaScript interpreter. We then pass in the JavaScript program we want to run via serial. Once passed in, we can execute a "save()" which would save the loaded JavaScript to flash. On restart, the Espruino ESP32 app would look at the well known flash memory area and see if a script was present and, if so, use that. This is the model used on other Espruino platforms. The puzzle is to try and determine how this mapping might be supported on the ESP32. On the ESP8266 we just assumed that the app would be stored at the "bottom" of flash memory and we saved our script in a buffer 50K from the top of memory.

[WiFive]

The Non-volatile storage library supports blobs, but probably not ideal. Perhaps spiffs or SD card support with fat.

[kolban]

Actually a blob of non-volatile storage sound like just what I need. However the semantics of this component are not clear to me yet. For example, if I use nvs_set_blob() I believe that I can save an array of bytes. Great!! Now let us think of subsequently retrieving them. The corresponding API is nvs_get_blob() and here is where the semantics get grey. I can specify the handle to be used for accessing the data and the handle can opened in either read/write or readonly mode.

If I open the handle in readonly, if I then execute an nvs_get_blob() am I given a pointer to the previously saved data in flash or must I always allocate a RAM space large enough to hold the blob which would then (presumably) be copied from flash into that buffer?

Ideally, if I open the handle read-only and ask for the blob, I am given a pointer reference into flash memory address space ... but it isn't at all clear to me if that is what the semantics mean.

[ESP_igrr]

nvs_get_blob can not, in general, give you the pointer to the address space in flash, for the following reason. Flash may be encrypted, but NVS partition is non-encrypted. All memory-mapped reads from flash memory go through decryption module if flash encryption is enabled, so it is not possible to read plaintext data from flash in this case.

Actually, the header doc for nvs_get_blob describes how you can obtain a buffer large enough to hold data. It even gives an example (nvs_get_str and nvs_get_blob have identical semantics):

All functions expect out_value to be a pointer to an already allocated variable
of the given type.
Additionally, nvs_get_str and nvs_get_blob support WinAPI-style length queries.
To get the size necessary to store the value, call nvs_get_str or nvs_get_blob
with zero out_value and non-zero pointer to length. Variable pointed to
by length argument will be set to the required length. For nvs_get_str,
this length includes the zero terminator. When calling nvs_get_str and
nvs_get_blob with non-zero out_value, length has to be non-zero and has to
point to the length available in out_value.
It is suggested that nvs_get/set_str is used for zero-terminated C strings, and
nvs_get/set_blob used for arbitrary data structures.

Example of using nvs_get_i32:
int32_t max_buffer_size = 4096; // default value
esp_err_t err = nvs_get_i32(my_handle, "max_buffer_size", &max_buffer_size);
assert(err == ESP_OK || err == ESP_ERR_NVS_NOT_FOUND);
// if ESP_ERR_NVS_NOT_FOUND was returned, max_buffer_size will still
// have its default value.

Example (without error checking) of using nvs_get_str to get a string into dynamic array:
size_t required_size;
nvs_get_str(my_handle, "server_name", NULL, &required_size);
char* server_name = malloc(required_size);
nvs_get_str(my_handle, "server_name", server_name, &required_size);

Example (without error checking) of using nvs_get_blob to get a binary data into a static array:
uint8_t mac_addr[6];
size_t size = sizeof(mac_addr);
nvs_get_blob(my_handle, "dst_mac_addr", mac_addr, &size);


If anything is not clear in this description, please let me know so that I can improve it.

Regarding your original question, i think filesystems like spiffs or FAT will be better fit for the problem you are solving, but as mentioned in the release notes we don't have filesystem integration yet. Once we do, you will be able to use standard C functions (fopen, fread, fprinf, and so on) to work with files. This feature will be part of 1.0 release.

[ESP_igrr]

Filesystem API has been merged into ESP-IDF master, see components/vfs/README.md for details.
Partition API has also been merged and will be on Github as soon as tests pass.