NVS not saving arrays inside blob correctly

[xpetept]

I'm using NVS to save an array of structures. My code is based on the ESP32 example project that saves an int and a blob. The code works and I get no errors but when I retrieve the data it does not match the data I saved. The integer values in the structure are saved correctly but the arrays"name" and "week_days" are not. I guess it's saving the pointer instead of saving the value but i don't know how to save the value correctly.

The structure I am trying to save is as follows:

Code: Select all

typedef struct Events {
    char *name;
    int hour;
    int minutes;
    int *week_days;
} Event;

Here is the code I use to save the array 'events':

Code: Select all

nvs_handle my_handle;
size_t required_size = 0; 
err = nvs_get_blob(my_handle, "events", NULL, &required_size);
if (err != ESP_OK && err != ESP_ERR_NVS_NOT_FOUND) return err;

Event* events = malloc(required_size + sizeof(event));
if (required_size > 0) {
    err = nvs_get_blob(my_handle, "events", events, &required_size);
    if (err != ESP_OK) return err;
}
required_size += sizeof(event);
events[required_size / sizeof(event) - 1] = event;
err = nvs_set_blob(my_handle, "events", events, required_size);
free(events);
if (err != ESP_OK) return err;
err = nvs_commit(my_handle);
if (err != ESP_OK) return err;
nvs_close(my_handle);

And here is the code I use to retrieve the saved 'events' array:

Code: Select all

nvs_handle my_handle;
size_t required_size = 0;
err = nvs_get_blob(my_handle, "events", NULL, &required_size);
Event* events = malloc(required_size);
err = nvs_get_blob(my_handle, "events", events, &required_size);
if (err != ESP_OK) return err;
for (int i = 0; i < required_size / sizeof(Event); i++) {
    printf("%d: %s\n", i + 1, events[i].name);
    printf("%d: %d\n", i + 1, events[i].hour);
    printf("%d: %d\n", i + 1, events[i].minutes);
    printf("%d: %d\n", i + 1, events[i].week_days[0]);
    printf("%d: %d\n", i + 1, events[i].week_days[1]);
}
free(events);
nvs_close(my_handle);

Can anyone see what I am doing wrong?

[ESP_igrr]

week_days member of Events structure is declared as a pointer, and NVS saves the contents of the structure, including the pointer value. NVS doesn't know whether this is a pointer to a single integer or an array, and what is the size of the array. Unlike languages like Java or C#, in C there is no difference between dynamic arrays and pointers. It is usually up to the programmer to keep track of dynamic array length (which is a major source of bugs).

I would suggest addressing this in one of the two ways:

1) Use an existing serialization library for C to convert your nested data into a flat buffer, then use NVS to store the contents of this buffer. IDF master branch currently comes with protobuf-c component, so that's at least one options. Converting your structure into JSON or XML is also a way to serialize it for storage.

2) Declare week_days as a zero-length array at the end of the structure, and store length of the array in the structure:

Code: Select all

typedef struct {
    char *name;
    int hour;
    int minutes;
    int week_days_count;
    int week_days[0];
} Event;

// create an event:
int week_days_count = 3; // for example
// allocate the structure plus space required to store week_days
size_t size = sizeof(Event) + sizeof(int)*week_days_count;
Event* evt = calloc(1, size);
evt->week_days_count = week_days_count;
// fill the rest of the fields


// store into NVS
err = nvs_set_blob(my_handle, "event", evt, sizeof(Event) + sizeof(int)*evt->week_days_count);

However if you need to store multiple such event structures in a single NVS block, then this approach becomes more tricky, as you need to keep track of the length of each structure. If this is the case, it will probably pay off to use a serialization library.

[zeboxer]

Hi! I have same issue with store array of struct to NVS.

I'm have struct with embedded other struct:

Code: Select all

// Struct for store webhook notification
typedef struct
{
    bool enable;
    char url[128];
    char username[20];
    char password[20];
} webhook_notification_s;

// Struct for store gpio events action
typedef struct
{
    uint8_t input_gpio;                      	// Input GPIO number
    uint8_t trigger_level;                    // trigger on high level or low level
    uint8_t output_gpio;                     // Output GPIO for relay
    webhook_notification_s webhook_url; // URL for HTTP notify
    uint32_t timeout_ms;                     // delay timeout in milliseconds
} gpio_action_t;

I have my functions for read and write struct to NVS :

Code: Select all

esp_err_t read_struct(const char *key, void **read_struct, size_t size)
{
    nvs_handle_t handle;
    esp_err_t err;

    err = nvs_open("storage", NVS_READWRITE, &handle);
    err = nvs_get_blob(handle, key, *read_struct, &size);
    
    DEBUG_NVS((err != ESP_OK) ? "[NVS] Read %s struct = Failed!\n" : "[NVS] Read %s struct  = Done\n", key);
    nvs_close(handle);
    return err;
}

esp_err_t write_struct(const char *key, void *write_struct, size_t size)
{
    nvs_handle_t handle;
    esp_err_t err;
    err = nvs_open("storage", NVS_READWRITE, &handle);
    err = nvs_set_blob(handle, key, write_struct, size);
    err = nvs_commit(handle);
    DEBUG_NVS((err != ESP_OK) ? "[NVS] Write %s struct = Failed!\n" : "[NVS] Write %s struct  = Done\n", key);
    nvs_close(handle);
    return err;
}

I create array with 8 items (size) and try to write:

Code: Select all

   // defined array gpio_actions[8]
    gpio_action_t *gpio_actions = (gpio_action_t *)malloc(8 * sizeof(gpio_action_t));
    for (int i = 0; i < 8; i++)
    {

        gpio_action_t *gpio_action_test = malloc(sizeof(gpio_action_t));
        webhook_notification_s *wh_test = malloc(sizeof(webhook_notification_s));

        wh_test->enable = true;
        strcpy(wh_test->url, "http://test.local/hello-webhook?test=1");
        strcpy(wh_test->username, "admin");
        strcpy(wh_test->password, "password");

        gpio_action_test->input_gpio = 10;
        gpio_action_test->output_gpio = 12;
        gpio_action_test->timeout_ms = 1000;
        gpio_action_test->trigger_level = i;
        
        memcpy(&gpio_action_test->webhook_url, wh_test, sizeof(webhook_notification_s));
        free(wh_test);
        
        memcpy(&gpio_actions[i], gpio_action_test, sizeof(gpio_action_t));
        free(gpio_action_test);
    }
    
    // debug prints
      for (int i = 0; i < 8; i++)
    {
        printf("-------%d ----\n", i);
        printf("gpio input: %u\n", gpio_actions[i].input_gpio);
        printf("gpio out : %u\n", gpio_actions[i].output_gpio);
        printf("gpio timeout: %d\n", (int)gpio_actions[i].timeout_ms);
        printf("gpio level: %u\n", gpio_actions[i].trigger_level);
        printf("gpio wh en: %d\n", gpio_actions[i].webhook_url.enable);
        printf("gpio wh url: %s\n", gpio_actions[i].webhook_url.url);
        printf("gpio wh user: %s\n", gpio_actions[i].webhook_url.username);
        printf("gpio wh password: %s\n", gpio_actions[i].webhook_url.password);
        printf("----------\n");
    }
    
    esp_err_t err = write_gpio_actions(gpio_actions);
    if (err!=ESP_OK) {
	    printf("--> %s\n", esp_err_to_name(err));	  
    }
    free(gpio_actions);

Write done, without errors. Debug prints values is OK.

Then i try to read:

Code: Select all

 
 gpio_action_t *gpio_actions_read = (gpio_action_t *)malloc(8 * sizeof(gpio_action_t));
    if (read_gpio_actions(gpio_actions_read) == ESP_ERR_NVS_NOT_FOUND)
    {
        ESP_LOGE(TAG, "failed to read gpio actions");
        free(gpio_actions_read);
        return;
    }
    // debug prints
    for (int i = 0; i < 8; i++)
    {
        printf("-------%d ----\n", i);
        printf("gpio input: %u\n", gpio_actions_read[i].input_gpio);
        printf("gpio out : %u\n", gpio_actions_read[i].output_gpio);
        printf("gpio timeout: %d\n", (int)gpio_actions_read[i].timeout_ms);
        printf("gpio level: %u\n", gpio_actions_read[i].trigger_level);
        printf("gpio wh en: %d\n", gpio_actions_read[i].webhook_url.enable);
        printf("gpio wh url: %s\n", gpio_actions_read[i].webhook_url.url);
        printf("gpio wh user: %s\n", gpio_actions_read[i].webhook_url.username);
        printf("gpio wh password: %s\n", gpio_actions_read[i].webhook_url.password);
        printf("----------\n");
    }

    free(gpio_actions_read);
    
  

In read section, debug prints output wrong value in some fields for first and last items.

Code: Select all

[NVS] Read gpa struct  = Done
-------0 ----
gpio input: 140			//<-- invalid value
gpio out : 202			//<-- invalid value
gpio timeout: 1000 		
gpio level: 135 			//<-- invalid value
gpio wh en: 63 			//<-- invalid value
gpio wh url: ���?://test.local/hello-webhook?test=1  //<-- invalid value, prefix with wrong characters
gpio wh user: admin
gpio wh password: password
----------
-------1 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 1
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------2 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 2
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------3 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 3
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------4 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 4
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------5 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 5
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------6 ----
gpio input: 10
gpio out : 12
gpio timeout: 1000
gpio level: 6
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------
-------7 ----
gpio input: 10
gpio out : 12
gpio timeout: 1070280396  //<-- invalid value
gpio level: 7
gpio wh en: 1
gpio wh url: http://test.local/hello-webhook?test=1
gpio wh user: admin
gpio wh password: password
----------

Could anyone explain why?

[ESP_rrtandler]

Hi, Perhaps there is some trouble in your read / write functions. Could you post also the definition of

Code: Select all

 write_gpio_actions()

and

Code: Select all

read_gpio_actions()

?

[MicroController]

Code: Select all

esp_err_t read_struct(const char *key, void **read_struct, size_t size)
{
...
    err = nvs_get_blob(handle, key, *read_struct, &size);

seems odd. Maybe you actually want it the more 'usual' way (pointer to read_struct as input):

Code: Select all

esp_err_t read_struct(const char *key, void *read_struct, size_t size)
{
...
    err = nvs_get_blob(handle, key, read_struct, &size);
[/quote]