Storing Logs

[gregstewart90]

If you can pack two int32 variables into an int64 and store that instead of a blob, that will be twice as efficient.

I definitely can pack them together. Should I make an array of the log entries and save that to the blob or should I try to rename each entry?

Code: Select all

BlobLog blob_logs[];
...
nvs_set_blob(my_handle, "log", &blob_logs, sizeof(blob_logs));

or

Code: Select all

nvs_set_i32(my_handle, "unique_name", new_blob_log); //unique_name will probably just be an incremented int.

With the former it is easier to save since I don't have to create unique names for the nvs location, and with the latter I will have to read many different locations when accessing the blob. Does the latter have an advantage?

[ESP_igrr]

Not rewriting what was already written is obviously more efficient. You will need less flash operations when using a separate key-value pair for each log entry.

[burkulesomesh43]

Howdy my friend,
If we take away the word "logs" from your story and simply say "We are storing data for subsequent retrieval" then I am presuming that the data has to be persisted (i.e. not just in RAM and hence survive a power down or crash). In which case my recommendation would be to use the FatFS component of the ESP-IDF. This provides a POSIX file system implemented on top of wear-leveling flash memory storage. You can then use POSIX file I/O to write your data (binary or text) as files to the file system. This may be as simple as appending data to a single "log file". When your application writes that data, it will be present on subsequent restarts and available for retrieval. You could implement simple log file management policies such as:

1. Rolling files ... when a file reaches a certain size (eg. 500K or 1MByte) then that file is closed, renamed and a new file started
2. Maximum number of files ... when a new file is started, if there are already 3 files in existence, you delete the oldest so that there are always no more than 3 files

You might want to create a queue/task mechanism such that your application writes its log record into a queue and then continues while a separate task performs the file I/O. This would suffer from the potential problem that log writing would then be asynchronous and the write to a log may not have completed before a subsequent failure and hence your log might not show the last message. Or you can write synchronously such that your logic forces a flush to file before continuing. You'll have to look at the nature of your solution to determine if the very small amount of time necessary to write a log record will interfere with your app logic.

I am going to store my logs in FATfs of esp32.
but esp32 has limitation to write to files around 10000 times. so if it exceeds as you said to write in too many files it damages flash.
I have no idea how to avoid that situation.
can you please explain?