Storing Logs

[gregstewart90]

I'm ready in my project to store logs on the esp32. The logs would be used to check for bugs and problems. I would like to know the best way to go about it. I've implemented some logs in the blob, but read-the-docs says I shouldn't store big things in the blob. What is the correct way to store logs that can be retrieved and sent back to my server?

[kolban]

Howdy my friend,
If we take away the word "logs" from your story and simply say "We are storing data for subsequent retrieval" then I am presuming that the data has to be persisted (i.e. not just in RAM and hence survive a power down or crash). In which case my recommendation would be to use the FatFS component of the ESP-IDF. This provides a POSIX file system implemented on top of wear-leveling flash memory storage. You can then use POSIX file I/O to write your data (binary or text) as files to the file system. This may be as simple as appending data to a single "log file". When your application writes that data, it will be present on subsequent restarts and available for retrieval. You could implement simple log file management policies such as:

1. Rolling files ... when a file reaches a certain size (eg. 500K or 1MByte) then that file is closed, renamed and a new file started
2. Maximum number of files ... when a new file is started, if there are already 3 files in existence, you delete the oldest so that there are always no more than 3 files

You might want to create a queue/task mechanism such that your application writes its log record into a queue and then continues while a separate task performs the file I/O. This would suffer from the potential problem that log writing would then be asynchronous and the write to a log may not have completed before a subsequent failure and hence your log might not show the last message. Or you can write synchronously such that your logic forces a flush to file before continuing. You'll have to look at the nature of your solution to determine if the very small amount of time necessary to write a log record will interfere with your app logic.

[gregstewart90]

I've found thi library that supports everything you have just described. I already have many units deployed which I can only update with OTA. Can I update the partition table with OTA? My current partition table looks like the following:

Code: Select all

# Name,   Type, SubType, Offset,   Size
nvs,      data, nvs,     0x9000,  0x4000
otadata,  data, ota,     0xd000,  0x2000
phy_init, data, phy,     0xf000,  0x1000
ota_0,    app,  ota_0,   0x10000,  1472K
ota_1,    app,  ota_1,   ,         1472K

My firmware binary sizes are about 1.2MB, so I need large oat partitions.

[gregstewart90]

(bump)

[kolban]

Out of curiosity, what would the new/desired partition table map look like as compared to the one at hand?

[ESP_Sprite]

Also, no, you cannot (should not) update the partition table over OTA. Even if you hack something together to do it anyway, your device may loose power or hang during the erase/write cycle of the partition table, leaving the device bricked.

[WiFive]

Seems like you could do this at relatively low risk in certain cases and if well tested before deploying. Maybe implement in a wake stub?

Probably not a good idea for a mass consumer product but for an in-house deployment may be ok.

[ESP_Sprite]

There's nothing stopping you from doing it, to be sure: the partition tables are only a few sectors in SPI flash, so you can use the normal SPI flash routines to erase and overwrite them. I'm just saying that it's an extremely bad idea to do on production, because the chance of bricking a device is definitely non-zero.

[gregstewart90]

So I see my two options are to keep doing everything in nvs storage or take a big risk with the partition tables. I'll have a new run of boards soon that I can flash with a different partition table. Is there anything wrong with using the blob? My current plan uses a struct with two int32_t variables to store all of the data for a single log entry. I can keep adding these to the blob, but does that cause a problem with flash longevity or anything else?

Code: Select all

typedef struct BlobLog{
	int32_t timestamp; //Time Since Epoch
	int32_t encoded_message;
} BlobLog;

BlobLog blob_log;

[ESP_igrr]

If you can pack two int32 variables into an int64 and store that instead of a blob, that will be twice as efficient.