Advice on Core Dumps + Flash Encryption

[paul-flowcoffee]

Hello, I've got a device with firmware in production. We have enabled flash encryption, and have cored dumps enabled. Unfortunately we didn't see anywhere in the documentation that the core dump partition needs to be explicitly marked as encrypted in the partition table...

At this point we have our bootloader, and 3 app partitions (a factory, and two OTA partitions), as well as a FATFS partition, and core dump partition, and a few NVS partitions, etc... For our (several thousand) devices in the wild now running this setup, we wanted to enable remote uploading of the core dumps to our servers, so we could do crash analysis...

Unfortunately when we enabled the code in a dev firmware, we quickly encountered the following error when trying to read out the core dump data at runtime:

E (4122) esp_core_dump_elf: Flash encryption enabled in hardware and core dump partition is not encrypted!
And I'm getting an ESP_ERR_NOT_SUPPORTED back from the coredump functions.

The error is being thrown by esp_core_dump_get_summary which we're just using as a test trying to get the summary info from the header initially and ship it via MQTT.

So I have a few questions that this raises:

• First off, is there any way we can retroactively apply the encryption flag to the core dump partition on an existing encrypted device?
• If we can't, then is there any way we can make this work (able to read data out to transmit to server from a core dump on an "unecrypted" partition?)
• Lastly, if the coredump partition is unencrypted, does that mean the core dumps are being stored in plain text despite the flash encryption setting? As a result I suspect this is a major security risk for us...

Here is the partition table we have configured:

Code: Select all

sec_cert,   0x3F,   ,           0x11000,    0x3000, ,  # Never mark this as an encrypted partition
nvs,        data,   nvs,        0x14000,    0x6000,
otadata,    data,   ota,        ,           0x2000,
phy_init,   data,   phy,        ,           0x1000,
rmaker,     data,   nvs,        ,           0x6000,
coredump,   data,   coredump,   ,           512K,
factory,    app,    factory,    0xB0000,    3M,     encrypted
ota_0,      app,    ota_0,      ,           3M,     encrypted
ota_1,      app,    ota_1,      ,           3M,     encrypted
storage,    data,   fat,        ,           6M,     encrypted

Also as a clarification, right now our only means to update a device in the field is our OTA process which sends an MQTT command to the device via a secure MQTT channel, asking it to fetch a new application image from an authenticated HTTPS server which it downloads and applies using the esp_https_ota workflow.

Thanks in advance for any ideas, suggestions and feedback, we're greatly appreciative.

[ESP_Mahavir]

Hello,

I see this error coming from the API `esp_core_dump_get_summary`, so the implementation of this API reads data from coredump partition using mmap and that shall result in garbage if partition is not written in an encrypted manner.

I do not see any other incompatibility in your case with coredump module even if the partition is not encrypted on the flash. Only `esp_core_dump_get_summary` API won't work in this scenario. Interim, this looks very similar to https://github.com/espressif/esp-insights/issues/2

Can you please share more information on your usage scenario (API, test code etc.) with coredump partition?

Note: We will definitely update the point about `encrypted` flag for coredump partition in our docs.

[ESP_Mahavir]

First off, is there any way we can retroactively apply the encryption flag to the core dump partition on an existing encrypted device?

You may check the following workaround in your code to fix the `encrypted` flag for coredump partition:

Code: Select all

__attribute__((constructor)) void fix_partition_status(void)
{
    ((esp_partition_t*)esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_COREDUMP, "coredump"))->encrypted = true;
}

Please consider this as a workaround. Clean fix would be to update the partition table but that is very difficult given power failure scenarios for on-field devices.

If we can't, then is there any way we can make this work (able to read data out to transmit to server from a core dump on an "unecrypted" partition?)

I think so. Except the API `esp_coredump_get_summary()`, other coredump functionality should work as expected.

Lastly, if the coredump partition is unencrypted, does that mean the core dumps are being stored in plain text despite the flash encryption setting? As a result I suspect this is a major security risk for us...

Yes. There is a narrow window from the crash where data is stored (in plain-text as its not marked as encrypted) on this partition to the point where data would be uploaded to the cloud. You may erase the partition after that using API `esp_core_dump_image_erase()`.