Encrypted OTA firmware

[zelea20]

I'm successfully flashing encrypted OTA firmware using this procedure:

• Patch the $(IDF_PATH) with the attached diff
  This will disable the https OTA and use normal http instead
  Also it prevents the double encryption when flashing OTA since the firmware binary is already encrpted
• Generate a unique flash encryption key
  $(ESPSECURE) generate_flash_encryption_key aes_key.bin
• Burn the key and protect it
  $(ESPEFUSE) burn_key flash_encryption aes_key.bin
• Compile your application WITHOUT enabling flash encryption on boot
• Encrypt the binary files (add in Makefile this rule)
  %.xbin: %.bin
  $(ESPSECURE) encrypt_flash_data -k aes_key.bin --flash_crypt_conf 0 -a 0 -o $@ $^
• Use esptool to program the encrypted binaries for the first time
  $(ESPTOOL) --chip esp32 --port /dev/ttyUSB0 --baud 115200 ...
• Burn the flash encryption fuse
  $(ESPEFUSE) burn_efuse FLASH_CRYPT_CNT
• For subsequent OTA firmware updates just place the new encrypted application binary on a website
  and in your code use some version numbering scheme and call 'esp_https_ota' when a newer version is found

I'm using the above procedure with just 2 OTA partitions (no factory one). The '--flash_crypt_conf 0' encrypts the binary regardless of its loading address (otherwise you'll need to generate different encrypted binaries for each address where it runs from). You also need to keep the FLASH_CRYPT_CONFIG fuses at 0. This is not diminishing the encryption, it was only adding 19 bits of entropy (with a known scheme) to the 256 bits of the AES key anyways. Even with known plaintext, reinserting chunks of encrypted 16 bytes at different (aligned) addresses is not a very useful attack vector.
The secure boot needs to be enabled for production (as explained in the ESP32 documents) so that your code remains protected from physical access to your device.
The advantages of this scheme is that you can use a unique AES key for all the devices and you don't have to bother with certificates and expiration dates or website authentication. In fact you can completely remove all the mbedTLS stuff and even SSL if that's not used elsewhere. Completely removing the mbedTLS component need extra patching though, just disabling it in menuconfig is not enough because there are still a lot of inter-dependencies. Another bonus is that this will significantly reduce your code size.

Code: Select all

diff --git a/components/app_update/esp_ota_ops.c b/components/app_update/esp_ota_ops.c
index 5e4dcb424..20d4a92f2 100644
--- a/components/app_update/esp_ota_ops.c
+++ b/components/app_update/esp_ota_ops.c
@@ -44,7 +44,7 @@
 
 typedef struct ota_ops_entry_ {
     uint32_t handle;
-    const esp_partition_t *part;
+    esp_partition_t *part;
     uint32_t erased_size;
     uint32_t wrote_size;
     uint8_t partial_bytes;
@@ -123,7 +123,7 @@ esp_err_t esp_ota_begin(const esp_partition_t *partition, size_t image_size, esp
         new_entry->erased_size = image_size;
     }
 
-    new_entry->part = partition;
+    new_entry->part = (esp_partition_t *)partition;
     new_entry->handle = ++s_ota_ops_last_handle;
     *out_handle = new_entry->handle;
     return ESP_OK;
@@ -146,7 +146,7 @@ esp_err_t esp_ota_write(esp_ota_handle_t handle, const void *data, size_t size)
             // must erase the partition before writing to it
             assert(it->erased_size > 0 && "must erase the partition before writing to it");
 
-            if(it->wrote_size == 0 && size > 0 && data_bytes[0] != 0xE9) {
+            if(0 && it->wrote_size == 0 && size > 0 && data_bytes[0] != 0xE9) {
                 ESP_LOGE(TAG, "OTA image has invalid magic byte (expected 0xE9, saw 0x%02x", data_bytes[0]);
                 return ESP_ERR_OTA_VALIDATE_FAILED;
             }
@@ -163,6 +163,7 @@ esp_err_t esp_ota_write(esp_ota_handle_t handle, const void *data, size_t size)
                     if (it->partial_bytes != 16) {
                         return ESP_OK; /* nothing to write yet, just filling buffer */
                     }
+		    it->part->encrypted = false;
                     /* write 16 byte to partition */
                     ret = esp_partition_write(it->part, it->wrote_size, it->partial_data, 16);
                     if (ret != ESP_OK) {
@@ -182,7 +183,7 @@ esp_err_t esp_ota_write(esp_ota_handle_t handle, const void *data, size_t size)
                     memcpy(it->partial_data, data_bytes + size, it->partial_bytes);
                 }
             }
-
+	    it->part->encrypted = false;
             ret = esp_partition_write(it->part, it->wrote_size, data_bytes, size);
             if(ret == ESP_OK){
                 it->wrote_size += size;
@@ -235,7 +236,7 @@ esp_err_t esp_ota_end(esp_ota_handle_t handle)
       .offset = it->part->address,
       .size = it->part->size,
     };
-
+    it->part->encrypted = true;
     if (esp_image_verify(ESP_IMAGE_VERIFY, &part_pos, &data) != ESP_OK) {
         ret = ESP_ERR_OTA_VALIDATE_FAILED;
         goto cleanup;
diff --git a/components/esp_https_ota/src/esp_https_ota.c b/components/esp_https_ota/src/esp_https_ota.c
index 9929a1856..5cffadf80 100644
--- a/components/esp_https_ota/src/esp_https_ota.c
+++ b/components/esp_https_ota/src/esp_https_ota.c
@@ -35,7 +35,7 @@ esp_err_t esp_https_ota(const esp_http_client_config_t *config)
         return ESP_ERR_INVALID_ARG;
     }
 
-    if (!config->cert_pem) {
+    if (0 && !config->cert_pem) {
         ESP_LOGE(TAG, "Server certificate not found in esp_http_client config");
         return ESP_FAIL;
     }
@@ -46,7 +46,7 @@ esp_err_t esp_https_ota(const esp_http_client_config_t *config)
         return ESP_FAIL;
     }
 
-    if (esp_http_client_get_transport_type(client) != HTTP_TRANSPORT_OVER_SSL) {
+    if (0 && esp_http_client_get_transport_type(client) != HTTP_TRANSPORT_OVER_SSL) {
         ESP_LOGE(TAG, "Transport is not over HTTPS");
         return ESP_FAIL;
     }

[francescofcf]

Thanks for your sharing.
Very useful to try and solve my problem.
I would like to start the procedure ota using esp32 as AP with DHCP and this involves the problem of not knowing the IP address of the HTTPS server and therefore I can not generate the "self-signed certificate and key" because I do not know the "Common Name", then I thought to disable the https OTA and use normal http instead.
Now I try to follow your example to understand how you've done to disable it.
Or have you any tips?

[ESP_Angus]

Hi zelea20,

Thanks for explaining your process so clearly. We'll add official support for pre-encrypted OTA updates in a future IDF update.

A couple of small points:

This will disable the https OTA and use normal http instead

Note that although the pre-encrypted image and secure boot signature check will prevent an attacker using the insecure transport protocol (HTTP) to tamper with the image, it doesn't prevent an attacker from launching (for example) a downgrade attack by MITMing the transport scheme and sending the device incorrect or outdated update metadata.

You can probably account for this problem in other ways, but it's worth considering.

The '--flash_crypt_conf 0' encrypts the binary regardless of its loading address (otherwise you'll need to generate different encrypted binaries for each address where it runs from). You also need to keep the FLASH_CRYPT_CONFIG fuses at 0. This is not diminishing the encryption, it was only adding 19 bits of entropy (with a known scheme) to the 256 bits of the AES key anyways. Even with known plaintext, reinserting chunks of encrypted 16 bytes at different (aligned) addresses is not a very useful attack vector.

It is strongly encouraged not to do this (leaving FLASH_CRYPT_CONFIG at zero), especially if there's any other data (apart from the app images) stored on the device and protected with encryption, and doubly so if any of that data may be externally sourced (ie under control of an attacker).

Forcing AES-ECB mode (by setting FLASH_CRYPT_CONFIG to zero) for the entire flash definitely "diminishes the encryption", even though there are still some challenges to attacking a device which is protected this way.

[fly135]

Thanks for your sharing.
Very useful to try and solve my problem.
I would like to start the procedure ota using esp32 as AP with DHCP and this involves the problem of not knowing the IP address of the HTTPS server and therefore I can not generate the "self-signed certificate and key" because I do not know the "Common Name", then I thought to disable the https OTA and use normal http instead.
Now I try to follow your example to understand how you've done to disable it.
Or have you any tips?

You can use "esp_https_ota" to work with HTTP or HTTPS, and with or without a certificate simply by commenting out the return on the checks for HTTPS and certificate. Not sure why they even put those checks in. It simply diminishes the usefulness of the code without adding any functionality.

John A

[ESP_Angus]

Not sure why they even put those checks in. It simply diminishes the usefulness of the code without adding any functionality.

I believe the idea was to make it difficult to ship devices which use an insecure protocol for their OTA updates. Is HTTPS really that hard these days?

[fly135]

Not sure why they even put those checks in. It simply diminishes the usefulness of the code without adding any functionality.

I believe the idea was to make it difficult to ship devices which use an insecure protocol for their OTA updates. Is HTTPS really that hard these days?

Probably not. But I still haven't wrapped my head completely around supporting certificates. If at some point in the future the endpoint for OTA downloads changes along with the certificate, wouldn't that prevent existing units from getting updates? Or even brick them if an update was required for communication with a back end server?


John A

[ESP_Angus]

Hi John,

Provided you have a copy of the private key and the certificate for the endpoint site (for the given endpoint DNS name), you should be able to deploy it on a new server without any trouble.

If you lose the key, then you will be in trouble. No different to if you lose control of the domain name that the device is connecting to. (You can always support fallbacks for both of these things.)

If you need to change the server to a totally new domain name, you have an opportunity to update the certificate at the same time. (Device without TLS will be just as bricked if the old server name stops working.)

Most IoT devices don't do certificate revocation or expiry (ESP-IDF disables certificate expiry by default), so the key & cert will stay valid until the device OTA updates to a new version which no longer trusts that cert.

Sometimes it can be useful to load a root CA cert or an intermediate cert in the certificate chain to the device, instead of the cert for a particular server. The device will then trust any certificate signed by that root or intermediate certificate (for example, if you want to support multiple endpoint servers without needing to put multiple certificates into the device.)

Perhaps we should write up a guide to certificate "best practices" for ESP-IDF & IoT deployments in general.

[luisonoff]

Perhaps we should write up a guide to certificate "best practices" for ESP-IDF & IoT deployments in general.

That would be great :). In my devices I am using some intermediate certificates I downloaded from somewhere, but not sure what servers they support, or how long they will be valid, or if one day suddenly they can be rejected, etc.

[fly135]

Sometimes it can be useful to load a root CA cert or an intermediate cert in the certificate chain to the device, instead of the cert for a particular server. The device will then trust any certificate signed by that root or intermediate certificate (for example, if you want to support multiple endpoint servers without needing to put multiple certificates into the device.)

Perhaps we should write up a guide to certificate "best practices" for ESP-IDF & IoT deployments in general.

That would be awesome. I find it hard to get a big picture understanding. I can imagine new units in inventory when out of necessity the OTA and back end servers get changed and then they can't sign in or download updates when they are unboxed and deployed.

That intermediate or root cert sounds interesting.

John A

[stoikos]

yes please!