Failed to enable secure boot / encrypted flash

[Markus Becker]

I just enabled secure boot together with encrypted flash for the very first time. While trying to follow documentation I must have missed something, as make monitor displays in a loop:

Code: Select all

rst:0x10 (RTCWDT_RTC_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:6944
ho 0 tail 12 room 4
load:0x40078000,len:0
load:0x40078000,len:20620
secure boot check fail
ets_main.c 371 
ets Jun  8 2016 00:22:57

I did the following:
used make menuconfig to enable the features, resulting in sdkconfig:

Code: Select all

<snip>
#
# Security features
#
CONFIG_SECURE_BOOT_ENABLED=y
CONFIG_SECURE_BOOTLOADER_ONE_TIME_FLASH=y
CONFIG_SECURE_BOOTLOADER_REFLASHABLE=
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
CONFIG_SECURE_BOOT_SIGNING_KEY="keys/secure_boot_signing_key.pem"
CONFIG_SECURE_BOOT_INSECURE=
CONFIG_FLASH_ENCRYPTION_ENABLED=y
CONFIG_FLASH_ENCRYPTION_INSECURE=
<snip>

Made and flashed the bootloader and the app:

Code: Select all

python /home/esp32/esp/esp-idf/components/esptool_py/esptool/espsecure.py generate_signing_key /home/esp32/tbox_v2/tbox-spp/keys/secure_boot_signing_key.pem
make bootloader
python /home/esp32/esp/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after no_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/esp32/tbox_v2/tbox-spp/build/bootloader/bootloader.bin
make flash

make monitor shows the 'secure boot check fail' message.

Fuses state is:

Code: Select all

espefuse.py v2.2.1
Connecting........_
Security fuses:
FLASH_CRYPT_CNT        Flash encryption mode counter                     = 1 R/W (0x1)
FLASH_CRYPT_CONFIG     Flash encryption config (key tweak bits)          = 15 R/W (0xf)
CONSOLE_DEBUG_DISABLE  Disable ROM BASIC interpreter fallback            = 1 R/W (0x1)
ABS_DONE_0             secure boot enabled for bootloader                = 1 R/W (0x1)
ABS_DONE_1             secure boot abstract 1 locked                     = 0 R/W (0x0)
JTAG_DISABLE           Disable JTAG                                      = 1 R/W (0x1)
DISABLE_DL_ENCRYPT     Disable flash encryption in UART bootloader       = 1 R/W (0x1)
DISABLE_DL_DECRYPT     Disable flash decryption in UART bootloader       = 1 R/W (0x1)
DISABLE_DL_CACHE       Disable flash cache in UART bootloader            = 1 R/W (0x1)
BLK1                   Flash encryption key                              
  = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -/- 
BLK2                   Secure boot key                                   
  = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -/- 
BLK3                   Variable Block 3                                  
  = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W 

Efuse fuses:
WR_DIS                 Efuse write disable mask                          = 384 R/W (0x180)
RD_DIS                 Efuse read disablemask                            = 3 R/W (0x3)
CODING_SCHEME          Efuse variable block length scheme                = 0 R/W (0x0)
KEY_STATUS             Usage of efuse block 3 (reserved)                 = 0 R/W (0x0)

Config fuses:
XPD_SDIO_FORCE         Ignore MTDI pin (GPIO12) for VDD_SDIO on reset    = 0 R/W (0x0)
XPD_SDIO_REG           If XPD_SDIO_FORCE, enable VDD_SDIO reg on reset   = 0 R/W (0x0)
XPD_SDIO_TIEH          If XPD_SDIO_FORCE & XPD_SDIO_REG, 1=3.3V 0=1.8V   = 0 R/W (0x0)
SPI_PAD_CONFIG_CLK     Override SD_CLK pad (GPIO6/SPICLK)                = 0 R/W (0x0)
SPI_PAD_CONFIG_Q       Override SD_DATA_0 pad (GPIO7/SPIQ)               = 0 R/W (0x0)
SPI_PAD_CONFIG_D       Override SD_DATA_1 pad (GPIO8/SPID)               = 0 R/W (0x0)
SPI_PAD_CONFIG_HD      Override SD_DATA_2 pad (GPIO9/SPIHD)              = 0 R/W (0x0)
SPI_PAD_CONFIG_CS0     Override SD_CMD pad (GPIO11/SPICS0)               = 0 R/W (0x0)
DISABLE_SDIO_HOST      Disable SDIO host                                 = 0 R/W (0x0)

Identity fuses:
MAC                    MAC Address                                       
  = 24:0a:c4:05:58:9c (CRC 66 OK) R/W 
CHIP_VERSION           Chip version                                      = 0 -/W (0x0)
CHIP_PACKAGE           Chip package identifier                           = 0 -/W (0x0)

Flash voltage (VDD_SDIO) determined by GPIO12 on reset (High for 1.8V, Low/NC for 3.3V).

The partition table in use is (no factory partition):

Code: Select all

otadata,  data, ota,       0x9000,   0x2000
conf,     data, spiffs,    0xb000,   0x5000
ota_0,    app,  ota_0,   0x10000, 0x180000
ota_1,    app,  ota_1,   0x190000, 0x180000
nvs,      data, nvs,     0x310000, 0x0f0000

Could it be, that turning on secure boot and encrypted flash this way fails, because there is no factory image partition?
Up to now, I had no issues to use 'make flash' to write ota_0 and then do ota updates to use ota_1->ota_0->ota_1...

Markus

[ESP_Angus]

Hi Markus,

What ESP-IDF version are you using? I was able to use the same settings you have here (including partition table) and enable secure boot on a fresh chip just now, running the current Github master branch (77eae33a7).

The "secure boot check fail" message you are getting indicates that the initial secure boot check on the bootloader.bin is failing. The partition table & apps are dealt with at a later stage, by the second stage bootloader, so these are not a factor in this message appearing.

The efuse summary indicates that the bootloader.bin region has been encrypted in place successfully (keys are set, and the relevant efuses are all burned). This shouldn't happen until after the bootloader has verified there's a path to boot a valid app, although as mentioned above it's failing at an earlier stage now.

The early log lines like "load:0x40078000,len:20620" suggest that the bootloader image is being read successfully from flash, as well, so it hasn't become corrupted or accidentally replaced with a plaintext copy.

Are you able to share your bootloader.bin and full sdkconfig with me (either privately via email or here, the bootloader binary only contains the public verification key so it won't compromise your private key by sharing it)?

[Markus Becker]

Hi Angus,

Thank you for the detailed and very helpful response and the test you made

What ESP-IDF version are you using? I was able to use the same settings you have here (including partition table) and enable secure boot on a fresh chip just now, running the current Github master branch (77eae33a7).

Code: Select all

esp32@esp32:~/esp/esp-idf$ git rev-parse --short HEAD
77eae33a

Should be fresh, git status indicates no local changes

Are you able to share your bootloader.bin and full sdkconfig with me (either privately via email or here, the bootloader binary only contains the public verification key so it won't compromise your private key by sharing it)?

Hmm, I have the sources for encrypted/secure mode in an extra branch of my repository, but lost the binary of the bootloader by switching the branch, make clean, make.

Up to now, I never cared about the serial flasher config, now reviewing sdkconfig I find

Code: Select all

CONFIG_ESPTOOLPY_BEFORE_RESET=y
CONFIG_ESPTOOLPY_BEFORE_NORESET=
CONFIG_ESPTOOLPY_BEFORE="default_reset"
CONFIG_ESPTOOLPY_AFTER_RESET=y
CONFIG_ESPTOOLPY_AFTER_NORESET=
CONFIG_ESPTOOLPY_AFTER="hard_reset"

This caused a reset after flashing the bootloader, before flashing the app - which is bad, right?

I did not erase the flash before starting, so after flashing the bootloader there was a valid but unsigned image in ota_0.

On the other hand that does not explain to me how that could cause the bootloader fail to be verified.

Would it be better to first flash the signed image and then the bootloader? My next try would be
Change CONFIG_ESPTOOLPY_AFTER to 'stay in bootloader', then:

Code: Select all

make bootloader
make app
make erase_flash
(flash a prebuild image to the spiffs partition)
make app-flash
flash the bootloader (python esptool.py ... bootloader.bin)
reset/powercycle the chip

Could that be right?

Unfortunately, I have to wait for fresh ESP32 Modules which might take a few days, before I can step ahead.

Markus

[ESP_Angus]

Hi Markus,

Thanks for the detailed update.

This caused a reset after flashing the bootloader, before flashing the app - which is bad, right?

I did not erase the flash before starting, so after flashing the bootloader there was a valid but unsigned image in ota_0.

No. These settings apply for most types of flashing, but if you check the "flash bootloader" command which is printed when secure boot is enabled, this already has "--after no_reset" in the options. This is specifically to avoid the situation you mention where the ESP32 resets before you want it to (for all other flash targets or printed flashing commands, you get the options set in the project configuration).

On the other hand that does not explain to me how that could cause the bootloader fail to be verified.

I'm not aware of a series of events which could lead to the problem you describe just from regular flashing plus unexpected resets.

The secure boot enable efuse (ABS_DONE_0) is only burned after the bootloader image is verified and a secure digest is generated and stored in the flash at offset 0x0. Any reset before the digest is in place should have no permanent effects. After the digest is in place and the efuse is burned, the ESP32 should only produce the "secure boot check fail" error if the bootloader or the digest are corrupted or changed.

Immediately after secure boot is enabled, flash encryption is enabled. Weird things can happen if the first boot is interrupted while the partitions are being encrypted in place, as you end up with only partially encrypted flash contents. However the efuse for flash encryption (FLASH_CRYPT_CNT) is only burned at the end of this process, and your results show this efuse was burned.

This indicates to me that the entire first boot (enabling secure boot then enabling flash encryption) completed without failure.

Hmm, I have the sources for encrypted/secure mode in an extra branch of my repository, but lost the binary of the bootloader by switching the branch, make clean, make.

Sorry to hear that. Do you have the full sdkconfig file that you built that bootloader with? Any chance you could share that, please?

Otherwise... I'm loath to mention "bad hardware", but a faulty flash chip, faulty flash power supply, or signal integrity problems on the SPI flash pins (are they connected to anything else or inserted in a breadboard?) could lead to this kind of problem.

What development board or hardware are you using? Did you ever experience a "bad flash" on this device before, when you had to re-flash before things worked?

[Markus Becker]

Hi Angus

Thanks for commenting, I really appreciate your help.

The board in use is self made. Around a ESP-WROOM-32 module there is a power supply for a wider supply voltage range, an I2C port expander, level IO, RS232, CAN, sd-card support among others.
Regarding reset, boot, flashing: The circuitry equals DEVKIT-C with a pin header for TX, RX, DTR, RTS that connects to an usb-serial adapter.
In seldom cases flashing aborted with errors. Firmware OTA can be done wireless or from a file on sd-card. Again, that failed sometimes. This all can have different reasons.

I think there are several very good reasons to have encrypted flash and secure boot in place, maybe we see one of those right now, as we do not need an attacker to change flash content in an irregular way, if we have a bug in our app

I'm now preparing the next try - to get most out of it, how should I adjust bootloader config (I saw a message, indicating bootloader could get too large)?

Last time, I did not 'make monitor' to avoid interrupting flash encryption half way. On the other hand monitoring output might well be the key to finding the cause...

Which settings and procedure would you suggest?

@Angus regarding sdkconfig: there is nothing secret in it, but I find it wrong to embed it into the message here, seem to be unable to attach it and can not reach you by email. Would you please email me, so I can answer? Sorry for my incompetence

Markus

[Markus Becker]

Finally the following procedure worked well:

Code: Select all

#!/bin/bash
echo erase....
make erase_flash
if (( $? > 0 )) ; then
  echo erase flash failed
  exit
fi
echo spiffs....
python /home/esp32/esp/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0xb000 build/spiffs.bin 
if (( $? > 0 )) ; then
  echo flash spiffs failed
  exit
fi
echo firmware....
make flash
if (( $? > 0 )) ; then
  echo flash firmware failed
  exit
fi
echo bootloader....
python <...>/esptool.py <...>build/bootloader/bootloader.bin
make monitor

To flash the bootloader I used the command printed out by make bootloader.
I used make menuconfig to set sdkconfig to:

Code: Select all

#
# Bootloader config
#
CONFIG_LOG_BOOTLOADER_LEVEL_NONE=
CONFIG_LOG_BOOTLOADER_LEVEL_ERROR=
CONFIG_LOG_BOOTLOADER_LEVEL_WARN=y
CONFIG_LOG_BOOTLOADER_LEVEL_INFO=
CONFIG_LOG_BOOTLOADER_LEVEL_DEBUG=
CONFIG_LOG_BOOTLOADER_LEVEL_VERBOSE=
CONFIG_LOG_BOOTLOADER_LEVEL=2
CONFIG_BOOTLOADER_VDDSDIO_BOOST=y

#
# Security features
#
CONFIG_SECURE_BOOT_ENABLED=y
CONFIG_SECURE_BOOTLOADER_ONE_TIME_FLASH=y
CONFIG_SECURE_BOOTLOADER_REFLASHABLE=
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
CONFIG_SECURE_BOOT_SIGNING_KEY="keys/secure_boot_signing_key.pem"
CONFIG_SECURE_BOOT_INSECURE=
CONFIG_FLASH_ENCRYPTION_ENABLED=y
CONFIG_FLASH_ENCRYPTION_INSECURE=

Note: Trying to set the bootloader log level to anything more detailed led to a binary size larger than 28672bytes which is the allowed maximum as far as I understand, be careful.

Conclusion: I did not really manage to find out what went wrong in the first attempt, but a hardware/flash related error could be possible. First attempt was on a rev.0 chip, successful attempt was on a rev.1 chip.

Thanks a lot for the support

Markus

[ESP_Angus]

Hi Markus,

Glad you got it working and thanks for posting the details of what worked for you.

Out of curiosity, were you flashing a SPIFFS partition the first (failed) time as well?

The fact you've had failed flashes over serial & OTA causes me to agree that an intermittent hardware fault of some kind is quite likely as a root cause.

Last time, I did not 'make monitor' to avoid interrupting flash encryption half way. On the other hand monitoring output might well be the key to finding the cause...

What you're doing (flashing bootloader with "--after no_reset" and then starting monitor) is probably the best, the firmware won't boot until "monitor" resets the board and then you can see the first boot output.

[Markus Becker]

Hi Angus,

Out of curiosity, were you flashing a SPIFFS partition the first (failed) time as well?

well, it was there. As I did not erase the flash, there was a somewhat unclear situation. The SPIFFS contains configuration data, the app never writes to it by now.

Best
Markus

[prasad.gj]

Hi ,

Enabled security features on ESP-WROOM-32 but getting an error.Anyone can help me to come out from the below issues,

Monitoring:

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4
load:0x3fff0034,len:7016
load:0x40078000,len:13072
ho 0 tail 12 room 4
load:0x40080400,len:3896
0x40080400: _init at ??:?

secure boot check fail
ets_main.c 371


Also have verified the signature,

espsecure.py verify_signature --version 1 --keyfile secure_boot_signing_key.pem build/blink.bin
espsecure.py v2.9-dev
Verifying 153920 bytes of data
Signature is valid

[minhbka]

Hi ,

Enabled security features on ESP-WROOM-32 but getting an error.Anyone can help me to come out from the below issues,

Monitoring:

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4
load:0x3fff0034,len:7016
load:0x40078000,len:13072
ho 0 tail 12 room 4
load:0x40080400,len:3896
0x40080400: _init at ??:?

secure boot check fail
ets_main.c 371


Also have verified the signature,

espsecure.py verify_signature --version 1 --keyfile secure_boot_signing_key.pem build/blink.bin
espsecure.py v2.9-dev
Verifying 153920 bytes of data
Signature is valid

Hi,
Did you fixed your issue,
I have similar issue to you.