Invalid magic byte on secure boot with custom partition table offset

[addr55]

Hello. I have an ESP-WROOM-32 ESP32 I'm trying to enable Secure Boot v2 on.

I've gone through menuconfig and selected "Enable secure boot version 2"
When I try to flash I get an error saying the bootloader is too large:

[100/101] cmd.exe /C "cd /D D:\Dev\esp-idf\hello_world\build\bootloader\esp-idf\esptool_py && D:\Software\Espressif\python_env\idf5.2_py3.11_env\Scripts\python.exe D:/Software/Espressif/frameworks/esp-idf-v5.2.1/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 D:/Dev/esp-idf/hello_world/build/bootloader/bootloader.bin"
FAILED: esp-idf/esptool_py/CMakeFiles/bootloader_check_size D:/Dev/esp-idf/hello_world/build/bootloader/esp-idf/esptool_py/CMakeFiles/bootloader_check_size
cmd.exe /C "cd /D D:\Dev\esp-idf\hello_world\build\bootloader\esp-idf\esptool_py && D:\Software\Espressif\python_env\idf5.2_py3.11_env\Scripts\python.exe D:/Software/Espressif/frameworks/esp-idf-v5.2.1/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 D:/Dev/esp-idf/hello_world/build/bootloader/bootloader.bin"
Error: Bootloader binary size 0x8560 bytes is too large for partition table offset 0x8000. Bootloader binary can be maximum 0x7000 (28672) bytes unless the partition table offset is increased in the Partition Table section of the project configuration menu.

I followed the suggestion of reducing verbose level but it didn't reduce the size enough
So my next step was to move the partition table offset to make room for the bootloader
By default, it was set to 0x8000, I increased it to 0x9600

Now the flashing process succeeds, but the device constantly error with this:

I (60) boot: ESP-IDF v5.2.1 2nd stage bootloader
I (60) boot: compile time Apr 29 2024 17:14:56
I (60) boot: Multicore bootloader
I (65) boot: chip revision: v3.1
I (69) boot.esp32: SPI Speed : 40MHz
I (74) boot.esp32: SPI Mode : DIO
I (78) boot.esp32: SPI Flash Size : 4MB
I (83) boot: Enabling RNG early entropy source...
I (88) boot: Partition Table:
I (92) boot: ## Label Usage Type ST Offset Length
I (99) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (106) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (114) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (122) boot: 3 factory factory app 00 00 00010000 00100000
I (129) boot: 4 ota_0 OTA app 00 10 00110000 00100000
I (137) boot: 5 ota_1 OTA app 00 11 00210000 00100000
I (144) boot: End of partition table
I (149) boot: Defaulting to factory image
E (153) esp_image: image at 0x10000 has invalid magic byte (nothing flashed here?)
E (161) boot: Factory app partition is not bootable
E (167) esp_image: image at 0x110000 has invalid magic byte (nothing flashed here?)
E (175) boot: OTA app partition slot 0 is not bootable
E (181) esp_image: image at 0x210000 has invalid magic byte (nothing flashed here?)
E (189) boot: OTA app partition slot 1 is not bootable
E (195) boot: No bootable app partitions in the partition table

If I disable Secure Boot entirely, everything works again.
What am I missing? Thanks

[ESP_adokitkat]

Hello.

Setting up Secure boot is by just switching it on in menuconfig won't work, you have to follow a set of steps to enable it properly. Please read through this documentation page: https://docs.espressif.com/projects/esp ... ot-v2.html

Did you follow the instructions there under "How To Enable Secure Boot V2" section"? There are also restrictions when Secure boot V2 is enabled properly, which you can find in "Restrictions After Secure Boot Is Enabled" section.

[addr55]

Hi, and thank you for your response.
I did read the documentation you provided. My apologies if I didn't make that clear initially

Here's my workflow:

Creating the project

idf.py create-project secureboot_test
cd secureboot_test

Writing some sample hello world code

nano ./main/secureboot_test.c

Enabling secure boot

menuconfig -> Component Config -> Hardware Settings -> Chip revision -> Minimum Supported ESP32 Revision (Rev v3.1 (ECO4))
menuconfig -> Security features -> Enable hardware Secure Boot in bootloader (version 2)

Creating the signing key

openssl genrsa -out secure_boot_signing_key.pem 3072

Attempting to build

idf.py build

It is at this point that I receive

Error: Bootloader binary size 0x9dd0 bytes is too large for partition table offset 0x8000. Bootloader binary can be maximum 0x7000 (28672) bytes unless the partition table offset is increased in the Partition Table section of the project configuration menu.

So I try optimizing the bootloader for size and increasing the partition table offset

menuconfig -> Bootloader config -> Bootloader log verbosity (Error)
menuconfig -> Partition table -> (0x10000) Offset of partition table

This allows me to do

idf.py build
idf.py bootloader

successfully but it creates the "Invalid magic byte" output I shared earlier

[ESP_Mahavir]

The error here points to no legit application binary on the flash. Can you please confirm that you are flashing application binary at either factory/ota_0/ota_1 partition offset?

[addr55]

Hey, it looks like I've partially sorted out the issue. I was using this command before:

idf.py flash

It worked fine when Secure Boot was disabled in menuconfig, but threw an "invalid magic byte" error when Secure Boot was enabled.
Here's what I'd see in the console while flashing, if it helps:

...
esptool.py --chip esp32 -p /dev/ttyUSB0 -b 460800 --before=default_reset --after=no_reset write_flash --flash_mode dio --flash_freq 40m --flash_size keep 0x20000 secureboot_test.bin 0x10000 partition_table/partition-table.bin 0x15000 ota_data_initial.bin
esptool.py v4.8.dev3
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ec:64:c9:85:1f:74
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00020000 to 0x00050fff...
Flash will be erased from 0x00010000 to 0x00010fff...
Flash will be erased from 0x00015000 to 0x00016fff...
Compressed 200704 bytes to 98033...
Writing at 0x0004644e... (100 %)
Wrote 200704 bytes (98033 compressed) at 0x00020000 in 2.7 seconds (effective 600.4 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 139...
Writing at 0x00010000... (100 %)
Wrote 3072 bytes (139 compressed) at 0x00010000 in 0.0 seconds (effective 506.8 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 31...
Writing at 0x00015000... (100 %)
Wrote 8192 bytes (31 compressed) at 0x00015000 in 0.0 seconds (effective 2761.6 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.
[100%] Built target flash
Done

Instead, I am now using

esptool.py write_flash 0x10000 build/secureboot_test.bin

This seems to produce a working application, though with some errors:

I (280) heap_init: Initializing. RAM available for dynamic allocation:
I (287) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (293) heap_init: At 3FFB2A70 len 0002D590 (181 KiB): DRAM
I (299) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (305) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (312) heap_init: At 4008BF4C len 000140B4 (80 KiB): IRAM
I (320) spi_flash: detected chip: generic
I (323) spi_flash: flash io: dio
E (327) secure_boot: Mismatch in secure boot settings: the app config is enabled but eFuse not
I (337) main_task: Started on CPU0
I (347) main_task: Calling app_main()
Hello from ESP
Hello from ESP
Hello from ESP

I'm unsure if this is the right way to do things, and I thought 0x10000 would be where the partition table lives, and the application itself would be a bit further down. But I can live with that for now