ESP32 flash verify fails after a while

[SpaeniDario]

Hi,
Currently struggling with a strange phenomenon:
Some of the used ESP32-WROOM-32E-N16 modules in a custom device stop flashing properly after they have worked properly multiple times (often 10+) before.

The flashing itself completes fine, but the verify using the MD5 checksum fails (see attachments). Once this happens, any subsequent flashing, even a different binary, fails.
The same binary then works fine with another module.

I normally use an Olimex ARM-USB-OCD-H debugger for flashing (@20Mhz or 3MHz Jtag, no difference), but as a sanity check I've also used serial flashing, as well as flashing a known good binary (ESP RF Test). The result is always that the verify fails.
Afterwards, the ESP still runs after a reset, but the running program is the last program which was flashed properly.

This tells me that the writes to the flash ic stopped working, but reads are still fine. How is this possible?

[ESP_Sprite]

Is your GPIO12 perhaps connected to something? That is a bootstrap pin that sets the flash voltage, if that is wrong you'll get issues like that.

[SpaeniDario]

GPIO12 is only connected to a debug header, no other external component. I'm measuring 0.0V on this pin.
Wouldn't flash reads also fail if GPIO12 was pulled high?

[ESP_Sprite]

0V is correct for that module. (And if the setting were wrong, you would get spurious results from any flash activity... the thing is that flash chips mostly work with the wrong supply voltage. Mostly.)


Can you run
esptool.py flash_id
esptool.py read_flash_status
on the broken module and post the output here?

[SpaeniDario]

Sure, flash_id:

Code: Select all

C:\Users\Embedded>esptool -p COM7 flash_id
esptool.py v4.7.0
Serial port COM7
Connecting......
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip is ESP32-D0WDR2-V3-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: b0:b2:1c:f3:3a:70
Uploading stub...
Running stub...
Stub running...
Manufacturer: 68
Device: 4017
Detected flash size: 8MB
Hard resetting via RTS pin...

read_flash_status:

Code: Select all

C:\Users\Embedded>esptool -p COM7 read_flash_status
esptool.py v4.7.0
Serial port COM7
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting......
Detecting chip type... ESP32
Chip is ESP32-D0WDR2-V3-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: b0:b2:1c:f3:3a:70
Uploading stub...
Running stub...
Stub running...
Status value: 0x4200
Hard resetting via RTS pin...

[ESP_Sprite]

Interesting. According to the datasheet, the CMP bit got set, which combined with the default values of all other things makes the entire flash write-protected, which would give you exactly the issues you saw.

I think you can reset that bit by doing a
esptool.py write_flash_status -non-volatile 0x0200
but you'd have to try that. I'm not entirely sure why that bit got set either; it's not something esp-idf normally messes with.

[SpaeniDario]

Hmm, changing the status as well as flashing a binary (ESP32_RFTest) worked, but then things got even wierder...

The actual command I've used was:

Code: Select all

esptool -p PORT write_flash_status --non-volatile 0x0200

I then verified that the setting has actually changed:

Code: Select all

C:\Users\Embedded>esptool -p COM7 read_flash_status
esptool.py v4.7.0
Serial port COM7
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting..........
Detecting chip type... ESP32
Chip is ESP32-D0WDR2-V3-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: b0:b2:1c:f3:3a:70
Uploading stub...
Running stub...
Stub running...
Status value: 0x0200
Hard resetting via RTS pin...

Afterwards I flashed the ESP32_RFTest binary, this time verify was successful. However, the ESP stopped being responsive and the serial is now sending garbage (hex data output from HTerm, no baudrate 300-256000 gives a reasonable output):

Code: Select all

04 00 00 00 24 80 00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 
00 00 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 00 80 00 00 00 00 00 02 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 04 
00 00 00 00 00 00 00 00 09 24 80 00 00 00 00 00 00 00 00 00 00 00 00 02 01 20 80 00 00 00 00 01 00 3E 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 00 00 00 80 00 80 00 00 00 00 00 00 00 00 00
00 00 00 02 09 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The data (approx 150chars@115200baud at once) is sent periodically every 2.8s, so I'm guessing the ESP is now constantly crashing/boot looping?
What's wierd is that reset to bootloader also stopped working, the same garbage data is sent instead of the usual "Waiting for download...".
I then tried erase_flash, intending to then flash the bootloader only. However, after erase_flash even reading the status got difficult (it works occasionally, if I hit reset just right) and flashing is impossible with the same error:

Code: Select all

C:\Users\Embedded>esptool -p COM7 read_flash_status
esptool.py v4.7.0
Serial port COM7
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting..............
Detecting chip type... ESP32
Chip is ESP32-D0WDR2-V3-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: b0:b2:1c:f3:3a:70
Uploading stub...
Running stub...
Stub running...

A fatal error occurred: Invalid head of packet (0x90): Possible serial noise or corruption.

[ESP_Sprite]

That is very curious, I never have seen anything like that. Can you tell me a bit more about the device you're running the ESP32 in? Does it have sources of EMI (relays, motors, tesla coils etc) nearby, what does the power supply look like, etc?

[SpaeniDario]

The device is used to measure a strain gauge.
Power supply is a pmic, powered by USB and/or a single Li-Ion cell. The output used for the ESP is an LDO@3.15V, max 220mA
The pmic has 2 more LDOs and a boost-converter for the TFT backlight and strain gauge supply.
Components include:
USB-C connected to an FT231XS through ESD protection chip
Front-end for the strain gauge
TFT display, RTC, external EEPROM, MEMS accelerometer

Additionally, the device has recently passed EMI testing without any problems.

[ESP_Sprite]

The output used for the ESP is an LDO@3.15V, max 220mA

That is something to look at: 220mA is really underdimensioned for an ESP32. I think the datasheet specifies a minimum of 500mA.