One byte in ESP32 flash memory is broken!!!

[thienluan]

Dear ESP32 supporter,

Our company is making a product using ESP32. We released nearly 1000 products to our customers using ESP32-WROOM.
At the first stage, we made firmware for ESP32-WROOM. We downloaded our firmware using ESP DOWNLOAD TOOL. And it worked very well. We did test with the full system for a long time (more than 2 months), and there is no any problem. And then, we released nearly 1000 products our customers.
However, among of them, we got reports from customers, there are several products that have the rebooting issue after working for 1 or 2 months. I attached the figure 1 for your reference.

reboot_log.JPG (68.02 KiB) Viewed 14839 times

There is no detail log for debugging. So we dumped the flash memory, written on the reboot ESP32-WROOM and compare with our bin file.
The difference between our bin file and the reboot ESP32 is just one or two bytes. And the symptom happened at different address per each reboot ESP32. I attached two figures for your reference.

compare.JPG (146.28 KiB) Viewed 14839 times

compare1.JPG (157.39 KiB) Viewed 14839 times

It seem that after working for a long time, one or two bytes in the flash memory of ESP32 is broken.
I would like to know why it happened, and please show me the solution to fix this problem.

Thank you very much!

[ESP_Sprite]

Can you give us a bit more information?
- What hardware are you using specifically? What's the power supply like? What specific version of Wroom are you using, with what amount of flash?
- What environment is the product used in? Anything like high temperatures, EMC, ... that could be a risk?
- What esp-idf / Arduino version are you using?
- What does you hardware do with respect to flash writes? Are you using NVS, OTA upgrades, SPIFFS, FatFS, ...? Do you store logs?
- Any chance you can share code and/or schematics somehow?

[thienluan]

- What hardware are you using specifically? What's the power supply like? What specific version of Wroom are you using, with what amount of flash?
. The figure below show the ESPWROM version, we are using. The power supply is 3.3V.

esp32.jpg (50.01 KiB) Viewed 14794 times

- What environment is the product used in? Anything like high temperatures, EMC, ... that could be a risk?
. We're using it to control air filter, ventilator, ... We did test in the high or low temperature environment, and we got the certification
- What esp-idf / Arduino version are you using?
. Arduino version 1.8.3 with ESP32 board version 1.0.1
- What does you hardware do with respect to flash writes? Are you using NVS, OTA upgrades, SPIFFS, FatFS, ...? Do you store logs?
. OTA upgrading function is included but not used yet. We used EEPROM to store some data. We did not store log.
- Any chance you can share code and/or schematics somehow?
. We can share in private inbox

Thanks

[thienluan]

Do we have any way to lock/unlock a region of esp flash memory from writing function?

[thienluan]

NO response?

[ESP_Sprite]

Apologies, this is not a common problem. I've looked into what you wrote plus the schematics a bit longer, and I don't think the flash itself is corrupted by some write from the ESP32. The main reason is that the writes seem to have turned a 0 into an 1. This is not possible without an entire page erase and then writing back the data in that page with that one bit flipped. That does not seem likely, especially that there is nothing in your firmware that I know has an usage pattern like this. The fact that the address of the corruption also seems random adds to the theory that the issue is not programmatic.

Do you know how hot the ESP32 in this circuit can potentially get? Is there anything else that ties the customers having symptoms together?

Edit: A few things more: From the schematic, it doesn't look like it, but is there any way the ESP32 can boot up with IO12 pulled high on reset? That would mess things up as it lets the flash run at 1.8V instead of 3.3V. Also, I see you have a RS485 line... while that does seem to be decoupled pretty well, is there any chance noise could leak in through that?

[Jujufr99]

Hi,

I have exactly the same problem with the ESP32-WROVER. I don't know why but some products, after some months of use, have a flash problem (some bytes modified without any reason). And the ESP32 resets in permanence. I can flash the module, the product is working but some days, weeks or months later, the problem could appear another time. I use Arduino.
Do you find the reason of the problem to help me?
Thank you very much.
Juju