Hardware Flash Corruption Issue

[Ritesh]

Just curious if your issues are related to a known issue with the flash chips manufactured by XMC?

If the output of:

Code: Select all

esptool.py flash_id

gives Manufacturer: 20,
that means you have a chip manufactured by "XMC".
And if the output of:

Code: Select all

esptool.py read_flash_status --bytes 3

gives Status value: 0xe37bfc,
that is indicative of corrupted status register.

If that's the case, head on over to:
https://github.com/espressif/esp-idf/issues/7994

Thanks for your prompt response.

We will check it and let you know results for the same

[Ritesh]

Just curious if your issues are related to a known issue with the flash chips manufactured by XMC?

If the output of:

Code: Select all

esptool.py flash_id

gives Manufacturer: 20,
that means you have a chip manufactured by "XMC".
And if the output of:

Code: Select all

esptool.py read_flash_status --bytes 3

gives Status value: 0xe37bfc,
that is indicative of corrupted status register.

If that's the case, head on over to:
https://github.com/espressif/esp-idf/issues/7994

Also I have checked issue with below link and found that it seems like issue which we are facing is different as per issue described and logs mentioned

https://github.com/espressif/esp-idf/issues/7994

Still we will check flash_id and flash status register and let you know updates for the same.

[scimon]

Hi @Ritesh,

have you taken a look at the bootloader partition of a bricked device? We are currently facing an issue which seems somehow related. I.e. devices in the field that were running for several months are suddenly stuck in a bootloop). In my case I was able to un-brick the failed devices by replacing the bootloader section (surely, a complete flash erase would have fixed the problem as well in my case).
According to the analysis of my problem, some bootloaders seem to be incompatible with firmware images that were compiled with certain compiler/IDF versions. Do your devices experience OTA updates from time to time?

If my issue is somehow related to yours, I'd like to share my data with you here.

[Ritesh]

Hi @Ritesh,

have you taken a look at the bootloader partition of a bricked device? We are currently facing an issue which seems somehow related. I.e. devices in the field that were running for several months are suddenly stuck in a bootloop). In my case I was able to un-brick the failed devices by replacing the bootloader section (surely, a complete flash erase would have fixed the problem as well in my case).
According to the analysis of my problem, some bootloaders seem to be incompatible with firmware images that were compiled with certain compiler/IDF versions. Do your devices experience OTA updates from time to time?

If my issue is somehow related to yours, I'd like to share my data with you here.

Hello,

Would you please share your data as well if seems to be related to me?

Yes. We are using OTA updates regularly whenever required like any bug fixes or any new feature updates.

Let me know if need anything else from my end.

[Ritesh]

Hello Espressif Team,

We have checked and facing issues into multiple sites almost 50 No of sites where product is deployed. So, We are completely stuck due to this type of issues as we need to get back all devices then manually reflash images and then only it can be able to recover.

We have also sent devices to Espressif Team directly but not received any update due to Covid last time.

Let me know if anyone has proper solution or root cause for the issue.

[Ritesh]

Hello Espressif Team,

Let us know if you have any updates regarding this issue because we are completely stuck and products are coming back due to this type of issues.

Because recovery mechanism is too difficult like need to open Product and then re-flash it but again don't know like when issue will happened again.

So, We need to find appropriate root cause for this issue and then need to apply solution to resolve it.

[ESP_Sprite]

At this point in time, your issue is kind-of muddled as you also seem to have run into and solved other issues at this point... Can I summarize the issue as follows:
[*]You get ESP32-Wrover modules that refuse to boot at some point. (Are they really the oldest Wrover modules, and not -E or other newer variants? Please be very specific wrt type number.)
[*]Reading the flash gives no differences between what is flashed and what should be in the flash.
[*]Erasing the flash and re-flashing the (exact same) image makes the thing work again.

There's also a bunch of questions people raised that you never answered:
[*]Can you scan a few QR codes on the modules and post what's in them here?
[*]What flash manufacturer ID do the modules have?

Also:
[*]The first and 2nd log you posted are different in that the 1st has a different calculated checksum every time while the 2nd has the same checksum. What generally is the behaviour you see?
[*]You said you were looking at the LDO powering the ESP. Any results from that research?

[Ritesh]

At this point in time, your issue is kind-of muddled as you also seem to have run into and solved other issues at this point... Can I summarize the issue as follows:
[*]You get ESP32-Wrover modules that refuse to boot at some point. (Are they really the oldest Wrover modules, and not -E or other newer variants? Please be very specific wrt type number.)
[*]Reading the flash gives no differences between what is flashed and what should be in the flash.
[*]Erasing the flash and re-flashing the (exact same) image makes the thing work again.

There's also a bunch of questions people raised that you never answered:
[*]Can you scan a few QR codes on the modules and post what's in them here?
[*]What flash manufacturer ID do the modules have?

Also:
[*]The first and 2nd log you posted are different in that the 1st has a different calculated checksum every time while the 2nd has the same checksum. What generally is the behaviour you see?
[*]You said you were looking at the LDO powering the ESP. Any results from that research?

Hello ESP_Sprite,

We got feedback from few customers like they are facing same issues like they faced earlier before 6 Months on site. So, We have asked to provide modules to us and then will give all required details again for that module.

Yes, After Erasing Flash or Re-flashing firmware images again then module starts to work again. We have also checked LDO Powering section but didn't find anything which can cause this type of issues.

I will check all details for rest of queries and will send on Monday once we receive that faulty module from one customer site.

Hope that would be fine for you.

[Ritesh]

Hello ESP_Sprite,

We have also sent one faulty modules with product to Espressif Team before 6 Months but they didn't check it yet as may be shipment has been lost into their office during last Covid.

SO, They are also checking for shipment and will let us know once they will get it.

[Ritesh]

Hello ESP_Sprite,

We will get one more sample of not working board within 2 days. So, I will send all required details of not working module before end of this week which will be helpful for you to understand it.

Hope that would be worked for you.