Help needed - New ESP32 PICO D4.1 design continously looping with RTCWDT_RTC_RESET

[deandob]

Hi,

I have a new board design based on the ESP32 PICO v4.1 development board and while I can program it, it goes into endless reboots at the end of the programming. I'm also fairly new to ESP32 having used a number of modules successfully but this is my first board design. Any hints on what else to look for would be very much appreciated as I'm stuck at this point having spent hours troubleshooting.

As this is a new board I suspect the issue is one of 4 possible issues:
- Error in the schematic. I triple checked the schematic for the D4.1 board and everything is the same as this reference design.
- Faulty soldering / components. Have made 2 of these boards, being very careful with the build (good PCB, using a stencil, solder paste, hot air station) and they both exhibit the same symptoms, so its unlikely to be a solder short/open circuit the PICO QFN pins
- Incorrect configuration. This I'm not sure about, I have selected the defaults in MENUCONFIG and the Arduino settings, its possible I'm missing a configuration setting that isn't default for the PICO. I need help here to identify any specific configuration I should be making in Arduino config or MENUCONFIG for the PICO D4.
- Bad power supply. I'm using a switching regulator to get a bit more efficiency compare to a LDO. On an oscilloscope the power traces are rock solid, no glitches or harmonics. I'm also using a USB cable that works with the other ESP32 modules I have.

A couple of points about the board:
- IO12 is open circuit (pull down for flash power 3.3v) as per PICO datasheet
- I have ceramic 10uf / 0.1uf caps close to the pins for decoupling but even if these were bad the PICO has internal power supply decoupling as well.
- None of the IO pins have significant current drain (the LCD backlight uses a transistor buffer).
- The board seems to program fine, with no errors on the console while flashing, so I'm assuming the PICO hasn't had problems with board assembly (eg. shorts on the pins).
- I tried IDF and Arduino to flash, doesn't make a difference
- In Arduino I tried a number of different boards in the Arduino tools settings, didn't make a difference (default tried was ESP32 PICO kit)
- The ESP32 does not get warm
- Tried erasing the flash from the IDF command line, no difference
- I only have a LCD connected and it was working fine with the same pins in a prototype using a WROOM module. These IO lines are used: 23, 18, 32, 5
- EN line is stable high when viewing on an oscilloscope

----------------------------
Code
void setup() {
// put your setup code here, to run once:
Serial.println("Init code");
}

void loop() {
// put your main code here, to run repeatedly:
Serial.println("In the loop");
}

------------------------------
Programmer settings (Arduino)
Board: ESP32 Dev Module (also tried ESP32 PICO Kit)
Upload speed: 115200
Flash frequency: 40Mhz
Flash mode: DIO
Flash size: 4Mb

-----------------------------
Console output during flashing
Sketch uses 174900 bytes (13%) of program storage space. Maximum is 1310720 bytes.
Global variables use 13340 bytes (4%) of dynamic memory, leaving 314340 bytes for local variables. Maximum is 327680 bytes.
esptool.py v2.5.0
Serial port COM8
Connecting....
Chip is ESP32-PICO-D4 (revision 1)
Features: WiFi, BT, Dual Core, Embedded Flash, VRef calibration in efuse
MAC: d8:a0:1d:64:ad:8c
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 8192 bytes to 47...

Writing at 0x0000e000... (100 %)
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.0 seconds (effective 5957.9 kbit/s)...
Hash of data verified.
Compressed 13696 bytes to 8946...

Writing at 0x00001000... (100 %)
Wrote 13696 bytes (8946 compressed) at 0x00001000 in 0.8 seconds (effective 138.5 kbit/s)...
Hash of data verified.
Compressed 175040 bytes to 92345...

Writing at 0x00010000... (16 %)
Writing at 0x00014000... (33 %)
Writing at 0x00018000... (50 %)
Writing at 0x0001c000... (66 %)
Writing at 0x00020000... (83 %)
Writing at 0x00024000... (100 %)
Wrote 175040 bytes (92345 compressed) at 0x00010000 in 8.2 seconds (effective 171.8 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 144...

Writing at 0x00008000... (100 %)
Wrote 3072 bytes (144 compressed) at 0x00008000 in 0.0 seconds (effective 1293.5 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

------------------------------
Serial port output (Arduino or IDF doesn't make a difference). Below is the output of the serial terminal after a reset.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DOUT, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:8488
load:0x40078000,len:10788
load:0x40080400,len:6832
entry 0x40080798
D (76) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)[0m
ets Jun 8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DOUT, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:8488
load:0x40078000,len:10788
load:0x40080400,len:6832
entry 0x40080798
D (77) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)[0m
ets Jun 8 2016 00:22:57

(the second block repeats every 500mSec)

[ESP_igrr]

Do you have another board with ESP32-PICO, which is known to work? Trying to flash the same binary to it can help determine whether the issue is in the board or in the binary.

[deandob]

I don't have PICO modules, I have been using the TTGO and LOLIN32 modules without troubles. I can flash the simple print program on the TTGO modules and it works, while my PICO boards continue to reboot. I have also tried flashing other working binaries from the TTGO boards and the new PICO boards still reset in a loop.

Is there anything special about the PICO (efuse, menuconfig settings) needed? Reading the datasheets the only thing I can pick up is keeping IO12 unconnected so that the flash has 3.3v (which it does with a voltmeter on VDD_SDIO).

I'm running out of ideas - there are a couple of older posts with other people having similar problems with the PICO in a reset loop https://www.esp32.com/viewtopic.php?f=1 ... ico#p26387 but no resolution documented.

Much appreciated any help as I'm on a project deadline and the PICO was supposed to be an easy implementation as all that is needed is to apply power....

[deandob]

Hi,

I'm out of ideas why this is happening, having checked everything from schematics, board assembly, compile options, power supply etc. It seems that the code is being loaded from flash as I can see the binary sequences on the flash pins on an oscilloscope.

Even if someone could outline the usual causes for RTCWDT_RTC_RESET looping on startup to give me some clues to what else to look at?

Any help gratefully appreciated - am about to jump back to STM devices as I can't get this working and have a project deadline to meet (already delayed).

Are there options for paid support from Espressif (or third parties) to help me sort this out?

[ESP_Angus]

If you set the bootloader to DIO mode instead of DOUT, and 20MHz instead of 40MHz, does it make any difference to the log output? (These settings are under the Serial Flasher Config menu for ESP-IDF menuconfig, or in the Arduino IDE menus.)

(The PICO should work at the settings you have, but it's not clear what the root cause is so it may be worth trying some other variations.)

Do you only have one prototype board built at the moment, or is this happening on multiple boards?

What ESP-IDF and Arduino version(s) are you using?

[deandob]

Angus, thanks for the reply.

I have tried DIO, QIO and DOUT modes, and have kept the flash at 20Mhz for all the recent tests. No luck.

I have 2 boards hand made, each displaying the same symptoms / serial output.

Some ideas that I need feedback on:
- Its still possible that there is a solder bridge or o/c under the QFN, I don't have x-ray facilities to check but I have been careful with the stencil and the amount of paste deposited should be OK (not too much or too little). However its still soldered manually with a manual reflow profile so there is a chance that there is a soldering fault, even across 2 boards. The blank PCBs have been checked and they are OK (no shorts or open circuits). And I assume that if I can flash the chip chances are the PCB/soldering/IC is OK - is this a fair assumption?

- I may have a faulty batch of PICO chips or even counterfeits (they were ordered from LCSC so should be OK....). I can order another batch from a different supplier but I think this is a long shot.

- Looking at the power supply its very stable but there is some ringing at 240Khz (80mV) which is attributable to the SMPS I'm using. I compared the power supply on one of the WEMO modules and its power supply isn't as stable (200mV ringing + other noise).

- Would it help if I disable WiFi and Bluetooth to see if this is the problem? There doesn't seem to be a way to do this in MENUCONFIG and from what I have read if you aren't including those functions in your code they will be disabled anyway.

- I am using the latest Arduino (1.8.7) & IDF with fresh installs from GitHub. If you think it is likely a software issue I can rebuild, but the issue is the same with both Arduino and IDF testing.

- What are the typical issues that cause the RTCWDT_RTC_RESET reboot loop? Typically hardware or software issues? Anything else I could try?

- It doesn't seem to get past the initial bootloader to run the program, and as the bootloader was flashed along with the program, the flash should be OK (I can see flash pins SDO/CLK etc. toggling on the oscilloscope after restart) but it doesn't get any further to launch the program. Are there any specific settings in MENUCONFIG that I should be setting for the PICO?

I can post the schematics and board layout if that helps.

[ESP_Angus]

I assume that if I can flash the chip chances are the PCB/soldering/IC is OK - is this a fair assumption?

For booting in dual I/O mode, this is probably a reasonably fair assumption. Quad I/O mode adds an additional two data lines which are not used in dual mode, but that shouldn't be relevant here.

The only other difference is that the SPI clock speed is higher when booting compared to flashing. Is there anything at all connected to GPIOs 6, 7, 8, 11 16, or 17 that might be causing interference or increasing the pin's capacitance? (These pins are shared with the internal SPI flash).

(Note these are GPIO numbers not package pin numbers, consult the PICO-D4 datasheet for the pin numbers on the package.)

- I may have a faulty batch of PICO chips or even counterfeits (they were ordered from LCSC so should be OK....). I can order another batch from a different supplier but I think this is a long shot.

I wouldn't expect problems with chips purchased from LCSC.

- Looking at the power supply its very stable but there is some ringing at 240Khz (80mV) which is attributable to the SMPS I'm using. I compared the power supply on one of the WEMO modules and its power supply isn't as stable (200mV ringing + other noise).

Yes, 80mV p/p seems pretty harmless.

- Would it help if I disable WiFi and Bluetooth to see if this is the problem? There doesn't seem to be a way to do this in MENUCONFIG and from what I have read if you aren't including those functions in your code they will be disabled anyway.

No, the chip seems to be crashing very early in the boot process, while the second stage bootloader is running and before the app has even been loaded.

- I am using the latest Arduino (1.8.7) & IDF with fresh installs from GitHub. If you think it is likely a software issue I can rebuild, but the issue is the same with both Arduino and IDF testing.

So, recent IDF master branch? Running "git describe --tags --dirty" in the IDF working directory will give you the exact version.

It looks like you might have already tried this, but can you try getting a simple example app in IDF (like getting-started/hello_world), run menuconfig and set the Bootloader log level to Verbose while leaving all other settings at default, then build and flash it (make flash or idf.py flash), and see if you get any additional log output?

- What are the typical issues that cause the RTCWDT_RTC_RESET reboot loop? Typically hardware or software issues? Anything else I could try?

Given the log output, it looks like the ROM bootloader successfully loads the second stage bootloader (indicating no large-scale problems with the flash communications) and then run the second stage bootloader. The second stage bootloader runs for a bit (at least one log line is partially printed) and then gets stuck/wedged and the whole system is reset by the RTC WDT due to inactivity.

This can be caused by hardware or software, but if the same software runs on other boards you have (and IDF examples certainly run on the PICO-KIT baords I have), then it is probably hardware related.

I can post the schematics and board layout if that helps.

Sure, if you can then there might be something useful there.

Can you also please post the output of "espsecure.py -p port summary", just to check nothing weird in the chip's efuse configuration?

[deandob]

Hi Angus,

Thank you for the detailed troubleshooting tips, very helpful. I've spent some time today diving deeper into this problem, summary below.

Please see the linked PDF for oscilloscope snapshots of the GPIO lines exposed for the internal flash. Definitely something not right with some of these waveforms (although my oscilloscope probes aren't very good). https://1drv.ms/b/s!ApUB68SilQqFhuMzLHlyS3dn1-D2HA

Re: GPIO's used for the internal flash (GPIOs 6, 7, 8, 11 16, or 17), they are all unconnected and not shorted.

IDF version from GIT describe: v3.3-dev-206-g0d7f2d77c

Also, I get a slightly different output when comparing the IDF flash to the Arduino flash.

Arduino flashed (board "ESP Dev module", upload speed 115200, 40Mhz, 4Mb, DIO, verbose) serial output when reset:
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:952
load:0x40078000,len:6084
load:0x40080000,len:7944
entry 0x40080310
ets Jun 8 2016 00:22:57

IDF flashed (upload 115200, 20Mhz, 4Mb, DIO, verbose) with getting started/hello_world serial output when reset (and I tried setting CPU frequency to 80Mhz, output same as when set to 160Mhz):
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:4
load:0x3fff0018,len:4
load:0x3fff001c,len:8488
load:0x40078000,len:10788
load:0x40080400,len:6832
entry 0x40080798
D (94) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)[0m
ets Jun 8 2016 00:22:57

----------------------------------
esp fuses (note I earlier burnt setting the VDD_SDIO to 3.3v just to be sure)
espefuse.py v2.6-beta1
Connecting....
EFUSE_NAME Description = [Meaningful Value] [Readable/Writeable] (Hex Value)
----------------------------------------------------------------------------------------
Security fuses:
FLASH_CRYPT_CNT Flash encryption mode counter = 0 R/W (0x0)
FLASH_CRYPT_CONFIG Flash encryption config (key tweak bits) = 0 R/W (0x0)
CONSOLE_DEBUG_DISABLE Disable ROM BASIC interpreter fallback = 1 R/W (0x1)
ABS_DONE_0 secure boot enabled for bootloader = 0 R/W (0x0)
ABS_DONE_1 secure boot abstract 1 locked = 0 R/W (0x0)
JTAG_DISABLE Disable JTAG = 0 R/W (0x0)
DISABLE_DL_ENCRYPT Disable flash encryption in UART bootloader = 0 R/W (0x0)
DISABLE_DL_DECRYPT Disable flash decryption in UART bootloader = 0 R/W (0x0)
DISABLE_DL_CACHE Disable flash cache in UART bootloader = 0 R/W (0x0)
BLK1 Flash encryption key
= 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W
BLK2 Secure boot key
= 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W
BLK3 Variable Block 3
= 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W

Efuse fuses:
WR_DIS Efuse write disable mask = 0 R/W (0x0)
RD_DIS Efuse read disablemask = 0 R/W (0x0)
CODING_SCHEME Efuse variable block length scheme = 0 R/W (0x0)
KEY_STATUS Usage of efuse block 3 (reserved) = 0 R/W (0x0)

Config fuses:
XPD_SDIO_FORCE Ignore MTDI pin (GPIO12) for VDD_SDIO on reset = 1 R/W (0x1)
XPD_SDIO_REG If XPD_SDIO_FORCE, enable VDD_SDIO reg on reset = 1 R/W (0x1)
XPD_SDIO_TIEH If XPD_SDIO_FORCE & XPD_SDIO_REG, 1=3.3V 0=1.8V = 1 R/W (0x1)
SPI_PAD_CONFIG_CLK Override SD_CLK pad (GPIO6/SPICLK) = 6 R/W (0x6)
SPI_PAD_CONFIG_Q Override SD_DATA_0 pad (GPIO7/SPIQ) = 17 R/W (0x11)
SPI_PAD_CONFIG_D Override SD_DATA_1 pad (GPIO8/SPID) = 8 R/W (0x8)
SPI_PAD_CONFIG_HD Override SD_DATA_2 pad (GPIO9/SPIHD) = 11 R/W (0xb)
SPI_PAD_CONFIG_CS0 Override SD_CMD pad (GPIO11/SPICS0) = 16 R/W (0x10)
DISABLE_SDIO_HOST Disable SDIO host = 0 R/W (0x0)

Identity fuses:
MAC MAC Address
= d8:a0:1d:64:ad:8c (CRC 80 OK) R/W
CHIP_VER_REV1 Silicon Revision 1 = 1 R/W (0x1)
CHIP_VERSION Reserved for future chip versions = 0 R/W (0x0)
CHIP_PACKAGE Chip package identifier = 5 R/W (0x5)

Calibration fuses:
BLK3_PART_RESERVE BLOCK3 partially served for ADC calibration data = 0 R/W (0x0)
ADC_VREF Voltage reference calibration = 1149 R/W (0x7)

Flash voltage (VDD_SDIO) set to 3.3V by efuse.

----------------------------
I'm assuming there really isn't that much that should go wrong with the PICO hardware wise as all the sensitive components (eg. xtal) are inside the SIP. But if there is an open connection on one of the power supply pins, could this be the cause?

Inspecting under a microscope shows that both boards that I have populated for testing should be OK as the chip is firmly squat down on the PCB, pads extruding from the IC look like they have reflowed properly with no sign of shorts or opens. I tried reflowing one of the boards again, and an additional 0.1uF ceramic close to the power pins, but no change in symptoms.

Is it fair to assume that the board layout for a PICO isn't as critical as with the standard ESP32 chips? I have followed all the guidance in the ESP32 hardware design guidelines - using a double sided board with as much of a ground plane below the PICO and around the PICO as possible, reasonable sized VCC and GND connections, decoupling ceramics close to the pins, rf traces short, lots of vias to the ground plane. However this board needs to be tiny due to product space constraints so there is one via in the VCC line but it should be OK for the amount of current being drawn.

If you think the problem is likely to be hardware, I'll make up another board (or send it away to be made by someone who has better facilities than I do), as there is always some doubt about the paste reflow underneath the IC body. The oscilloscope waveforms do look suspect though - not sure exactly why.

Again, thanks for the help, its appreciated.

[deandob]

Hi Angus / ESP team,

I hate to push this, but we still need help to identify the source of the reboots at startup, as this is holding up our team. As mentioned previously, we can pay for support if needed to expedite (our product is expected to sell 20K+ units of ESP32 PICO and this product is our evaluation to see if its worth switching from STM....).

See my posting from yesterday above with answers to Angus's questions about our configuration, especially check the linked PDF with waveforms from the internal flash, seems like there is some unwanted (significant) capacitance somewhere effecting the waveforms but it only shows up occassionally, especially when comparing to the waveforms from an ESP module with an external flash which is much more stable.

As the internal flash pins are not connected (we do utilise GPIO9 and 10 but these should be OK), I'm thinking we either have a bad batch of chips or one of the VCC power pins hasn't soldered correctly under the QFN chip.

Input please.

[ESP_igrr]

If you can do a few experiment with the 2nd stage bootloader, it could help narrow down the cause of the issue. Inside IDF, in https://github.com/espressif/esp-idf/bl ... der_init.c there is a bootloader_main function. Try adding additional log messages between various calls, such as flash_gpio_configue, bootloader_clock_configure, etc. Add some delay (ets_delay_us(10000);) after each log message to make sure they are printed fully before proceeding to the next function.
Given that the observed watchdog is RTC watchdog, and not TG watchdog (which has shorter timeout), it is likely that the digital part of the chip is not working. This may be a result of some issue during clock switch, e.g. that the PLL does not lock.
If log messages confirm that the execution does not proceed past bootloader_clock_configure, next step would be to modify some RTC code to not enable the PLL and instead keep running at XTAL frequency.