ESP32 Manufacturing QC issues

[PaulNi]

Great idea WiFive, sadly no clock issues with good or bad modules, generated a 10 and 100Khz pwm they are all rock solid and spot on frequency wise.

My guess will be power, and if it isn't, the only possible source of variability other than from the chip itself will be from in-package components.

If it isn't burdensome, can you try transplanting chip from one module to another and seeing if it changes anything?

After that, I would've referred to an RF lab to see if it may be something with an RF tract. Check antenna swr with chip off in between good and bad modules.

And after that, the only remedy will be sandpaper, acid, and an electron microscope.

[Foxabilo]

I did a quick check of power, both good and bad pull ~150mA when receiving, but I will put the LDO through its paces and check for ripple and transients. I'll see if Angus has any more thoughts after reading my posts and then get to more... old school methods, your idea about swapping the esp chips is a good one, should not be a huge job to rework them across, not like I am short on spares if I mess up.
The fact that a bad unit will, after a little use, jump up a power of 2 in speed reminds me of LNA's with switching gain from my GPS days, I just wish I could decipher the strange difference in the Wireshark captures but unfortunately HTTP protocol level stuff is not my bag and I don't have anyone on staff that is, pitty. Well, I won't call the university up just yet to reserve some electron microscopy time but I sure would like to put this issue to bed and get on with making green bits of paper rather than magic smoke.

Cheers for the suggestions Paul and please if anyone has an idea or something to try just shout up, at this point nothing is a silly thing to try.

[WiFive]

From the Wireshark log it looks like for the good one the ack is coming in a few ms but the other the ack is taking 100ms possibly due to retries.

Have you reproduced this with different access point types?

Also does your test firmware do full rf calibration every boot?

[ESP_Angus]

Hi Foxabilo,

Thanks for the extra details. If you're using ESP32-WROOM modules then this is enough information to go on, I wanted to clarify if any of the RF circuit was custom or from a third party module vendor but it seems like this part is all from Espressif.

Looking at the TCP/IP packet capture, it does look like ACKs from the ESP32 to the other side are being delayed or lost somehow - the ACKs are coming several TCP packets late, and by the time the ESP32 sends the ACK then the available receive window size is always maximum again, implying that the ESP32 TCP/IP stack has processed the previously received bytes already but hadn't managed to "ACK" this yet. Whereas in the "good" capture the ACKs come more directly and you can see the available window size reduce as data is queued up in the TCP/IP stack for processing by the application layer.

An unusual thing is that there are no duplicate or redundant ACK packets in the "bad" capture, and they're consistently with the full window size (at last for the ones visible in the screen grab). When debugging RF problems I would expect a few retries and some random window size values, as WiFi has it's own ack/retry layer underneath the IP layer.

Suggest looking into three things, if possible:

- Check that no other variation in the ESP32 firmware may be causing unusual timing problems or resource starvation inside the firmware itself. The only thing I can think of is variation between SD cards, if you take the SD writes out of the equation (so the ESP32 just reads packet data and discards it), is there any difference? This is less likely, but worth ruling out. If the firmware is doing anything else concurrently with this transfer, try disabling that as well.

- Try to get a raw wifi "monitor mode" capture of the Wi-Fi data. Some details can be found here. Essentially this gives a grab of all the raw Wi-Fi frames on the same channel as the network you're debugging. You won't be able to decode the Wi-Fi content due to WPA encryption (assuming Wi-Fi network is not open), you just see the metadata around each frame. However if there are RF problems then you'll see frame retransmits happening at the Wi-Fi layer, that are not visible at the TCP/IP layer.

- Try forcing a full RF calibration on each boot: https://docs.espressif.com/projects/esp ... alibration

[Foxabilo]

From the Wireshark log it looks like for the good one the ack is coming in a few ms but the other the ack is taking 100ms possibly due to retries.

Have you reproduced this with different access point types?

Also does your test firmware do full rf calibration every boot?

Hi WiFive,

Yes I have various WiFi environments for product tests, I'm sure you're the same when you have a new product you carry a demo rig in your car for impromptu testing, I've tried it on everything from free wifi at MacDonald's to my parents ancient BT box to smartphones, so I think I've got that base covered, I just saw about the wifi calibration thing in Angus' post too, never knew it could do that, I'll 100% try that. Thankyou for your reading on the Wireshark logs your interpritation makes complete sense.

[Foxabilo]

Hi Angus,

Thank you for that reply, I'll get right on the calibration option, never even knew that was a thing, so I am crossing my fingers that's it, I'll get the Raw Wifi capture environment setup today.

The SD card write gets skipped over if the unit does not detect the SD card but I have physically commented that code out to make sure it was not creating issues, the speed goes from about 450KB/s to about 680KB/s when not writing data.

There are indeed some duplicate ACK's and retry's in the log file, just not visible in the screenshot I posted.

Right, I'll get on with trying the various bits mentioned and let you know the results.

Cheers All.

[Foxabilo]

Hurray!

I erased the devices Flash to force an RF calibration and job done, all the bad units are now performing at full speed.

Thank you very much people, excellent diagnostic skills and thank you for a pleasant support experience, staffers and dedicated enthusiasts alike well done.

Thanks for all the fish!

Foxabilo

[Foxabilo]

Ohh, almost forgot, I am working in the PlatformIO Arduino environment, any way to get the Audrino implementation to force a calibration on every re-boot?

[rudi ;-)]

..

- Try forcing a full RF calibration on each boot: https://docs.espressif.com/projects/esp ... alibration

edit stable link (404)
.. working full RF calibration last link

Hurray!

I erased the devices Flash to force an RF calibration and job done, all the bad units are now performing at full speed.

Hurray!

well done!

best wishes
rudi

[Foxabilo]

Bit of an update, about half of the units with low speed responded positively to having the Flash reset to cause an RF recalibration but the half is stubbornly refusing to play ball and remain at the slow speed, I will try the idea mentioned about swapping CPU's from a good board to a bad board to see if the remaining issue is RF. Power is good with low ripple and no capturable transients on a 1GHz scope.