Ethernet goes mad

[dmitryga]

Hi everybody!

I've noticed that after some time (several hours, sometimes - 1-2 days) ethernet interface goes into very strange state:
- network is fluded with STP requests
- no valid input and output traffic from ESP (no pings, no TCP/UDP requests, nothing)
Only complete reboot could resolve this issue.
Ethernet is built on LAN8720A chip (similar to Olimex board) and works fine all other time.

After 2 or 3 issues I've connected PC with Wireshark to other end of patch cord and recorded ethernet frames for a while. I see only STP requests and nothing more. No DHCP requests, no ICMP, no other useful frames. Only STP every 30 ms.
Here is one of such requests (all of them are absolutely identical):


Analyzing this frame I've also noticed another strange thing. Source MAC address has its bytes in reverse order (highlighted on the screenshot). Correct MAC is 30:AE:A4:7E:64:17.

How can I correct this behavior or maybe there is workaround?

[ESP_Angus]

Hi Dmitry,

Thanks for posting these details. I have a couple more questions to clarify:

- What IDF version do you have?
- Is the ethernet stable for you, before this particular behaviour is triggered?
- What is the ESP32 doing when it goes into this state? Are any errors logged on the serial console?

Angus

[dmitryga]

1. IDF version is 3.1rc1. The same behavior was with versions 3.0.2 and 3.0.3 (possible it was on versions 3.0.0 and 3.0.1, I can't remember or device was not working for so long in those times).
2. Yes, ethernet is stable. When I've noticed this problem first time, I've tried to connect device to another switch (3 different switches were used, different outlets, cables, even cat6 shielded patch cords). Result was the same.
3. No direct errors about ethernet were logged. Only errors about TCP/IP (unable to resolve IP, unable to send/receive data, etc) were logged. Moreover all other components of ESP32 were working fine: BLE, SPI, RS232.

[ESP_Angus]

Thanks for the quick follow-up. How much free heap is available at the time the ethernet malfunctions?

[dmitryga]

Free heap size is not constantly monitoring, but during normal operation its size is about 40-60 KB (depends on active web server connections). Also I've noticed, that heap fragmentation is large, for example, I'm not able to allocate 20KB buffer even when esp_free_heap_size shows that there is 40-50 KB of free heap.

[dmitryga]

I'm back with update about my problem with ethernet.
I've took "magic" value 5704 (0x1648) from STP packet and made search across ESP-IDF sources. And I've found only one occurrence inside ethernet\emac_dev.c file in emac_enable_flowctrl function.
So I've decided to disable flow control during ethernet initialization and it works!
It took 2 weeks, I had 3 devices running, and for all this time I have not encountered any ethernet problem.

[ESP_morris]

Nice work， I'm wondering if there's a circle in your network topology? So the pause frame is just walking around the circle, like the snowball effect.
The bug of the wrong MAC address has been fixed in the latest master branch weeks ago.

[dmitryga]

There are no loops in our network. We're using switches with STP support which are shutting down looped ports. This problem is reproducible even when ESP ethernet port is connected directly to PC ethernet port with a single link.

Thank you for information about MAC address fix. I'll check if we are using appropriate ESP-IDF build.