No RMII Traffic Despite Properly Configured PHY

[khaaaaaan]

Hello all,

I am working on a custom board based on the ESP32-PICO-D4 SOC. Unfortunately, I am having a devil of a time getting the RMII link to function. Specifically, I see the PHY (KSZ8081) initialize correctly and output a 50MHz reference clock on REF_CLK but the EMACCSTATUS_REG register never shows a valid speed or duplex (always stuck at 2.5MHz/half).

Some things I have checked:

• The MDI-side link initializes correctly and autonegotiates with my Ethernet switch.
• The MDIO/SMI interface is functional and I am able to read/write/dump registers via a custom command.
• The PHY is correctly outputting a 50MHz sinusoidal waveform (via attached 25MHz crystal). If I intentionally misconfigure the PHY clock then this waveform halts.
• REF_CLK from the PHY is connected to GPIO 0, which is the only GPIO which supports clock input.
• The MAC is configured with:

  Code: Select all

  #define ETHERNET_PHY_TXCLK_GPIO 0
  
  interface = EMAC_DATA_INTERFACE_RMII
  clock_config.rmii.clock_mode = EMAC_CLK_EXT_IN
  clock_config.rmii.clock_gpio = ETHERNET_PHY_TXCLK_GPIO

• A custom ping command I wrote shows no errors but no frames actually egress the device. I have added debug to LWIP and friends, this actually gets stuck at the first ARP request which is consistent with the MAC not passing traffic.

What I think might be going on:

I added some debug prints to my custom command so I can dump a couple of key registers on command. Everything seems to be configured correctly EXCEPT that, as mentioned above, EMACCSTATUS_REG is always 0x00000000. This sure seems to indicate that the MAC hasn't properly configured the input clock. A sample of these registers is below:

Code: Select all

cmd_smi_do_read: EMACCONFIG_REG=0x0000CC0C, EMACDEBUG_REG=0x00000000
cmd_smi_do_read: EMACCSTATUS_REG=0x00000000, EMACWDOGTO_REG=0x00000000
cmd_smi_do_read: EMAC_EX_CLKOUT_CONF_REG=0x00000024, EMAC_EX_OSCCLK_CONF_REG=0x01001253, EMAC_EX_CLK_CTRL_REG=0x00000001
cmd_smi_do_read: EMAC_EX_PHYINF_CONF_REG=0x00008000, EMAC_PD_SEL_REG=0x00000000

Also please see Ethernet init code in attached sources. Waveforms and schematics available upon request.

So, in short, can anybody see what I am doing wrong? Are there any other registers or functions I can call to get additional information about the system?

Thanks in advance!

[ESP_ondrej]

What IDF version do you use? Could you please provide logs?

[khaaaaaan]

ESP-IDF version is 4.4.5 (I haven't had a chance to make the jump to the new 5 series yet ).

I have attached a log demonstrating the PHY link. Most interesting is probably the end where I have some debug code in the SMI read command (smi -r) to dump the internal MAC registers, which seems to indicate the lack of RMII link. Note that IP is statically assigned.

[ESP_ondrej]

I checked the internal documentation and this EMACCSTATUS_REG register is optional and is present only when the MAC is configured for the SGMII, RGMII, or SMII PHY interface. Therefore you cannot get correct data from it when in RMII mode.

If you need to know PHY speed or duplex, use:

Code: Select all

esp_eth_ioctl(eth_handle, ETH_CMD_G_DUPLEX_MODE, &exp_duplex);
esp_eth_ioctl(eth_handle, ETH_CMD_G_SPEED, &exp_speed);

For link status, please monitor `ETHERNET_EVENT_CONNECTED` in event handler you register during initialization.

[ESP_ondrej]

For further debugging, try to switch PHY to loopback mode. You need to write simple application. The best approach is to not use LwIP for this. Just initialize Ethernet without ESP-NETIF init and register your own Rx callback function using `esp_eth_update_input_path`. Then check with oscilloscope if you see some signal between PHY and ESP on RMII Tx/Rx pins.

[khaaaaaan]

I checked the internal documentation and this EMACCSTATUS_REG register is optional and is present only when the MAC is configured for the SGMII, RGMII, or SMII PHY interface. Therefore you cannot get correct data from it when in RMII mode.

Well, that certainly answers some questions of mine! Ok, disregarding this register moving forward.

For link status, please monitor `ETHERNET_EVENT_CONNECTED` in event handler you register during initialization.

I do see this event fire in my logs so it appears this is working. Just for my own edification, what does this event specifically signify? E.g. does this event track the PHY MDI link status (Clause 22 register 0x01, bit 2) or the RMII link status? Trying to see if I can equate this event to knowing the internal MAC state machine is happy.

For further debugging, try to switch PHY to loopback mode. You need to write simple application. The best approach is to not use LwIP for this. Just initialize Ethernet without ESP-NETIF init and register your own Rx callback function using `esp_eth_update_input_path`. Then check with oscilloscope if you see some signal between PHY and ESP on RMII Tx/Rx pins.

I see, this will take a bit of time to put together and verify.

Relatedly, are there any kind of MAC hardware counters? I like to use these to identify if the MTL queues are doing their thing though I didn't see any obvious registers in the reference manual. Are there any other places I can check for MTL / DMA status beyond DMASTATUS_REG and EMACDEBUG_REG? I already checked the PHY datasheet but it doesn't look like there are the usual histogram counter regs.

Thanks again for the help!

[ESP_ondrej]

I do see this event fire in my logs so it appears this is working. Just for my own edification, what does this event specifically signify?

It signify that link between KSZ8081 and your PC was negotiated and is up, i.e. PHY MDI link status, check details here. In any case, this tells you that the PHY is able to communicate with its counterpart (the PC) and that PHY MII Management Interface (MDC/MDIO) is functional. However, it seems there might be issue with RMII data path.

I don't think it is needed to get much deeper into EMAC, the driver driver should be stable enough. You can try to increase debug log in EMAC driver to see, if you receive any data (https://github.com/espressif/esp-idf/bl ... esp.c#L243). If I were you, I would focus at the RMII data path at this point (check it is correctly connected and if signal is present). For example, I have experience that the PHY was able to communicate with the ESP32 in one way only. The root cause turned out to be fallen/not assembled termination resistor on RMII data path. Issue one wouldn't expect

[khaaaaaan]

It signify that link between KSZ8081 and your PC was negotiated and is up, i.e. PHY MDI link status, check details here. In any case, this tells you that the PHY is able to communicate with its counterpart (the PC) and that PHY MII Management Interface (MDC/MDIO) is functional. However, it seems there might be issue with RMII data path.

Makes sense. In this case, I am very comfortable saying the MDI-side link is functioning as expected which the ESP is correctly able to determine.

I don't think it is needed to get much deeper into EMAC, the driver driver should be stable enough. You can try to increase debug log in EMAC driver to see, if you receive any data (https://github.com/espressif/esp-idf/bl ... esp.c#L243). If I were you, I would focus at the RMII data path at this point (check it is correctly connected and if signal is present). For example, I have experience that the PHY was able to communicate with the ESP32 in one way only. The root cause turned out to be fallen/not assembled termination resistor on RMII data path. Issue one wouldn't expect

Oof, yeah, I have debugged failures like that; no fun (I once had one where two PHYs from different production runs treated a 1.8V input as either high or low with all of the strange strapping behavior you might expect from that ).

And, up until just recently I would have agreed that the issue is the RMII link. That said, I had some more time to work on this and now I am not thinking so. Some interesting findings:

• I enabled remote (analog) loopback on the PHY; I do see broadcast pings from my test host returned. Given this, I am comfortable saying the MDI-side link is just fine.
• I enabled local (digital) loopback on the PHY; unfortunately pings to my local IP are always returned, even after I disable this loopback. I am calling this one a push as I suspect either LWIP or the MAC driver layer is performing the loopback in software and I haven't yet had time to write a direct Ethernet initialization and RX callback test shim yet.
• I probed RXD[1] and TXD[0] pins while sending bidirectional traffic. Unfortunately, my personal scope setup is somewhat modest and I don't have a differential probe so, really, all I am looking for is structure in the data. And structure I do find, see attached waveforms. Given this, I am fairly certain that RMII is functioning as expected.
• I looked into that log print in emac_esp32_receive your previous message. In the interest of expedience, I just switched it to an ESP_LOGE call rather than enabling full debug logs (which is going to be my next test). Unfortunately, I never see this message printed.
• I tried adding a print to emac_isr_default_handler with the goal of observing TX / RX interrupts and the DMA ring in general but I never see my print hit. This seems like nonsense since I DO receive ARP requests from the ESP32 on my dev machine (which sends an ARP reply which is never fully received by the ESP causing the next ping cycle to start with another ARP request). Given that TX seems to be working fine I think I must be missing something in my test methodology, though I couldn't find any other promising ISRs in the MAC layer that read from the DMA status register.

So, given the above, I am absolutely comfortable saying that there could still be a problem with my board but I am struggling to see how given the scope waveforms and analog loopback test results. I am also expecting that I am missing something in my probing of the MAC given the last two points above (particularly the last one).

The one thing that I have not yet been prove or disprove is if the ESP MAC is actually receiving the data from the PHY. I know the PHY is jiggling the RXD[1:0] lines but maybe the internal MAC FIFO is rejecting the data due to clock skew or something like that? Do you know of any way to answer this question on the ESP side?

Can you think of anything else I might be missing here? My next plan is going to be to kick on full debug logs for everything and see what that gets me.

[ESP_ondrej]

I enabled local (digital) loopback on the PHY; unfortunately pings to my local IP are always returned, even after I disable this loopback. I am calling this one a push as I suspect either LWIP or the MAC driver layer is performing the loopback in software and I haven't yet had time to write a direct Ethernet initialization and RX callback test shim yet.

Yes, lwIP knows it is local address so it loop backs it just right in the software. You can try to send some broadcast message. That should be looped back and received.

I tried adding a print to emac_isr_default_handler with the goal of observing TX / RX interrupts and the DMA ring in general but I never see my print hit. This seems like nonsense since I DO receive ARP requests from the ESP32 on my dev machine (which sends an ARP reply which is never fully received by the ESP causing the next ping cycle to start with another ARP request). Given that TX seems to be working fine I think I must be missing something in my test methodology, though I couldn't find any other promising ISRs in the MAC layer that read from the DMA status register.

You don't see Tx interrupt in emac_isr_default_handler because interrupts are not enabled for Tx path. If you are able to see message from ESP on your test computer then it really indicates there is issue at RMII Rx path. That's actually good sign, the issue is quite isolated.
Try the following options:
- check if the Rx[0] is not connected to Rx[1] by mistake. Do it in schematics and on the real PCB.
- switch the PHY to 10 mbps
- try to replace termination resistors with different value

You also may have some troubles with you PCB design. The RMII is high speed interface and care needs to be taken to properly layout it - https://resources.pcb.cadence.com/blog/ ... r-ethernet

[khaaaaaan]

Well, I found the issue: there was a cracked solder joint under the RXD0 pin on the ESP. Infuriatingly, I was probing RXD1 which is why we didn't see anything untoward in the waveforms I posted. Reworking this and the surrounding connections now has RMII functioning as expected. How embarrassing.

Thanks again for the assistance in this matter; it really did make the difference.