Auto download fails in virtual machines possibly due to USB lag

[RMandR]

When trying to program ESP32 from a virtual machine (guest operating system), auto-download does not work.

Pressing and releasing the Boot button does start the download however, so this not an issue of serial port set up etc. I have tried the following boards without auto-download success:

- ESP32 DevKitC V4
- ESP WROVER KIT 4.1
- ESP PROG board (this one has an FTDI chip)
- Adafruit Huzzah32 feather

Running the on the host operating system however, seems to work fine (have only tried it on DevKitC)

Is there a workaround for this?

-a

[ESP_Angus]

Hi RMandR,

What virtualization software and guest/host OS combination are you using? It seems like some are more susceptible than others.

I don't know of any workaround for this. Managing timing on a "native" OS with the Python pyserial abstraction layer, then the OS serial abstraction layer, then the OS serial port driver and finally USB timing behaviour is already quite complex, and adding additional abstraction layers on top of this can make it very hard to get the timing right.

You could try adding more capacitance to the EN pin (a capacitor between EN and GND, maybe in the 1uF-10uF range) to delay the ESP32 coming out of reset by a little, giving the GPIO0 pin more time to drive low.

You could also try changing the way the serial port is virtualized - many Virtual Machine programs can either virtualise the host's serial port (so the serial driver is in the host OS), or can virtualise the host's USB device (so the serial driver is in the guest OS). It's possible one of these combinations will work more reliably that the other.

Angus

[Markus Becker]

Hi RMandR,

We use the toolchain within Ubuntu guest on Windows host. We use VirtualBox virtualization (with usb virtualization).

In our environment we did not manage to get the auto-program logic to work reliable. Latencies introduced by usb virtualization caused the logic to fail in our case. Changing capacitor values did not help either.

esptool.py uses pyserial. Unfortunately, with pyserial there is no way to toggle both used control signals (RTS and DTR) together in one step.

Using fcntl and termios, I found a way to get the stuff to work very reliable for me. Unfortunately this only works on Linux. With my limited python competence I did not manage to make the changed script compatible to the other platforms (where fcntl and termios don't exist). Anyway, maybe someone capable is out there, so I leave the few changes I made to esptool.py here.

... additional imports

Code: Select all

import fcntl
import termios

Code: Select all

# Values to set DTR together with RTS in one call
    TIOCMSET = getattr(termios, 'TIOCMSET', 0x5418)
    TIOCM_DTR_SET_RTS_CLEAR = 0x002
    TIOCM_DTR_SET_RTS_CLEAR_str = struct.pack('I', TIOCM_DTR_SET_RTS_CLEAR)
    TIOCM_DTR_CLEAR_RTS_SET = 0x004
    TIOCM_DTR_CLEAR_RTS_SET_str = struct.pack('I', TIOCM_DTR_CLEAR_RTS_SET)

Code: Select all

# detect platform	
plat = sys.platform.lower()
        if plat[:5] == 'linux':   
            self._is_linux = True
        else:
            self._is_linux = False	

Code: Select all

def _setDTR_and_inverted_RTS(self, state):
        """ Set DTR to state and RTS to NOT state """
        # PySerial can not set multiple signals in one call, as it should be 
        # for the auto-program logic to work best.
        # In some cases (like running esptool within a virtual machine) latencies
        # can cause the ESP chip to fail entering download boot.
        # fcntl.ioctl() is used to set DTR and RTS in one call. This is platform specific. 
        if (self._is_linux):
            if state:
                fcntl.ioctl(self._port.fileno(), self.TIOCMSET, self.TIOCM_DTR_SET_RTS_CLEAR_str)
            else:
                fcntl.ioctl(self._port.fileno(), self.TIOCMSET, self.TIOCM_DTR_CLEAR_RTS_SET_str)
        else:
            self._port.setDTR(state)
            self._port.setRTS( not state)	

Original mechanism in esptool.py:

Code: Select all

        if mode != 'no_reset':
            self._setDTR(False)  # IO0=HIGH
            self._setRTS(True)   # EN=LOW, chip in reset
            time.sleep(0.1)
            if esp32r0_delay:
                # Some chips are more likely to trigger the esp32r0
                # watchdog reset silicon bug if they're held with EN=LOW
                # for a longer period
                time.sleep(1.2)
            self._setDTR(True)   # IO0=LOW
            self._setRTS(False)  # EN=HIGH, chip out of reset
            if esp32r0_delay:

Changed to:

Code: Select all

       if mode != 'no_reset':
            self._setDTR_and_inverted_RTS( False) # IO0=HIGH, EN=LOW, chip in reset
            time.sleep(0.1)
            if esp32r0_delay:
                # Some chips are more likely to trigger the esp32r0
                # watchdog reset silicon bug if they're held with EN=LOW
                # for a longer period
                time.sleep(1.2)
            self._setDTR_and_inverted_RTS( True) # IO0=LOW, EN=HIGH, chip out of reset
            if esp32r0_delay:

Best,
Markus

[ESP_Angus]

Hi Markus,

Thanks for sharing that!

make the changed script compatible to the other platforms (where fcntl and termios don't exist). Anyway, maybe someone capable is out there, so I leave the few changes I made to esptool.py here.

In 2016 Windows users were having some problems with this as well, so I wrote some equivalent code for the win32 API:
https://github.com/espressif/esptool/co ... 81b312R308

This is not in esptool now, because the capacitor fix turned out to solve the problem for most users. But I believe this code works (possibly some Windows drivers split the operation into two again in the kernel, but I don't believe they do.)

The comment in my code says that there's no equivalent POSIX call, but I think possibly the code you've written is POSIX compliant - so it could work on macOS as well.

Meaning that it may be possible to patch around the timing issue for all supported esptool.py platforms. Although it does involve using pyserial private APIs, which may not be a great long-term plan.

[Markus Becker]

Hi Angus,

Meaning that it may be possible to patch around the timing issue for all supported esptool.py platforms. Although it does involve using pyserial private APIs, which may not be a great long-term plan.

Totally aggree.

I think I don't use pyserial private api (.fileno() instead of ._port_handle), but platform dependent code in esptool.py should be avoided anyways of course.

Best
Markus

[ESP_Angus]

I agree you're safe API-wise. On Windows there's no public API to get the win32 handle for the port, though. (And, yes, platform-dependent code in the application layer can quickly get messy.)

[RMandR]

Hi Agnus, and Markus,

Thank you for your input. It helps to know I'm not the only one.

The Virtualization software is VMWare Workstation. The guest OS is Windows so no fcntl. I did try the win32 mod Angus pointed out from github, and out of the box, did not work. I haven't tried hooking up the lines to the scope and actually measure the lag.

I also tried added capacitance on EN without success.

The last thing to do is to use a straight up FTDI chip without the "auto_download" circuit and modify Esptool.py to generate the correct signal.

Regards,

Armand

[ESP_Angus]

The last thing to do is to use a straight up FTDI chip without the "auto_download" circuit and modify Esptool.py to generate the correct signal.

This should actually "just work" without modifications if the FTDI is wired up as shown here: https://github.com/espressif/esptool/wi ... bootloader

The reason for the "auto download" circuit is to not hold the chip in reset if DTR & RTS are asserted simultaneously, because most common serial programs do that by default. And this behaviour is the root cause of the timing issues, because esptool has to assert DTR (pull IO0 low) before it releases RTS (release EN to high to take the chip out of reset), and this creates a very brief window where both DTR & RTS are asserted - not a problem for the "simple" circuit, but causes the auto download circuit to take the chip out of reset early (and possibly in the wrong mode).

[rpoelvogels]

You could also try changing the way the serial port is virtualized - many Virtual Machine programs can either virtualise the host's serial port (so the serial driver is in the host OS), or can virtualise the host's USB device (so the serial driver is in the guest OS). It's possible one of these combinations will work more reliably that the other.

Hi,

I had the same problem as the OTP and can confirm that virtualizing the serial port instead of the USB device works reliably.

Kind regards,

Remco Poelstra

[RMandR]

Hi RMandR,

Using fcntl and termios, I found a way to get the stuff to work very reliable for me. Unfortunately this only works on Linux. With my limited python competence I did not manage to make the changed script compatible to the other platforms (where fcntl and termios don't exist). Anyway, maybe someone capable is out there, so I leave the few changes I made to esptool.py here.

<snip>
Best,
Markus

Wanted to say: thanks Markus!
It's been a couple of years, but your solution is the only one that has worked so far.

-a