MSYS2 is slow on Windows 10

[pdelprat]

Hello,

In my environnement, the compilation is extremely slow. It can be speed up a little if I desactivate Windows Defender.

I installed ubuntu for windows 10 to compare and the result is better by a factor of 3 !!!

But, on ubuntu, it's more difficult to configure, like COM1 is /dev/ttyS1 and I need to change access security. And I'm not able to setup openocd correctly, purhaps by misunderstanting.

If someone have a idea of how to speed up msys2 or be able to setup ubuntu for windows to work with openocd ...

Have a good day,
kind regards,
Pascal

[wobblyboots]

I concur, the pauses (not sure what it's doing) do slow down the development process.

kind regards
Ian.

[ESP_Angus]

Hi Pascal & Ian,

We are aware of this (poor Windows build performance) and we're working on a solution, which will probably be available in IDF v3.1 (TBC). We'll announce it on here once it's available.

The root causes are that GNU make (which IDF currently uses for its build system) does a lot of fork() operations and also does a lot of scanning of the filesystem. These two operations are fast in Linux/OSX but slow in Windows. Plus, MSYS itself is a translation layer which adds some overhead.

Some things you can do to help Windows build performance right now are:

• Disable Windows Defender real-time scanning under the C:\MSYS2 directory and any other directories used in the build.(*)
• Run the build with "make -jN" where N is a large-ish number (try double the number of CPUs in your system).
• Install ccache with "pacman -S ccache" and configure IDF to use it (one way to configure it to set the "Compiler toolchain path/prefix" under "SDK tool configuration" to "ccache xtensa-esp32-elf-")

Still, you will probably not get the same performance as a Linux VM,. However these should help.

(*) Note that this is a risk you'll have to trade-off against. We don't plan to ship malware or viruses in the MSYS2-based environments we supply, but you'll have to make sure not to download anything untrusted into the MSYS2 directories.

But, on ubuntu, it's more difficult to configure, like COM1 is /dev/ttyS1 and I need to change access security. And I'm not able to setup openocd correctly, purhaps by misunderstanting.

In Ubuntu (and most Linux distros) by default, normal users don't have permissions for serial ports or generic USB devices (for openocd). You can run the following commands in a terminal to create the relevant permissions:

Code: Select all

sudo usermod -aG plugdev,dialout $USER
cd /path/to/openocd-esp32
sudo cp contrib/99-openocd.rules /etc/udev/rules.d

You'll need to log your out and also restart udev before these rules take effect. Exactly how to restart udev varies between Ubuntu versions I think, but rebooting will definitely get you there.

Once this is done, serial & openocd should work without any special steps.

(The first command adds your user to plugdev and dialout groups, "dialout" group is for serial access without "sudo". The last command copies some openocd-specific "udev" device rules. These rules make various debugger interfaces available to all users in the "plugdev" group without "sudo".)

[papaluna]

Thanks for the tip ESP_Angus to exclude MSYS2 and the project folders in Windows Defender. The time to run a make under Windows is now comparable to the time to run a make in a VM running on the same Windows box.

[martinB]

It would also help if the makefile in the esp-idf process was a bit more intelligent about what gets rebuilt. Yesterday I was loading code onto both WROVER and WROOM boards - they come up as different COM ports on my laptop. I went into menuconfig and changed the serial flasher configuration and make decided to recompile everything - component libraries and my source code. I doubt that, in practice, it was necessary to recompile anything.

[ESP_igrr]

The inclusion of the serial port configuration into sdkconfig was an oversight, and we plan to remove this local configuration from sdkconfig. Currently if you set ESPPORT and ESPBAUD environment variables, then these variables will override settings from sdkconfig. So you can upload to different boards just by doing 'export ESPPORT=COMx` and then running `make flash`. Or even `make flash ESPPORT=COMx`. In both cases this will not cause full rebuild.

[martinB]

Thanks for that!

How about reducing the number of components that are included in the build? My build directory currently contains compiled versions of everything in the ESP32 distribution irrespective of whether I am using it or not.

[ESP_igrr]

You can set COMPONENTS variable in your project makefile (before including project.mk) to the list of components you need to build.

http://esp-idf.readthedocs.io/en/latest ... -variables


Edit: we are planning to introduce some major changes to the build system which will make more explicit dependency tracking possible... Preview is coming in a couple of weeks.

[PaulFreund]

For everybody on Windows 10 I can wholeheartedly recommend the Windows Subsystem for Linux (WSL). Especially as my work PC I can't make exceptions to Windows Defender.

Compile time dropped from >15 minutes to <2 minutes. Also starting the flasher and monitor are a lot faster.

Setup is pretty straightforward:

• Run the powershell command from here: https://docs.microsoft.com/en-us/window ... tall-win10
• Install Debian (my distro of choice) from Microsoft Store
• Start the Debian App and follow the linux setup guide: https://esp-idf.readthedocs.io/en/lates ... setup.html
• Windows partitions can be accessed in /mnt, for example C: = /mnt/c/
• Serial terminals are mapped COM13 = /dev/ttyS13 they have to be made user accessible with

  Code: Select all

  sudo chmod 666 /dev/ttyS13

• I have my project on C:\someproject and in Visual Studio code I configured the commandline with:

  Code: Select all

  "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\bash.exe",

• If you want to have access to esp-idf files form windows you can clone it to your drive and make a link with (for example)

  Code: Select all

  ln -s /mnt/c/esp-idf ~/esp/esp-idf

[squonk11]

Since compilation in windows was too slow I also installed the Windows Subsystem for Linux (WSL). This works really fast. Thank you for the hint so far.
But I am having a problem: when I invoke the terminal via "make monitor" I can not exit the terminal using the key "Ctrl+]". How can I accomplish that?