Flash is too slow

[Haptesthai]

Hi,

I have recently converted to ESP-IDF from the ADF, and trying to get over the overwhelming feeling.

I realised two things, however. The first is that, I now seem to be able to flash programs much slower than the Arduino IDE. I tried replacing the IDF's esptool.py with the one I found to see if that was the cause, but no changes, still. You can see below the console output for the two.

Secondly, I am still in my "playing" phase, so creating lots of project directories (i.e. copying from the examples directory). Every time I build a project for the first time, it takes a good amount of time building all the components in the ESP-IDF, inside the "build" directory inside the project. This also occupies hundres of MBs with redundant files. What am I doing wrong? I am on Linux.

Many thanks in advance.

ESP-IDF Flash Process (36 seconds, VSCode):

Code: Select all

esptool.py v2.6
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... ESP32
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
MAC: a4:cf:12:0b:9f:64
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 3072 bytes to 103...
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 1858.9 kbit/s)...
Hash of data verified.
Compressed 24960 bytes to 15250...
Wrote 24960 bytes (15250 compressed) at 0x00001000 in 1.3 seconds (effective 148.6 kbit/s)...
Hash of data verified.
Compressed 634832 bytes to 401400...
Wrote 634832 bytes (401400 compressed) at 0x00010000 in 35.4 seconds (effective 143.4 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Arduino IDE Flash process (6 seconds):

Code: Select all

esptool.py v2.6
Serial port /dev/ttyUSB0
Connecting......
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
MAC: a4:cf:12:0b:9f:64
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 8192 bytes to 47...
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.0 seconds (effective 41291.6 kbit/s)...
Hash of data verified.
Compressed 15856 bytes to 10276...
Wrote 15856 bytes (10276 compressed) at 0x00001000 in 0.1 seconds (effective 902.2 kbit/s)...
Hash of data verified.
Compressed 650704 bytes to 392441...
Wrote 650704 bytes (392441 compressed) at 0x00010000 in 5.9 seconds (effective 889.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 128...
Wrote 3072 bytes (128 compressed) at 0x00008000 in 0.0 seconds (effective 8388.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

[ESP_Angus]

Hi Haptesthai,

The default serial port flashing baud rate is set very conservatively in ESP-IDF when using the GNU Make build system (default in IDF v3.x). We increased it significantly in ESP-IDF v4.0 when using the CMake build system, but it looks like you're using v3.x.

You can increase it in the "menuconfig" of a project by setting this value:
https://docs.espressif.com/projects/esp ... oolpy-baud

Or you can set the ESPBAUD environment variable to override it for all projects. The ESP-IDF default is 115200, the arduino default is 921600. Depending on your hardware 2000000 or 3000000 may even work, but even at 921600 the bottleneck is usually writing to the flash chip not sending the data over serial.

[ESP_Angus]

it takes a good amount of time building all the components in the ESP-IDF, inside the "build" directory inside the project. This also occupies hundres of MBs with redundant files. What am I doing wrong? I am on Linux.

You're not doing anything wrong. Unlike in Arduino, the project configuration menu means that each project can configure ESP-IDF very differently at build time. However, this means ESP-IDF has to be recompiled whenever the config changes (or when a new project is created). In Arduino ESP-IDF is compiled once with a static config, so there are less files to compile even for a "clean" build.

The second build should be significantly faster as only the changed files are recompiled (unless the config is changed).

It is possible to install and enable ccache in order to speed up builds of source files that turn out to be the same. This can be enabled on the command line or by setting an environment variable when using the CMake-based build system in ESP-IDF V4.0 and newer:
https://docs.espressif.com/projects/esp ... py-options

For GNU Make build system it's a bit fiddlier to set up ccache, you need to either set the compiler toolchain prefix to "cmake xtensa-esp32-elf-" in the compiler settings in menuconfig, or set up symlinks. Suggest skipping this for now unless you're sure you'll be doing a lot of "clean" builds with the GNU Make build system.

Also, if you're building on the command line with "make" then make sure to enable parallel make, ie "make -j4" if you have a quad core system.

If you're using VSCode then you may find the ESP-IDF VScode extension useful: https://github.com/espressif/vscode-esp ... -extension (it's based on the newer CMake build system as well.)

[Haptesthai]

It is possible to install and enable ccache in order to speed up builds of source files that turn out to be the same. This can be enabled on the command line or by setting an environment variable when using the CMake-based build system in ESP-IDF V4.0 and newer:
https://docs.espressif.com/projects/esp ... py-options

For GNU Make build system it's a bit fiddlier to set up ccache, you need to either set the compiler toolchain prefix to "cmake xtensa-esp32-elf-" in the compiler settings in menuconfig, or set up symlinks. Suggest skipping this for now unless you're sure you'll be doing a lot of "clean" builds with the GNU Make build system.

Hi ESP_Angus,

Thank you for your reply.

About the BAUD rate, I see different menus for "make menuconfig" and "idf.py menuconfig". Only the former has an option to set the flash baud rate. idf.py gives a somewhat limited number of options (I'm on ESP-IDF v.4.2). Is that normal? (I solved this by setting the BAUD_RATE env. variable, so it's not really an issue, anyway.)

As for "ccache", thank you for the recommendation. It sped up my builds by a lot. A question, now that I'm working with simple sample projects to try out different components/features of ESP32, I have a lot of project folders, and they total a few GBs. Is there a way (like using symlinks to a common build directory) to use a common cache for every project folder? The only difference between those folders is main.c, and config parameters, though I'd be fine using a single config (Arduino-like) for all those toy projects.

Lastly, I see that even a simple app that prints "Hello World" over serial takes 140+ MB when built, is there a reason why everything is built even when they're not needed?

Haptesthai.