Trying to limit the number of components being built and failing!

[RobMeades]

The build time (on Windows) for my code when using ESP-IDF (CMake version) is about 5 times longer than any other chipset/platform (NRF52 GCC and SES, NRF53 Zephyr and STM32F4); whatever I do I can't seem to get it down. This is on a CI machine so it is always doing clean builds for all of my chipset/platforms and doing all of them in parallel so controlling -j to match the number of cores is not a realistic solution: ALL the cores are always maxed out in any case.

A better solution is to remove unnecessary components by specifying:

Code: Select all

set(COMPONENTS "esp32" "esptool_py" "unity")

...but I don't seem to be able to make this work; despite needing nothing other than these few components I always end up getting the list below, which is close to 900 source files.

My build metadata is attached: can anyone spot what I'm doing wrong?

app_trace app_update asio bootloader bootloader_support bt cbor coap console cxx driver efuse esp-idf esp-tls esp32 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_http_client esp_http_server esp_https_ota esp_https_server esp_local_ctrl esp_ringbuf esp_rom esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter ubxlib_runner ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa

[ESP_Angus]

Hi Rob,

To limit the components in the build, you need to set the COMPONENTS variable in the top-level project file, before calling project(). It's not necessary to set it in each individual component CMakeLists file.

(However, if individual components set COMPONENT_REQUIRES in their own components files, these additional components will be automatically added to the build.)

In your case this looks like runner/CMakeLists.txt is the project, so try setting it there.

If it still doesn't work, please us know which ESP-IDF version you're using.

[RobMeades]

Hi, and thanks for the very swift response.

In your case this looks like runner/CMakeLists.txt is the project, so try setting it there.

Interesting. I thought I was doing that since runner/CMakeLists.txt has the line:

Code: Select all

set(COMPONENTS "")

...which I believed should empty the list. However I've just tried replacing this with:

Code: Select all

set(COMPONENTS "esp32" "esptool_py" "unity")

...and that does indeed reduce the component list to the below; still 557 source files but better than 900. Is there any way of getting rid of any more of this stuff? This is with ESP-IDF release/v4.1 (#5d258af93). I really only need the bootloader, basic chipset drivers and FreeRTOS, I'm not doing anything else [no WIFI in this].

app_trace app_update bootloader bootloader_support cxx driver efuse esp32 esp_common esp_eth esp_event esp_netif esp_ringbuf esp_rom esp_wifi espcoredump esptool_py freertos heap log lwip mbedtls newlib nvs_flash partition_table perfmon pthread soc spi_flash tcpip_adapter unity vfs wpa_supplicant xtensa

[RobMeades]

Actually, it now fails to link, it is no longer able to find the stuff in the main sub-directory. I've added main explicitly to EXTRA_COMPONENT_DIRS; what else might I be doing wrong? Revised build metadata and the build output are attached.

Code: Select all

C:/esp32/esp-idf-latest/components/esp32/cpu_start.c:552: undefined reference to `app_main'

EDIT: now fixed, question above about whether I can reduce the number of files further still stands: all I need is the processor to be up with RTOS, nothing more.

[mylenedipenta]

Similar problem. I'm testing configuration options with a small test program that literally has a single printf statement, and compiling 966 components every time I need to do a clean build. Any updates on this?

[mylenedipenta]

Very helpful twitter convo with @esp_igrr helped me sort this out. TL:DR -- "not really, but also yes, sort of."

1: There are lots of things that simply can't be removed from the build. See the API Reference on "Build System" for the list of mandatorily-built components.

These components, and any dependencies they have, can't not be built.
https://docs.espressif.com/projects/esp ... quirements

2: There are some counter-intuitive components included in the minimum build. Here's @esp_igrr:

"In the past esp_wifi.h header file used to live in one of the "common" components, "esp32". To avoid regressions, when it was moved to esp_wifi component, esp_wifi was added as a dependency of "esp32". "

Since esp32 is always required, that means esp_wifi will always get built. There are a few other non-intuitive things in my build list (nsv_flash, tcpip_adapter). I assume they're stuck in there for similar reasons.

3. This affects the build time, but I don't think it affects the binary size. For me, unused code gets eliminated at link time. @esp_igrr mentioned enabling CCache; it think it's turned on by default in my VSCode plugin installation, so I haven't tested this.

4. To reduce the build time: include this line, in the *project* folder's CMakeLists.txt (NOT the main folder, but the project top-level folder):

Code: Select all

set(COMPONENTS esptool_py main)

NOTE: this doesn't preclude you from building/linking in other components. You're going to include those by listing them in the *component* folder CMakeLists.txt, and their dependencies will get included too.

Be careful about renaming your main folder! A folder named "main", containing a file named main.c, gets treated differently by CMake/ESP-IDF than other components. (See the link to the Build System API reference, linked above, for details).

All this line does is force CMake to trim components that aren't referenced by your code in any way. For reference, my top-level (project folder) CMakeLists.txt looks like this:

Code: Select all

cmake_minimum_required(VERSION 3.5)

set(COMPONENTS esptool_py main)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(main)

5. Wondering why ESP-IDF doesn't just assume this behaviour as default? Here's @esp_igrr:

There are a few reasons why even a simple ESP-IDF project builds a lot of files. IDF was developed mainly for IoT related products, and while we did make it extensible, not much effort was put into making it to scale down to simple applications.

Most real applications built for ESP chips will require these components anyway, so there is not much to gain, aside from build time, by leaving them out. We do see that trimming the build is needed sometimes, for example to be able to build many different test apps quickly.

Build time, code size, performance can all be improved once they get to a working application or demo; rather than having them figure out why the can't use some function or can't include some header file from the very start.

I was glad to learn this. It helps me understand, when something seems to work inconsistently, that I should consider if it makes the life of a new dev easier. If it does, it's likely not a bug. I was starting to doubt whether I had even understood CMake at all, until I understood this.

OVERALL

Adding the set(COMPONENTS...) line to my project-level CMakeLists.txt reduced built components from 966 to 613, and brought the build time down under 1 min. Not ideal, but understandable, and an acceptable compromise for the greater ease that will result once my projects are out of the "trivial driver" stage.