Preview release: CMake-based build system for ESP-IDF

[ESP_Angus]

But, What if we want to continue GNU based system forever with latest update inline with CMAKE based system? can we get that type of support at that time? or once CMAKE build system will be released as stable release then GNU make build system will not be maintained means not updated after that?

After IDF V4.0 we'll continue to support GNU Make (fix bugs, etc), however totally new build system features may not be backported to GNU Make - to use them you may need to move to CMake.

At some point in the future we may announce removal of all support for GNU Make. This will definitely not be before IDF V5.0 at the very earliest, and whether (and when) it happens will depend on what we learn as we roll out the CMake-based build system to a bigger audience. If it does happen, plenty of notice will be given.

FWIW: Porting an IDF project from GNU Make to CMake is not a particularly complex process, most of the IDF build system features look the same to the component author in both systems - just with different syntax. There is even an automated tool to create a CMake-based project from a GNU Make based project. More details can be found in the docs.

[ESP_Angus]

As folks have noticed already, experimental CMake support has now been merged for the ESP-IDF V3.1 release and ESP-IDF master branch. Thanks to everyone who used the preview branch, reported issues, and made suggestions. The feature/cmake preview branch will be deleted soon.

I've updated the first post to reflect these changes.

CMake support is still "experimental" in V3.1 and master branch. We plan for CMake-based build system to be feature complete in ESP-IDF V3.2, and become the default build system in ESP-IDF V4.0.

[p-rimes]

This new CMake build system is awesome! I'm slowly porting my component.mk files, and really enjoying everything about it (my component dependencies are now much more clear -- it won't even build without all dependent components included correctly in COMPONENT_REQUIRES). As well, ccache is a godsend... how did I not know about this tool before? Can ccache be used with the original esp-idf build system as well, or is that CMake-only?

Finally, a few questions have come up that I am unable to solve simply in CMake:
1. For generated sources, it seems that adding them the recommended way in the new docs only affects the `${COMPONENT_NAME}.a` output target (`add_custom_target(logo DEPENDS logo.h)`), but I am not sure how to have one source file eg Bar.c inside the component that requires the generated file(s) Foo.h, Baz.h before compiling Bar.c? It seems like a race condition -- sometimes the generated files happen first, but other times they are not ready when Bar.c is compiled.

2. How to add project-wide definitions to CFLAGS? Currently I am using a `project_include.cmake` file, with `add_definitions()` in there. Is that correct? (I used to use `Makefile.projbuild`: `CFLAGS += "-Dfoo"` in the old build system)

2(b). I can't figure out how to define macro *functions* (eg FOO(x)=(x*2)) at all. At the file, component, or project-level, this doesn't seem possible with CMake? (the warning message suggests modifying the source code to define it at the top but I'd prefer not to modify the source files)

3. How to create wildcard file rules? Previously I used a `Makefile.componentbuild` with custom makefile targets using `%`. Now I think I need to create CMake functions in a `project_include.cmake` file, and then call those functions from within other components CMakeLists.txt? It works, but it is kinda clunky and CMake syntax is quite tedious compared to a Makefile rule.

[ESP_Angus]

Hi p-rimes,

Glad you're enjoying it!

how did I not know about this tool before? Can ccache be used with the original esp-idf build system as well, or is that CMake-only?

It can, it's just a little fiddler. Two ways to do it:

- Set the compiler prefix in the toolchain config to "ccache xtensa-esp32-elf-"

- Create a directory of symlinks to ccache where the links all have the name of the compiler, then put this directory in the PATH variable before the "real" toolchain directory. ie, my laptop has:

Code: Select all

$ ls -l ~/bin/ccache_local/
total 0
lrwxrwxrwx 1 gus gus 15 Oct  5  2016 xtensa-esp32-elf-cc -> /usr/bin/ccache
lrwxrwxrwx 1 gus gus 15 Oct  5  2016 xtensa-esp32-elf-cpp -> /usr/bin/ccache
lrwxrwxrwx 1 gus gus 15 Oct  5  2016 xtensa-esp32-elf-g++ -> /usr/bin/ccache
lrwxrwxrwx 1 gus gus 15 Oct  5  2016 xtensa-esp32-elf-gcc -> /usr/bin/ccache
$ echo $PATH
/home/gus/bin/ccache_local: ... :/home/gus/bin/xtensa-esp32-elf/bin: ...
$ which xtensa-esp32-elf-gcc
/home/gus/bin/ccache_local/xtensa-esp32-elf-gcc

1. For generated sources, it seems that adding them the recommended way in the new docs only affects the `${COMPONENT_NAME}.a` output target (`add_custom_target(logo DEPENDS logo.h)`), but I am not sure how to have one source file eg Bar.c inside the component that requires the generated file(s) Foo.h, Baz.h before compiling Bar.c? It seems like a race condition -- sometimes the generated files happen first, but other times they are not ready when Bar.c is compiled.

There is a mistake in the docs, the missing line is this one:

Code: Select all

target_include_directories(${COMPONENT_NAME} PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

Which tells cmake to include headers in the current "binary" (ie target build) directory when building the component. This is how it finds logo.h at build time.

Regarding the specific source file compile time dependencies, CMake does some "magic" here in order to determine what header files each source file depends on, and will automatically infer the dependency. Provided the dependency exists between targets in CMake (with add_dependency), and the target_include_directories() includes the directory where the header is generated, CMake should ensure everything is done in the correct order.

2. How to add project-wide definitions to CFLAGS? Currently I am using a `project_include.cmake` file, with `add_definitions()` in there. Is that correct? (I used to use `Makefile.projbuild`: `CFLAGS += "-Dfoo"` in the old build system)

Yes, these two are equivalent.

2(b). I can't figure out how to define macro *functions* (eg FOO(x)=(x*2)) at all. At the file, component, or project-level, this doesn't seem possible with CMake? (the warning message suggests modifying the source code to define it at the top but I'd prefer not to modify the source files)

I'm not familiar with this, what's the Makefile equivalent of what you were doing before?

3. How to create wildcard file rules? Previously I used a `Makefile.componentbuild` with custom makefile targets using `%`. Now I think I need to create CMake functions in a `project_include.cmake` file, and then call those functions from within other components CMakeLists.txt? It works, but it is kinda clunky and CMake syntax is quite tedious compared to a Makefile rule.

I think this should be possible without being too clunky. Can you give some examples of the Make functionality you're trying to reproduce?

[p-rimes]

It can, it's just a little fiddler.

Ah, I see! ccache is just a regular tool, not part of cmake. Well I look forward to supercharging all my toolchains this way!

1. For generated sources, it seems that adding them the recommended way in the new docs only affects the `${COMPONENT_NAME}.a` output target (`add_custom_target(logo DEPENDS logo.h)`), but I am not sure how to have one source file eg Bar.c inside the component that requires the generated file(s) Foo.h, Baz.h before compiling Bar.c? It seems like a race condition -- sometimes the generated files happen first, but other times they are not ready when Bar.c is compiled.

There is a mistake in the docs, the missing line is this one:

Code: Select all

target_include_directories(${COMPONENT_NAME} PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

Which tells cmake to include headers in the current "binary" (ie target build) directory when building the component. This is how it finds logo.h at build time.

Regarding the specific source file compile time dependencies, CMake does some "magic" here in order to determine what header files each source file depends on, and will automatically infer the dependency. Provided the dependency exists between targets in CMake (with add_dependency), and the target_include_directories() includes the directory where the header is generated, CMake should ensure everything is done in the correct order.

Doesn't seem to be the case for me. I have to run build 4 or 5 times to get all the files generated. As well, I am generating header files back into the component directory, and I check these in to git (an anti-pattern, but frustratingly necessary right now), so that snippet doesn't apply to my case, I think (I already have my "src/gen" folder in `COMPONENT_ADD_INCLUDEDIRS`, which I think is the same but not PRIVATE?). After some research (aka stackoverflow, because CMake documentation seems ... partial?), it seems like I need to also apply `OBJECT_DEPENDS` at the source file level:
https://stackoverflow.com/questions/293 ... ther-files

FYI, In component.mk, I was doing it like this:

Code: Select all

#components/network_manager/component.mk
src/ntp_actor.o: $(COMPONENT_PATH)/src/gen/network_manager_generated.h

2(b). I can't figure out how to define macro *functions* (eg FOO(x)=(x*2)) at all. At the file, component, or project-level, this doesn't seem possible with CMake? (the warning message suggests modifying the source code to define it at the top but I'd prefer not to modify the source files)

I'm not familiar with this, what's the Makefile equivalent of what you were doing before?

Oddly these same two functions have plagued me on so many platforms...
This is what I was doing in component.mk:

Code: Select all

#components/osc/component.mk
lib/tinyosc/tinyosc.o: CFLAGS += \
→ -D'htonll(x)=((((uint64_t)htonl(x) & 0xFFFFFFFF) << 32) + htonl((x) >> 32))' \
→ -D'ntohll(x)=((((uint64_t)ntohl(x) & 0xFFFFFFFF) << 32) + ntohl((x) >> 32))'

3. How to create wildcard file rules? Previously I used a `Makefile.componentbuild` with custom makefile targets using `%`. Now I think I need to create CMake functions in a `project_include.cmake` file, and then call those functions from within other components CMakeLists.txt? It works, but it is kinda clunky and CMake syntax is quite tedious compared to a Makefile rule.

I think this should be possible without being too clunky. Can you give some examples of the Make functionality you're trying to reproduce?

Actually I do think this is OK. What I lost in the terseness of the Makefile, CMake definitely makes up for it by allowing me to abstract all the steps (targets, deps, ADDITIONAL_CLEAN, etc) inside a cmake function. Just another scripting language, I can vaguely see that it could be very declarative and clean.

[p-rimes]

However, the largest hurdle right now seems to be adding a new flash target (in my case, a fatfs partition image that gets flashed to an offset set in a Kconfig `CONFIG_FATFS_PARTITION_BASE_ADDR`). Not really sure where to hook in my own partitions into the flashing.

I switched from idf.py to using ninja in the build dir directly (I think I prefer this familiar workflow, instead of idf.py). At least then I can create my own targets for ninja without plumbing them through idf.py e.g.

Code: Select all

ninja build flash flash_fatfs monitor

But when running ninja this way, it seems that the port and baud settings are not populated correctly eg when running just `ninja flash`.
(it tries to run the following command, with no args for `-p` or `-b`:)

Code: Select all

esptool.py --chip esp32 -p -b write_flash --flash_mode dio 

(I am talking about the instructions here, followed by `ninja build flash`:)
https://github.com/espressif/esp-idf/bl ... e-directly

[ESP_Angus]

Ah, I see! ccache is just a regular tool, not part of cmake. Well I look forward to supercharging all my toolchains this way!

Yes, I think ccache predates CMake by at least a decade. It's just easier to "automatically" detect tools like this with a CMake build flow.

Doesn't seem to be the case for me. I have to run build 4 or 5 times to get all the files generated. As well, I am generating header files back into the component directory, and I check these in to git (an anti-pattern, but frustratingly necessary right now), so that snippet doesn't apply to my case,

Yes, I see. I think CMake makes some internal distinction between "generated" and "not generated" files, so I can see this not working as expected. I think the approach you mention should work. Otherwise, the best two suggestions I can think of are:

• Generate the file in the build directory. If you still need to check it into git, copy it from the build directory to the git repo directory as part of the add_custom_command() (it's possible to chain multiple COMMANDs in the same invocation). It may be necessary to pass "BEFORE" option to target_include_directories(... ${CMAKE_CURRENT_BINARY_DIR}) so that CMake's dependency generation sees the "generated" header file in the build dir, not the one already in the source dir. The goal is to have CMake totally ignore the one in the source directory when determining the order to build things.
  EDIT: The above won't work if the source file and the "generated" header are in the same directory as each other, and included via #include "" with quotes. This syntax always looks in the source directory first. Have to either be in different directories, or use #include <> with angle brackets.
• If header generation doesn't take too long: Generate the header each time CMake runs, using the execute_process() function. This way it's always done before the ninja/make build pass even starts. CMake has a "copy_if_changed" command (you can grep for it in IDF) which can be used to avoid overwriting the header each time CMake runs, to avoid unnecessary recompiles. Append the input file(s) to CMAKE_CONFIGURE_DEPENDS (also used heavily in IDF) so CMake will automatically re-run if generation needs to be done again.

This is what I was doing in component.mk:

Code: Select all

#components/osc/component.mk
lib/tinyosc/tinyosc.o: CFLAGS += \
→ -D'htonll(x)=((((uint64_t)htonl(x) & 0xFFFFFFFF) << 32) + htonl((x) >> 32))' \
→ -D'ntohll(x)=((((uint64_t)ntohl(x) & 0xFFFFFFFF) << 32) + ntohl((x) >> 32))'

Oh wow... that's an adventurous use of the C preprocessor!

Best alternatives I can think of are:

• Find whatever C header tinyosc.o expects to find htonll inside (arpa/inet.h?), and add a "wrapper" header with the same name/path to a private include directory for this component only. Then the content of that header looks like:

  Code: Select all

  #pragma once
  #include_next <arpa/inet.h>
  inline static uint64_t hontll { blah; }
  // etc 
  

  The build system should pass private component header directory paths before public ones on the command line.
• Put the functions in a platform-specific header file somewhere and pass the option -include path/to/header.h to the compiler. This option causes gcc to include that header as if it was included at the top of the source file itself. Can do this in CMake with target_compile_options().

[ESP_Angus]

However, the largest hurdle right now seems to be adding a new flash target (in my case, a fatfs partition image that gets flashed to an offset set in a Kconfig `CONFIG_FATFS_PARTITION_BASE_ADDR`). Not really sure where to hook in my own partitions into the flashing.

I don't think this has been considered as a use case, but I will consider it now!

There is a feature about to land which allows extending idf.py with project-specific commands. But I think in this case of building the list of flashing targets, it'd be good if we can support it in a generic way.

Will try to think of something.

But when running ninja this way, it seems that the port and baud settings are not populated correctly eg when running just `ninja flash`.
(it tries to run the following command, with no args for `-p` or `-b`:)

Code: Select all

esptool.py --chip esp32 -p -b write_flash --flash_mode dio 

This is a bug, thanks for pointing it out.

[p-rimes]

However, the largest hurdle right now seems to be adding a new flash target (in my case, a fatfs partition image that gets flashed to an offset set in a Kconfig `CONFIG_FATFS_PARTITION_BASE_ADDR`). Not really sure where to hook in my own partitions into the flashing.

I don't think this has been considered as a use case, but I will consider it now!

There is a feature about to land which allows extending idf.py with project-specific commands. But I think in this case of building the list of flashing targets, it'd be good if we can support it in a generic way.

Will try to think of something.

Great! May I say that I am personally interested in a pure CMake workflow (using cmake / ninja directly in a build dir), so if I knew how to work around that port/baud bug I would be solved. I also think working out of a build directory is very close to allowing the possibility of multiple build dirs for the same project. Right now (I think) this isn't possible with esp-idf, since sdkconfig is stored in the root dir. But if I could run `ninja menuconfig` from within a build dir -- applied to sdkconfig in that same dir -- then I could have lots of different build dirs on the go (I do this frequently for e.g. {unit_test,home,prod} or {wifi1,wifi2}, and it is clunky right now). Using make/ninja from within a Release-esp32 and Debug-esp32 folder is a common workflow on some x86 cmake projects, so that would be ideal for me for ESP32, too.

[ESP_Angus]

I would be solved. I also think working out of a build directory is very close to allowing the possibility of multiple build dirs for the same project. Right now (I think) this isn't possible with esp-idf, since sdkconfig is stored in the root dir.

You can override the SDKCONFIG path (using make variable named SDKCONFIG in the GNU Make system or with a CMake cache override in the CMake system) to have multiple configs. However, there's no way to get the behaviour you're describing by default right now.

Config management (being able to better slice and dice configs for a project's needs) is on the list of things we want to look at in future IDF versions, so describing these use cases is very useful.