Preview release: CMake-based build system for ESP-IDF

[p-rimes]

Doesn't seem to be the case for me. I have to run build 4 or 5 times to get all the files generated. As well, I am generating header files back into the component directory...

Yes, I see. I think CMake makes some internal distinction between "generated" and "not generated" files, so I can see this not working as expected. I think the approach you mention should work....

This is still not working for me, even though I think I have done things correctly... (set PROPERTIES GENERATED TRUE on the generated files, created a custom *_TARGET, populated a new variable with each file generated into a new variable *_OUTPUTS that gets injected into the PARENT_SCOPE, use that variable to set OBJECT_DEPENDS on the source files in that component that use the generated files)

I still need to re-run the build a few times to get everything generated. Here is an example CMake function (it will generate a `Foobar_generated.h` for each `Foobar.fbs` file in the function args), and another snippet for how I use it:

Code: Select all

function(FLATBUFFERS_GENERATE_GENERATED_H schema_generated_h)
  set(GENERATED_OUTPUTS)
  foreach(FILE ${ARGN})
    get_filename_component(SCHEMA ${FILE} NAME_WE)
    set(OUT "${COMPONENT_PATH}/src/gen/${SCHEMA}_generated.h")
    list(APPEND GENERATED_OUTPUTS ${OUT})

    add_custom_command(
      OUTPUT ${OUT}
      COMMAND flatc
      ARGS
        --cpp -o "${COMPONENT_PATH}/src/gen/"
        -I "${PROJECT_PATH}/esp32-network-lib/uuid"
        --scoped-enums
        --gen-mutable
        --gen-name-strings
        --reflect-types
        --reflect-names
        "${FILE}"
      DEPENDS "${FILE}"
      COMMENT "Building flatbuffers C++ header for ${FILE}"
      WORKING_DIRECTORY "${COMPONENT_PATH}"
    )
  endforeach()

  set_source_files_properties("${GENERATED_OUTPUTS}" PROPERTIES GENERATED TRUE)

  set_property(
    DIRECTORY "${COMPONENT_PATH}"
    APPEND PROPERTY
    ADDITIONAL_MAKE_CLEAN_FILES "${GENERATED_OUTPUTS}"
  )

  set(${schema_generated_h}_OUTPUTS ${GENERATED_OUTPUTS} PARENT_SCOPE)
  add_custom_target(${schema_generated_h}_TARGET DEPENDS ${GENERATED_OUTPUTS})
endfunction()

Code: Select all

FLATBUFFERS_GENERATE_GENERATED_H(uuid_generated_h uuid.fbs)

set_property(
  SOURCE src/uuid.cpp
  APPEND PROPERTY
  OBJECT_DEPENDS "${uuid_generated_h_OUTPUTS}"
)
add_dependencies(${COMPONENT_NAME} uuid_generated_h_TARGET)

I think I have done everything right, however I still get errors like these (which seems to only happen for other components going across dependencies, even though the dependencies are already listed in COMPONENT_REQUIRES):

Code: Select all

In file included from ../requests/src/request_manager.cpp:11:
../uuid/src/uuid.h:13:28: fatal error: uuid_generated.h: No such file or directory
compilation terminated.

e.g. here the `requests` component fails since the uuid_generated.h file from the `uuid` component is not generated in time. I don't seem to have any issues for files within the same component anymore (the `OBJECT_DEPENDS` part I added seems to be working inside the component, then?)

Oh wow... that's an adventurous use of the C preprocessor!

Best alternatives I can think of are:...

Oh yeah! `-include ${COMPONENT_PATH}/src/foo.h` is an excellent approach! I've hunted for an option like that before, so that's another thing I learned from this thread!

I also have some additional CMake questions:

• Are the empty settings `set(COMPONENT_REQUIRES)` and/or `set(COMPONENT_SRCS)` required in every `CMakeLists.txt`, even when they are empty?
• How can we override the C++ std setting (e.g. to use `-std=c++14`)
• When is it necessary to prefix `${COMPONENT_PATH}/` to file paths, or are they always relative to the compiling component?

[p-rimes]

I really wish there were some CMake functions provided by esp-idf, that could automate certain programming steps.
I've looked at the `flasher_args.json` stuff and... it's not really CMake friendly.

In my case I have a FATfs image which is generated, and should be flashed to a specific offset. Right now I have to create a `build/flash_MYTARGET_args` file with all the correct write_flash settings. I could generate this _args file as part of the build, but I'm really just looking for something like:

# (CMake variable, or setting in a Kconfig:)
set(FATFS_OFFSET 0x12345678)
esptool_py_custom_flash_target(fatfs_image_flash ${FATFS_OFFSET} "fatfs_factory.img")

[Deouss]

This is very helpful thread. I proposed to create a new board section just for CMake to keep better track about CMake development status. Wonder how many people would like that

[p-rimes]

Cool CMake-related commit here!
https://github.com/espressif/esp-idf/co ... 91ce116d2d

For CMake-only, we no longer need to specify serial flasher device path/baud rate (previously was set in `menuconfig` -> `Serial flasher config`), a serial device will be auto-detected by esptool.py when run from default `ninja flash` target, and for specific settings can be overridden at flash-time by environment variables `ESPPORT`, and `ESPBAUD`.

Now changing serial device does not trigger a full rebuild, different devices can be programmed from the same build folder just by changing the settings. This is excellent for automation, release archiving, and/or CI-type purposes.

Thank you @ESP_Angus, this is a very appreciated change.

[ESP_Angus]

Now changing serial device does not trigger a full rebuild, different devices can be programmed from the same build folder just by changing the settings. This is excellent for automation, release archiving, and/or CI-type purposes.

Thank you @ESP_Angus, this is a very appreciated change.

Glad you like it!

For the record, this works since ESP-IDF V1.0 with GNU Make as well, ie you can do "make flash ESPPORT=xyz ESPBAUD=460800" and it will use that port & baud rate without triggering a full rebuild.

The main difference with CMake is taking the sdkconfig defaults out of the system entirely.

Are the empty settings `set(COMPONENT_REQUIRES)` and/or `set(COMPONENT_SRCS)` required in every `CMakeLists.txt`, even when they are empty?

No, shouldn't be. The main difference to GNU Make is that none of these variables have default values any more, so unset == empty rather than a default.

How can we override the C++ std setting (e.g. to use `-std=c++14`)

Do this in a component CMakeLists.txt file with component_compile_options, ie component_compile_options(-std=c++14)

This means the compiler command line will have two -std= options, but the component-specific ones will come after the project-wide ones and override them. (If a source-file-specific compile option is passed, this comes last of all.)

When is it necessary to prefix `${COMPONENT_PATH}/` to file paths, or are they always relative to the compiling component?

Generally relative to the compiling component. The only exception (I think) is defining custom commands with add_custom_command(), as these are run in the component build directory by default so paths in the command definition may need it.

This is very helpful thread. I proposed to create a new board section just for CMake to keep better track about CMake development status. Wonder how many people would like that

This is a good point. The thing is, so far we've kept all the ESP-IDF discussion in one place (this single forum) rather than making sub-forums for individual parts of IDF.

I agree that it is probably time to break out of this thread though, and start specific threads in the IDF forum for individual CMake-related questions. I think marking them with CMake in the title is probably enough to keep track, for now.

[p-rimes]

How can we override the C++ std setting (e.g. to use `-std=c++14`)

Do this in a component CMakeLists.txt file with component_compile_options, ie component_compile_options(-std=c++14)

This means the compiler command line will have two -std= options, but the component-specific ones will come after the project-wide ones and override them. (If a source-file-specific compile option is passed, this comes last of all.)

Ah, I did in fact mean at a project-wide level here. I found that it didn't work to e.g. `add_compile_options(-std=c++14)` in the top-level `CMakeLists.txt` (the esp-idf default `-std=` option did come after it, overriding my setting). However what did work (project-wide) was the following in top-level `CMakeLists.txt`:

Code: Select all

set(CMAKE_CXX_STANDARD 14)

This seems to put the `-std=` arg after the esp-idf ones (and component ones as well). Is there a reason for `esp-idf` to use `-std=` directly, instead of the `CMAKE_CXX_STANDARD` variable? Seems like there would only be one `-std=` arg emitted then (so if I override that var elsewhere, project or component, it won't matter where the arg gets placed)

[ESP_Angus]

Is there a reason for `esp-idf` to use `-std=` directly, instead of the `CMAKE_CXX_STANDARD` variable?

The reason is: I didn't realise that CMAKE_<lang>_STANDARD existed until just now.

I think there's no reason we can't change the CMake-based build system to use this variable instead. However, ESP-IDF only supports compiling ESP-IDF's internal source files with the IDF's default C/C++ standards. If you override it on the entire project (rather than just in some particular components), then you're on your own...

[p-rimes]

Is there a reason for `esp-idf` to use `-std=` directly, instead of the `CMAKE_CXX_STANDARD` variable?

The reason is: I didn't realise that CMAKE_<lang>_STANDARD existed until just now.

I think there's no reason we can't change the CMake-based build system to use this variable instead. However, ESP-IDF only supports compiling ESP-IDF's internal source files with the IDF's default C/C++ standards. If you override it on the entire project (rather than just in some particular components), then you're on your own...

Yikes! I have been compiling my CMake prototypes with -std=c++14 everywhere (seems to work OK, but I don't want to do that if it isn't supported).

Can we get some option or CMake function to configure the `-std=` level for everything within a project, but excluding esp-idf sources? Would be a lot easier than manually overriding in each component to C++14.

[ESP_Angus]

Yikes! I have been compiling my CMake prototypes with -std=c++14 everywhere (seems to work OK, but I don't want to do that if it isn't supported).

There isn't any reason that I know of why it won't work, the standards tend to be backwards compatible (although we build with std=gnu11 which adds some GNU-specific extensions). But we also can't support every possible combination of compiler flags - the ones which the ESP-IDF build & config system sets are the ones that we test with, everything else is (de facto) untested.

Can we get some option or CMake function to configure the `-std=` level for everything within a project, but excluding esp-idf sources? Would be a lot easier than manually overriding in each component to C++14.

I will think about this, but probably not. Our mid-term plan is to introduce package management for ESP-IDF and move a lot of the less commonly used components outside of the core IDF repo. This means the easiest level for specific settings to be introduced will be at the component level.

Something we probably can do is change the default C++ version to C++14 (or, more likely, gnu++14) in ESP-IDF V4.0.

[Oleg Endo]

I've just tried the latest GCC 8.2 toolchain and latest IDF master branch with cmake. Something's off, apart from the few build failures (GCC's new warnings turned in to errors ... doesn't really turn off even if selected so in the menuconfig).

When building the hello world example app using cmake, flashing works fine but the resulting code does not run. If it starts, it prints garbage and then crashes and gets restarted by the WDT. However, building the same app using the old build system (and the new GCC 8.2 toolchain), works fine.

Am I missing something here?