Compatibility between "normal CMake" and ESP-IDF

[ESP_Angus]

Angus,

I've reached a point where I no longer can hold off the software side of the project. As such I'm going to start converting my framework into components and I just want to ask if there are any major changes coming to the CMake build system that, if known, can prevent double work.

As always, thanks for listening.

There are some changes coming, we have someone working on the use cases of better generic CMake integration. But we don't have a timeline for this to be available yet, sorry.

[permal]

There are some changes coming, we have someone working on the use cases of better generic CMake integration. But we don't have a timeline for this to be available yet, sorry.

Even without a timeline that is good news. Thank you.

[permal]

Hi,

My quest to compile both my application and framework code for both Linux and IDF continues. Today I tried to write a CMakeLists.txt that is aware if it is being compiled for IDF or native Linux.

Issue #1: idf.py does not define IDF_PLATFORM. Easily fixed by updating it:

Code: Select all

cmake_args = ["cmake", "-G", args.generator, "-DPYTHON_DEPS_CHECKED=1", "-DIDF_PLATFORM=1"]

Issue #2: The IDF cmake function "project" in project.cmake does not define IDF_PLATFORM. Also easily fixed:

Code: Select all

execute_process(COMMAND "${CMAKE_COMMAND}"
...
-D "ESP_PLATFORM=1"
...
WORKING_DIRECTORY "${PROJECT_PATH}")

Issue #3: When CMake is run in script mode, i.e. via the -P switch as in #2, there are certain functionalities of CMake that cannot be used. Among these are project, target_compile_definitions and add_library, the latter actually being referred to in the documentation on how to write IDF components in "pure" CMake for IDF.

So, I end up with the following error and subsequent failure.

Executing "cmake -G 'Unix Makefiles' -DPYTHON_DEPS_CHECKED=1 -DIDF_PLATFORM=1 --warn-uninitialized /home/permal/electronics/IO-Card-G3/software"...
Warn about uninitialized values.
-- Found Git: /usr/bin/git (found version "2.17.1")
CMake Error at main/CMakeLists.txt:6 (add_library):
add_library command is not scriptable
Call Stack (most recent call first):
/home/permal/esp/esp-idf/tools/cmake/scripts/expand_requirements.cmake:169 (include)
/home/permal/esp/esp-idf/tools/cmake/scripts/expand_requirements.cmake:207 (expand_component_requirements)


CMake Error at /home/permal/esp/esp-idf/tools/cmake/project.cmake:74 (include):
include could not find load file:

/home/permal/electronics/IO-Card-G3/software/build/component_depends.cmake
Call Stack (most recent call first):
CMakeLists.txt:10 (project)

Issue #4: CMakeLists.txt in "main" needs to be duplicated in top-level CMakeLists.txt when compiling for Linux. This is because IDF expects the main app to be in the "main" folder and as such the "add_executable()" call when compiling for Linux needs to either refer to files in "main" folder like this:

project(g3)
include_directories(externals/smooth/include)
add_subdirectory(externals/smooth)
add_executable(g3 main/main.cpp)
target_link_libraries(g3 smooth pthread)

or, we can use an empty cpp file like this and link in the main library:

project(g3)
include_directories(externals/smooth/include)
add_subdirectory(main)
add_subdirectory(externals/smooth)
add_executable(g3 empty.cpp)
target_link_libraries(g3 main smooth pthread)

The first alternative is no good because of duplication. The second is slightly better, but still rather ugly. I'd prefer removing the requirement of putting files in "main" so that the top-level c-list can be used both for Linux and IDF.

For reference, the top-level project is here: https://github.com/PerMalmberg/IO-Card- ... eLists.txt
and the submodule is here: https://github.com/PerMalmberg/Smooth/b ... eLists.txt

Issue 1,2 and 4 are at least solvable or has a workaround, but number 3 still prevents me from getting any further. What's your take on this Angus?

Re the IDF_PLATFORM, should I create a PR for that or is it already on its way?

[permal]

Followup on last nights adventure.

IDF_PLATFORM should of course be ESP_PLATFORM

Issue #5: The fact that the main application in an IDF app must be in the "main" folder breaks standard CMake functionality in target_link_libraries() since it is not possible to link a sibling library to another sibling, i.e. "main" can not have a dependency
on "other library". This forces us to replicate the "main" CMakeLists.txt in the top-level CMakeLists.txt like so:

Code: Select all

project(g3)
add_subdirectory(externals/smooth)
add_executable(g3 main/main.cpp) # <-- Pointing to main app
target_link_libraries(g3 smooth pthread)

I had hopes of extracting the contents of main/CMakeLists.txt to an app.cmake file I could include from the original and also from the top-level, but I could't get that to work; include() can't find it when run via idf.py.[/s]
Update: Turns out you actually can use target_link_libraries() on siblings.

I've now opted to manage two paths in the CMake files; one for native Linux and one to build against the IDF. It's not pretty, but at least I can compile for both. Next up is to make all the test projects functional, I'll keep you updated on what other problems I find.

Edit: PR to support dual-platform CMakeLists.txt files.

[permal]

Hello again,

Stumbled on another issue: register_component() tries to create a target based on the folder from which the call is made. In my case I have a folder named "test" which contains the file common_project.cmake with the following contents:

Code: Select all

get_filename_component(PROJECT ${CMAKE_CURRENT_SOURCE_DIR} NAME)

message(STATUS "Building: ${PROJECT}")

set(SOURCES
        ${PROJECT}.cpp
        ${PROJECT}.h)

if (${ESP_PLATFORM})
    set(COMPONENT_SRCS ${SOURCES})
    set(COMPONENT_REQUIRES smooth)
    set(COMPONENT_ADD_INCLUDEDIRS ${COMPONENT_PATH})
    register_component()
else ()
    project(${PROJECT})
    add_library(${PROJECT} ${SOURCES})
    target_link_libraries(${PROJECT} smooth)
    target_include_directories(${PROJECT} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
endif ()

This file acts as a template for all my test projects. By including it in the respective CMakeLists.txt in each test project (test/logging; test/hello_world etc) this it becomes the entire CMakeLists.txt for a test project.

Code: Select all

include(${CMAKE_CURRENT_LIST_DIR}/../common_project.cmake)

However, since the register_component() uses CMAKE_CURRENT_LIST_FILE as a base for the project name, it gets it wrong and tries to create a project/component named "test", which CMake doesn't like at all. The the name could have been anything and it would still have been a problem since it uses the name of the directory from which it is called and thus would have tried to create multiple projects with the same name.

Changing the second call to get_filename_component() in components.cmake / register_component() to this resolves this issue:

Code: Select all

get_filename_component(component_dir ${CMAKE_CURRENT_LIST_FILE} DIRECTORY)
get_filename_component(component ${CMAKE_CURRENT_SOURCE_DIR} NAME)

I don't know what changes you have coming for the CMake build system, but please keep this scenario in mind. The already existing PR includes this change too since I haven't branched my fork.

[toshi38]

Hey guys,

I"m also running into similar problems with the

Code: Select all

project

issues. My setup is something like this:

I have my project included under `contrib` or similar folder, it has a CMakeLists.txt that uses the handy DownloadProject cmake plugin (https://github.com/Crascit/DownloadProject) to add some libraries... things like libcurl etc. These are then fetched and added to my build.

As a result while I can of course edit my project (although I'd rather not) I'd really like to avoid updating these other libraries as it seems like a really slippery slope to have to go through updating CMakeLists.txt files through all subdependencies in order to get around these build issues. Is there a current recommended way to handle this?

@permal have you got your builds working in some way now? I saw some comments about changing project to idf_project or similar but wasn't sure if you'd managed to resolve things by doing this? I'll likely look at trying to dig into this tomorrow, but if someone has some tips it could save me some time, and open me up to making a PR faster (If a PR would be welcome?)

[permal]

@toshi38 Yes, I 've managed to get my projects to build, but it is not pretty. As I wrote above, I opted to maintain two sections in each CMakeLists.txt - one for native compilation and one for IDF. Its double work and I've hit several blockers due to how IDF uses CMake. Also, I've had to modify the IDF buildsystem slightly to support this method, see this PR: https://github.com/espressif/esp-idf/pull/2601

You can have a look at my solution here if you want: https://github.com/PerMalmberg/Smooth/t ... t_to_cmake

I'm currently merging my test projects into that repo so its all a bit WiP at the moment. Using the root CMakeLists.txt you should be able to compile for both IDF and native Linux.

@Espressif - Any chance you can let us know what is coming?

[toshi38]

@permal Thanks!

I've found a super simple work around for the `project` and sub project issues. That said I'm not a cmake guru so still trying to evaluate what _other_ side effects this approach has before I consider making a PR.

Inside the idf cmake file project.cmake I modified their `project` macro to have something like this:

Code: Select all

macro(project name)
    if(NOT PROJECT_INITIALIZED)
        message("*** *** First project call, initializing: " ${name})
        #.... all of the current stuff
    else()
        message("*** *** Just adding the project not initializing")
        # Just add the project as we've already set everything up!
        _project(${name} ASM C CXX)
    endif()

The problem I have now is that the various "components" added via `register_component()` are not being found by my dependencies using `find_package()` for example pthreads (which clearly exists). I think perhaps I need to add some proper package registration... still digging.

Code: Select all

  find_package(Threads REQUIRED)

BTW where in Sweden are you located? I'm in Malmö!

[permal]

@toshi38 Considering that we end up changing the IDF files shows that it doesn't yet meet the real-world requirements. That said, it is till in preview and it is this kind of feedback Espressif is looking for, so I guess we can't really complain.

I'm located just north of Stockholm.

[js]

It's not just that it does not integrate with CMake - it does not integrate with anything. Even if it would integrate with CMake, that would still not be sufficient, as not all software uses CMake.

The problem is that the toolchain is actually incomplete - if you try to use xtensa-esp32-elf-gcc to compile an empty .elf, it will fail, because it will not use the right linker script and there are no C runtime files to be found.

The toolchain should either be extended to be a full toolchain, like for other embedded targets, or there should be a target in esp-idf to build the missing files, so that at the end, there's a fully working compiler. Then, it will be possible to just ./configure --host=xtensa-esp32-elf to build other libraries.