Compatibility between "normal CMake" and ESP-IDF

[permal]

Angus,

This is the link command:

Code: Select all

[ 98%] Linking CXX executable main
cd /home/permal/electronics/IO-Card-G3/software/externals/smooth/build/test/main && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/main.dir/link.txt --verbose=1
/home/permal/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++  -mlongcalls  -nostdlib CMakeFiles/main.dir/generated_test.cpp.obj  -o main ../../lib/smooth/libsmooth.a ../hello_world/libhello_world.a ../../lib/smooth/libsmooth.a -Wl,--start-group ../../lib/smooth/esp-idf/vfs/libvfs.a ../../lib/smooth/esp-idf/esp_ringbuf/libesp_ringbuf.a ../../lib/smooth/esp-idf/driver/libdriver.a ../../lib/smooth/esp-idf/esp_event/libesp_event.a ../../lib/smooth/esp-idf/ethernet/libethernet.a ../../lib/smooth/esp-idf/lwip/liblwip.a ../../lib/smooth/esp-idf/tcpip_adapter/libtcpip_adapter.a ../../lib/smooth/esp-idf/app_update/libapp_update.a ../../lib/smooth/esp-idf/spi_flash/libspi_flash.a ../../lib/smooth/esp-idf/mbedtls/libmbedtls.a ../../lib/smooth/esp-idf/mbedtls/mbedtls/library/libmbedtls.a ../../lib/smooth/esp-idf/mbedtls/mbedtls/library/libmbedx509.a ../../lib/smooth/esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a ../../lib/smooth/esp-idf/soc/libsoc.a ../../lib/smooth/esp-idf/log/liblog.a ../../lib/smooth/esp-idf/heap/libheap.a ../../lib/smooth/esp-idf/freertos/libfreertos.a -Wl,--undefined=uxTopUsedPriority ../../lib/smooth/esp-idf/newlib/libnewlib.a -L /home/permal/esp/esp-idf/components/newlib/lib -lc -lm ../../lib/smooth/esp-idf/micro-ecc/libmicro-ecc.a ../../lib/smooth/esp-idf/bootloader_support/libbootloader_support.a ../../lib/smooth/esp-idf/nvs_flash/libnvs_flash.a ../../lib/smooth/esp-idf/pthread/libpthread.a ../../lib/smooth/esp-idf/smartconfig_ack/libsmartconfig_ack.a ../../lib/smooth/esp-idf/wpa_supplicant/libwpa_supplicant.a ../../lib/smooth/esp-idf/xtensa-debug-module/libxtensa-debug-module.a ../../lib/smooth/esp-idf/esp32/libesp32.a -L /home/permal/esp/esp-idf/components/esp32/lib -lcoexist -lcore -lespnow -lmesh -lnet80211 -lphy -lpp -lrtc -lsmartconfig -lwpa2 -lwpa -lwps -L /home/permal/electronics/IO-Card-G3/software/externals/smooth/build/lib/smooth/esp-idf/esp32 -T esp32_out.ld -T /home/permal/electronics/IO-Card-G3/software/externals/smooth/build/lib/smooth/esp-idf/esp32/esp32.common.ld -L /home/permal/esp/esp-idf/components/esp32/ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.libgcc.ld -T esp32.rom.spiram_incompatible_fns.ld /home/permal/esp/esp-idf/components/esp32/libhal.a -lgcc -u call_user_start_cpu0 -u ld_include_panic_highint_hdl ../../lib/smooth/esp-idf/cxx/libcxx.a -lstdc++ -u __cxa_guard_dummy ../../lib/smooth/esp-idf/app_trace/libapp_trace.a -lgcov ../../lib/smooth/esp-idf/asio/libasio.a ../../lib/smooth/esp-idf/jsmn/libjsmn.a ../../lib/smooth/esp-idf/coap/libcoap.a ../../lib/smooth/esp-idf/console/libconsole.a ../../lib/smooth/esp-idf/nghttp/libnghttp.a ../../lib/smooth/esp-idf/esp-tls/libesp-tls.a ../../lib/smooth/esp-idf/esp_adc_cal/libesp_adc_cal.a ../../lib/smooth/esp-idf/tcp_transport/libtcp_transport.a ../../lib/smooth/esp-idf/esp_http_client/libesp_http_client.a ../../lib/smooth/esp-idf/esp_http_server/libesp_http_server.a ../../lib/smooth/esp-idf/esp_https_ota/libesp_https_ota.a ../../lib/smooth/esp-idf/openssl/libopenssl.a ../../lib/smooth/esp-idf/esp_https_server/libesp_https_server.a ../../lib/smooth/esp-idf/expat/libexpat.a ../../lib/smooth/esp-idf/wear_levelling/libwear_levelling.a ../../lib/smooth/esp-idf/sdmmc/libsdmmc.a ../../lib/smooth/esp-idf/fatfs/libfatfs.a ../../lib/smooth/esp-idf/freemodbus/libfreemodbus.a ../../lib/smooth/esp-idf/json/libjson.a ../../lib/smooth/esp-idf/libsodium/liblibsodium.a ../../lib/smooth/esp-idf/mdns/libmdns.a ../../lib/smooth/esp-idf/mqtt/libmqtt.a ../../lib/smooth/esp-idf/protobuf-c/libprotobuf-c.a ../../lib/smooth/esp-idf/protocomm/libprotocomm.a ../../lib/smooth/esp-idf/spiffs/libspiffs.a ../../lib/smooth/esp-idf/ulp/libulp.a ../../lib/smooth/esp-idf/unity/libunity.a ../../lib/smooth/esp-idf/wifi_provisioning/libwifi_provisioning.a 
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(pm.o): In function `pm_set_sleep_cycles':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:60: undefined reference to `rtc_slowck_cali'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(pm.o): In function `pm_rtc_clock_cali':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:68: undefined reference to `rtc_slowck_cali'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(pm.o):(.text.pm_set_sleep_mode_full+0x10): undefined reference to `rtc_slp_prep'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(pm.o): In function `pm_set_sleep_mode_full':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:237: undefined reference to `rtc_slp_prep'
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:265: undefined reference to `rtc_slp_prep'
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:308: undefined reference to `rtc_slp_prep'
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:358: undefined reference to `rtc_slp_prep'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(pm.o):/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/pm.c:370: more undefined references to `rtc_slp_prep' follow
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(rtc.o): In function `rtc_soc_clk_ck12m':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/rtc.c:230: undefined reference to `rtc_init_clk'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(rtc.o): In function `rtc_init_full':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/rtc.c:333: undefined reference to `rtc_init_clk'
/home/permal/esp/esp-idf/components/esp32/lib/librtc.a(rtc.o): In function `rtc_slp_prep_lite_12M':
/home/qgu/git_tree/chip7.1_rtc/board_code/app_test/pp/rtc/rtc.c:448: undefined reference to `rtc_slp_prep'
/home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/lib/libstdc++.a(thread.o):(.literal._ZNSt6thread20hardware_concurrencyEv+0x0): undefined reference to `sysconf'
/home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/lib/libstdc++.a(thread.o): In function `std::thread::hardware_concurrency()':
/builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/src/c++11/thread.cc:160: undefined reference to `sysconf'
collect2: error: ld returned 1 exit status
test/main/CMakeFiles/main.dir/build.make:143: recipe for target 'test/main/main' failed
make[2]: *** [test/main/main] Error 1
make[2]: Leaving directory '/home/permal/electronics/IO-Card-G3/software/externals/smooth/build'
CMakeFiles/Makefile2:258: recipe for target 'test/main/CMakeFiles/main.dir/all' failed
make[1]: *** [test/main/CMakeFiles/main.dir/all] Error 2
make[1]: Leaving directory '/home/permal/electronics/IO-Card-G3/software/externals/smooth/build'
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

I doubt it is useful, but here are the last few thousand lines of output of that make-command.

I'm using today's master branch, 7fa98593bc179ea50a1bc8244d5b94bac59c9a10 of IDF, with all submodules updated
The code I'm compiling is this branch of my C++ framework: https://github.com/PerMalmberg/Smooth/t ... -new-cmake
simply do these steps to reproduce.

Code: Select all

mkdir build
cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=$IDF_PATH/tools/cmake/toolchain-esp32.cmake -DIDF_TARGET=esp32
make

The same code compiles and links when IDF isn't used as a library.

[ESP_Angus]

Thanks permal. Can reproduce, will fix.

[ESP_Angus]

permal, a workaround is to call this from the project CMakeLists.txt:

Code: Select all

target_link_libraries(${CMAKE_PROJECT_NAME}.elf "-Wl,--gc-sections")

(Currently this is added by the IDF project, so doesn't happen in a non-IDF project, but it's a required flag if linking the wifi libraries.)

[permal]

permal, a workaround is to call this from the project CMakeLists.txt:

Code: Select all

target_link_libraries(${CMAKE_PROJECT_NAME}.elf "-Wl,--gc-sections")

(Currently this is added by the IDF project, so doesn't happen in a non-IDF project, but it's a required flag if linking the wifi libraries.)

Nice, that brought me a step further. I'm now battling an issue where IDF doesn't play nicely if the main executable isn't generated by the top-level project (mine is in test/main). Not sure this'll remain an issue for me, but thought you should know.

Btw, how's the fix for menuconfig coming?

[permal]

Ok, so after getting the tests/examples of my framework to compile, link and run it was time to get it all working with a larger application, which in practice means adding another CMake layer. Like this:

Code: Select all

CMakeLists.txt
|--- main
|       |-- CMakeLists.txt
|
|--- Framework
	|-- CMakeLists.txt
	        |---- IDF as lib

I got this compile and link by adding these lines to the top CMakeLists.txt file:

Code: Select all

set(IDF_BUILD_ARTIFACTS ON)
set(IDF_PROJECT_EXECUTABLE g3)
set(IDF_BUILD_ARTIFACTS_DIR ${CMAKE_CURRENT_LIST_DIR}/build)

However I once again ran into problems with idf.py not wanting to play nice with this setup, just as it didn't with the main app being in a sub directory to the top level CMakeLists.txt.

Code: Select all

[100%] Linking CXX executable g3
[100%] Built target g3
Scanning dependencies of target app
[100%] Generating ../../../../../g3.bin
esptool.py v2.6-beta1
Traceback (most recent call last):
  File "/home/permal/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2865, in <module>
    _main()
  File "/home/permal/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2858, in _main
    main()
  File "/home/permal/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2631, in main
    operation_func(args)
  File "/home/permal/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 2175, in elf2image
    e = ELFFile(args.input)
  File "/home/permal/esp/esp-idf/components/esptool_py/esptool/esptool.py", line 1725, in __init__
    with open(self.name, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'g3'
externals/smooth/lib/smooth/esp-idf/CMakeFiles/app.dir/build.make:60: recipe for target 'g3.bin' failed
make[2]: *** [g3.bin] Error 1
CMakeFiles/Makefile2:749: recipe for target 'externals/smooth/lib/smooth/esp-idf/CMakeFiles/app.dir/all' failed
make[1]: *** [externals/smooth/lib/smooth/esp-idf/CMakeFiles/app.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

The main project is being built from

Code: Select all

/home/permal/electronics/IO-Card-G3/software/build

using command

Code: Select all

cmake .. -DCMAKE_TOOLCHAIN_FILE=$IDF_PATH/tools/cmake/toolchain-esp32.cmake -DIDF_TARGET=esp32 && make -j8

It seems idf.py expects the g3 binary to be located at externals/smooth/lib, which is where my framework's top level CMakeLists.txt is located, not the binary that was just produced which is located at

Code: Select all

/home/permal/electronics/IO-Card-G3/software/build/main/g3

Considering it compiles and link, it feels as if it should be possible to make some small adjustment to IDF so that it can find the binary. Any thoughts on this Angus?

The entire project is available here, within the software folder: https://github.com/PerMalmberg/IO-Card-G3.git

[permal]

Weekend - got some time to look at this again.

IDF makes the assumption that the binary is placed in the top level project where as mine is <root>/main/

Placing the binary where IDF expects it works ofc:

Code: Select all

set_target_properties( ${PROJECT_NAME}
        PROPERTIES
        RUNTIME_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/.."
        )

However, this is a bit messy since it ads artificial requirements on the project instead of the tooling handling it.

We already have to specify this

Code: Select all

set(IDF_PROJECT_EXECUTABLE g3)

How about replacing that with something along these lines, allowing us to specify where the main project is located?

Code: Select all

idf_set_project_executable(g3 ${CMAKE_CURRENT_BINARY_DIR}/main)

[chodzikman]

Hi!

I am very happy with the direction you took - it is possible to use esp-idf as a library and use other CMake-managed libraries with ease, but my prefered use case is a bit more advanced:

I want to wrap esp-idf with my own abstraction layer (or in other words, implement my cross-platform abstraction layer for ESP32 with esp-idf) and there is a still loads of issues with your internal component trickery.

I would need to plainly link separate components from ESP-IDF to my wrapper libraries, creating executable in top-level CMakeLists.txt. Do you have any ideas on working it around for now, and do you have getting rid of this component trickery on your roadmap?

Nonetheless, great work so far!

[permal]

I want to wrap esp-idf with my own abstraction layer (or in other words, implement my cross-platform abstraction layer for ESP32 with esp-idf) and there is a still loads of issues with your internal component trickery.

That is what I've done with my framework. I've just gotten rid of 99.9% of the component trickery using IDF as a library which my library/abstraction layer links against. I only have one component left with the sole purpose to get configuration items into menuconfig.

[chodzikman]

Nice! I took a look into your repo - it looks like we have very similar goal (yet I have approached it slightly different) maybe we could merge some of our work in the future.

Anyway - Angus, could you please advice which way should we go? I am ready to get involved in the process a bit more, and for example frequently integrate your WIP into my project and/or give you my recommendations on what to avoid.

Also - is there still a blocking point with toolchain that prevents CMake build system perform just like ones for arm-none-eabi-gcc? Is this only caused by lack of library that provides 'syscalls'? If so, maybe instead of toolchain's

Code: Select all

set(CMAKE_EXE_LINKER_FLAGS "-nostdlib" CACHE STRING "Linker Base Flags")

there should be linking flag to stdlib developed inside IDF?

In general, I think, that your goal should be
1. (crucial) drop dependencies on target name inside your cmake trickery - maybe simple workaround would be to provide macro that should be executed in top-level CMakeLists.txt with all necessary variables provided as arguments
2. keeping whole project definition in plain CMake - 0 config/code generators etc - KConfig is not necessary if you can use plain CMake to opt in/out for components, and you have done quite significant progress since I have started to observe it.

Right now the best I can get is

Code: Select all

Scanning dependencies of target idf_component_mbedtls
[ 97%] Building C object esp-idf/mbedtls/CMakeFiles/idf_component_mbedtls.dir/mbedtls.c.obj
[ 98%] Linking CXX static library libmbedtls.a
[ 98%] Built target idf_component_mbedtls
[ 98%] Generating idf_component_mbedtls.sections_info
[ 98%] Built target idf_component_mbedtls_sections_info
[ 98%] Built target ldgen_section_infos
[100%] Built target ldgen_esp32.common.ld_script
[100%] Built target ldgen
Scanning dependencies of target Beerbob_FW_ESP32.elf
make[3]: *** No rule to make target 'Beerbob_FW_ESP32-NOTFOUND', needed by 'Beerbob_FW_ESP32.elf'.  Stop.
make[3]: *** Waiting for unfinished jobs....
[100%] Building CXX object CMakeFiles/Beerbob_FW_ESP32.elf.dir/main/application.cpp.obj
make[2]: *** [CMakeFiles/Makefile2:133: CMakeFiles/Beerbob_FW_ESP32.elf.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:496: esp-idf/CMakeFiles/app.dir/rule] Error 2
make: *** [Makefile:242: app] Error 2

It it helps you, you can use my project as reference:
https://gitlab.com/OpenBrewery/Firmware ... b_fw_esp32

and track my progress on branch feature/bsp

[chodzikman]

Hi,

Bumping topic up:
Could you possibly relate to this (probably) last major issue?