Using idf.cmake in a custom CMake project with multiple executables

[itavero]

In our company we are working on a mono-repo approach for our new platform products.
For these products some of them include an ESP32-S3 or an ESP32-C3.
The exact chip isn't that important, as for each architecture we'll need a different toolchain thus a different CMake configure step.

Within a single CMake project I want to define multiple executable targets and have these be build (all for the same MCU type/architecture).
It's unclear to me how I should use the CMake commands provided by including idf.cmake (as described in the documentation on using ESP-IDF in your own CMake project) when having multiple executables.
As these executables will be for different products, they will likely have different sdkconfigs, partition tables, etc.
The API of idf_build_process seems to be too limited or unclear to me.
Somehow it seems to assume that there's only one executable to build, which does not make sense to me.

Is there an example available where this is actually being used?
Or can someone point me in the right direction?

[itavero]

Briefly scanned this GitHub issue: https://github.com/espressif/esp-idf/issues/3640

If I understand correctly this is currently not possible without duplicating and extending a lot of the scripting. That's a real pitty. We have other MCUs as well for which we create reusable software components. So far for all MCUs we've been able to use consistent a Modern CMake approach. Having to do something totally different for the Espressif MCUs is disappointing to say the least.

[itavero]

Diving a bit into the CMake scripts in ESP-IDF, it seems that the main problem is in tools/cmake/build.cmake.
All options and such are added to a single __idf_build_target target.

If we could extend this to optionally be able to pass a custom name (which could simply be used as a suffix to this), I think the problem is solvable.
Unfortunately, there's several other CMake files in that same directory that of course depend on the functions defined in build.cmake, so all these functions should also be extended with the ability to pass a custom name if needed.

Probably there's more work with regards to components, as of course you might want to add a single component to multiple executables.

All in all, it won't be a small change (though the impact of such a change can be limited).

[ESP_jakob]

Hello,

we're sorry you had a tough time with the IDF build system. It shouldn't be difficult for users to use it. I have a few questions, maybe just to better understand the issue here:

IDF projects usually build one executable. An IDF project basically bundles configurations, sources and other things that are unique for a single application (see also https://docs.espressif.com/projects/esp ... l#concepts). Is it not possible to create different IDF projects for different applications within your repository?

If you need re-usable software, would it be possible to create IDF components for this software or wrap IDF components if it's a 3rd-party library?

[itavero]

This single repository contains our embedded code base for multiple products.
Some products use Espressif MCUs, some products use other ARM Cortex-M33 MCUs or a combination of an ARM and ESP.
Currently we are setting this up as one big CMake project, with individual executable targets for specific products (which of course only apply to specific MCUs) as well as library targets for reusable software components.

We want to define all our software components in such a way that they can be used for both MCUs. So this would be an add_library call in CMake (we're not 100% sure yet if we'll use static libraries or object libraries, we'll have to see what the impact on binary size is in the end).
So, for the components that I might need for an application that is build for an ESP, ideally I do not want to have to do something specific in case we are compiling for ESP (aside from configuring compile flags correctly of course).
I just want to link it to my executable target in the end.

As we will have multiple products that will be very similar (e.g. both using an ESP32-S3 and perhaps even the same IDF components), I would like to be able to compile them in one configuration.
That way I only need to build my relevant reusable components once in my CI pipeline and on my development machine, I also only need one CMake configuration for each MCU architecture, which limits the amount of switching I have to do while developing software for a group of products.

Hopefully this clarifies the use case a bit.

[itavero]

Another update from my side. While trying to integrate the idf.cmake as we would expect, we ran in to some issues:

• The CMake script for esptool_py incorrectly assumed that the target name always matches the name of the binary it produces (see PR #12558)
• It seems to matter where idf.cmake is included, as the targets it defines seem to "disappear" on a higher level. (see this post on the CMake Discourse)

We still need to figure out what is needed to change in the ESP-IDF to support multiple boards (sdkconfigs) and multiple executable targets, but I first want to have the idf.cmake integrated as we'd expect it for a single executable.

[nctnico]

I just want to chime in to 'say' I'm running into the same problem. I want to create a testing, encrypted and certification build for the same project. All this would need is an option to select an sdkconfig file and set C / C++ compiler flags to a build target. I'm rather surprised that this isn't possible out of the box as standard Eclipse CDT has supported this workflow for over 15 years.

[itavero]

The annoying part we ran into, is that ESP-IDF also includes and preconfigures all its components (based on what's in the sdkconfig if I recall correctly).

Effectively this would mean that we'd probably have to create an sdkconfig-specific target for each component that is added.
To be honest, for our workflow I would be fine with that, but it's a lot of hassle to change their scripts (especially with the risk of Espressif not merging it back into ESP-IDF and we having to do the same stuff over and over again every time we wish to use a new ESP-IDF release).

It would be really awesome if someone at Espressif would take a critical look at their build system and see if they can improve it for these kinds of use cases.
Just the fact that they are doing a lot of "non-standard" / non-modern CMake stuff really bugs me.

Unfortunately, at the same time we're busy developing new products with these MCUs that have to hit the market in time, so I don't really have time to waste on getting this to work properly.