cmake and sha.h

[Reiner1210]

Hello,

The last 2 days I spent in upgrading my code to use the new netif instead of tcp adapter and the new event loop. Ok now it works.

Now I want to try to use cmake. For other Linux projects I always used GNU make.
How do I create CMake.txt files from my old make files ???
All my components lives in a single directory called mycomponents and the Applications have no main directory instead they have a src directory.

My normal root Makefile looks like this:

Code: Select all

PROJECT_NAME := time-switch

EXTRA_COMPONENT_DIRS += src $(abspath ../mycomponents)

include $(IDF_PATH)/make/project.mk

My CMakeLists.txt looks like this

Code: Select all

cmake_minimum_required(VERSION 3.5)

set(EXTRA_COMPONENT_DIRS src $ENV{IDF_PATH}/../mycomponents)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(time_switch)

In the src dir there is the following CMakeLists.txt

Code: Select all

idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS "."
                    EMBED_FILES "index.html" "update.html" "timeswitch.css" "reset.css" "timeswitch.js" "update.js" "config.html" "led_green_on.png" "led_green_off.png" "led_x_on.png")

idf.py gives me an error

Code: Select all

keFiles/__idf_src.dir/main.c.obj.d -o esp-idf/src/CMakeFiles/__idf_src.dir/main.c.obj   -c ../src/main.c
../src/main.c:26:10: fatal error: esp32/sha.h: No such file or directory
 #include "esp32/sha.h"
          ^~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Any idea how to glue the sha.h header in it ???

[ESP_Angus]

Hi Reiner,

In the src dir there is the following CMakeLists.txt

Code: Select all

idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS "."
                    EMBED_FILES "index.html" "update.html" "timeswitch.css" "reset.css" "timeswitch.js" "update.js" "config.html" "led_green_on.png" "led_green_off.png" "led_x_on.png")

idf.py gives me an error

Code: Select all

keFiles/__idf_src.dir/main.c.obj.d -o esp-idf/src/CMakeFiles/__idf_src.dir/main.c.obj   -c ../src/main.c
../src/main.c:26:10: fatal error: esp32/sha.h: No such file or directory
 #include "esp32/sha.h"
          ^~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Any idea how to glue the sha.h header in it ???

Something that's new in the CMake build system is that components have to declare their dependencies. This helps optimise the build process.

So, for example, if your component's source file includes the sha.h header from mbedtls component, then add the words "PRIV_REQUIRES mbedtls" inside the idf_component_register() function of the component CMakeLists.txt.

You can read more about this here, as well as some examples:
https://docs.espressif.com/projects/esp ... quirements

BTW, if possible would recommend including "mbedtls/sha.h" instead and calling the mbedTLS APIs. These will use the hardware accelerated SHA implementation if hardware acceleration is enabled in menuconfig (this is enabled by default in v4.1), so it's the same as including esp32/sha.h. However the esp32/sha.h header is part of the mbedTLS "port" layer and it may change at some point in the future, whereas the mbedTLS API should stay stable.

[Reiner1210]

Thanks I will try it.
But I think it is a painfull wayalways add dependencies.
So I prefer the normal make build system.
Build time is not so important than flexibility I tink.

Reiner

[Reiner1210]

It does not work

I have added
PRIV_REQUIRES "sha"

CMake Error at /home/reiner/esp/esp-idf/tools/cmake/build.cmake:185 (message):
Failed to resolve component 'sha'.
Call Stack (most recent call first):
/home/reiner/esp/esp-idf/tools/cmake/build.cmake:216 (__build_resolve_and_add_req)
/home/reiner/esp/esp-idf/tools/cmake/build.cmake:425 (__build_expand_requirements)
/home/reiner/esp/esp-idf/tools/cmake/project.cmake:357 (idf_build_process)
CMakeLists.txt:8 (project)


What is with the freertos/FreeRTOS.h Header must I also lit it ?

So I think the Cmake stuff is very bad and breaks existing projects .......

By the way I do not know if esp32 is used in an industrial and professional enviroment. But breaking existing code with new IDF releases will be a no go !

Reiner

[ESP_Angus]

I have added
PRIV_REQUIRES "sha"

CMake Error at /home/reiner/esp/esp-idf/tools/cmake/build.cmake:185 (message):
Failed to resolve component 'sha'.
Call Stack (most recent call first):
/home/reiner/esp/esp-idf/tools/cmake/build.cmake:216 (__build_resolve_and_add_req)
/home/reiner/esp/esp-idf/tools/cmake/build.cmake:425 (__build_expand_requirements)
/home/reiner/esp/esp-idf/tools/cmake/project.cmake:357 (idf_build_process)
CMakeLists.txt:8 (project)

The component that contains the "sha.h" header is called "mbedtls" not "sha". If you put "PRIV_REQUIRES mbedtls", it should work.

So I think the Cmake stuff is very bad and breaks existing projects .......

By the way I do not know if esp32 is used in an industrial and professional enviroment. But breaking existing code with new IDF releases will be a no go !

I'm sorry that you had a bad experience transitioning to CMake and I appreciate that the new build system is more complex and some things which "just worked" before require additional effort.

For the record, we take not breaking existing code very seriously.

The GNU Make build system is still supported in IDF v4.x, you can keep using an existing GNU Make based project from an earlier IDF release without converting it to CMake. There may be some minor adjustments needed due to Breaking Changes between v3.x and v4.x, as shown in the Breaking Changes section of the Release Notes: https://github.com/espressif/esp-idf/re ... v4.0-beta2

If you want to avoid any breaking changes between v3.x and v4.x, it's possible to keep using ESP-IDF v3.3.x. This is a Long Term Support release and will be supported until Feb 2022.

For more details see:
https://github.com/espressif/esp-idf/bl ... _POLICY.md
https://docs.espressif.com/projects/esp ... sions.html

[Reiner1210]

Thanks for your answer.

Well now I get a little bit farer now "nvs_flash.h" is missing.
Is there a tool which will create the dammed CMakeLists.txt files from old make files ?
Is there a rule how to find the right PRIV_REQUIRES entries ? How to get a component from header ?
What about my own components which stays outside of the project in a common folder ?
I can imagine that other people will have the same problems.
If you promise to support make in the future it will also be very nice

Ok I fixed the nvs_flash.

Now I have a component nvs_helper in a folder outside of the project. How to include it ?

Code: Select all

set(EXTRA_COMPONENT_DIRS "src $ENV{IDF_PATH}/../mycomponents")

is set in the cmake file in the project root

Code: Select all

PRIV_REQUIRES "mbedtls" "nvs_flash" "nvs_helper"

does not work. Do I need a Cmakefile in my shared componennt directory ?

Perhaps you should write a tutorial with examples what to put where and a description of the mystery of your keywords like
PRIV_REQUIRES - I think this is not a cmake macro


Best regards
Reiner

[ESP_Angus]

Hi Reiner,

Thanks for bearing with us on this.

Now I have a component nvs_helper in a folder outside of the project. How to include it ?

Code: Select all

set(EXTRA_COMPONENT_DIRS "src $ENV{IDF_PATH}/../mycomponents")

is set in the cmake file in the project root

Code: Select all

PRIV_REQUIRES "mbedtls" "nvs_flash" "nvs_helper"

does not work. Do I need a Cmakefile in my shared componennt directory ?

This looks like it should work, provided the line setting EXTRA_COMPONENT_DIRS is before the line with "project()". Can you please post the output from "idf.py reconfigure"? It will include the component paths that CMake found.

Perhaps you should write a tutorial with examples what to put where and a description of the mystery of your keywords like
PRIV_REQUIRES - I think this is not a cmake macro

There is some information about this in the docs, including a sample project layout:
https://docs.espressif.com/projects/esp ... le-project

It also includes the section about component requirements that I linked earlier, which includes a description of how REQUIRES and PRIV_REQUIRES are used. You're right these are not built-in CMake features:
https://docs.espressif.com/projects/esp ... quirements

I know this docs page is a bit jumbled though, we have a plan to reorganize it soon. If there's anything else you think is missing or unclear, please let us know.

[Reiner1210]

Hello,

Your hard effort is highly appreciated, so I will give you here all inormations:

Host system is Linux 64 Bit (Linux Mint)

Output of the export script

Code: Select all

Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements from /home/reiner/esp/esp-idf/requirements.txt are satisfied.
Added the following directories to PATH:
  /home/reiner/esp/esp-idf/components/esptool_py/esptool
  /home/reiner/esp/esp-idf/components/espcoredump
  /home/reiner/esp/esp-idf/components/partition_table/
  /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin
  /home/reiner/esp/tools/xtensa-esp32s2-elf/esp-2019r2-8.2.0/xtensa-esp32s2-elf/bin
  /home/reiner/esp/tools/esp32ulp-elf/2.28.51.20170517/esp32ulp-elf-binutils/bin
  /home/reiner/esp/tools/openocd-esp32/v0.10.0-esp32-20190708/openocd-esp32/bin
  /home/reiner/esp/python_env/idf4.1_py2.7_env/bin
  /home/reiner/esp/esp-idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

IDF_PATH is

Code: Select all

/home/reiner/esp/esp-idf

Directory Layout:

Code: Select all

/home/reiner/esp

is the root of all my projects.

Code: Select all

├── esp-idf
├── mycomponents
├── time-switch

Inside the mycomponents directory it looks like this

Code: Select all

.
└── components
    ├── am2302_helper.c
    ├── bh1750_helper.c
    ├── bmx80_helper.c
    ├── component.mk
    ├── esp_helper.c
    ├── file_helper.c
    ├── http_helper.c
    ├── i2c_helper.c
    ├── include
    │   ├── am2302_helper.h
    │   ├── bh1750_helper.h
    │   ├── bmx80_helper.h
    │   ├── esp_helper.h
    │   ├── file_helper.h
    │   ├── http_helper.h
    │   ├── http_status.h
    │   ├── i2c_helper.h
    │   ├── ledc_helper.h
    │   ├── list_helper.h
    │   ├── log_buffer.h
    │   ├── myboard.h
    │   ├── myhttp_server.h
    │   ├── noip_helper.h
    │   ├── nvs_helper.h
    │   ├── ota_helper.h
    │   ├── partition_helper.h
    │   ├── pbank_helper.h
    │   ├── sd_helper.h
    │   ├── sd_log.h
    │   ├── sensor_helper.h
    │   ├── sht3x_helper.h
    │   ├── str_helper.h
    │   ├── terminal_helper.h
    │   ├── tsl2561_helper.h
    │   ├── tsl2591_helper.h
    │   ├── uv_index.h
    │   ├── veml6040_helper.h
    │   ├── veml6070_helper.h
    │   ├── veml6075_helper.h
    │   └── wifi_helper.h
    ├── ledc_helper.c
    ├── list_helper.c
    ├── log_buffer.c
    ├── myhttp_server.c
    ├── noip_helper.c
    ├── nvs_helper.c
    ├── ota_helper.c
    ├── partition_helper.c
    ├── pbank_helper.c
    ├── sd_helper.c
    ├── sd_log.c
    ├── sensor_helper.c
    ├── sht3x_helper.c
    ├── str_helper.c
    ├── terminal_helper.c
    ├── tsl2561_helper.c
    ├── tsl2591_helper.c
    ├── veml6040_helper.c
    ├── veml6070_helper.c
    ├── veml6075_helper.c
    └── wifi_helper.c

The components.mk here is empty and there is no CMakeLists.txt - should be there one - and if yes with which content ?

Now lets go inside the time_switch project

Code: Select all

.
├── build
├── CMakeLists.txt
├── Makefile
├── sdkconfig
├── sdkconfig.defaults
├── sdkconfig.old
└── src
    ├── CMakeLists.txt
    ├── component.mk
    ├── config.html
    ├── index.html
    ├── Kconfig.projbuild
    ├── led_green_off.png
    ├── led_green_on.png
    ├── led_x_on.png
    ├── main.c
    ├── reset.css
    ├── timeswitch.css
    ├── timeswitch.js
    ├── update.html
    └── update.js

The CMakeLists.txt in the project root looks like this

Code: Select all

# The following five lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)

set(EXTRA_COMPONENT_DIRS "src" "$ENV{IDF_PATH}/../mycomponents")

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(time_switch)

The Makefile looks like this

Code: Select all

#
# This is a project Makefile. It is assumed the directory this Makefile resides in is a
# project subdirectory.
#

PROJECT_NAME := time-switch

EXTRA_COMPONENT_DIRS += src $(abspath ../mycomponents)

include $(IDF_PATH)/make/project.mk

Now lets go into the src subfolder:
The component.mk looks like this

Code: Select all

#
# "main" pseudo-component makefile.
#
# (Uses default behaviour of compiling all source files in directory, adding 'include' to include path.)
COMPONENT_EMBED_FILES := index.html update.html timeswitch.css reset.css timeswitch.js update.js config.html \
	led_green_on.png led_green_off.png led_x_on.png

The CMakeLists.txt looks like this:

Code: Select all

idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS "."
                    EMBED_FILES "index.html" "update.html" "timeswitch.css" "reset.css" "timeswitch.js" "update.js" "config.html" "led_green_on.png" "led_green_off.png" "led_x_on.png"
                    PRIV_REQUIRES "mbedtls" "nvs_flash" "nvs_helper"
                    )

I include headers like this (from the main.c file:

Code: Select all

#include <stdio.h>
#include <string.h>

#include <time.h>
#include <dirent.h>
#include <math.h>

#include "freertos/FreeRTOS.h"
#include "freertos/task.h"


#include "esp32/sha.h"

#include "esp_log.h"
#include "nvs_flash.h"
#include "esp_task_wdt.h"

#include "nvs_helper.h"
#include "wifi_helper.h"
#include "myhttp_server.h"
#include "http_helper.h"
#include "file_helper.h"
#include "ota_helper.h"
#include "list_helper.h"
#include "str_helper.h"
#include "http_status.h"

#include "bmx80_helper.h"
#include "sensor_helper.h"

#include "noip_helper.h"

#include "log_buffer.h"

#include "cJSON.h"

#include "mdns.h"

#include "sdkconfig.h"

In the components I include (e.g. nvs_helper.h

Code: Select all

#ifndef _NVS_HELPER_H
#define _NVS_HELPER_H

#include "nvs_flash.h"

#ifdef __cplusplus
extern "C" {
#endif

// Read
void nvs_read_uint8(nvs_handle nvs, const char* key, uint8_t* value, const uint8_t def_value);
void nvs_read_int8(nvs_handle nvs, const char* key, int8_t* value, const int8_t def_value);

void nvs_read_uint16(nvs_handle nvs, const char* key, uint16_t* value, const uint16_t def_value);
void nvs_read_int16(nvs_handle nvs, const char* key, int16_t* value, const int16_t def_value);

void nvs_read_uint32(nvs_handle nvs, const char* key, uint32_t* value, const uint32_t def_value);
void nvs_read_int32(nvs_handle nvs, const char* key, int32_t* value, const int32_t def_value);

void nvs_read_uint64(nvs_handle nvs, const char* key, uint64_t* value, const uint64_t def_value);
void nvs_read_int64(nvs_handle nvs, const char* key, int64_t* value, const int64_t def_value);

void nvs_read_string(nvs_handle nvs, const char* key, char* str, size_t size, const char* def_str);

esp_err_t nvs_read_float(nvs_handle nvs, const char* key, float* value, const float def_value);

esp_err_t nvs_read_blob(nvs_handle nvs, const char* key, void** data, size_t* size);

// Write
esp_err_t nvs_write_uint8(nvs_handle nvs, const char* key, const uint8_t value);
esp_err_t nvs_write_int8(nvs_handle nvs, const char* key, const int8_t value);

esp_err_t nvs_write_uint16(nvs_handle nvs, const char* key, const uint16_t value);
esp_err_t nvs_write_int16(nvs_handle nvs, const char* key, const int16_t value);

esp_err_t nvs_write_uint32(nvs_handle nvs, const char* key, const uint32_t value);
esp_err_t nvs_write_int32(nvs_handle nvs, const char* key, const int32_t value);

esp_err_t nvs_write_uint64(nvs_handle nvs, const char* key, const uint64_t value);
esp_err_t nvs_write_int64(nvs_handle nvs, const char* key, const int64_t value);

esp_err_t nvs_write_string(nvs_handle nvs, const char* key, char* str);

esp_err_t nvs_write_float(nvs_handle nvs, const char* key, const float value);

esp_err_t nvs_write_blob(nvs_handle nvs, const char* key, const void* data, size_t size);

// Other
esp_err_t nvs_smart_init(void);

#ifdef __cplusplus
}
#endif

#endif // _NVS_HELPER_H

in the nvs_helper.c I use:

Code: Select all

#include "esp_err.h"
#include "esp_log.h"

#include "nvs_flash.h"

#include "str_helper.h"

static const char* tag = "nvs_helper";

// Read
// 8
void nvs_read_uint8(nvs_handle nvs, const char* key, uint8_t* value, const uint8_t def_value)
{
	esp_err_t err;
	if ((err = nvs_get_u8(nvs, key, value)) != ESP_OK)
	{
		ESP_LOGE(tag, "nvs_get_u8 failed with '%s (%d)'", esp_err_to_name(err), err);
		*value = def_value;
	}
}
....

Now I deleted all from the build directory and idf.py reconfigure gives an error:

Code: Select all

Executing action: reconfigure
Running cmake in directory /home/reiner/esp/time-switch/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 --warn-uninitialized -DCCACHE_ENABLE=0 /home/reiner/esp/time-switch"...
Warn about uninitialized values.
-- Found Git: /usr/bin/git (found version "2.17.1") 
-- IDF_TARGET not set, using default target: esp32
-- The C compiler identification is GNU 8.2.0
-- The CXX compiler identification is GNU 8.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
-- Check for working C compiler: /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
-- Check for working C compiler: /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++
-- Check for working CXX compiler: /home/reiner/esp/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Project version: 1
-- Building ESP-IDF components for target esp32
CMake Error at /home/reiner/esp/esp-idf/tools/cmake/build.cmake:185 (message):
  Failed to resolve component 'nvs_helper'.
Call Stack (most recent call first):
  /home/reiner/esp/esp-idf/tools/cmake/build.cmake:216 (__build_resolve_and_add_req)
  /home/reiner/esp/esp-idf/tools/cmake/build.cmake:425 (__build_expand_requirements)
  /home/reiner/esp/esp-idf/tools/cmake/project.cmake:357 (idf_build_process)
  CMakeLists.txt:8 (project)


-- Configuring incomplete, errors occurred!
See also "/home/reiner/esp/time-switch/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1

I have attatched the output of the log file

[ESP_Angus]

Hi,

Thanks for all this detail and sorry for the long time getting back to you.

I think the main source of confusion here is this: a header file is not a component. A component is a collection of one more source files and header files, in a directory.

To understand more clearly, take a look at the sample project layout in the docs which shows two components:
https://docs.espressif.com/projects/esp ... le-project

So, where you have a directory 'mycomponents/components/' with files including a component.mk file and a number of .c and .h files, the ESP-IDF GNU Make build system sees a single component with the name "components" (the name of the directory becomes the name of the component).

This is quite confusing but it should be able to work, although you might want to consider renaming the directory in the future.

If you want to use this "components" component with CMake build system, add a CMakeLists.txt file alongside the component.mk file.

The CMakeLists.txt file can't be empty, it'll need to contain something like this:

Code: Select all

idf_component_register(INCLUDE_DIRS .
                            SRCS
                            am2302_helper.c
                            bh1750_helper.c
                            bmx80_helper.c
                            esp_helper.c
                            etc etc
                            )

Then the CMake build system will also see a component named "components" (provided the EXTRA_COMPONENT_DIRS is set to incloude the "mycomponents" directory in your project, as shown).

Any other component in your project that wants to include header files from the component named "components" will need to add "REQUIRES components" in their own CMakeLists.txt files, in the same place that "REQUIRES mbedtls" was used. (To use multiple components, write it like "REQUIRES mbedtls components".

Does that make sense?

[Reiner1210]

Hello,

thanks a lot for your patient!

Now I have taken the hello world example renamed the main directory to src and try to include a simple component (no other components required). The glue is that in the CMakeLists.txt the component name is NOT the component it is the folder name.

You have given me the hint in you last post. So I will try it with more complex components .....

Regards
Reiner