Create tar archive automatically in build process

[ghost07]

Hi, how I should configure CMakeLists.txt to automatically generate tar archive in build process?

I've tried this script, but I can't figure out, which target I am supposed to use.

Code: Select all

idf_component_register(
    SRCS "embedded_vfs.c" "microtar.c"
    INCLUDE_DIRS "include"                    

    EMBED_FILES 
    embedded.tar
)

# execute_process(COMMAND rm embedded.tar)
# execute_process(COMMAND tar -cf embedded.tar --owner dev --group dev -C files . --transform 's/^\.//')

add_custom_command(
    OUTPUT embedded.tar
    COMMAND tar -cf "embedded.tar" -C files
)
target_add_binary_data(PROJECT_DIR "embedded.tar" BINARY)

This script reports error:

Cannot specify sources for target "PROJECT_DIR" which is not built by this project.

I've tried ${PROJECT_NAME}.elf and embedded_files (component's name) as well, but neither have worked.
And it seems that execute_process didn't run at all (when it wasn't commented, of course).

[MicroController]

Try

Code: Select all

target_add_binary_data( ${COMPONENT_LIB} "path/to/embedded.tar" BINARY )

[ghost07]

Then I get another error:

-- Configuring done
CMake Error in components/embedded_files/CMakeLists.txt:
Cannot find source file:

components/embedded_files/embedded.tar

Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
.hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc


-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.
FAILED: build.ninja

CMakeLists.txt:

Code: Select all

idf_component_register(
    SRCS "embedded_vfs.c" "microtar.c"
    INCLUDE_DIRS "include"
)

add_custom_target(my_process 
    COMMAND tar -cf "embedded.tar" -C files
    COMMENT "Creating tar archive..."
)
target_add_binary_data(${COMPONENT_LIB} "embedded.tar" BINARY DEPENDS my_process)

I've added COMMENT directive, and I don't see the comment text nowhere in ouptut, so I think it doesn't execute.
Before I tried add_custom_command() instead of add_custom_target(), with COMMENT as well, and there were no comment text in the output either.

[MicroController]

Make sure you get the path(s) right (the build takes place in the "build" directory), and maybe try specifying the BYPRODUCT of the tar command.

[ghost07]

To ensure paths are correct I added WORKING_DIRECTORY ${COMPONENT_DIR} directive. After I removed EMBED_FILE / target_add_binary_data() - it actually generated the file, but failed in linking step.

If I put back EMBED_FILE in idf_register_component() or target_add_binary_data(), and the file doesn't exists, I get error posted in previous reply, even though there is a custom command / target to generate that specific file. If file exists, it embeds the file as is, and generates updated version after, so I would have to build 2 times in order to get executable binary with up-to-date file.

I am not sure what to pass as an argument to BYPRODUCT. I tried the folder name, but that resulted in error:

/bin/sh: 1: cd: can't cd to [project path]/build/esp-idf/embedded_files/files

I've managed to automatically generate file using execute_process(), which is not recommended, but at least it works.

Code: Select all

execute_process(
    COMMAND_ECHO STDOUT
    WORKING_DIRECTORY ${COMPONENT_DIR}
    COMMAND tar -cf embedded.tar --owner dev --group dev -C files . --transform "s/^\\.\\///"
)

If someone would know how to generate file the right way (in build step, instead of in configure step - execute_process()), I would appreciate any suggestions.

[MicroController]

I am not sure what to pass as an argument to BYPRODUCT.

The page I linked to says:

BYPRODUCTS

Specify the files the command is expected to produce ... It is also useful when other build rules (e.g. custom commands) depend on the byproducts. Ninja requires a build rule for any generated file on which another rule depends

If you could slow down on changing multiple elements in your CMakeList and modify it one step at a time until you figure the effect of that step out we could better advise you of the next step to try.

[ghost07]

Alright, I've tried to pass output file name (embedded.tar) to BYPRODUCTS, but it haven't solved mentioned problems:

1) File has to already exist
This is inconvenient to version control (git), since I don't want to track this file because it is generated from files that are already tracked

2) File is generated (updated) after it is embedded to executable binary, so I have to run build twice

CMakeLists.txt:

Code: Select all

idf_component_register(
    SRCS "embedded_vfs.c" "microtar.c"
    INCLUDE_DIRS "include"
)

add_custom_target(my_process
    BYPRODUCTS embedded.tar
    WORKING_DIRECTORY ${COMPONENT_DIR}
    COMMAND tar -cf embedded.tar -C files .
)
target_add_binary_data(${COMPONENT_LIB} "embedded.tar" BINARY DEPENDS my_process)

[MicroController]

1) is easily solved: You can tell git to ignore arbitrary files/directories via a .gitignore file. If you use a GUI/IDE for git, it should let you select the file(s) and create/update .gitignore for you.

2) is ... a real bummer. I think your latest CMakeLists should actually do the trick according to the documentation.

Edit:
I just read "Place [target_add_binary_data] after the project() line in your project CMakeLists.txt file". Out of desperation, can you try and move the tar command and target_add_binary_data(...) to the project's CMakeLists.txt, i.e. not the one inside the "main" source folder but the one in main's parent (project) directory (the one which contains the "project(...)" declaration)?

[ghost07]

Thanks for idea, but unfortunately still the same.
I think it may do something with cache, because when I alter tar source folder (add, remove, modify some file), the first build is almost instant, and second one takes some time.
It doesn't make any difference if I put add_custom_target() before or after target_add_binary_data().

[project]/CMakeLists.txt:

Code: Select all

# The following five lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project([project name])

target_add_binary_data(${PROJECT_NAME}.elf ${PROJECT_DIR}/components/embedded_files/embedded.tar BINARY DEPENDS my_process)

add_custom_target(my_process
    BYPRODUCTS embedded.tar
    WORKING_DIRECTORY ${PROJECT_DIR}/components/embedded_files
    COMMAND tar -cf embedded.tar -C files .
)

For now, I think I will stick with the execute_process() approach.

I am just curious why example from documentation doesn't work. I totally forgot to specify my IDF version, which is 4.4, however, embedding generated file is documented since version 4.3.

EDIT:
Ok, execute_process() is apparently even worse, because cmake doesn't re-run configuration if I don't modify actual builded source code (only modify some file in tar source folder, which is probably not tracked by cmake).

[MicroController]

Another random idea: You could try

Code: Select all

add_dependencies(${PROJECT_NAME}.elf my_process)

in an attempt to get the project build to consistently execute the tar command.