clarification about unit testing
-
- Posts: 51
- Joined: Wed Aug 30, 2017 12:36 pm
clarification about unit testing
Hello, everyone!
Sorry in advance for likely stupid questions but..
I am trying to write some general tests for my C code, but I don't clearly understand how to create unit-tests and perform testing in the project structure.
So, I have a project, which is located in ${IDF_PATH}/my_proj.
Also I have my custom component schedule in path ${IDF_PATH}/my_proj/components/schedule.
Following the Unit Testing in ESP32 chapter, I have created
${IDF_PATH}/my_proj/components/schedule/test
- test_schedule.c
- component.mk
Now, I need to write my test code, but it requires to include unity.h which is a component of the unit-test-app project, and I am confused with this.
So, what is the right way:
a) to copy (or symlink?) my schedule folder from my_proj/components to unit-test-app/components
b) to move my schedule component under ${IDF_PATH}/components
c) to move unit-test-app/components/unity to my_proj/components/unity
?
And another question. What approach could you recommend to perform testing on building host? After all, it's not always needed to test some general stuff on ESP and it's much faster to run such tests on building host.
Sorry in advance for likely stupid questions but..
I am trying to write some general tests for my C code, but I don't clearly understand how to create unit-tests and perform testing in the project structure.
So, I have a project, which is located in ${IDF_PATH}/my_proj.
Also I have my custom component schedule in path ${IDF_PATH}/my_proj/components/schedule.
Following the Unit Testing in ESP32 chapter, I have created
${IDF_PATH}/my_proj/components/schedule/test
- test_schedule.c
- component.mk
Now, I need to write my test code, but it requires to include unity.h which is a component of the unit-test-app project, and I am confused with this.
So, what is the right way:
a) to copy (or symlink?) my schedule folder from my_proj/components to unit-test-app/components
b) to move my schedule component under ${IDF_PATH}/components
c) to move unit-test-app/components/unity to my_proj/components/unity
?
And another question. What approach could you recommend to perform testing on building host? After all, it's not always needed to test some general stuff on ESP and it's much faster to run such tests on building host.
Re: clarification about unit testing
Unit tests always run in the context of the unit-test-app project (ie not your project.)atlascoder wrote: Now, I need to write my test code, but it requires to include unity.h which is a component of the unit-test-app project, and I am confused with this.
To make it see your "schedule" directory, you should be able to do something like this:
Code: Select all
make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=/path/to/my_proj/components TEST_COMPONENTS=schedule
You can probably wrap this command into some custom "make unit-test" and "make unit-test-flash" targets in your project's own Makefile.
If this seems a little fiddly, it's because the original use case was testing IDF's internal components. If you have any ideas/suggestions about how we can make it more streamlined for use cases like yours, we'd love to hear them.
This is a good approach (testing as much of your firmware logic on the host as possible). IDF itself doesn't provide much direct support for this. However some components have "host-side-test" support, look at the nvs_flash, mdns, heap and wear_levelling components for some ideas.And another question. What approach could you recommend to perform testing on building host? After all, it's not always needed to test some general stuff on ESP and it's much faster to run such tests on building host.
Exactly how to structure such tests probably depends on what your firmware does, and what layer you want to test at.
-
- Posts: 51
- Joined: Wed Aug 30, 2017 12:36 pm
Re: clarification about unit testing
Thanks for this informative answer, ESP_Angus!
I think, the code with EXTRA_COMPONENT_DIRS is much suit for my case!
And I will try to make some make target to build unit-test-app with my project's components.
As an idea about testing, just one thing has confused me - it's usage of the separate project for testing. If the testing project could be generated within an app project structure - it would be more convenient and coherent, I think. In example, I use template for a project as usual, an if I need to create some tests - I just run some script, or make target, that generates/includes necessary components and code in my project's structure along with it. And I get targets for building unit-test-app from my project's structure.
Yes, I understand that the most relevant tests should be executed in production-clone context. I don't know if used testing frameworks are able to run tests on building host, but it's more sophisticated thing because it may require to implement stubs/mocks for contextual things, and understood, it can't be easy to get.
I think, the code with EXTRA_COMPONENT_DIRS is much suit for my case!
Code: Select all
make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=/path/to/my_proj/components TEST_COMPONENTS=schedule
As an idea about testing, just one thing has confused me - it's usage of the separate project for testing. If the testing project could be generated within an app project structure - it would be more convenient and coherent, I think. In example, I use template for a project as usual, an if I need to create some tests - I just run some script, or make target, that generates/includes necessary components and code in my project's structure along with it. And I get targets for building unit-test-app from my project's structure.
Yes, I understand that the most relevant tests should be executed in production-clone context. I don't know if used testing frameworks are able to run tests on building host, but it's more sophisticated thing because it may require to implement stubs/mocks for contextual things, and understood, it can't be easy to get.
Re: clarification about unit testing
I understand.atlascoder wrote: As an idea about testing, just one thing has confused me - it's usage of the separate project for testing. If the testing project could be generated within an app project structure - it would be more convenient and coherent, I think. In example, I use template for a project as usual, an if I need to create some tests - I just run some script, or make target, that generates/includes necessary components and code in my project's structure along with it. And I get targets for building unit-test-app from my project's structure.
For what it's worth, you can make your own unit test project that uses the IDF "unity" component by setting EXTRA_COMPONENT_DIRS to include tools/unit-test-app/components. But you'll need to write the "main" routine of the test app yourself, then.
Unity can be used on the host (it's a platform-agnostic C library, see https://www.throwtheswitch.org/unity ). But you'd probably need a separate copy of it that gets included when building host-side tests. And, as you say, you need the additional mock hardware layers that are relevant to whatever you want to test.atlascoder wrote: Yes, I understand that the most relevant tests should be executed in production-clone context. I don't know if used testing frameworks are able to run tests on building host, but it's more sophisticated thing because it may require to implement stubs/mocks for contextual things, and understood, it can't be easy to get.
-
- Posts: 51
- Joined: Wed Aug 30, 2017 12:36 pm
Re: clarification about unit testing
Ok, I've got it worked.
I just added reference to unit-test-app/components/unity to my project's includes - it allowed to me edit tests' code within my project in Eclipse.
Then i added a phony target for building unit-test-app project with EXTRA_COMPONENTS_DIR, as you, Angus, suggested:
And then I also create another make target with command for flashing unit-test-app.
Now, I can easy add and edit my test code and run my app or the unit-test-app conveniently. Thanks for help!
***
And another question is mocking.. I dived deeply and met projects Ceedling, CMock and Fake Function Framework.
Ceedling http://www.throwtheswitch.org/ceedling/ allows to run tests on a development host. It generates directories and staff for new one project or for an existing one. I didn't try, because I found that it uses `build` directory and it would clash with existing one (http://www.electronvector.com/blog/add- ... h-ceedling), and I would not like to rebuild everything when switching projects. But this is very matching to idea expressed by me here about generating code.
CMock - is a mocking feature using the Unity. It has specific requirements to directory structure, which is different from esp-idf, particularly, all code files are under `src` dirs. But I guess it could be adopted to use with the unit-test-app project.
And Fake Function Framework (FFF) is simplest feature requiring for just one header file inclusion!
Unfortunately, I am not too proficient it building systems and all these mocking stuff requires for neat tuning of building scripts, as far I see. I've tried FFF and stuck into build error `multiple definition if xxx`. I know that to resolve it, I need to exclude mocked function from build (and its components as whole), but it's not seems easy yet.
ESP_Angus, probably you have some advice for further steps in using of mocks?
I just added reference to unit-test-app/components/unity to my project's includes - it allowed to me edit tests' code within my project in Eclipse.
Then i added a phony target for building unit-test-app project with EXTRA_COMPONENTS_DIR, as you, Angus, suggested:
Code: Select all
make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=${IDF_PATH}/my_project/components TEST_COMPONENTS='schedule'
Now, I can easy add and edit my test code and run my app or the unit-test-app conveniently. Thanks for help!
***
And another question is mocking.. I dived deeply and met projects Ceedling, CMock and Fake Function Framework.
Ceedling http://www.throwtheswitch.org/ceedling/ allows to run tests on a development host. It generates directories and staff for new one project or for an existing one. I didn't try, because I found that it uses `build` directory and it would clash with existing one (http://www.electronvector.com/blog/add- ... h-ceedling), and I would not like to rebuild everything when switching projects. But this is very matching to idea expressed by me here about generating code.
CMock - is a mocking feature using the Unity. It has specific requirements to directory structure, which is different from esp-idf, particularly, all code files are under `src` dirs. But I guess it could be adopted to use with the unit-test-app project.
And Fake Function Framework (FFF) is simplest feature requiring for just one header file inclusion!
Unfortunately, I am not too proficient it building systems and all these mocking stuff requires for neat tuning of building scripts, as far I see. I've tried FFF and stuck into build error `multiple definition if xxx`. I know that to resolve it, I need to exclude mocked function from build (and its components as whole), but it's not seems easy yet.
ESP_Angus, probably you have some advice for further steps in using of mocks?
-
- Posts: 51
- Joined: Wed Aug 30, 2017 12:36 pm
Re: clarification about unit testing
Howdy, again!
And yet, I'd like to return to mocking.. I am trying to build only certain components from ESP-IDF and my project in order to represent excluded components with fake functions.
I use the command:
But I have following error:
The sdkconfig.h present under build/include in both: my_project and unit-test-app project. I have tried to add -I build/include to the make arguments. but no luck.
How could I build unit-test-app with selected set of components, or probably, exclude a set of components?
And yet, I'd like to return to mocking.. I am trying to build only certain components from ESP-IDF and my project in order to represent excluded components with fake functions.
I use the command:
Code: Select all
make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=${IDF_PATH}/my_proj/components TEST_COMPONENTS="workcycle" COMPONENTS="freertos esp32 newlib esptool_py nvs_flash spi_flash log tcpip_adapter lwip main xtensa-debug-module driver bt soc unity workcycle"
Code: Select all
esp-idf/components/freertos/include/freertos/FreeRTOSConfig.h:73:23: fatal error: sdkconfig.h: No such file or directory
How could I build unit-test-app with selected set of components, or probably, exclude a set of components?
Re: clarification about unit testing
Just want to add: I also found this very confusing. The documentation starts off by showing how to add your own unit tests in a component, but it turns out it's only meant for ESP-IDF's internal components, so who exactly is the documentation targeting?
That aside, how could I achieve this via CMake?
That aside, how could I achieve this via CMake?
Re: clarification about unit testing
We have recently added example of creating unit tests and unit test projects: https://github.com/espressif/esp-idf/bl ... /README.md
Re: clarification about unit testing
I wrote an example showing how FFF (Fake Function Framework) can be used with the Unity component of ESP-IDF to write unit tests for your application with test doubles (mocks).
https://gitlab.com/deltalejo/esp32-unity-fff-demo
https://gitlab.com/deltalejo/esp32-unity-fff-demo
Who is online
Users browsing this forum: No registered users and 99 guests