ESP32 Forum

I'm having difficulty compiling the IDF for the ESP32-S2 against a non-LWIP stack. I need the added performance that's lost in the LWIP/socket paradigm and when departing. Performance is significantly higher (approx 600 Hz round trip challenge/response over websockets) on the ESP32.

(1) There are two changes that need to be made to the Makefile-based ESP-IDF to support this. Both smartconfig_ack as well as mbedtls_sockets need to be disabled. Right now, I'm commenting these files out, but I'm not sure how I could add this as features to the user config. Would such an option be something you would accept back into the IDF?

(2) I'm having a very difficult time porting this at all to the CMake/idf.py system, so this cannot be ported to the ESP32-S2. Because it requires several packages to not be built or to be overridden, it was easy to do with the Makefile-based system, but with the idf.py system this process of disabling seems much more sophisticated than simply adding folders to the components folder and adding empty CMakeLists.txt files. Is there any clear and concise mechanism to do this with the idf.py instead of Makefile?



BTW: (2) I intend to mirror the answer, whatever it is in this github issue: https://github.com/espressif/esp-idf/issues/1615

I realize it may be helpful to be more specific about my issues in using the idf.py.

If I "remove" components by placing the `CMakeLists.txt` files in folders in my project's `component` folder, the individual modules are removed, but in such a way that other, needed components are excluded from build flags. For instance, even though the esp_wifi is not excluded by me, something happens to cause it to be excluded from the build and `CFLAGS` of the build system.

If I add a`CMakeLists.txt` file in my component, 'cnip' that looks like the following: NOTE: Adding additional features to REQUIRES does not add it to the build for all components, only this one. How can one component indicate the need for all components being compiled to include another component. For instance, without esp_wifi headers, things fall apart and it is not implicitly defined here.

And, my project CMakeLists.txt,
This is rather awkward, and I have difficulty properly including other components for the whole project. I end up needing the include_directories line to fix that.

There's other things I'm working through but this is the bulk of the confusion.

Additionally, I am seeing significantly worse performance on the ESP32-S2, almost 5ms round-trip PSH/ACK/PSH/ACK round trips on websockets! That's going to take some effort to debug. Is there any known issue or change in the wireless path that would increase latency?

Hi cnlohr,

Will try to give a more complete answer later, but if there are no symbol name conflicts between lwip and your new TCP/IP library then it may be possible to compile lwip but not link it.

• The only coupling between lwip and the wifi libraries should be via the tcpip_adapter component. If you copy this component into your project's components directory, you can change it to hook up your TCP/IP library instead - or alternatively keep it all as-is and don't call any functions from tcpip_adapter, and hook up the Wi-Fi library directly to your IP library.
• In the CMake-based build system, all components (except "main") need to declare their dependencies. So if you declare a component depends on your IP stack component and esp_wifi, you'll be able to include headers from there.

Acknowledge this is a bit hacky and wasteful (compile a bunch of source and not link it), but I think a better solution may take a little while to put together.

It's also a mistake that smartconfig_ack is somehow linked directly to LWIP, will need to fix this in the Wi-Fi libraries.

EDIT: Just saw your new post, will reply on a couple of points:

NOTE: Adding additional features to REQUIRES does not add it to the build for all components, only this one.

This is intentional, we're trying to clean up the dependency hierarchy between components rather than the free-for-all we had before. Although there are still a number of layering violations.

How can one component indicate the need for all components being compiled to include another component. For instance, without esp_wifi headers, things fall apart and it is not implicitly defined here.

set(APPEND COMPONENTS ${COMPONENTS} "cnip" )
set(APPEND COMPONENT_DIRS ${COMPONENT_DIRS} "components")

I think the root cause is these lines. The "COMPONENTS" list is actually empty at this point, so by "appending" to it it gets set to "cnip" only. This tells ESP-IDF to only build the minimal set of components[*], plus cnip. This excludes esp_wifi from the build.

You shouldn't need these two lines at all, the build system should search the project "components" dir automatically and add any components which it finds there.

However you might have some success doing something like "set(COMPONENTS cnip esp_wifi)" to get a very minimal project[*].

In either case you still need to add "REQUIRES esp_wifi" in your cnip CMakeLists, so that the cnip component can include esp_wifi headers.

[*] There is a list of common components, see __COMPONENT_REQUIRES_COMMON, which are included no matter what and which all components depend on. This - unfortunately for you - includes lwip component for standard socket headers, we're discussing if there's a way to not do this. Suggest not messing with this value though, declare explicit dependencies instead if you can.

Additionally, I am seeing significantly worse performance on the ESP32-S2, almost 5ms round-trip PSH/ACK/PSH/ACK round trips on websockets! That's going to take some effort to debug. Is there any known issue or change in the wireless path that would increase latency?

S2beta is still a preview so there are a number of things unoptimized at this point. The fact you're using a custom IP stack makes it a bit hard to compare apples to apples.

If you have a way to capture raw Wi-Fi frames then this will help narrow down the problem to Wi-Fi layer issues (lots of retransmits, etc), or issues with latency in the ESP32-S2 firmware.

Do you have the latest PHY (Marlin3)? You can tell by reading the chip silkscreen - there are 3 ESP32-S2beta PHY revisions and Wi-Fi performance is best in the latest (Marlin3).

Thank you for your response.

1: Re: Dependency on lwip: I still cannot seem to figure out how to build smartconfig_ack and components/mbedtls/port/net_sockets, or rather build without these components. My stack does not provided BSD-like functionality.

2: Re: Needing COMPONENT_DIRS: Well, I'll be. I guess I was doing something wrong earlier. Now that I've switched things back around, it seems not to be needed in my base project! That's awesome. [RESOLVED]

3: Re: Not needing header includes. That is still a negatory. Even if I add esp_wifi to cnip, other subsystems like esp_event do require it. It seems to only be "brought in" for cnip.

4: Re: __COMPONENT_REQUIRES_COMMON: I believe I have circumvented this by overriding/disabling the following components in my project: asio, esp_http_server, libsodium, smartconfig_ack***, esp_https_ota, lwip, tcpip_adapter**, coap, esp_https_server, mdns, tcp_transport, esp_local_ctrl, mqtt, wifi_provisioning, esp-tls, nghttp, esp_http_client, esp_websocket_client and protocomm.
*** = Even though I am overriding this, it still is attempting to build
** = I have my own copy of this.

5: Re: Performance: Performance is significant. I was more inquiring as to if there was a deliberate architectural shift which would impair performance. I noticed that the performance changes significantly depending on the FLASH Settings. It would appear some functions in the critical path are not in IRAM despite my best efforts. I don't believe I have retries, as it's more of my ping-baseline is at around 1.2-2.4ms. Let's not discuss this any further in this thread though. Performance is almost sufficient for this test. [RESOLVED]

6: It is a MARLIN3.

Also - I have another question - how can you define two separate modules to rely on each other? Or, more importantly, a module which expects an application to implement a function. It "looks" like you guys may have made an effort to make this impossible, for instance, how tcpip_adapter is triggered from events instead of function calls, but events are much slower than function calls. Is there any mechanism to allow this? Or is it now prohibited and all dependences must be unidirectional? If not, how would I define the cmakelists REQUIRES? EDIT I misunderstood something - this issue doesn't seem to be inter-module, however, the question does stand for final app linkage.

Hi cnlohr,

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

1: Re: Dependency on lwip: I still cannot seem to figure out how to build smartconfig_ack and components/mbedtls/port/net_sockets, or rather build without these components. My stack does not provided BSD-like functionality.

What I mean is: build with all of the normal IDF components, including LWIP. If your code doesn't call any functions that use any of these things, they shouldn't end up linked into the final binary. So you waste some initial compile time (annoying), but it shouldn't make any difference to the final binary.

The only time this definitely won't work is if you need to replace an IDF symbol with a same-named one in your custom component (shouldn't be needed to swap out LWIP because everything depends on what functions the main app calls), or if a symbol that you want to link in happens to have the same name as a global symbol in one of the IDF components.

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

2: Re: Needing COMPONENT_DIRS: Well, I'll be. I guess I was doing something wrong earlier. Now that I've switched things back around, it seems not to be needed in my base project! That's awesome. [RESOLVED]

Yay!

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

3: Re: Not needing header includes. That is still a negatory. Even if I add esp_wifi to cnip, other subsystems like esp_event do require it. It seems to only be "brought in" for cnip.

Ah... you are probably the first person to really test the validity some of the per-component requirements in ESP-IDF. The root cause is: esp_event includes wifi headers directly, but currently esp_wifi is not in its REQUIRES list even though should be.

In a normal IDF project, it will not matter because the public header requirements chain goes esp_event -> tcpip_adapter -> esp_wifi. Because your project overrides tcpip_adapter, there is no longer a path from esp_event to esp_wifi.

The correct fix is to add esp_wifi to the REQUIRES list of esp_event, will submit a patch for this internally but you can also change this in IDF now.

If you don't include "esp_event_legacy.h" then the modified tcpip_adapter component shouldn't matter, as this is the only header that uses tcpip_adapter.

Probably the other errors will have a similar root cause.

(BTW, this is another thing you won't have to worry about if you can get away with a "build all IDF components, but don't link anything you don't need" approach. However we're very happy to fix these dependency declaration problems in IDF as well, as our long term goal is to allow this kind of approach.)

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

4: Re: __COMPONENT_REQUIRES_COMMON: I believe I have circumvented this by overriding/disabling the following components in my project: asio, esp_http_server, libsodium, smartconfig_ack***, esp_https_ota, lwip, tcpip_adapter**, coap, esp_https_server, mdns, tcp_transport, esp_local_ctrl, mqtt, wifi_provisioning, esp-tls, nghttp, esp_http_client, esp_websocket_client and protocomm.
*** = Even though I am overriding this, it still is attempting to build
** = I have my own copy of this.

What does "attempting to build" mean in this context? You may need to add a dummy idf_component_register() line in the CMakeLists.txt so the build system see a component by this name, for resolving dependencies.

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

5: Re: Performance: Performance is significant. I was more inquiring as to if there was a deliberate architectural shift which would impair performance. I noticed that the performance changes significantly depending on the FLASH Settings. It would appear some functions in the critical path are not in IRAM despite my best efforts. I don't believe I have retries, as it's more of my ping-baseline is at around 1.2-2.4ms. Let's not discuss this any further in this thread though. Performance is almost sufficient for this test.

OK, yes you'll probably need to bring this up with the Wi-Fi team.

No deliberate shift of this kind, in fact the flash cache settings have been changed with the goal of improving performance - but probably there is a lot of benchmarking and tweaking which is yet to be done. The default flash cache settings are probably not the ones which give absolute best performance, at this time.

You probably know this, but note that Wi-Fi will retry at the MAC layer, totally transparently to the IP stack. The only way to tell this is happening is to look at a raw Wi-Fi frame capture. Edit: Never mind, I re-read your posts and realised you're seeing 1.2-2.4ms ping times but 5ms round trips on a WebSocket. This sounds like a tuning issue with the IP stack.

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

6: It is a MARLIN3.

Great!

cnlohr wrote: ↑

Fri Nov 08, 2019 9:10 am

Also - I have another question - how can you define two separate modules to rely on each other? Or, more importantly, a module which expects an application to implement a function. It "looks" like you guys may have made an effort to make this impossible, for instance, how tcpip_adapter is triggered from events instead of function calls, but events are much slower than function calls. Is there any mechanism to allow this? Or is it now prohibited and all dependences must be unidirectional? If not, how would I define the cmakelists REQUIRES? EDIT I misunderstood something - this issue doesn't seem to be inter-module, however, the question does stand for final app linkage.

A "REQUIRES B" clause in component A means "A's public interface includes headers from B". A clause "PRIV_REQUIRES B" means "A's source files include headers from B", or "A links against symbols from B", or both of those. More details here: https://docs.espressif.com/projects/esp ... quirements

(Because REQUIRES is for the public headers, this is why the relationship is transitive across multiple components. PRIV_REQUIRES is not transitive for header paths.)

The esp_event refactor was intended to remove tight coupling from places where it wasn't needed. We benchmark for performance regressions pretty heavily and nothing much came up, although most wifi events don't happen at a particularly high frequency (packets still go via function calls). If you see a performance regression that can be pinned on the event mechanism then please let us know.

Maybe I misunderstand - how am I going to get lwip to compile along-side mine if tcpip_adapter's involved? Should I specifically be using the esp_idf's tcpip_adapter.h but my own tcpip_adapter.c?

I am discovering the value of fullclean. I've found several sticking points which have been resolved by this. It seems no dummy idf_component_register() is required, just liberal use of fullclean. There is a HUGE annoyance with this. Doing a fullclean unsets the retarget to esp32s2beta, which when reset clears out most settings in my sdkconfig file. Overwriting the skdconfig file makes the system angry at automatic component removals (i.e. esp_eth no longer needed as a function of sdkconfig) which requires manual editing of CMakeCache.txt. This feels like a bug?

All of your other points have been incredibly helpful to provide background! My only remaining thought was some of these trade-offs with with events/function calls should probably be structured more in the direction of configurable static-functionality being changed rather than dynamic functionality being available. I.e. the changes to esp_event. By having a very simple, flat caller/callee call structure, even if it isn't _as_ automatic, it makes it a lot easier for other aspects like dead code removal, etc. to happen effectively, without the compiler worrying about some path being possible that truly isn't.


I want to wrap up this thread, and I intend to post back to the github issue, but I want to make sure we tidy all this up before I go post something inaccurate. Some take-aways:
* Most of the time, you do not need to override the functionality.
* You can override components the same way you used to, with a "components" folder in a project.
* Dummy or modified components need a CMakeLists.txt which can optionally contain a idf_component_register.
* Whenever modifying overrides or configurations, be sure to always issue an idf.py fullclean.
* No modifications are needed to the project's CMakeLists.txt unless you want to separately include overriden module's headers, which can be done with include_directories( $ENV{IDF_PATH}/... )


PSS I believe I found my issue with slow incoming packet processing. It seems ppRxPkt is in flash and I worry other critical path functions are also in flash instead of ram. Is there yet an option to control the placement of binary blob functions? (NOTE: this is NOT critical to my immediate task at hand, but will need to be addressed at some point.)

cnlohr wrote: ↑

Sat Nov 09, 2019 10:53 am

Maybe I misunderstand - how am I going to get lwip to compile along-side mine if tcpip_adapter's involved? Should I specifically be using the esp_idf's tcpip_adapter.h but my own tcpip_adapter.c?

Yes, you can still replace tcpip_adapter component with your own - and make whatever changes you need, while also removing any calls to LWIP. If nothing calls LWIP in your project, it will be compiled but not linked.

Or you should be able to choose not to use tcpip_adapter at all, and instead provide the event hooks & function hookup via some other code you create yourself - but this will probably end up looking a lot like tcpip_adapter.

cnlohr wrote: ↑

Sat Nov 09, 2019 10:53 am

I am discovering the value of fullclean. I've found several sticking points which have been resolved by this. It seems no dummy idf_component_register() is required, just liberal use of fullclean. There is a HUGE annoyance with this. Doing a fullclean unsets the retarget to esp32s2beta, which when reset clears out most settings in my sdkconfig file. Overwriting the skdconfig file makes the system angry at automatic component removals (i.e. esp_eth no longer needed as a function of sdkconfig) which requires manual editing of CMakeCache.txt. This feels like a bug?

When overriding or adding components, this will require a full re-run of CMake but CMake won't detect this from a normal build (unless you also edit one of the CMakeLists that it knows about). This is a limitation of CMake, they advise against "auto detection" of paths like this (for this reason) but we still do it at the component level.

Does "idf.py reconfigure" work, rather than fullclean? It should do.

Regarding the IDF_TARGET value being lost when you fullclean, we're tracking this internally as a bug. It's scheduled to be fixed before the v4.1 release.

cnlohr wrote: ↑

Sat Nov 09, 2019 10:53 am

All of your other points have been incredibly helpful to provide background! My only remaining thought was some of these trade-offs with with events/function calls should probably be structured more in the direction of configurable static-functionality being changed rather than dynamic functionality being available. I.e. the changes to esp_event. By having a very simple, flat caller/callee call structure, even if it isn't _as_ automatic, it makes it a lot easier for other aspects like dead code removal, etc. to happen effectively, without the compiler worrying about some path being possible that truly isn't.

Appreciate your suggestion. This tradeoff of more runtime configurability vs restrictions on what can be static linked is something we've debated a bit as well.

cnlohr wrote: ↑

Sat Nov 09, 2019 10:53 am

I want to wrap up this thread, and I intend to post back to the github issue, but I want to make sure we tidy all this up before I go post something inaccurate. Some take-aways:
* Most of the time, you do not need to override the functionality.
* You can override components the same way you used to, with a "components" folder in a project.
* Dummy or modified components need a CMakeLists.txt which can optionally contain a idf_component_register.
* Whenever modifying overrides or configurations, be sure to always issue an idf.py fullclean.
* No modifications are needed to the project's CMakeLists.txt unless you want to separately include overriden module's headers, which can be done with include_directories( $ENV{IDF_PATH}/... )

They also sound correct to me except the last two:

- Should be able to do "idf.py reconfigure" instead of fullclean, as mentioned above. Specifically this is needed if you add a new component (whether or not it's overriding an existing component).

- You shouldn't really need the "include_directories( IDF_PATH )" hack if you put an idf_component_register in your overriden component, and add that component to the REQUIRES list of any component that needs to use its headers. If you're still finding a case where the only solution is include_directories(), it's probably a bug.

cnlohr wrote: ↑

Sat Nov 09, 2019 10:53 am

PSS I believe I found my issue with slow incoming packet processing. It seems ppRxPkt is in flash and I worry other critical path functions are also in flash instead of ram. Is there yet an option to control the placement of binary blob functions? (NOTE: this is NOT critical to my immediate task at hand, but will need to be addressed at some point.)

An optimisation to move additional RX code to IRAM is in review now, so should be merged soon.

In general, expect the S2 Beta wifi performance to improve over the next couple of months.