Gripes on ESP32/ESP-IDF

[michprev]

(Mod note: Changed the title of this topic; it was 'ESP32 not ready for production')

Hello guys,
I've been working with ESP32 for about one year. I have built quadcopter firmware based on ESP-IDF but soon understood that it will be impossible to make it fully working.
Whole IDF is written inefficient and it is quite unreadable. Everything is based on FreeRTOS which makes it really heavy. It would make better sense to write low level drivers without FreeRTOS and make an optional higher level api with FreeRTOS - something like ESP-IDF.
There are no naming conventions. Drivers for peripherals are written very differently come with many assumptions. There are two types of files decribing registers (*_reg and *_struct). And many many more...

There are so many bugs reported on github. ESP-IDF is released for public since august 2016 and there are still bugs in drivers for SPI or I2C.

Because of this I started working on my own SDK. Unfortunatelly docs are so incomplete that it is nearly impossible to do that.
Debugging tools (openocd and gdb) are incomplete - in my opinion it should be the first thing that should be done (before starting working on IDF).

There are also so many hardware bugs that Espressif probably does not want to admit (and maybe because of that docs are not complete). Some of them are not critical but they still exist and should be easily discovered by tests before market launch.

As a conclusion I would rather pay more than have partially working hardware with incomplete docs.

[plajjd]

My company is seriously considering using the ESP32 for a production product (> 10K units annually). Your post concerns me.

I would appreciate input and feedback from others in the community as to the "production readiness" of the ESP32 and the development tools. I also would love to have success stories in addition to concerns.

I also wonder if Espressif has examples of large volume products in the market based on ESP8266 and/or ESP32? Even if I had to sign an NDA, it would very much help my confidence if I could know of other companies using these chips.

Thanks!

[urbanze]

esp8266 is a toy inferior to the esp32 and yet is rather used commercially as the famous sonoff and etc. I am also using esp32 for commercial products in the industrial sector and I am betting my chips in this great mcu, even with some inconveniences.

[WiFive]

A drone is sort of the ultimate test of a platform. It has to read and process tons of realtime sensor data, generate control signals, and maintain a robust wireless link. It can't crash or things fall out of the sky. If you can build a drone on a platform, you can probably build anything.

There should be a rock solid core with wireless, kernel os, and hw drivers. It should be bomb proof. This requires more of a bottom up approach in contrast to the current top down patch it until it works approach. I agree it is not there yet and hopefully the refactoring on mainline lwip and freertos and moving all non critical components out of the main repo and creating a component/package manager will help. They have lots of new developers and maybe they will hire a third party team to audit the wifi code if they are going to keep it closed source. It seems fragile.

You are also right that any limitations should be clearly documented and not hidden. There should be a "so you want to build a [drone]" series of articles that point out best practices and pain points.

[michprev]

Making quadcopters is just my hobby.
I think that ESP32 fills a huge gap on market - it is powerful and there are ready to use modules (DevKitC V4). I also love the fact that it uses Xtensa architecture.

But it hurts to see how many features / configuration of SPI hardware have been planned and are not working or not documented.
For example there are special commands for working with SPI flash that work even on HSPI (and probably VSPI too).

There is SPI master driver I've been working on a few days. It is incomplete and it is also very probable that it is not 100% working but I believe it is a bit more readable and mainly faster. I believe that drivers like this may be the right way. Unfortunatelly it cannot fix hardware issues - CS pre-transmission delay, dummy bits in full-duplex, DMA with half-duplex...

[mikemoy]

Hello guys,
Everything is based on FreeRTOS which makes it really heavy. It would make better sense to write low level drivers without FreeRTOS and make an optional higher level api with FreeRTOS - something like ESP-IDF.

FreeRTOS is not heavy at all. To say "It would make better sense to write low level drivers without FreeRTOS " is really not true.
FreeRTOS, makes running many threads at one very simple. You can even set a section of code as CRITICAL if you need to something timing intensive so there are no task switching, not to mention you can also run certain things on just one of the cores, to get even better performance. In the config you can even set FreeRTOS to only run on one core, and not the other. So it's quite flexible to get most job done without a problem.

If you doing quad copter stuff, you should really be coding in assembler for many things anyways. which you can do that do here.
You timing requirements for things of this nature are really tight, so you already kinda barking up the wrong tree if you in C with it.

But ya, they need to start really attacking some of the other issues.

[michprev]

FreeRTOS is not heavy at all. To say "It would make better sense to write low level drivers without FreeRTOS " is really not true.
FreeRTOS, makes running many threads at one very simple. You can even set a section of code as CRITICAL if you need to something timing intensive so there are no task switching, not to mention you can also run certain things on just one of the cores, to get even better performance. In the config you can even set FreeRTOS to only run on one core, and not the other. So it's quite flexible to get most job done without a problem.

I am not of the same opinion. FreeRTOS works only with tick frequency. Once you need to wait for something inside a driver you will wait for 1 ms with 1000 Hz tick frequency. With higher tick frequency you're wasting more CPU power. Also there is no need to use FreeRTOS inside hardware drivers. If you want to use FreeRTOS anyway you can always do something like

Code: Select all

while (!Do_SPI_Transmission_Without_FreeRTOS() != STATUS_OK)
	vTaskDelay(whatever);

STM does not use FreeRTOS in its drivers at all. As far as I know NXP offers "busy waiting" driver, DMA driver and FreeRTOS driver.

You cannot use CRITICAL section around a call to ESP-IDF SPI driver.

I am aware of the option running FreeRTOS only on the first core. It is also probably the best option for me. But you have to write everything yourself because FreeRTOS is everywhere. But with docs like this it is sometimes painful.

If you doing quad copter stuff, you should really be coding in assembler for many things anyways. which you can do that do here.
You timing requirements for things of this nature are really tight, so you already kinda barking up the wrong tree if you in C with it.

Almost all open source quadcopter projects use C and still offer very good performance. As long as you go with good drivers for the chip you are using there is no need to go for assembler.

[hassan789]

FreeRTOS works only with tick frequency. Once you need to wait for something inside a driver you will wait for 1 ms with 1000 Hz tick frequency.

In FreeRTOS you can get microsecond-level responsiveness...
You can study this: https://www.freertos.org/a00124.html

I have written hobby code for quadrotor UAV using FreeRTOS, and there was no problem. Can you provide a concrete example of your biggest issue?

[ESP_Sprite]

FIY, mod action: I'm gonna change the title of this topic. While I get your gripes, they are based on a very specific bit of the tech that we have; the fact that we actually do have millions of chips being used in the field kind-of disproves that statement, and I'd rather not have others panicing and thinking you found a gaping fundamental flaw or something.

FWIW, FreeRTOS is a pre-emptive RTOS and as such not bound to the tick time: if you have an interrupt or task or whatever that unblocks something on the other core. Especially for something reasonably slow (doesn't need sub-uS of reaction time) as controlling quadcopter, I'd really like to see an example of why the standard top-half-bottom-half working would be too slow. (Fair note: I'm not a quadcopter expert, and this is not hyperbole: I really would like to see an example here.)

With regards to the I2C and SPI peripherals: I agree with you there. These bits of hardware have been expanded from their ESP8266 counterparts too fast, with too little testing. They can still do most of the things that we want them to do, but it takes some babysitting and specific register fiddling, and we haven't found every single gotcha for every single combination of modes yet; we're working on them and I would say that for a lot of use cases the drivers work petfectly fine. I 100% agree these things should be documented in the TRM, but you have to look at the reality of things: very few people, especially in a commercial setting, want/need to twiddle with the registers themselves, and thus fixing the drivers takes precedence. I'd also disagree with us hiding these things: the drivers and everything is open-source, so you can see exactly what things we need to do to get things working.

With regards to the separation between lower-level drivers and higher-level (FreeRTOS or something else based): This is on our ToDo list. We are aware that this would be helpful so people porting everything to a different OS. However, again, you have to keep in mind our priorities are partially based on the use cases of our customers, so it's not something that will be implemented tomorrow.

With regards to the WiFi/BT drivers not being portable: I disagree there. Both the WiFi and BT drivers have a portability layer where you can effectively hook your own RTOS primitives, and they should work just fine in that fashion. I know of at least one OS that has been ported to it this way (not sure if that has an NDA, so I can't really say more, though.)

In general: we're pretty available here on the forum, on Github, IRC and elsewhere. Even if 99% of our users and customers are perfectly happy using esp-idf/FreeRTOS and the abstraction layers it provides, we are not against people trying to work things out for themselves. It hurts to see you come on the forum and effectively slam our efforts and FreeRTOS, partially with things that are not true, instead.

[michprev]

Ad. FreeRTOS:
I don't have hardware by myself now so I cannot give some real data to show if it's too heavy or not. I don't want to blame you for using FreeRTOS generally - I ment this thread to be about state of ESP32 hardware, docs and IDF. I did not want it to be about quadcopters. I knew that I will have to give big efforts to make it working.

I have written hobby code for quadrotor UAV using FreeRTOS, and there was no problem. Can you provide a concrete example of your biggest issue?

I can give an example tomorrow. How often do you read gyro & accel data? Are you sure that you catch the new data every single time (you don't miss every second for example)? How powerful was the CPU you were using?

I would say that for a lot of use cases the drivers work petfectly fine.

Because there is a ton of workarounds and limitations. SPI CS pre-transmission setup time is quite a huge issue (it is working only in half duplex mode which is not very frequent). It means that you have to control CS signal manually (together with WP and HOLD signals).

I 100% agree these things should be documented in the TRM

Firstly it should be in "ECO and Workarounds for Bugs in ESP32" document. People will not look deeply into ESP-IDF when considering using ESP32. ADC issue should be there - I don't know about it very much but from I've read this is a big one.

very few people, especially in a commercial setting, want/need to twiddle with the registers themselves

That quite surprises me. I would expect hobbyists being comfortable with ESP-IDF (as many of them are "Arduino users" that do not care about performance anyway) while commercial users often have special requirements and need to go deeper - but that was just my guess.

and thus fixing the drivers takes precedence

Don't get me wrong but for me you are fixing them way too long. As WiFive said they should be bomb proof. Right now they're certainly not as well as whole 2 years.

I'd also disagree with us hiding these things: the drivers and everything is open-source, so you can see exactly what things we need to do to get things working.

As I said people will not go deeply into ESP-IDF when considering buying ESP32. Also media will write only about those bugs mentioned in ECO document. As far as I know ROM code is not available.

With regards to the separation between lower-level drivers and higher-level (FreeRTOS or something else based): This is on our ToDo list. We are aware that this would be helpful so people porting everything to a different OS. However, again, you have to keep in mind our priorities are partially based on the use cases of our customers, so it's not something that will be implemented tomorrow.

I don't even understand why current drivers look like this. It is much easier to write drivers like SPI one I have posted here. As it not uses FreeRTOS there probably won't be so many bugs and it is much much faster. It took a week to make something like this.

As I said I wanted to make my custom SDK - SPI driver is a small part of it. Once it would be at least partially functional I would even share it with community. But TRM is almost unusable in this. The best (and only) source is ESP-IDF. But still you have to keep trying to see how to exactly works. And that takes a lot of time.
I can give a concrete example here:
Let's look at https://github.com/espressif/esp-idf/bl ... i_struct.h.
The first are flash commands. These are not documented at all. Thanksfully there are bugs in ROM code that handles SPI flash so we have this file https://github.com/espressif/esp-idf/bl ... om_patch.c (it still does not cover all uses that hardware offers).
There is no way how to know which hardware options are still applicable and which are override/ignored. Does changing "sio" bit do anything while using flash_rdid? And what about flash_read? (which by the way does not work with fast read modes).
What is the difference between "cs_hold_delay" and "hold_time"? (You don't have to answer me, I have hunderds of questions like this)

In general: we're pretty available here on the forum, on Github, IRC and elsewhere. Even if 99% of our users and customers are perfectly happy using esp-idf/FreeRTOS and the abstraction layers it provides, we are not against people trying to work things out for themselves. It hurts to see you come on the forum and effectively slam our efforts and FreeRTOS, partially with things that are not true, instead.

Before ESP32 I've been using STM32 chips so I certainly come with too high expectations. But I don't mind coding a lot of things myself and as I said I would even pay twice the price (or more) to have complete documentation and bug-free hardware.