Amount of jitter on LEVEL3 interrupt?

[bbulkow]

Hi,
I'm doing a fairly standard thing of using the RMT interface to control LEDs. You have a simple test program that does the same, there are multiple libraries.

However, I've also got a very simple web server, and accepting TCP requests. That code is also part of your standard distribution, and not being hit very hard. Like, having a few browser windows. The REST requests coming in are not serving flash objects, they are simply returning time values using a REST interface.

The timing required to control 800Khz WS8211 with a single RMT buffer is about 35us. That's the amount of time using the double-buffer technique - the interrupt handler has to be called every 35us, and can tolerate about 10 to 20us of jitter. This is running at the highest priority that will accept C code ( even though the function itself is not doing anything other than math from pointers ).

I am finding with this simple use case that there is often (a few times a minute) 40 to 50us of jitter in this interrupt, on either core. I can only imagine it's the wifi interrupt, because that's all that's happening in the system. This is enough to cause the RMT driver to run dry, which creates visual artifcats.

In this case, I would prefer to drop the IP packet. Having glitchy LEDs is bad, and TCP is a protocol that accepts retransmits ( and wifi is lossy anyway ).

The test case is very simple, because it uses three different documented ESP-IDF interfaces ( HTTP server, Wifi module, RMT test code ). It can even be observed without LEDs, because one can measure the jitter.

The questions:

* Do you expect jitter amounts in the 40 to 50us range on an IRAM_ADDR LEVEL3 interrupt, just for a two-line response to a HTTP rest request ( like 404 not found )? Or should this be considered a bug? The test case is not hard to reproduce.

* Is there a way to decrease the priority of WIFI interrupts? I have crawled through the code for a few hours and couldn't find anything obvious. I found a very old forum post where someone said they needed to do that in a similar case, but reading everywhere I haven't found where the priority levels for the different Wifi interrupts are held. A great feature would be a CONFIG for higher and lower priority wifi.

I have two other paths.

One is to use more RMT buffer space, reducing the number of channels to 4, which given the measurements I have, might solve the problem with little code. However, I will never be sure of a larger delay coming and creating a glitch, because any number of people could put a web browser at the control interface. There is a limit on the number of TCP connections, so perhaps the problem won't be bad. In this particular application I am only controlling 500-ish LEDs thus can still run good frame rates with 4 pins.

The second is to write an assembly language shim and raise the priority level of the interrupt code which translates the RGB values to RMT's format. Since the code literally does nothing other than pointers and copies it is very safe. I wish there was an easy way to notate that a particular bit of code is safe, instead of having to actually write in assembly language, but I think you have an example of calling C from assembly (which carefully states in the comments that particular C call is safe) so I can make a shim without having to literally code the logic in assembly.

In the case of raising the priority, the documentation implies I must not use the usual routine of using a xSemaphoreGiveFromISR to wake the task. Is there a safe way of signaling a task from a high priority interrupt (say, 5). I have some other ways to check completion, like using combination of a timer and a shared memory address, but if there is a better way using the scheduling system it would be nice to know.

I wonder if you could also give an opinion on which of these is more likely to succeed.

Thank you.

[bbulkow]

I have reworked my code to use MEM_BLOCK_NUM in RMT, and have removed the LED glitches. This is because the measured jitter was about 50us, and by changing MEM_BLOCK_NUM to 2, I could absorb over 120us of jitter, so it works.

I still believe that since the ESP-IDF sample code glitches very badly with a single Wifi HTTP web server, there needs to be investigation.

* Please provide a way to deprioritize Wifi, or improve the Wifi stack to not be so greedy. 50 us is a lot.

* Please provide a better description of higher priority interrupts, perhaps with another sample. I'm glad you have a sample, but noting how to signal a task of completion from a high priority interrupt, and how to call a trivial safe C function, would make the system far more usable.

Thanks.

[bbulkow]

This issue is very serious, and I would hope there could be some attention.

Let me ask the simple question:

how do I lower the priority of the wifi system to LEVEL 2 so to make the LEVEL 3 interrupts more responsive?

Is wifi at a particular interrupt in all versions? Is there pseudo code for finding the wifi interrupt? How many interrupts are there? ( there seems to be a NMI and a lower "medium level" interrupt?

I have searched a bit through the code and found no obvious breadcrumbs for where the interrupt might be registered, what priority it's set to, and how to change it.

Espressif, please help.

[ESP_Sprite]

As an alternative, can you just run your timing-sensitive stuff on the other core?

[bbulkow]

As an alternative, can you just run your timing-sensitive stuff on the other core?

Thanks for taking the time to answer.

Since there are various Wifi interrupts and server tasks, and no promise that I've read from ESP-IDF regarding which core is used for what in the ESP-IDF system (other than the general app / sys split), how would I select the core, other than trying both cores and seeing which one happens to have lower jitter with the particular build of ESP-IDF I have? Both cores seem to have a lot of jitter, and I'm also publishing a library ( https://github.com/bbulkow/FastLED-idf ), so I'd like to have a solution that works on all 4.x versions of ESP-IDF ( see the version specific code I've already had to write for RMT, not pleasant ), and guidance on where to run tasks.

There is a menuconfig for the Wifi _task_ core (default 0), right, but my interrupts are running above task priority at LEVEL 3 interrupt so should interrupt that task, and I am running on the "other core" compared to the Wifi _task_. But this question is about interrupts.

Again, the question is how to I find the Wifi interrupts and potentially lower priority.

There are API calls for finding tasks, and IRQs can be iterated.

Is that the proposed solution, just dig through and find the IRQ handler pointer that decompiles to the wifi interrupts(s), figure out which cores they are on, and either avoid those cores and/or change the priority?

[ESP_Sprite]

If you have the menuconfig WiFi core set to 0, and initialize the WiFi stack on core 0 (not sure if that's needed, can't look into the WiFi code at the moment), I'm pretty sure that it should never allocate any interrupts on core 1. Same goes for the BT stack.

Wrt lowering the prio of WiFi interrupts: there's no 'nice' way to do this, but if you want to prove that lowering the WiFi int prio works, the interface between the WiFi libs and ESP-IDF is in components/esp_wifi/esp32/esp_adapter.c , there's also set_isr_wrapper there. It's a bit iffy, but you might be able to install the WiFi interrupt into a lower-level interrupt there. You'd also need to use the interrupt matrix to route the physical WiFi interrupt there, though. (You need to use a different interrupt number as the levels for the various interrupts are fixed on Xtensa.) If this turns out to solve the issue, I can create an internal issue to make the WiFi interrupt level configurable.

[bbulkow]

Thanks for the response. It turned out that a good solution was to use the RMT mem_blocks parameter to increase the amount of buffering available to the system. Since that worked I did not go back and attempt to play with the wifi priority. Now that I see your report and the proper breadcrumb to try it I might, thank you.

[BB5000]

@bbulkow
A big thanks for bringing up this topic and finding a work around that at least alleviates the effects (if it doesn't completely solve it).
Applying your work around at least in my case, and as far as I could witness so far, did the trick.

Prior to finding this thread I already change the core for WIFI like ESP_Sprite also proposed here, but that did not remove the glitches.

I have to admit, that your solution still leaves some doubts and I am not sure if there are certain circumstance (e.g. like you mention under heavy WIFI activity) in which the problems might return.
Thus a clear explanation why switching the memblock will always solve the problem or an alternative solution e.g. with an altered prioritization between RMT and the WIFI-handler (perhaps from Espressif side) would be highly appreciated.