Increase UDP WiFi TX buffer count

[rickard.eklof]

Hi,

I'm having issues when sending a UDP data stream from the ESP32 using SDK 2.1.

I've tried alot of different settings in menuconfig to improve the performance. I've tried both static and dynamic TX buffers. I tried increasing the number of buffers to the max value 64. I also tried to set it to 0 (manually in the sdkconfig.h since menuconfig does not allow a setting of 0 even though the Wifi manual specifies that 0 should be allowed and then unlimited buffers can be allocated).

Why can I not set CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM to 0 for unlimited allocations?

This is what my initialization logs tell me about allocated buffers.

Code: Select all

I (282) wifi: Init dynamic tx buffer num: 64
I (292) wifi: Init dynamic rx buffer num: 32
I (292) wifi: wifi driver task: 3ffcde84, prio:23, stack:4096
I (292) wifi: Init static rx buffer num: 25
I (292) wifi: Init dynamic rx buffer num: 32

This is my loop for sending data (15 bytes sent as fast as possible). Using an oscilloscope I see that Pin15 gets set after the loop is run ~35 times. Eventually all data is sent, but there seems to be an issue with the buffers since I set them to 64 which should allow all my packets to be sent without sendto() returning ENOMEM.

Code: Select all

for(int i = 0; i < 50; ++i)
{
	int res = sendto(txSocket, &data[0], 15, 0, (struct sockaddr *)&sRemote, sizeof(sRemote));
	while (res == -1)
	{
		gpio_set_level(15, 0);
		ESP_LOGE(TAG, "Error sending: %i", errno);
		ESP_LOGE(TAG, "Error sending: %s", strerror(errno));
		const TickType_t xDelay = 1 / portTICK_PERIOD_MS;
		vTaskDelay(xDelay);
		res = sendto(txSocket, &data[0], 15, 0, (struct sockaddr *)&sRemote, sizeof(sRemote));
		gpio_set_level(15, 1);
	}
}

This is the table of processes and priorities. All my processes are 3 or 2 (very very low priority). That way I was hoping that the wifi-driver will be prioritized above my processes and thus empty the send queue faster than I can fill it. That is not the case though.

Why does not the wifi driver send the packets faster than I can insert them?

Is there a way to increase the priority or speed of the wifi driver?

Code: Select all

main            R       1       2080    4
IDLE            R       0       612     5
IDLE            R       0       552     6
tiT             B       18      1624    11
Tmr Svc         B       1       1696    7
pmT             B       21      2152    14
HandleSPIRXTask B       3       3688    18
HandleSPITXTask B       2       3688    20
SPIRXParsePacke B       2       3680    19
rtT             B       22      2984    15
ipc1            B       24      576     2
eventTask       B       20      3288    12
UDP_ReceiveTask B       3       3328    22
UDP_HandleTXTas B       3       3680    24
UDP_HandleRXTas B       2       3672    23
wifi            B       23      1964    13
ipc0            B       24      628     1

Thanks
Rickard

[ESP_Angus]

Hi Rickard,

Sorry for the slow response.

Hi,

I'm having issues when sending a UDP data stream from the ESP32 using SDK 2.1.

I've tried alot of different settings in menuconfig to improve the performance. I've tried both static and dynamic TX buffers. I tried increasing the number of buffers to the max value 64. I also tried to set it to 0 (manually in the sdkconfig.h since menuconfig does not allow a setting of 0 even though the Wifi manual specifies that 0 should be allowed and then unlimited buffers can be allocated).

Why can I not set CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM to 0 for unlimited allocations?

I checked with the WiFi team and the documentation which says CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER can be set to zero is a mistake. The RX buffer can be set this way for unlimited (until memory runs out) dynamic receive buffers but the TX buffers always have a fixed limit. We'll fix the config documentation.

This is my loop for sending data (15 bytes sent as fast as possible). Using an oscilloscope I see that Pin15 gets set after the loop is run ~35 times. Eventually all data is sent, but there seems to be an issue with the buffers since I set them to 64 which should allow all my packets to be sent without sendto() returning ENOMEM.

EDIT: There were some statements here about dynamic TX buffers being used for management frames but this is not the case. Checking with WiFi team for more clarification.

I was able to reproduce your results (with 64 dynamic TX buffers) on a busy network, but not on every iteration (100ms delay between iterations). I didn't take a full packet capture, but possibly sending ICMP port unreachable responses or other replies to unsolicited traffic may be occupying some of the TX buffers.

Why does not the wifi driver send the packets faster than I can insert them?

Is there a way to increase the priority or speed of the wifi driver?

If you can improve reception to the WiFi AP then you can probably increase the rate at which the Wifi level TX buffer queue is cleared.

If your AP suports 802.11n AMPDU (frame coalescing), then this may also help when sending lots of small UDP packets. This is enabled by default but you can try increasing "WiFi AMPDU TX BA window size " to the maximum supported value (32).

There is also the situation that the TCP/IP task (which is pushing data frames to the WiFi driver) can run on either core, which means it will often be running on a different core to the WiFi task. This means that even though the WiFi task is higher priority, it may not have cleared the driver-level TX queue yet (due to RF network environment constraints). But this doesn't block the TCP/IP task from running on the other CPU and trying to send data frames to the driver.

I've spoken to the WiFi team about allowing for some very limited duration blocking & retrying at the driver layer (low_level_output() to esp_wifi_internal_tx()) if the TX buffers are full. This will prevent most situations where the driver layer ERR_MEM is relayed all the way back to the caller of send(). We don't have an ETA for this feature, unfortunately.

In the meantime, if you need to send a lot of UDP packets in a short period of time then testing for ERR_MEM and retrying after a short delay (as you are doing) is the best solution.

[rickard.eklof]

Hi Angus,
Thanks a lot for your answer.

Our goal is to write a real time communicating system via Wifi. Therefore latency is prioritized higher than throughput.

If your AP suports 802.11n AMPDU (frame coalescing), then this may also help when sending lots of small UDP packets. This is enabled by default but you can try increasing "WiFi AMPDU TX BA window size " to the maximum supported value (32).

Q1. Is this a setting in the ESP32 SDK or in the router?

There is also the situation that the TCP/IP task (which is pushing data frames to the WiFi driver) can run on either core, which means it will often be running on a different core to the WiFi task. This means that even though the WiFi task is higher priority, it may not have cleared the driver-level TX queue yet (due to RF network environment constraints). But this doesn't block the TCP/IP task from running on the other CPU and trying to send data frames to the driver.

Q2. Is there a way to ensure that both TCP/TP task and the wifi task are running on the same core?

I've spoken to the WiFi team about allowing for some very limited duration blocking & retrying at the driver layer (low_level_output() to esp_wifi_internal_tx()) if the TX buffers are full. This will prevent most situations where the driver layer ERR_MEM is relayed all the way back to the caller of send(). We don't have an ETA for this feature, unfortunately.

OK, but it still won't solve our problem.

Since we prioritize latency over throughput, I believe the AMPDU is not desired in our application. I will try to undefine it and check the results.

Q3. Is there a way to flush the TX wifi buffer, forcing the wifi hw to send the data?

Q4. How long time does it take for the wifi stack + hw to send a packet (assuming AMPDU is turned off)?

Again, thanks alot for your quick response.
Rickard

[ESP_Angus]

If your AP suports 802.11n AMPDU (frame coalescing), then this may also help when sending lots of small UDP packets. This is enabled by default but you can try increasing "WiFi AMPDU TX BA window size " to the maximum supported value (32).

Q1. Is this a setting in the ESP32 SDK or in the router?

It has to be enabled in the ESP32 configuration and supported in the Access Pointer (router). It's a mandatory part of 802.11n so if 802.11n is supported & enabled then it should be supported.

I don't know the exact circumstances under which the ESP32 will combine transmitting frames using AMPDU, but I can find out.

Q2. Is there a way to ensure that both TCP/TP task and the wifi task are running on the same core?

There's no supported way, but you can experiment with the implementation of sys_thread_new() in components/lwip/port/freertos/sys_arch.c - currently this creates an unpinned task but you could experiment with having it call xTaskCreatePinnedToCore().

Note that this is very unlikely improve WiFi latency, it just may move the "buffer bloat" from the Wifi layer to the TCP/IP task (similar to adding a short blocking period to TX in the TCP/IP task).

Since we prioritize latency over throughput, I believe the AMPDU is not desired in our application. I will try to undefine it and check the results.

It's worth experimenting to verify, but I think AMPDU can only be helpful for latency if there are multiple data frames queued for transmission at the 802.11 layer.

Q3. Is there a way to flush the TX wifi buffer, forcing the wifi hw to send the data?

Q4. How long time does it take for the wifi stack + hw to send a packet (assuming AMPDU is turned off)?

The Wifi driver will be transmitting Wifi data frames as fast as it can. The root problem here is that Wifi is not a hard real-time network, there are many things which can effect latency. These include:

• Current 802.11 modulation scheme & protocol version in use (negotiated based on link quality & AP capabilities). This determines the maximum data rate under ideal conditions (ie what you would get if the AP and the ESP32 were sealed in an RF shielded box).
• Other WiFi traffic on the same network (WiFi uses RTS/CTS frames to negotiate access to the network with the AP & other stations).
• Data loss due to link quality & interference. 2.4GHz is an increasingly noisy spectrum, with innumerable ways for data frames to be corrupted over the air. When this happens, the 802.11 stack has to retransmit the data frames after a (spec-mandated) timeout.
• Power saving sleep modes, if enabled, which increase the latency between client transmissions (these are disabled by default so probably OK on the ESP32 side here.)

Moving the AP to a quieter channel, or improving the link quality (reducing distance, removing obstacles, redesigning device for better antenna, etc) will all help.

If your primary concern is minimizing latency, it may be worth making some packet captures from a target device on the network (or, even better, investing in a WiFi USB dongle that can use "monitor mode" to capture raw 802.11 traffic). Compare the end-to-end latency & inter-packet jitter that you are getting with your application's requirements.

If your requirements allow non-IP networking, you may also be interested in ESP-NOW which allows you to send connectionless data without the overhead of a TCP/IP stack. You still have a lot of the overhead of the 802.11 protocol, though.

[rickard.eklof]

Hi again Angus,
Thanks for your well written response.

I'm observing some strange results in my testing. I've tested on two separate ESP32 Thing boards and the results are similar. I'm getting a lot of dropped UDP packages when sending from the ESP32. The RX side works much better.

From a UDP client I send 1000 UDP messages very fast (~1ms between packets) to the ESP32 and it receives all of them.

When sending from the ESP32, I cannot even successfully send 100 messages (100 bytes each) to the client. I've experimented with different delays but even though I add delays of 100ms I still get dropped packets.

I've enabled AMPDU. This is the code I'm using to send data. It is being looped 100 times. If I set burstLimit = 1 and xDelay = 100ms, the loop should send 1 message, wait 100ms then send another packet. It mostly does this but it drops packets when e.g. i = 2, i = 8, etc. sendto() never returns any error (delays is always 0). Looking in Wireshark I see that the dropped messages are between two AMPDU transmissions. See the two pictures below.

Usually there are around 10% packet drops when using this scenario. It seems way too much to be caused by Wifi link quality since I can send 1000 packets from the client to the ESP32 without problems.

I also tried the same scenario with AMPDU disabled but the number of dropped packets are similar (10%).

Code: Select all

if (burst++ >= burstLimit)
{
	vTaskDelay(xDelay);
	burst = 0;
}
	
int res = sendto(rxSocket, data, count, 0, (struct sockaddr *)&sRemote, sizeof(sRemote));
uint16_t delays = 0;
while (res == -1)
{
	delays++;
		
	TickType_t xDelay = 20 / portTICK_PERIOD_MS;
	vTaskDelay(xDelay);
	res = sendto(rxSocket, data, count, 0, (struct sockaddr *)&sRemote, sizeof(sRemote));
}

1.png (16.03 KiB) Viewed 18083 times

2.png (11.65 KiB) Viewed 18083 times

[rickard.eklof]

I've done some more testing with a second ESP32 used as a Wifi packet sniffer to capture the radio data. I see the same problem in the sniffed packets. Some of the 100 packets I send are just missing, they never get sent from the ESP32.

I also tried different settings in menuconfig such as static TX buffers, 16 buffer, 64 buffers, the same result every time.

I've also tried on two separate ESP32 thing devices, and two different routers. Same result.

I've traced the code through the lwip-stack all the way to esp_wifi_internal_tx(), and there is not any error reported anywhere.

I do however get the following error very often (even when the device is idling):
Line 335 in etharp.c

Do you have any suggestions for me to try?

Rickard

[rickard.eklof]

Hi,
We still need information about this issue in order to move forward with our product.

The issue is simple and should be reproducible. Could you implement the following scenario and tell me if you experience the same issues or not:

- Setup an Access Point without access to Internet
- Boot ESP32 in station mode, connecting to the access point, setting up an UDP server
- Connect a UDP client to the same access point.
- Send a message from the UDP client to the ESP32
- The ESP32 sends 100 messages back to the UDP client with a 100 ms delay between each send.

I see 10% dropped packets in the scenario above. Using a second ESP32 as wifi sniffer, the dropped packets are not even transmitted from the ESP32.

Thanks
Rickard

[Magnetuz]

Hi @rickard.eklof.

Did you find the solution for this problem? I am having the same issue with UDP packets loss/not been transmitted.