UDP packet delays

[kasper]

Hi,

I'm seeing some significant delays in the delivery of UDP packets from the esp32s3. I have testet this in two different wifi environments/networks using wireshark (see below)

The graph shows time between consecutive UDP packets (with encapsulated RTP packets).

Noisy wifi environment
The esp32 is placed close to the AP.
Here the biggest measured delay is 388ms, but I have mesured up to twice that, delays often hits between 200-300ms as seen.

Screenshot from 2022-05-05 14-49-00.png (218.24 KiB) Viewed 8245 times

Less noisy wifi environment
Esp is further away from AP
Measuring up to 462ms here, though most are below 100ms. Delays again often hit between 200-300ms.

Screenshot from 2022-05-06 09-40-07.png (208.62 KiB) Viewed 8245 times

The packages seems to be lumped together when leaving the esp (or elsewhere in the network?) when there has been a big delay.
Packets 56225-56253 (15 packets - others are fragmentation packets) arrive at the laptop at the same time after a delay of 194ms. The esp is set to send every 10ms.

screenshot from 2022-05-06 09-50-56.jpg (260.7 KiB) Viewed 8245 times

These are my sdkconfig settings:

Code: Select all

CONFIG_LWIP_IRAM_OPTIMIZATION=y
CONFIG_LWIP_MAX_UDP_PCBS=64
CONFIG_LWIP_UDP_RECVMBOX_SIZE=64

CONFIG_ESP32S3_INSTRUCTION_CACHE_32KB=y
CONFIG_ESP32S3_INSTRUCTION_CACHE_SIZE=0x8000
CONFIG_ESP32S3_INSTRUCTION_CACHE_8WAYS=y
CONFIG_ESP32S3_ICACHE_ASSOCIATED_WAYS=8
CONFIG_ESP32S3_INSTRUCTION_CACHE_LINE_32B=y
CONFIG_ESP32S3_INSTRUCTION_CACHE_LINE_SIZE=32
CONFIG_ESP32S3_DATA_CACHE_32KB=y
CONFIG_ESP32S3_DATA_CACHE_SIZE=0x8000
CONFIG_ESP32S3_DATA_CACHE_8WAYS=y
CONFIG_ESP32S3_DATA_CACHE_LINE_32B=y
CONFIG_ESP32S3_DATA_CACHE_LINE_SIZE=32

CONFIG_ESP32_WIFI_ENABLED=y
CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=2
CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=8
CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER=y
CONFIG_ESP32_WIFI_TX_BUFFER_TYPE=1
CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER_NUM=128
CONFIG_ESP32_WIFI_AMPDU_TX_ENABLED=y
CONFIG_ESP32_WIFI_TX_BA_WIN=32
# CONFIG_ESP32_WIFI_AMPDU_RX_ENABLED is not set
CONFIG_ESP32_WIFI_NVS_ENABLED=y
CONFIG_ESP32_WIFI_TASK_PINNED_TO_CORE_0=y
CONFIG_ESP32_WIFI_SOFTAP_BEACON_MAX_LEN=752
CONFIG_ESP32_WIFI_MGMT_SBUF_NUM=32
CONFIG_ESP32_WIFI_IRAM_OPT=y
# CONFIG_ESP32_WIFI_RX_IRAM_OPT is not set
CONFIG_ESP32_WIFI_ENABLE_WPA3_SAE=y
CONFIG_ESP_WIFI_SOFTAP_SUPPORT=y

Are these delays and lumped together packets expected behavior, or can I do something to try to minimize it?


Thanks!
/K

[ESP_YJM]

Hi Kasper,
Thank you for providing such a professional analysis. I think this may not UDP delay, this maybe the WIFI layer delay. When the ESP device want to send data, it need follow WIFI RTS/CTS protocol. If the environment is bad, it may delay some time to send out data. Besides, AP DTIM value also affects the station. And RTP protocol is a multicast packet，AP DTIM value more bigger and delay more longer.
You can change a AP and reduce the station assoicated with AP or choose a clean environment to re-test.

[kasper]

Hi ESP_YJM,

Thanks for the kind reply!
I'm playing around with the TX buffer size in the WiFi menuconfig, and I get significantly reduced overall delay when setting the TX buffers to static and a low buffer count.
However when I try to set the 'Max number of Wifi static TX buffers' above 32 the esp crashed with:

Code: Select all

ESP_ERROR_CHECK failed: esp_err_t 0x101 (ESP_ERR_NO_MEM) at 0x4038017c

The range should be up to 64

Do you have any idea what could cause this?

Thanks again!

/Kasper

[ESP_YJM]

Hi Kasper
The error is that not enough memory. You need optimize some memory.

[kasper]

Thanks for the quick reply ESP_YJM!

Here is my idf.py size output:

Code: Select all

Total sizes:
Used stat D/IRAM:  169203 bytes ( 101133 remain, 62.6% used)
      .data size:   16941 bytes
      .bss  size:   70040 bytes
      .text size:   81195 bytes
   .vectors size:    1027 bytes
Used Flash size :  539608 bytes
      .text     :  439279 bytes
      .rodata   :  100073 bytes
Total image size:  638771 bytes (.bin may be padded larger)

Is it correct that the static WiFi buffers are still allocated from the heap? And that the remaining 37,4% of the DRAM is heap?

Shouldn't I be able to set the 'CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM' higher than 32?
The Esp crashes at runtime when I set it to 33

Thanks!
/K

[ESP_YJM]

The remaining 37,4% heap is DRAM and IRAM. DRAM occupy more. Static WiFi buffers are allocated from the DRAM. I'm not sure that CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM whether can set up to 64. In your test, it will be crash when set to 33. So maybe it can't set, or maybe other reason. I am not a wifi expert. But you can turn to dynamic buffer to test, it can be set to 64 https://github.com/espressif/esp-idf/bl ... s.esp32#L6 and iperf example use this.

[kasper]

I can set the dynamic buffers to 128 no problem, but I would very much like them to be static for performance reasons, and as large as possible. Menuconfig says range: 1-64

Screenshot from 2022-05-20 13-23-50.png (82.36 KiB) Viewed 7920 times

[kasper]

Would it be possible to place the WiFi TX buffers in the PSRAM? I'm using the ESP32-S3-WROOM-1

[ESP_YJM]

Yes, we support PSRAM. Also you can enable PSRAM and set static TX buffer to over 32 and try.

[kasper]

Hi ESP_YJM,

I have now tried with an ESP32-S3-DevKitC-1-N8R8 with 8MB of PSRAM and I still can't set the static wifi buffer to more than 32 without it crashing at boot.

Can you point me in direction of someone who has deep insigt into the esp32 WiFi stack?

Thanks,
Kasper Nyhus