HTTP server can't handle many requests

[marclee]

Configuration:
- ESP32-S2-WROVER (4MB) with ESP-IDF V5.0
- Wifi is running in mode WIFI_MODE_APSTA
- HTTP-Server is running
- different other processes are running too

Everything works fine as long as HTTP server is not overloaded. HTTP requests from a web browser are processed well. Dynamic data as well as files from SPIFFS file system. Files can be downloaded using wget command.

But if more than one HTTP clients request data from HTTP server is starts getting in trouble.
There are different behaviours:

- HTTP-Clients waits in an endless loop for data.
- ESP32 blocks other processes that crash or watchdog kills them.
- HTTP server stucks.

Error and warning messages:

httpd_txrx: httpd_sock_err: error in send : 104
httpd_resp_send_chunk: ESP_ERR_HTTPD_RESP_SEND

httpd_txrx: httpd_sock_err: error in send : 11
httpd_resp_send_chunk: ESP_ERR_HTTPD_RESP_SEND

task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
task_wdt: - IDLE (CPU 0)
task_wdt: Tasks currently running:
task_wdt: CPU 0: tiT
task_wdt: Print CPU 0 (current core) backtrace

Sometime an other process crashes: spi_device_queue_trans: ret = ESP_ERR_NO_MEM

How can I make the ESP32 more stable? Too many requests shouldn't lead to crashes.

I have tried to reduce and increase HTTP server config variable max_open_sockets.

Desired behaviour: ESP32 should just ignore HTTP requests that can't be handled.

[ESP_YJM]

HTTP server only support handle requests one by one. If one requests callback cost too much time, it will block other requets. I think you can try add thread in every requests callback, that will not block HTTP server to handle many requests. Note that you need thread synchronization for some value and status(Not sure what will happend).

[marclee]

Thank you for your answer. I know that only one request can be handled at once. So how should new requests be handled while another request is processed? Could you explain what you mean with "add thread" and "thread synchronization"? Is there any example?

No external event should be eligible to let the ESP32 get stuck. If there are more requests than ESP32 can candle, it should never lead to stucking or crashing. ESP32 should just ignore further requests. At the moment to many request lead to complete stucking or crashing of some (random) task.

[ESP_Sprite]

I'm honestly wondering if there's something else going on... the http server does have its limits, but it's not been unstable to me. Are you sure you're not e.g. subtly having memory corruption or overflowing a stack somewhere? You should be able to turn on checks for that in menuconfig fairly easily.

[ESP_YJM]

Hi marclee
`Add thread`, you can use queue or task to not block http server task to receive.You can creat a queue and when http server receive a request, you can throw this request to queue, so it will not block http server to receive another requests.
"thread synchronization" is a complex issue and you maybe know what it is when you testing function.

[boarchuz]

when http server receive a request, you can throw this request to queue, so it will not block http server to receive another requests.

From recent experience, the server will purge the request upon return from the handler, so this is not really viable. It's also not clear what the lifetime is of the request data passed to the handler so using anything outside of that context seems unsafe.

Would it be worth opening a feature request for a multi-threaded httpd? (Or an example, if this is already possible and I'm doing something wrong.) AsyncWebServer in Arduino shows us this can result in significantly better performance and responsiveness.

[ESP_YJM]

Hi boarchuz
Yes, you are right. One http server use the same hd, so after return, the old hd will be replaced by the new one. It will be wrong.
So need make every requests use different hd and free it in uri callback handle. This more like a feature for improving HTTP server performance, need internal evaluation.

[marclee]

I'm honestly wondering if there's something else going on... the http server does have its limits, but it's not been unstable to me. Are you sure you're not e.g. subtly having memory corruption or overflowing a stack somewhere? You should be able to turn on checks for that in menuconfig fairly easily.

I was thinking the same but couldn't find any mistake in my code yet.

Today I tried following:
I reduced .max_open_sockets to 1. So only one open socket is allowed.

From Linux console I started several connection requests simultaneously in a loop. It works and does not make any problem. Requests are handled successively without any error. But if requests come from firefox web browser, more than one request at the same time leads inevitable to fail behaviour. First ESP32 stucks, than after a few seconds, the following error messages are shown on console:

Code: Select all

What's the difference between requests from wget an firefox web browser?
I (43951) http-server.c: Sending file: /js/bundle.js (301894 bytes)...
W (43961) httpd_txrx: httpd_sock_err: error in send : 104
E (43961) http-server.c: File sending failed!
E (43971) http-server.c: httpd_resp_send_chunk: ESP_ERR_HTTPD_RESP_SEND
W (43971) httpd_txrx: httpd_sock_err: error in send : 104
W (43981) httpd_txrx: httpd_resp_send_err: 500 Internal Server Error - Failed to send file.
W (43991) httpd_txrx: httpd_resp_send_err: error calling setsockopt : 22
W (44001) httpd_txrx: httpd_sock_err: error in send : 128

It also seems that after successful transmission firefox does not close the connection. Socket stays blocked and the web server can't be reached by wget anymore.

[marclee]

Meanwhile I figured out that httpd_resp_set_hdr(req, "Connection", "close"); was set at the wrong place in code, so that it had no effect. As a result, connections weren't closed after completing requests. Sockets kept blocked.

Even though, blocked sockets shouldn't lead to fail behaviour. Even if all sockets are blocked or busy, ESP32 shouldn't stuck. I guess there's another problem.

Code: Select all

E (23474) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (23474) task_wdt:  - IDLE (CPU 0)
E (23474) task_wdt: Tasks currently running:
E (23474) task_wdt: CPU 0: httpd
E (23474) task_wdt: Print CPU 0 (current core) backtrace

[pawcuq]

Meanwhile I figured out that httpd_resp_set_hdr(req, "Connection", "close"); was set at the wrong place in code, so that it had no effect. As a result, connections weren't closed after completing requests. Sockets kept blocked.

Even though, blocked sockets shouldn't lead to fail behaviour. Even if all sockets are blocked or busy, ESP32 shouldn't stuck. I guess there's another problem.

Code: Select all

E (23474) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (23474) task_wdt:  - IDLE (CPU 0)
E (23474) task_wdt: Tasks currently running:
E (23474) task_wdt: CPU 0: httpd
E (23474) task_wdt: Print CPU 0 (current core) backtrace

Could you please show exactly where you put httpd_resp_set_hdr(req, "Connection", "close");? I have same problem here...