ESPAsyncWebServer Fails When Serving Large Files

catotonic · Postby **catotonic** » Mon Mar 18, 2019 11:41 am

I am very close to having everything working on my application. I only have two issues left and maybe only one.
The first issue is my lack of understanding of using the AsyncWebServer and FreeRTOS. I am trying to keep all my communications running on core 0, but at this time I am not sure what is happening. I will write another post about ASWS in the future, but because I don't understand, I am not sure if my big problem is affected by how I am implementing my second issue which is much more serious.
My big problem is that I have a SPA(single page application) with several large(200K-2M)files that I need to send from the SD Card to the client. I am able to send My index.htm file to the client just fine, but when is loads it then asks for several files.

Code: [Select all] [Expand/Collapse]

<html>
<head>
  <meta charset=utf-8>
  <meta name=viewport content="width=device-width,initial-scale=1">
  <title>1X Verifier</title>
  
<link href=/static/css/app.css rel=stylesheet></head>
<body><div id=app></div>
<script type=text/javascript src=/static/js/app.js></script>
< script type=text/javascript src=/static/js/manifest.js></script >
<script type=text/javascript src=/static/js/vendor.js></script>
 
</body>
</html>

GeSHi © Codebox Plus Extension

At most I am able to send 98% of the css file over, before it stops but none of the others.
I have tried several pattern, but none work.
I don't think FreeRTOS is the issue, I am able to Serial.write() 200K files within the server.on request handler.
Below is one of the simpler patterns that I have tried.

Code: [Select all] [Expand/Collapse]

server.on("/static/css/app.css",[](AsyncWebServerRequest *request){
  Serial.println("Sending /static/css/app.css");
   // request->send(200, "text/html", "<html><body><h1>Hello Heat!</h1></body></html>");
  sd1File = SD.open("/static/css/app.css"); // Is the file there?
  if (sd1File) {
    Serial.println("File /static/css/app.css found");
    //Serial.write(sd1File.read()):
    request->send(sd1File, "/static/css/app.css", "text/css");
  }
  sd1File.close();  
  });

GeSHi © Codebox Plus Extension

I see other methods and classes that might work, but I have no idea on how to implement them.
Suggestions, of the method in what class to use would help, an example would be great.
Thanks

catotonic · Postby **catotonic** » Mon Mar 18, 2019 7:35 pm

I was hoping my previous post would save me from blindly trying to use undocumented functions. So now I would like to ask for help on a specific function. I would like to know if it is the best to use and hopefully how to use it.
I tried to implement:

Code: [Select all] [Expand/Collapse]

request->send("text/css", 128, [](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {
//Write up to "maxLen" bytes into "buffer" and return the amount written.
  //index equals the amount of bytes that have been already sent
  //You will not be asked for more bytes once the content length has been reached.
  //Keep in mind that you can not delay or yield waiting for more data!
  //Send what you currently have and you will be asked for more again
  return sd1File.read(sendBuffer, 256);
}
 

GeSHi © Codebox Plus Extension

I am not that good at C++ to understand what:

Code: [Select all] [Expand/Collapse]

[](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {}

GeSHi © Codebox Plus Extension

Means.
I assume the [](u.... is somehow returning a AsyncWebServerRequest object, that somehow points to a function.
I have not run across the size_t{} or even the typedef{} pattern before.
So I took a wild guess and wrote the following code:

Code: [Select all] [Expand/Collapse]

   uint8_t sendbuff[256] = {0};
    fs::File sd1File.read(sendBuffer, 256);
    request->send("text/css", 128, [](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {
  return sd1File.read(sendBuffer, 256);
});

GeSHi © Codebox Plus Extension

But my code promptly crashed.
I have been very pleased with all the examples and explanations I have found online from the support staff, contractors, and amazing volunteer coders in this community. I have learned a lot about coding. But the code above is well over my head and I wouldn't even know how to search google for it. Hopefully someone will flame me and tell me this is coding, and that I should know such simple techniques.
That is as long as they also give me the solution.

igorlab · Postby **igorlab** » Fri Aug 09, 2019 6:54 am

"Houston we have a problem"

catotonic, hello!
Did you find the solution to use ESPAsyncWebServer with SD card?

Thanks in advance for reply

Igor

catotonic · Postby **catotonic** » Mon Aug 12, 2019 4:40 pm

Hello Igor,
Yes, I did find a solution. I just went with the ESP32WebServer and WebSocketServer. They are not as elegant as the the ESPAsyncWebServer and I liked the ESPAsync socket server better, but the ESP32WebServer can chunk and server files up to 500KB without timing out or crashing. Slow, but it gets the job done. It will be interesting when I get finished with my final Vue JS application. It is over 2 MB in size. I might be back at the forums begging for help

lbernstone · Postby **lbernstone** » Tue Aug 13, 2019 1:35 am

A common thing I see with people using the AsyncWebServer to serve files, is they don't allocate enough file buffers when they begin the backend file system. SD.begin is defined as
bool begin(uint8_t ssPin=SS, SPIClass &spi=SPI, uint32_t frequency=4000000, const char * mountpoint="/sd", uint8_t max_files=5);

Async by it's nature allows you to request all the files at once, and this can be depleted very quickly if you have a large composite web page or a lot of simultaneous clients. Try with more max_files.

catotonic · Postby **catotonic** » Tue Aug 13, 2019 10:46 am

@Ibernstone
Thanks for the input.
It is something to try, although the Vue JS application being served is no more than four large files served once and there is only one person using it at at time. Without trying it, it doesn't sound like it would explain why it crashed serving four files, except for the possibilities that they require more memory than available or more time than the watchdog timer will allow. I am not sure, but it looked as if the ESP32Webserver had a way of breaking up the files and serving them in chunks. I will probably go back an try it again because the ESP32WebServer is very slow.

rsavage · Postby **rsavage** » Tue Aug 13, 2019 3:05 pm

I had a similar problem, I was using ESPAsyncWebServer and have a web page that draws graphs and has buttons that allow a user to download raw data in the form of large (up to 1 MB) text files. When I first tried implementing the server only about 98% of a text files would make it to the client. I found that if I removed the "sdfile.close" statement then it would work. Never figured out why it works but it does and I have not had any issues leaving the statement off (it works without closing them). So far my server has been up and running for about a month with no crashes. Snip-it of code that I am using is shown below.

server.on("/MCD.txt", HTTP_GET, [](AsyncWebServerRequest *request){
File file11 = SD.open("/data/MCD.txt");
if(!file11){
//Serial.println("MCD File Failed to open file for reading");
return;
}
request->send(file11, "/MCD.txt", "text/xhr");
Serial.print("Recieved MCD.txt request from client IP ");
Serial.println(request->client()->remoteIP());
});

In terms of web browsers, I did find that if I used Safari it would cause the web server to lock up. Using Chrome or Internet Explorer I have no issues and the server is stable. The only difference that I can see is that Safari makes multiple request for files while the other two browsers make the request one time and then wait for the data to be sent. Safari for some reason doesn't wait long enough for the server to respond then when the data has not arrived yet it makes more request and causes the the server lock up? I tried to
find some type of HTML statement that I could include in my "index.html" file that would cause the (Safari) browser to wait but have
not found anything, I am a newbie when it comes to web page programming.

catotonic · Postby **catotonic** » Tue Aug 13, 2019 6:29 pm

@rsavage
Thank you for the suggestions. Sounds promising. I plan on specifying Chrome. Hopefully Chrome for Apple does not have the same issues a Safari.

rsavage · Postby **rsavage** » Wed Aug 14, 2019 8:53 pm

I have test my AsybcWebServer using Chrome on iPhone and no issues, it work without any problems.

ullixesp · Postby **ullixesp** » Sat Jun 12, 2021 2:58 pm

Has this finally come to a positive conclusion? I am facing a similar problem with a ESP32 and Watch Dog Timer triggered reboots when attempting to download files >130k. A lot of details reported on github:

Bug: Using server.serveStatic causes WDT crashes almost every 2nd time for files >150K
https://github.com/me-no-dev/ESPAsyncWe ... issues/984

I want to use only the Flash of 4MB on an ESP32 Dev.

ESPAsyncWebServer Fails When Serving Large Files

ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Serving Many Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Re: ESPAsyncWebServer Fails When Serving Large Files

Who is online

About Us

Extra

Information