Hi there,
I am using the HTTP client to do HTTP requests. One request should return a JSON that might be multiple megabytes large. I want to handle the case that it is larger than the available RAM. How do I do that properly?
The HTTP client docs (https://docs.espressif.com/projects/esp ... lient.html) do not specify much about the data_len attribute of the esp_http_client_event struct. Since I couldn't find a KConfig setting for some static buffer size for the HTTP client, I assume the memory for the event payload is allocated on the heap. What happens if malloc fails? Will I get an HTTP_EVENT_ERROR?
Also, I cannot change the API endpoint to use chunked encoding.
Thanks in advance!
HTTP Client: What happens if the server response is too large?
-
- Posts: 9
- Joined: Mon Nov 27, 2023 8:11 am
-
- Posts: 9
- Joined: Mon Nov 27, 2023 8:11 am
Re: HTTP Client: What happens if the server response is too large?
Ok.. once again, after having searched a while, I find the obvious answer just after having written the post here. Hoping that it might nonetheless spare someone else some time, I do not delete my question and post the simple answer:
The esp_http_client_config_t has a buffer_size attribute (that defaults to DEFAULT_HTTP_BUF_SIZE = 512). Larger responses lead to multiple HTTP_EVENT_ON_DATA. I thought there was only one HTTP_EVENT_ON_DATA for non-chunked HTTP requests, which is not the case.
The esp_http_client_config_t has a buffer_size attribute (that defaults to DEFAULT_HTTP_BUF_SIZE = 512). Larger responses lead to multiple HTTP_EVENT_ON_DATA. I thought there was only one HTTP_EVENT_ON_DATA for non-chunked HTTP requests, which is not the case.
-
- Posts: 1692
- Joined: Mon Oct 17, 2022 7:38 pm
- Location: Europe, Germany
Re: HTTP Client: What happens if the server response is too large?
Re: HTTP Client: What happens if the server response is too large?
Yes, thats how it works.TobiasUhmann wrote: ↑Wed Apr 17, 2024 11:52 amOk.. once again, after having searched a while, I find the obvious answer just after having written the post here. Hoping that it might nonetheless spare someone else some time, I do not delete my question and post the simple answer:
The esp_http_client_config_t has a buffer_size attribute (that defaults to DEFAULT_HTTP_BUF_SIZE = 512). Larger responses lead to multiple HTTP_EVENT_ON_DATA. I thought there was only one HTTP_EVENT_ON_DATA for non-chunked HTTP requests, which is not the case.
Remember that wifi MTU is around 1500, so you have "to read in chunks" anyway, just it has nothing to do with http chunked encoding.
When you receive multiple on data events you have to concatenate it somehow, either in ram or in file on flash or sd card.
Re: HTTP Client: What happens if the server response is too large?
Worth noting that it is also possible to parse json in chunks, keeping the state of the decoder between chunks.
So it's easily possible to process a 1MB JSON file downloded over HTTP with the available RAM, as long as the fields you need from the json are themselves small.
So it's easily possible to process a 1MB JSON file downloded over HTTP with the available RAM, as long as the fields you need from the json are themselves small.
Re: HTTP Client: What happens if the server response is too large?
Can you share how to parse big (not completed) JSON?Hello1024 wrote: ↑Thu Apr 25, 2024 7:41 pmWorth noting that it is also possible to parse json in chunks, keeping the state of the decoder between chunks.
So it's easily possible to process a 1MB JSON file downloded over HTTP with the available RAM, as long as the fields you need from the json are themselves small.
Re: HTTP Client: What happens if the server response is too large?
Looks like this project does it:
https://github.com/kazuho/picojson/blob ... reaming.cc
Your code will have to be pretty customized to pull out the values needed sadly. That example can already parse json bigger than RAM as long as it is in the form of:
At any point in time, only one 'line' of the json above is loaded in memory (although it doesn't rely on linebreaks).
You'll need to implement your own istream_iterator using blocking calls to esp_http_client_read(), and if you don't want it to block, I suggest using a new FreeRTOS task.
A pretty big task I'm afraid.
https://github.com/kazuho/picojson/blob ... reaming.cc
Your code will have to be pretty customized to pull out the values needed sadly. That example can already parse json bigger than RAM as long as it is in the form of:
Code: Select all
[
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
]
You'll need to implement your own istream_iterator using blocking calls to esp_http_client_read(), and if you don't want it to block, I suggest using a new FreeRTOS task.
A pretty big task I'm afraid.
-
- Posts: 1692
- Joined: Mon Oct 17, 2022 7:38 pm
- Location: Europe, Germany
Re: HTTP Client: What happens if the server response is too large?
https://github.com/skeeto/pdjson
sound promising.
https://github.com/squix78/json-streaming-parser
looks like it could be made to work without Arduino libs.
https://github.com/GOB52/gob_json
seems to be available for both Arduino and non-Arduino.
sound promising.
https://github.com/squix78/json-streaming-parser
looks like it could be made to work without Arduino libs.
https://github.com/GOB52/gob_json
seems to be available for both Arduino and non-Arduino.
Who is online
Users browsing this forum: No registered users and 182 guests