HTTP Client: What happens if the server response is too large?

[TobiasUhmann]

Hi there,

I am using the HTTP client to do HTTP requests. One request should return a JSON that might be multiple megabytes large. I want to handle the case that it is larger than the available RAM. How do I do that properly?

The HTTP client docs (https://docs.espressif.com/projects/esp ... lient.html) do not specify much about the data_len attribute of the esp_http_client_event struct. Since I couldn't find a KConfig setting for some static buffer size for the HTTP client, I assume the memory for the event payload is allocated on the heap. What happens if malloc fails? Will I get an HTTP_EVENT_ERROR?

Also, I cannot change the API endpoint to use chunked encoding.

Thanks in advance!

[TobiasUhmann]

Ok.. once again, after having searched a while, I find the obvious answer just after having written the post here. Hoping that it might nonetheless spare someone else some time, I do not delete my question and post the simple answer:

The esp_http_client_config_t has a buffer_size attribute (that defaults to DEFAULT_HTTP_BUF_SIZE = 512). Larger responses lead to multiple HTTP_EVENT_ON_DATA. I thought there was only one HTTP_EVENT_ON_DATA for non-chunked HTTP requests, which is not the case.

[MicroController]

How do I do that properly?

1) esp_http_client_open(...)
2) repeat esp_http_client_read(...) until all data read.

[chegewara]

Ok.. once again, after having searched a while, I find the obvious answer just after having written the post here. Hoping that it might nonetheless spare someone else some time, I do not delete my question and post the simple answer:

The esp_http_client_config_t has a buffer_size attribute (that defaults to DEFAULT_HTTP_BUF_SIZE = 512). Larger responses lead to multiple HTTP_EVENT_ON_DATA. I thought there was only one HTTP_EVENT_ON_DATA for non-chunked HTTP requests, which is not the case.

Yes, thats how it works.
Remember that wifi MTU is around 1500, so you have "to read in chunks" anyway, just it has nothing to do with http chunked encoding.
When you receive multiple on data events you have to concatenate it somehow, either in ram or in file on flash or sd card.

[Hello1024]

Worth noting that it is also possible to parse json in chunks, keeping the state of the decoder between chunks.

So it's easily possible to process a 1MB JSON file downloded over HTTP with the available RAM, as long as the fields you need from the json are themselves small.

[chegewara]

Worth noting that it is also possible to parse json in chunks, keeping the state of the decoder between chunks.

So it's easily possible to process a 1MB JSON file downloded over HTTP with the available RAM, as long as the fields you need from the json are themselves small.

Can you share how to parse big (not completed) JSON?

[Hello1024]

Looks like this project does it:

https://github.com/kazuho/picojson/blob ... reaming.cc

Your code will have to be pretty customized to pull out the values needed sadly. That example can already parse json bigger than RAM as long as it is in the form of:

Code: Select all

[
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
{"x": "hello", "y": "test", "other_ignored": {whatever other data....},
]

At any point in time, only one 'line' of the json above is loaded in memory (although it doesn't rely on linebreaks).

You'll need to implement your own istream_iterator using blocking calls to esp_http_client_read(), and if you don't want it to block, I suggest using a new FreeRTOS task.

A pretty big task I'm afraid.

[MicroController]

https://github.com/skeeto/pdjson
sound promising.

https://github.com/squix78/json-streaming-parser
looks like it could be made to work without Arduino libs.

https://github.com/GOB52/gob_json
seems to be available for both Arduino and non-Arduino.