esp_http_client_read() returning wrong data

[Pakuee]

Hi all,

I've been playing around with the esp_http_client_example, specifically the https_with_url() function. My goal is it to download a JSON file, however I've noticed some strange behaviour wenn trying to read the date from esp_http_client_read(). For small files (~100bytes) this seems to work fine, but as soon as I try to retrieve larger char arrays the output is scrambled.

For example this file will output:

Code: Select all

 
Armada (@martuishere) and Javier Usobiaga (@htmlboy)


/* SITE */
        Last update:2012/02/04

        Language: Català / Czech / Deutsch / English / Castellano / Japanese / Dutch / Russian / Chinese
        Doctype:HTML5
        IDE: Sublime Text, Notepad++, FileZilla, PhotoshopTranslator: Daniel Kršiak
        Twitter: @krsiakdaniel
        Location: Czech Republic

        ZH Translator: Ana Villalba
        Location: Spain

        JA Translator: Clémence Haure
        Location: Spain

        FR Translator: Thibaud Desodt
        Location: Belgium

        Media Queries by: Marta��?humanstxt.org

Which is only the last portion of the file and also duplicated at the end. I'm not sure if the mistake is on my side in the way I retrieve the data from the function or if it is a bug.
Here is the code I used:

Code: Select all

#include <string.h>
#include <stdlib.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_log.h"
#include "esp_system.h"
#include "nvs_flash.h"

#include "app_wifi.h"

#include "esp_http_client.h"

#define MAX_HTTP_RECV_BUFFER 512
static const char *TAG = "HTTP_CLIENT";

extern const char server_root_cert_pem_start[] asm("_binary_server_root_cert_pem_start");
extern const char server_root_cert_pem_end[]   asm("_binary_server_root_cert_pem_end");

esp_err_t _http_event_handler(esp_http_client_event_t *evt)
{
    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            ESP_LOGD(TAG, "HTTP_EVENT_ERROR");
            break;
        case HTTP_EVENT_ON_CONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_CONNECTED");
            break;
        case HTTP_EVENT_HEADER_SENT:
            ESP_LOGD(TAG, "HTTP_EVENT_HEADER_SENT");
            break;
        case HTTP_EVENT_ON_HEADER:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key, evt->header_value);
            break;
        case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            if (!esp_http_client_is_chunked_response(evt->client)) {
                // Write out data
                // printf("%.*s", evt->data_len, (char*)evt->data);
            }

            break;
        case HTTP_EVENT_ON_FINISH:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_FINISH");
            break;
        case HTTP_EVENT_DISCONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_DISCONNECTED");
            break;
    }
    return ESP_OK;
}


static void https_with_url()
{
    esp_http_client_config_t config = {
        .url = "http://humanstxt.org/humans.txt",
        .event_handler = _http_event_handler
    };


    esp_http_client_handle_t client = esp_http_client_init(&config);
    esp_err_t err = esp_http_client_perform(client);

    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTPS Status = %d, content_length = %d",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
    } else {
        ESP_LOGE(TAG, "Error perform http request %s", esp_err_to_name(err));
    }

    char charbuffer[2000];

    int data_len = esp_http_client_read(client, &charbuffer, esp_http_client_get_content_length(client));
    printf("data length %i\n", data_len);

    printf("%s", charbuffer);

    esp_http_client_cleanup(client);
}




static void http_test_task(void *pvParameters)
{
    app_wifi_wait_connected();
    ESP_LOGI(TAG, "Connected to AP, begin http example");

    https_with_url();

    ESP_LOGI(TAG, "Finish http example");

    vTaskDelete(NULL);

}

void app_main()
{
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);
    app_wifi_initialise();

    xTaskCreate(&http_test_task, "http_test_task", 8192, NULL, 5, NULL);

}

Thank you for your time!

[chegewara]

Probably data amount is bigger than buffer size and you have to read and concatenate data in this event:

Code: Select all

case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            if (!esp_http_client_is_chunked_response(evt->client)) {
                // Write out data
                // printf("%.*s", evt->data_len, (char*)evt->data);
            }
            break;
         

[Pakuee]

I've been trying to get this to work for the past week, but I'm unable to concatenate the data between the "ON_DATA" events without the ESP crashing due to some memory allocation issue.
Do you happen to have an example that shows how to assemble the buffer properly? I've seen similar questions around other forums, but non show a concrete answer.

Thanks for your time!

[pdemianczuk]

I've similar problem, anyone knows something about that?

[pdemianczuk]

Hi, I probably solve the problem. I update IDF to v4.1-dev-1533 from IDF4.0-dec-667. I had to update also toolchain (to esp-2019r2, compiler to 8.2.0), and I havn't problem with memory.
I found out some https://github.com/espressif/esp-idf/co ... 165cf29087
fix, but I am not sure is this fix the problem.
Generally, You should check is the component is updated.
I dunno know why, my app had a problem but the most people wasnt.

[BB5000]

Hi ESPers.
About a year after the last message in this thread I happened to run into very that problem.
Sorry to say, but the above mentioned solutions were not helpful. Updating the IDF or even playing with esp_http_client_config_t parameters such as buffer_size or user_data lead to no solution.

I could however successfully solve the issue by changing over to reading the HTTP result as a stream. This actually makes a lot of sense for receiving larger results. The procedure is described in the ESP programming guide: https://docs.espressif.com/projects/esp ... lient.html
The difference is that you don't use

Code: Select all

esp_http_client_perform(client)

but rather just open it

Code: Select all

esp_http_client_open(client, 0)

and then read from the stream in you own buffer, where of course you control the size.
If you want you can copy&paste the code from the ESP-IDF /examples/protocols/esp_http_client/main/esp_http_client_example.c in function http_perform_as_stream_reader(void)

Hope this helps someone saving some minutes and rather using them for a cup of coffee and a chat.
Bernd