esp_http_client - How to POST Chunked Data?

[mikea101]

I need to be able to POST large amounts of data to a server. That is, the size of the data exceeds the ESP32 memory size (to big to malloc)

However, I cannot find any esp_http_client documentation or examples to do this as a chunked transfer.

[boarchuz]

Use the stream functions of esp_http_client:
https://docs.espressif.com/projects/esp ... ttp-stream

Needs this:

Code: Select all

esp_http_client_set_header(client, "Transfer-Encoding", "chunked");

eg.

Code: Select all

esp_http_client_write("5", 1); // length
esp_http_client_write("\r\n", 2);
esp_http_client_write("Hello", 5); // data
esp_http_client_write("\r\n", 2);
esp_http_client_write("6", 1); // length
esp_http_client_write("\r\n", 2);
esp_http_client_write(" World!", 7);  // data
esp_http_client_write("\r\n", 2);
esp_http_client_write("0", 1);  // end
esp_http_client_write("\r\n", 2);
esp_http_client_write("\r\n", 2);

[mikea101]

Thanks boarchuz, this is the way to do it. I have this working and there are some details/corrections I can add:

You do NOT need to do "esp_http_client_set_header(client, "Transfer-Encoding", "chunked");"

Instead, you do this:

Code: Select all

esp_http_client_open(client, -1);

When you open with a write_len of -1, it will set the "Transfer-Encoding" header to "chunked" automatically. It will also force the http method to POST. (see http_client_prepare_first_line() in the source code https://github.com/espressif/esp-idf/bl ... p_client.c)

The resulting code:

Code: Select all

	esp_http_client_config_t config = {
		.url = "http://httpbin.org/post",
		.method = HTTP_METHOD_POST, // This is NOT required. write_len < 0 will force POST anyway
	};

	esp_http_client_handle_t client = esp_http_client_init(&config);

	esp_err_t err = esp_http_client_open(client, -1); // write_len=-1 sets header "Transfer-Encoding: chunked" and method to POST
	if (err != ESP_OK) {
		ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));
		return;
	}

	// Post some data
	esp_http_client_write("5", 1); // length
	esp_http_client_write("\r\n", 2);
	esp_http_client_write("Hello", 5); // data
	esp_http_client_write("\r\n", 2);
	esp_http_client_write("6", 1); // length
	esp_http_client_write("\r\n", 2);
	esp_http_client_write(" World!", 7);  // data
	esp_http_client_write("\r\n", 2);
	esp_http_client_write("0", 1);  // end
	esp_http_client_write("\r\n", 2);
	esp_http_client_write("\r\n", 2);

	// After the POST is complete, you can examine the response as required using:
	//	esp_http_client_fetch_headers() for content length
	//	esp_http_client_get_status_code()
	//	esp_http_client_read()

	esp_http_client_close(client);
	esp_http_client_cleanup(client);

Note to Espressif: the documentation for esp_http_client_open() should explain that a negative value of write_len is how chunked POST is to be performed.

Use the stream functions of esp_http_client:
https://docs.espressif.com/projects/esp ... ttp-stream

Needs this:

Code: Select all

esp_http_client_set_header(client, "Transfer-Encoding", "chunked");

eg.
<snip>

[Snedig]

I wanted to post a reply for anyone else doiung this as I had a lot of trouble with sending chunked encoding, so hopefully it will help someone else out:)

The issue that was tripping me up was that my transfers all seemed to fail, and it turns out I was sending the length as decimal numbers - they need to be hexadecimal.

Also of note, if doing mixed transfers with the same handle, remember to delete any used headers before exiting the function, the transfer-encoding header seems to need to be deleted manually before using the handle for something else.

Here is some example code that I'm using - to send the end sequence, use transmit_chunk(""):

Code: Select all

int transmit_chunk(char *string) {
  int written_length = 0;
  int total_length = 0;
  char length_string[11] = "";

  if (strlen(string) > 0) {
    bzero(length_string, sizeof(length_string));
    snprintf(length_string, sizeof(length_string), "%X", strlen(string));

    written_length = esp_http_client_write(http_client, length_string, strlen(length_string));

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    written_length = esp_http_client_write(http_client, "\r\n", 2);

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    ESP_LOGD(UART_TAG, "Sent: %s", length_string);

    written_length = esp_http_client_write(http_client, string, strlen(string));

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    written_length = esp_http_client_write(http_client, "\r\n", 2);

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    ESP_LOGD(UART_TAG, "Sent: %s", string);
  } else {
    written_length = esp_http_client_write(http_client, "0", 1); // end

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    esp_http_client_write(http_client, "\r\n", 2);

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    esp_http_client_write(http_client, "\r\n", 2);

    if (written_length > 0) {
      total_length += written_length;
    } else {
      return -1;
    }

    ESP_LOGD(UART_TAG, "Sent http end chunk");
  }
  return total_length;
}