esp_tls_conn_read is adding extra characters to the data I receive

[HenryM]

I'm POST'ing data to a server and getting a response from it. Here's what I expect to get:

Code: Select all

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Server: Kestrel
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=0c1a6972157af12ca575fb7a0ce73c735c8fb0151334f0283fcc042a338a2cb6;Path=/;HttpOnly;Domain=mydomain.azurewebsites.net
Date: Thu, 04 Oct 2018 21:56:30 GMT
Connection: close

{"id":9,"deviceID":12,"addressID":0,"userID":0,"deviceSerialNumber":0,"firmwareVersion":0,"lastFillUp":"0001-01-01T00:00:00","currentLevel":0,"status":"Checked Device","firstConnected":"2018-10-04T16:03:33.7168214","lastConnected":"2018-10-04T21:56:30.0917821+00:00"}

But here's what I'm actually getting:

Code: Select all

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Server: Kestrel
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=0c1a6972157af12ca575fb7a0ce73c735c8fb0151334f0283fcc042a338a2cb6;Path=/;HttpOnly;Domain=mydomain.azurewebsites.net
Date: Thu, 04 Oct 2018 21:56:30 GMT
Connection: close

10b
{"id":9,"deviceID":12,"addressID":0,"userID":0,"deviceSerialNumber":0,"firmwareVersion":0,"lastFillUp":"0001-01-01T00:00:00","currentLevel":0,"status":"Checked Device","firstConnected":"2018-10-04T16:03:33.7168214","lastConnected":"2018-10-04T21:56:30.0917821+00:00"}
0

As far as I can tell, esp_tls_conn_read is stuffing 10b and 0 into the data stream. It also seems to appear that 10b is, in hex, the number of bytes of the following chunk of received data. I've sometimes received the data broken into smaller pieces, and that hex number always indicates the size of the following chunk.

How do I get it to not stuff my data with those byte lengths?

Here's my full code for this function:

Code: Select all

static int EthernetPost( char* url, char* content, char* rsp, int max_rsp_len )
{
    int ret = -1;
    static char header[512];
    int bytes_read = 0;

    sprintf(header, "POST %s HTTP/1.1\r\n", url);
    sprintf(header, "%sHost: %s\r\n", header, WEB_SERVER);
    strcat(header, "Connection: close\r\n");
    strcat(header, "Content-Type: application/x-www-form-urlencoded\r\n");
    sprintf(header, "%sContent-Length: %u\r\n\r\n", header, strlen(content));

    assert(strlen(header) < sizeof(header));

    esp_tls_cfg_t cfg = {
        .cacert_pem_buf  = server_root_cert_pem_start,
        .cacert_pem_bytes = server_root_cert_pem_end - server_root_cert_pem_start,
    };

    struct esp_tls *tls = esp_tls_conn_http_new(url, &cfg);

    if (tls)
    {
        bool error = false;
        size_t written_bytes = 0;

        while ((written_bytes < strlen(header)) && !error)
        {
            ret = esp_tls_conn_write(tls,
                                     header + written_bytes,
                                     strlen(header) - written_bytes);
            if (ret >= 0)
            {
                ESP_LOGI(TAG, "%d bytes written", ret);
                written_bytes += ret;
            }
            else if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE)
            {
                ESP_LOGE(TAG, "esp_tls_conn_write  returned 0x%x", ret);
                error = true;
            }
        }

        written_bytes = 0;
        while ((written_bytes < strlen(content)) && !error)
        {
            ret = esp_tls_conn_write(tls,
                                     content + written_bytes,
                                     strlen(content) - written_bytes);
            if (ret >= 0)
            {
                ESP_LOGI(TAG, "%d bytes written", ret);
                written_bytes += ret;
            }
            else if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE)
            {
                ESP_LOGE(TAG, "esp_tls_conn_write  returned 0x%x", ret);
                error = true;
            }
        }

        // Get the response
        bool done = false;
        while (!done && !error && (bytes_read < max_rsp_len))
        {
            ret = esp_tls_conn_read(tls, (rsp + bytes_read), (max_rsp_len - bytes_read));

            if (ret == MBEDTLS_ERR_SSL_WANT_WRITE  || ret == MBEDTLS_ERR_SSL_WANT_READ)
                continue;

            if (ret < 0)
            {
                ESP_LOGE(TAG, "esp_tls_conn_read returned -0x%x", -ret);
                error = true;
            }
            else if (ret == 0)
            {
                ESP_LOGI(TAG, "connection closed, bytes read: %d", bytes_read);
                done = true;
            }
            else
            {
                bytes_read += ret;
            }
        }
    }

    esp_tls_conn_delete(tls);

    if (ret < 0)
        return ret;
    else
        return bytes_read;
}

[WiFive]

Code: Select all

Transfer-Encoding: chunked

https://developer.mozilla.org/en-US/doc ... r-Encoding

That data is coming from the server

[ESP_Angus]

Hi Henry,

You may find the esp_http_client component helpful. It supports chunked encoding:
https://docs.espressif.com/projects/esp ... lient.html

Angus

[HenryM]

https://developer.mozilla.org/en-US/doc ... r-Encoding

That data is coming from the server

That would explain it, thanks!

Hi Henry,

You may find the esp_http_client component helpful. It supports chunked encoding:
https://docs.espressif.com/projects/esp ... lient.html

Angus

Excellent, I'll take a look at it.

[HenryM]

I opted to remove the chunk lengths from the response myself. I think using the http client may very well be a better option, but I'm very time constrained at the moment. If I encountered any more issues with these server communications, then I'll look into the http client. But for now, this is working.

In case someone else finds this post and wants to see what I did...

Code: Select all

/** ****************************************************************************
 * @brief The HTTP buffer data may be chunked.  If chunked, the chunk will
 * begin with a hex string followed by a \r\n.  At the end of the chunk may
 * be another \r\n.  This function removes the hex string as well as the two
 * instances of \r\n.
 *
 * @param[in] pChunk   Pointer to the beginning of the chunk
 *
 * @returns -1 if there was an error, else the number of bytes that were
 *          removed during the dechunking process.
 ******************************************************************************/
static int EthernetDechunk( char* pChunk )
{
    int bytes_removed = -1;
    int chunk_size_length = 0;

    if (pChunk)
    {
        while (((*pChunk != '\n') && (*pChunk != '\r')) && *pChunk)
        {
            pChunk++;
            chunk_size_length++;
        }

        if ((*pChunk == '\r') || (*pChunk == '\n'))
        {
            while ((*pChunk == '\r') || (*pChunk == '\n'))
            {
                pChunk++;
                chunk_size_length++;
            }

            // Do a memmove on the string, length is +1 to ensure we get a termninating NULL
            memmove(pChunk - chunk_size_length, pChunk, strlen(pChunk) + 1);
            bytes_removed = chunk_size_length;

            // If there is a final \r\n, remove it
            pChunk += (strlen(pChunk) - 2);
            if (((*pChunk == '\r') && (*(pChunk + 1) == '\n')) ||
                ((*pChunk == '\n') && (*(pChunk + 1) == '\r')))
            {
                *pChunk++ = 0;
                *pChunk = 0;
                bytes_removed += 2;
            }
        }
    }

    return bytes_removed;
}

/** ****************************************************************************
 * @brief Performs an HTTPS POST operation to a web server.
 *
 * @param[in] url       String representing the URL to which the data is to be
 *                      posted.  Example:  https://subdomain.server.com/Path
 * @param[in] content   URL encoded content to provide to the server
 *                      Example: deviceId=12&name=Bob%20Douglas
 * @param[out] rsp      Pointer to a buffer in which the response is to be
 *                      stored.  The HTTP header that is sent to the server
 *                      is also built and stored in this buffer temporarily,
 *                      so it must be of sufficient size to hold the header.
 *                      The header size will change depending on the URL.
 * @param[in] max_rsp_len   The length of the rsp buffer
 *
 * @returns <0 if there was an error, else the number of bytes that were
 *          received into the rsp buffer.
 ******************************************************************************/
int EthernetPost( char* url, char* content, char* rsp, int max_rsp_len )
{
    int ret = -1;
    int bytes_read = 0;
    char* header = rsp;

    sprintf(header, "POST %s HTTP/1.1\r\n", url);
    sprintf(header, "%sHost: %s\r\n", header, WEB_SERVER);
    strcat(header, "Connection: close\r\n");
    strcat(header, "Content-Type: application/x-www-form-urlencoded\r\n");
    sprintf(header, "%sContent-Length: %u\r\n\r\n", header, strlen(content));

    assert(strlen(header) < max_rsp_len);

    esp_tls_cfg_t cfg = {
        .cacert_pem_buf  = server_root_cert_pem_start,
        .cacert_pem_bytes = server_root_cert_pem_end - server_root_cert_pem_start,
    };

    struct esp_tls *tls = esp_tls_conn_http_new(url, &cfg);

    if (tls)
    {
        bool error = false;
        size_t written_bytes = 0;

        while ((written_bytes < strlen(header)) && !error)
        {
            ret = esp_tls_conn_write(tls,
                                     header + written_bytes,
                                     strlen(header) - written_bytes);
            if (ret >= 0)
            {
                written_bytes += ret;
            }
            else if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE)
            {
                ESP_LOGE(TAG, "esp_tls_conn_write  returned 0x%x", ret);
                error = true;
            }
        }

        memset( rsp, 0, max_rsp_len );

        written_bytes = 0;
        while ((written_bytes < strlen(content)) && !error)
        {
            ret = esp_tls_conn_write(tls,
                                     content + written_bytes,
                                     strlen(content) - written_bytes);
            if (ret >= 0)
            {
                written_bytes += ret;
            }
            else if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE)
            {
                ESP_LOGE(TAG, "esp_tls_conn_write  returned 0x%x", ret);
                error = true;
            }
        }

        // Get the response
        bool done = false;
        bool chunked = false;
        bool header_checked = false;
        int chunks = 0;
        while (!done && !error && (bytes_read < max_rsp_len))
        {
            ret = esp_tls_conn_read(tls, (rsp + bytes_read), (max_rsp_len - bytes_read));

            if (ret == MBEDTLS_ERR_SSL_WANT_WRITE  || ret == MBEDTLS_ERR_SSL_WANT_READ)
                continue;

            if (ret < 0)
            {
                ESP_LOGE(TAG, "esp_tls_conn_read returned -0x%x", -ret);
                error = true;
            }
            else if (ret == 0)
            {
                ESP_LOGD(TAG, "connection closed, bytes read: %d", bytes_read);
                done = true;
            }
            else
            {
                chunks++;
                if (!chunked && !header_checked)
                {
                    // If the data comes back from the server in chunks, the first response will contain
                    // the header at minimum and it will be followed up  with a chunk of data.  We'll check
                    // the header to see if it is chunked.  If so, subsequent reads, but not this one, will
                    // need to extract the chunk size data from the response buffer.  This response will need
                    // to search for the "\r\n\r\n" which signifies the beginning of the first chunk and
                    // extract the chunk length indicator.
                    chunked = (strstr(rsp, "Transfer-Encoding: chunked") != 0);
                    header_checked = true;
                    if (chunked)
                    {
                        char* pChunk = strstr(rsp, "\r\n\r\n");
                        if (pChunk)
                            pChunk += 4;
                        int bytes_removed = EthernetDechunk( pChunk );
                        if (bytes_removed < 0)
                        {
                            error = true;
                            ESP_LOGE(TAG, "EthernetDechunk returned %u", bytes_removed);
                        }
                        else
                            ret -= bytes_removed;
                    }
                }
                else if (chunked)
                {
                    // bytes_read has not been incremented yet, so (rsp + bytes_read) currently points to
                    // data that indicates the chunk size.  The chunk size consists of a hex number followed
                    // by a \r\n.  We need to find the data after the \r\n and move it to (rsp + bytes_read)
                    // then decrement bytes_read by the length of the data that was removed
                    char* pChunk = (rsp + bytes_read);
                    int bytes_removed = EthernetDechunk( pChunk );
                    if (bytes_removed < 0)
                    {
                        error = true;
                        ESP_LOGE(TAG, "EthernetDechunk returned %u", bytes_removed);
                    }
                    else
                        ret -= bytes_removed;
                }
                bytes_read += ret;
            }
        }

        ESP_LOGI(TAG, "POST completed - done:%u error:%u bytes_read:%d max_rsp_len:%d chunks:%d",
            done, error, bytes_read, max_rsp_len, chunks);
    }

    esp_tls_conn_delete(tls);

    if (ret < 0)
        return ret;
    else
        return bytes_read;
}