HTTP requests (GET, POST etc.) on ESP32

[andrew_p]

I'm playing with code examples provided on esp-idf github and fee like missing something.

As of now I'm mostly interested in POST requests, there are no examples on github, but I was able to modify https example and successfully sent POST message to external server (created a data record on a server), but got a problem: ESP32 doesn't process a response from the server, it just freezes and no log messages fired. And if I comment the part of the code, which processes the response, no issues appeared. Can somebody help with this or just provide a correct example how to send POST requests?

This is my POST request:

Code: Select all

POST https://xxxx.xxxx.com/xx/xxxxx HTTP/1.1
Host: xxxx.xxxx.com
Accept: application/json
Content-Type: application/json
Authorization: Basic aW90LxxxxxaW90LmRhdGE3MDc=
Cache-Control: no-cache
Content-Length: 30

{"u_payload": "ping from iot"}

I was not able to correctly modify http code example to send POST request - ESP32 log says socket send success, but there is errno=128 and no information comes to the external server. Any ideas on how to deal with that?

Code: Select all

I (7605) IoT data: ... socket send success
I (10805) IoT data: ... done reading from socket. Last read return=0 errno=128

[ESP_krzychb]

Hi andrew_p,

Do you mean http as in the subject or https requests as per example you used?

If http, then I have used 03_http_request example and adopted it to perform GET requests to:

• * https://openweathermap.org/
  * https://thingspeak.com/

Respective esp-idf components are as follows:

• * weather
  * thingspeak
  * http (common to both above)

In both cases I do not have any problems processing responses from serer.

To deal with issue you report, I would first check your application working with some common servers like above.

Krzysztof

[JanJansen47]

I'm also new (one day) on the ESP32 platform. The examples I tried run, except the 03_http_request.

If I compare the response as seen by the ESP32 with the response as seen by the Firefox webtools, ESP32 adds a 0 at the end and tells me there is errno 128. On top of this there is a long delay (seconds) before the process starts repeating.

I tried to debug but cannot find a hint for the long delay. I don't doubt my webpage I'm using this one for a couple a years (Synology NAS).

I will try the suggestions and see if this helps. (PS I started using Google.com with the same issues)

[ESP_krzychb]

I tried to debug but cannot find a hint for the long delay.

Hi JanJansen47,

I have also seen server waiting couple of dozen seconds before closing connection.

The GET request of example code 03_http_request:

Code: Select all

static const char *REQUEST = "GET " WEB_URL " HTTP/1.1\n"
    "Host: "WEB_SERVER"\n"
    "User-Agent: esp-idf/1.0 esp32\n"
    "\n";

does not contain Connection: close, so your server may be keeping connection open until timeout.

For my purposes I have modified the code as follows:

Code: Select all

static const char *REQUEST = "GET " WEB_URL " HTTP/1.1\n"
    "Host: "WEB_SERVER"\n"
    "Connection: close\n"
    "User-Agent: esp-idf/1.0 esp32\n"
    "\n";

This way the server (at least these I have listed in my previous post) close connection immediately after sending reply.

[andrew_p]

Hi andrew_p,

Do you mean http as in the subject or https requests as per example you used?

I was referring to the 'official' esp-idf examples 04_https_request and 03_http_request. They both do GET request and I don't have any problems with GET, I'm trying to do POST and cannot get it work.

When I do POST with 04_https_request and specify

Code: Select all

Content-Length: 30

It does come through and create a record with json payload on a remote server, but then ESP32 freezes after parsing a server response. I mean I see the response from the server in a terminal, but then it freezes. In other case - If I do not specify Content-Length then the request will be sent without payload, but data record will be created on a remote server anyway and ESP32 continues to run the program (after parsing the server response) without any problem.

UPDATE:
Connection: close header DOES the trick!!! 04_https_request example now works fine with POST request and ESP32 parses the server response without any problems.

[andrew_p]

OK, we have a success with a POST request in 04_https_request example. But I don't need SSL handshake with the remote server, since I'm using basic authentication, so I don't need that part and want to send POST as simple as possible. So I'm playing with 03_http_request and it doesn't work.. honestly, I don't even know if it's possible to send POST via websockets or not.

Here is my log

Code: Select all

I (199640) IoT data: Starting again!
I (199640) IoT data: Connected to AP
I (199650) IoT data: DNS lookup succeeded. IP=103.23.XX.XXX
I (199650) IoT data: ... allocated socket
I (199910) IoT data: ... connected
I (199910) IoT data: sending request=[POST https://XXXX.XXXX.com/XX/XXX/XXXXXXXX HTTP/1.1
Host: XXXXXX.XXXXXX.com
Accept: application/json
Content-Type: application/json
Authorization: Basic XXXXXXXXXXXXXXXXXXdGE3MDc=
Cache-Control: no-cache
Connection: close
Content-Length: 30

{"u_payload": "ping from iot"}]
I (199940) IoT data: ... socket send success
HTTP/1.0 302 Found
Location: https://XXXXXX.XXXXXX.comhttps://XXXX.XXXX.com/XX/XXX/XXXXXXXX
Server: BigIP
Connection: close
Content-Length: 0

I (200210) IoT data: ... done reading from socket. Last read return=0 errno=128

Again, just trying to find the simplest way to send POST request..

[kolban]

An HTTP POST request occurs when your ESP32 forms a TCP/IP connection to a remote server and sends a well formed HTTP protocol header request. If all goes well, the partner will return an HTTP protocol response message. At a high level, your algorithm will be:

1. Create a socket connection to the partner.
2. Send an HTTP request message.
3. Receive an HTTP response message.
4. Close the socket.

Beyond this we have notions such as TLS/SSL for encryption of the transmitted data, various settings of HTTP headers, encodings of content, ability to keep HTTP connections open for subsequent requests ... etc etc etc.

HTTP is a protocol (not a very complex one ... but a protocol none-the-less) and has to be properly constructed for transmission and properly parsed on response. One can perform this logic in ones own code or can leverage pre-existing libraries that have been pre-written to make some of these common tasks simpler.

It may help in your posts to explain which areas of the above are the ones that are causing trouble.

[andrew_p]

Neil, sorry for any confusion, I haven't worked with a low-level technologies stack since I graduated awhile ago.

Here is a detailed explanation of what I'm trying to achieve:

1. I have a remote server, which provides RESTful API over https with basic authentication (user name/password).
2. I want ESP32 to send a POST request to my remote server. I DO want to send a request over encrypted (https) connection, but there is no such thing as a certificate-based authentication on my remote server, it's just a username-password authentication.

I understand that there might be multiple ways how to achieve that (external libraries etc), I'm just trying to find the most simplest option, like 04_https_request example but maybe even simpler (if it's possible).

P.S. Neil, your book is awesome!

[kolban]

Thanks for the kind words.

The mbedtls library exists "outside" of the ESP-IDF. I found the following article quite helpful:

https://tls.mbed.org/kb/how-to/mbedtls-tutorial

Checking my notes, I seem to have a sample found on p135 of the latest release (2017-01-01) that might form the basis for a discussion.

[JanJansen47]

Thanks for your swift response. Indeed when the client closes the connection it's not waiting for the server. And the response is as expected. Never noticed the difference.