curl/libcurl networking unreliable? Could it be more modular?

[mgleason_3]

It may be easiest to show an example from a video - with apologies to Neil Kolban who made the video and is an awesome asset to the ESP32 community...

https://www.youtube.com/watch?v=tp4OC_kGMgU.

For those who don't have access to the video, Neil's trying to demonstrate using libcurl to make a RESTFUL api call from the ESP32 to a backend server via HTTPS. It fails the first time but works the second.

Jump to 11:40. He gives up on the first try around 12:09, re-compiles, runs it again and it does work the second time at 12:30.

The problem is I'm seeing the same behavior all the time. It may fail 3-times in a row. Each time I hit the reset button (I'm not re-compiling), and reboot and it fails... Then maybe the forth time it'll work... Putting it in a re-try loop doesn't seem to help - once it's failed, it always seems to need a reboot.

Same thing with HTTP (rather than HTTPS). So, the question is:

Is It possible to reliably make a RESTFUL HTTP or HTTPS calls and get a response on the ESP32?

A small rant as well - I wonder if it's possible to provide examples which are more like libraries? e.g. The wifi examples have multiple inter-related functions and structs and a bunch of stuff in main. Personally, I'd be happy if I could just call a single function (let's call it "get_connected("wifi")") that would make sure wifi is initialized, connected to the AP, has an IP, etc and returns SUCCESS if everything worked or FAILURE otherwise.

[ESP_Angus]

Hi mgleason,

The problem is I'm seeing the same behavior all the time. It may fail 3-times in a row. Each time I hit the reset button (I'm not re-compiling), and reboot and it fails... Then maybe the forth time it'll work... Putting it in a re-try loop doesn't seem to help - once it's failed, it always seems to need a reboot.

Are you seeing this behaviour with the curl library or with other code? We're not aware of any widespread problems with ESP32 networking in general (as per the title of your post.) But it's possible there's something in curl which is timing out wrong, or getting stuck in a bad state.

A small rant as well - I wonder if it's possible to provide examples which are more like libraries? e.g. The wifi examples have multiple inter-related functions and structs and a bunch of stuff in main. Personally, I'd be happy if I could just call a single function (let's call it "get_connected("wifi")") that would make sure wifi is initialized, connected to the AP, has an IP, etc and returns SUCCESS if everything worked or FAILURE otherwise.

Some convenience functions like this are planned.

There is a trade-off here, which is that as well as demonstrating basic functions we want the examples to be "fledged out" enough that a person can copy one to a totally new directory (outside of IDF) and start a new project by editing it. For most non-trivial projects, the extra boilerplate in "main" (event callbacks for state changes, WiFi configuration tweaks, etc.) will be necessary - even thought they're not really used in the basic "Hello World" style examples.

But, that said, you are totally right that there is some room to bundle more of that up into some convenience functions (at least for most examples.)

[WiFive]

"fledged out"

Fully-fledged or fleshed out maybe. Or is it an Aussie thing?

[ESP_Angus]

"fledged out"

Fully-fledged or fleshed out maybe. Or is it an Aussie thing?

Oops, yes I meant fleshed out. I blame early morning lack of coffee.

[mgleason_3]

Are you seeing this behaviour with the curl library or with other code?

Good question. My recollection was that I had the same issue running the http example (esp's version without curl). However, I just updated the esp-idf and re-ran the http example and it does work consistently as far as I can tell. This version prints out more error information than I recalled from when I ran it a while ago. From time-to-time I get:

.[0;31mE (15115) example: DNS lookup failed err=202 res=0x0.[0m
.[0;32mI (16115) example: Connected to AP.[0m
.[0;31mE (23015) example: DNS lookup failed err=202 res=0x0.[0m
.[0;32mI (24015) example: Connected to AP.[0m

Possibly curl isn't re-trying when theres a DNS lookup failure?

There is a trade-off here, which is that as well as demonstrating basic functions we want the examples ...

Yea, it would be great to have a Fully-fledged and simple to use rest library that you could call to make get, post, etc. calls that would just go off and do it (or return an error). I really don't need to touch the code required for every step that's needed to get there (connected to the AP, got an IP, DNS lookup succeeded).

For most non-trivial projects, the extra boilerplate in "main"...will be necessary

Not sure I totally agree, but I may be mis-interpreting. I created simple files to pull the wifi stuff out of main. With it I can:

Code: Select all

if (get_connected() == EC_SUCCESS){
    xTaskCreatePinnedToCore(&testcurl, "testcurl", 10000, NULL, 5, NULL, 0);
}

This is the code:

wifi_utils.h

Code: Select all

#ifndef _wifi_utils_h
#define _wifi_utils_h

#include "esp_wifi.h"
#include "freertos/FreeRTOS.h"
#include "freertos/event_groups.h"
#include "freertos/task.h"
#include "esp_event_loop.h"
#include "esp_log.h"


/* log tag */
static char tag[] = "wifi";

typedef enum {
    EC_SUCCESS,
    EC_FAILURE,
    EC_OVERFLOW,
    EC_UNDERFLOW,
    EC_INCONVERTIBLE
} result_error_code;

void initialise_wifi(void);
result_error_code get_connected ();
static esp_err_t event_handler(void *ctx, system_event_t *event);

#endif

wifi_utils.c

Code: Select all

#include "wifi_utils.h"

/* The examples use simple WiFi configuration that you can set via
   'make menuconfig'.

   If you'd rather not, just change the below entries to strings with
   the config you want - ie #define EXAMPLE_WIFI_SSID "mywifissid"
*/
#define EXAMPLE_WIFI_SSID CONFIG_WIFI_SSID
#define EXAMPLE_WIFI_PASS CONFIG_WIFI_PASSWORD

/* FreeRTOS event group to signal when we are connected & ready to make a request */
static EventGroupHandle_t wifi_event_group;

/* The event group allows multiple bits for each event,
   but we only care about one event - are we connected
   to the AP with an IP? */
const int CONNECTED_BIT = BIT0;


static esp_err_t event_handler(void *ctx, system_event_t *event)
{
    switch(event->event_id) {
    case SYSTEM_EVENT_STA_START:
        esp_wifi_connect();
        break;
    case SYSTEM_EVENT_STA_GOT_IP:
        xEventGroupSetBits(wifi_event_group, CONNECTED_BIT);
        break;
    case SYSTEM_EVENT_STA_DISCONNECTED:
        /* This is a workaround as ESP32 WiFi libs don't currently
           auto-reassociate. */
        esp_wifi_connect();
        xEventGroupClearBits(wifi_event_group, CONNECTED_BIT);
        break;
    default:
        break;
    }
    return ESP_OK;
}


void initialise_wifi(void)
{
    tcpip_adapter_init();
    wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK(esp_event_loop_init(event_handler, NULL));
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
    wifi_config_t wifi_config = {
        .sta = {
            .ssid = EXAMPLE_WIFI_SSID,
            .password = EXAMPLE_WIFI_PASS,
        },
    };
    ESP_LOGI(tag, "Setting WiFi configuration SSID %s...", wifi_config.sta.ssid);
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
    ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());
}


result_error_code get_connected () {
    EventBits_t evg_bits;

    evg_bits = xEventGroupGetBits(wifi_event_group);

    if (!(evg_bits & CONNECTED_BIT)) {
        ESP_LOGI(tag, "get_connected: not connected, trying esp_wifi_connect()");
        esp_wifi_connect();
    }

    evg_bits = xEventGroupWaitBits(wifi_event_group, CONNECTED_BIT, false, true, portMAX_DELAY);
    
    if (evg_bits & CONNECTED_BIT) {
		ESP_LOGI(tag, "get_connected: connected!");
		return EC_SUCCESS;
	} else {
        ESP_LOGI(tag, "get_connected: ERROR: Couldn't connect.  Cancelling push notification");
        return EC_FAILURE;
    }
}

Unfortunately, by updating esp-idf, I've broken the curl compile again on my Mac. What a pain! But, on the other hand, maybe going with the native IDF is a better strategy at this point.

Or, maybe I should switch over to Mongoose OS? I wish there was something that would provide guidance for all the development choices (esp-idf, Arduino, Mongoose) and why you would want to choose one vs another. My goal is to create a product I can sell to end users (vs one I'm creating as a hobby for myself). So, it needs to be end-user configurable, etc... Any thoughts?

[kolban]

One of the areas I've been tinkering with is the use of C++ classes to encapsulate ESP32 and ESP-IDF functions. My thinking here goes along the lines:

* C++ is fully supported by ESP-IDF
* C++ classes can be used by C programmers without them having to "learn" how to write new classes themselves

One of the classes that has been tinkered with is a RESTClient which encapsulates REST APIs making them much easier to use. See http://www.neilkolban.com/esp32/docs/cp ... lient.html.

[mgleason_3]

@kolban - thanks for the comments! The concept of using C++ classes to encapsulate the ESP-IDF sounds promising.

But, a few things kept me from using your RESTClient:

• Is there a version which doesn't use lib curl? The two I've seen appear to be based on libcurl rather than the base ESP-IDF - at least if I look at https://github.com/nkolban/esp32-snippe ... Client.cpp or https://github.com/nkolban/esp32-snippe ... Client.cpp.
• As I mentioned at the top, libcurl has been unable to work reliably - maybe that's changed or there's a timeout I need to lengthen?
• Compiling libcurl on my Mac breaks whenever there's an ESP-IDF update. Unfortunately, I don't have the expertise to fix the problems and relied on your help in the past (thank you!) - but this just isn't sustainable.
• My lack of knowledge of c++/fear of c++. Though as you point out, that shouldn't really hold me back since I can simply call it from C.

[kolban]

A few months ago I needed to make REST client calls from an ESP32 and looked around at the possibilities.

1) RAW sockets and parse/construct HTTP myself? - No ... why re-invent the wheel?
2) Packages C/C++ libraries ...
a) libcurl
b) Mongoose Networking

I studied both. The libcurl seems much richer and I liked the API structure. The port of libcurl (for me) wasn't onerous and seemed to pretty much work out of the box. So I then stuck with libcurl.

With basic REST under my belt, I wen't off on a C++ kick and needed to then call REST from C++ so "simply" wrapped libcurl. The caller of the REST methods doesn't know that the RESTClient class is implemented on top of libcurl so as long as the class fulfills its contract, all is well (for me).

As for reliability ... for me ... so far so good but I have "NOT" stressed it heavily. What I'd suggest is that we (as a community) pick a library (eg. libcurl) as the majority community choice and focus on one. The fact that I have libcurl running "dis-incentives" me from studying others. However, I'll go with the majority. If there is an alternative HTTP library that the majority want to use that isn't libcurl, I'll switch over to that.

As for ESP-IDF changes breaking libcurl ... not aware of any issues but if there are, I'll be happy to spend time trying to study them and write them up for the benefit of all. I happen to be a "windows" man however, when it comes to ESP32, I do all my work on Ubuntu running under VirtualBox.

I treat C++ as an opportunity rather than a hindrance. C++ has been in the top 5 programming languages by almost any metric for a good long while and will probably stick around for a while longer. As such, skills developed in C++ won't do any harm. It may not be a fashionable language but it seems to do the job.

[loboris]

I've made an example of using libcurl with various protocols.
viewtopic.php?f=18&t=1833

[mgleason_3]

OK, created a Docker image based on Ununtu 16.04 so I could remove any unknowns and hopefully get away from the compilation problems. It's brand-new install with the latest git clones of esp-idf, curl, etc...

Unfortunately, compiling produces the same errors as in this issue: https://github.com/nkolban/esp32-snippets/issues/13

Code: Select all

/esp/project/components/curl/lib/curl_ntlm_wb.c: In function 'Curl_ntlm_wb_cleanup':
/esp/project/components/curl/lib/curl_ntlm_wb.c:91:9: error: implicit declaration of function 'kill' [-Werror=implicit-function-declaration]
         kill(conn->ntlm_auth_hlpr_pid, SIGTERM);
         ^
/esp/project/components/curl/lib/curl_ntlm_wb.c: In function 'ntlm_wb_init':
/esp/project/components/curl/lib/curl_ntlm_wb.c:187:6: error: implicit declaration of function 'socketpair' [-Werror=implicit-function-declaration]
   if(socketpair(AF_UNIX, SOCK_STREAM, 0, sockfds)) {
      ^
/esp/project/components/curl/lib/curl_ntlm_wb.c:187:17: error: 'AF_UNIX' undeclared (first use in this function)
   if(socketpair(AF_UNIX, SOCK_STREAM, 0, sockfds)) {
                 ^
/esp/project/components/curl/lib/curl_ntlm_wb.c:187:17: note: each undeclared identifier is reported only once for each function it appears in
cc1: some warnings being treated as errors
/esp/esp-idf/make/component_wrapper.mk:176: recipe for target 'lib/curl_ntlm_wb.o' failed
make[1]: *** [lib/curl_ntlm_wb.o] Error 1
/esp/esp-idf/make/project.mk:386: recipe for target 'curl-build' failed
make: *** [curl-build] Error 2