[Video] - ESP32 and working with Lib CURL

[kolban]

The popular Lib CURL library is an open source implementation providing easier access to a variety of protocols including HTTP, HTTPS, mail, FTP and more. This video illustrates a recipe for building libcurl for the ESP32 and also samples of C++ classes that further encapsulate some function.

https://www.youtube.com/watch?v=tp4OC_kGMgU

[gregstewart90]

Kolban, Thanks for showing us how to get curl working. I'm working on a project that requires me to use https. I'm connecting to a server with a self signed cert. I use

Code: Select all

curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);

to get passed any warnings. The code below works on my mac. I tried three different ways to make it work on the ESP32. If I use xTaskCreatePinnedToCore(&login_test, "login_test", 10000, NULL, 5, NULL, 0) I get a watch dog timer not feed error. If I just call login_test directly I get a stack overflow. If I call login_test from another place in my code, it runs but doesn't do anything. The stack overflow and watch dog timer errors occur after "Before Perform" is printed.

For testing, I disabled https on the server, and it worked if I commented out the curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L); lines.

Code: Select all

#include "freertos/FreeRTOS.h"
#include "esp_wifi.h"
#include "esp_system.h"
#include "esp_event.h"
#include "esp_event_loop.h"
#include "nvs_flash.h"
#include "driver/gpio.h"
#include "esp_log.h"
#include "curl/curl.h"
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <time.h>
#include "curl.h"

#define SSID "SSID"
#define PASSWORD "PASSWORD"

static char tag[] = "curl";


static void print_cookies(CURL *curl)
{
	CURLcode res;
	struct curl_slist *cookies;
	struct curl_slist *nc;
	int i;

	printf("Cookies, curl knows:\n");
	res = curl_easy_getinfo(curl, CURLINFO_COOKIELIST, &cookies);
	if(res != CURLE_OK) {
		fprintf(stderr, "Curl curl_easy_getinfo failed: %s\n",
				curl_easy_strerror(res));
		exit(1);
	}
	nc = cookies, i = 1;
	while(nc) {
		printf("[%d]: %s\n", i, nc->data);
		nc = nc->next;
		i++;
	}
	if(i == 1) {
		printf("(none)\n");
	}
	curl_slist_free_all(cookies);
}

void login_test() {
	printf("Begining Login Test**\n");

	CURLcode ret;
	CURL *hnd;

	hnd = curl_easy_init();
	curl_easy_setopt(hnd, CURLOPT_URL, "https://192.168.1.20/");
	curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
	curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.50.3");
	curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
	curl_easy_setopt(hnd, CURLOPT_COOKIEFILE, "");
	curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);


	printf("Before Perform\n");
	ret = curl_easy_perform(hnd);
	printf("After Perform\n");
	print_cookies(hnd);


	struct curl_httppost *post1;
	struct curl_httppost *postend;
	struct curl_slist *slist1;

	post1 = NULL;
	postend = NULL;
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "uri",
			CURLFORM_COPYCONTENTS, "/",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "username",
			CURLFORM_COPYCONTENTS, "admin",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "password",
			CURLFORM_COPYCONTENTS, "passpass",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "country",
			CURLFORM_COPYCONTENTS, "56",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "ui_language",
			CURLFORM_COPYCONTENTS, "en_US",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "lang_changed",
			CURLFORM_COPYCONTENTS, "no",
			CURLFORM_END);
	slist1 = NULL;
	slist1 = curl_slist_append(slist1, "Expect:");

	curl_easy_setopt(hnd, CURLOPT_URL, "https://192.168.1.20/login.cgi");
	curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
	curl_easy_setopt(hnd, CURLOPT_HTTPPOST, post1);
	curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.50.3");
	curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
	curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
	curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

	ret = curl_easy_perform(hnd);

	curl_formfree(post1);
	post1 = NULL;
	curl_slist_free_all(slist1);
	slist1 = NULL;


	post1 = NULL;
	postend = NULL;
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "essid",
			CURLFORM_COPYCONTENTS, "SSID 123",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "security",
			CURLFORM_COPYCONTENTS, "wpa2aes",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "wpa_key",
			CURLFORM_COPYCONTENTS, "password",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "change",
			CURLFORM_COPYCONTENTS, "Change",
			CURLFORM_END);
	curl_formadd(&post1, &postend,
			CURLFORM_COPYNAME, "action",
			CURLFORM_COPYCONTENTS, "change",
			CURLFORM_END);
	slist1 = NULL;
	slist1 = curl_slist_append(slist1, "Expect:");


	curl_easy_setopt(hnd, CURLOPT_URL, "https://192.168.1.20/link.cgi");
	curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
	curl_easy_setopt(hnd, CURLOPT_HTTPPOST, post1);
	curl_easy_setopt(hnd, CURLOPT_REFERER, "https://192.168.1.20/link.cgi");
	curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.50.3");
	curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);
	curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
	curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

	ret = curl_easy_perform(hnd);

	curl_easy_setopt(hnd, CURLOPT_URL, "https://192.168.1.20/signal.cgi");
	curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
	curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.50.3");
	curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
	curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
	curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

	ret = curl_easy_perform(hnd);



	curl_easy_cleanup(hnd);
	hnd = NULL;
	curl_formfree(post1);
	post1 = NULL;
	curl_slist_free_all(slist1);
	slist1 = NULL;

	printf("Completed All\n");

}

esp_err_t wifi_event_handler(void *ctx, system_event_t *event)
{
	if (event->event_id == SYSTEM_EVENT_STA_GOT_IP) {
		ESP_LOGI(tag, "Got an IP ... ready to go!");
		
		//Calling login_test() here causes a stack overflow
		login_test();
		
		//Calling xTaskCreatePinnedToCore causes the watch dog timer to be called
//		xTaskCreatePinnedToCore(&login_test, "login_test", 10000, NULL, 5, NULL, 0);
		return ESP_OK;
	}
	return ESP_OK;
}

int app_main(void)
{
	nvs_flash_init();
	system_init();
	tcpip_adapter_init();
	ESP_ERROR_CHECK( esp_event_loop_init(wifi_event_handler, NULL) );
	wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
	ESP_ERROR_CHECK( esp_wifi_init(&cfg) );
	ESP_ERROR_CHECK( esp_wifi_set_storage(WIFI_STORAGE_RAM) );
	ESP_ERROR_CHECK( esp_wifi_set_mode(WIFI_MODE_STA) );
	wifi_config_t sta_config = {
			.sta = {
					.ssid = SSID,
					.password = PASSWORD,
					.bssid_set = 0
			}
	};
	ESP_ERROR_CHECK( esp_wifi_set_config(WIFI_IF_STA, &sta_config) );
	ESP_ERROR_CHECK( esp_wifi_start() );
	ESP_ERROR_CHECK( esp_wifi_connect() );
	ESP_LOGI(tag, "Registering test vfs");

	return 0;
}

Any ideas?

[kolban]

Howdy. When I worked on the libcurl recipe to get it working on the ESP32 my focus was on how to get it compiled cleanly. Unfortunately that doesn't mean that I am any kind of a libcurl internals or usage expert. Once it compiled, I ran some sniff tests including ensuring I could make simple HTTP and HTTPS calls. Those, I am glad to say, worked.

You describe two primary errors. Let's look at each in turn. The first was a call to your function in the WIFI event handler resulted in a stack overflow. That one we can explain quite easily.

You are compiling your application using the Espressif ESP-IDF framework. If you run "make menuconfig" you will find the settings for your environment. If we then navigate to "Component config -> ESP32-specific" we will find a setting called "Event loop task stack size". This is the size of the stack allocated to the thread (task) that is handling event processing. We will see that, by default, it is only 2048 bytes. That's pretty small. When you run libcurl in the event handler, it runs out of stack. However, in my opinion, the solution is NOT to increase the stack size, but rather to do what you were planning and spawn a new task to do the work outside of the event handler. Especially if it is going to take some time to do the work. As such, running out of stack calling libcurl directly in your event handler is not unexpected and probably not a great idea.

The next puzzle is the watch dog timer error. When a task runs, it is anticipated that either the task will end within a relatively short period of time "OR" the task will block and return control back to the ESP32 "OR" the task will 'claim' that it knows what it is doing and inform the ESP32 environment that it is still alive and working fine. This last item is known as "feeding the watchdog". See http://esp-idf.readthedocs.io/en/latest ... /wdts.html.

If you have no opportunity to feed the watchdog because you are calling "black box code" that just takes a long time to run, then you can disable the watchdog (again see the article). You can also switch off watch dog processing as a whole from "make menuconfig":

"Component config -> ESP32-specific"

and unchecking "Task watchdog".

[gregstewart90]

Kolban, Thanks for your help! Everything was very clear. I was able to disable the watch dog timer, for now, to debug my problem. Once I did, the program just sat there indefinitely. I added the verbose flag to give me more to work with and the following was the output.

Code: Select all

* timeout on name lookup is not supported
*   Trying 192.168.1.20...
* TCP_NODELAY set
* Connected to 192.168.1.20 (192.168.1.20) port 443 (#0)
* Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed
* mbedTLS: Connecting to 192.168.1.20:443
* mbedTLS: Set min SSL version to TLS 1.0

I ran the same code on my computer, and I got

* TLS 1.2 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
* Server certificate: UBNT-68:72:51:60:4D:4F

Any ideas?

[kolban]

Unfortunately, I'm no great hero on libcurl. Now if you can create a minimal example that I can run on my machine that would, for example, target an external (Internet) HTTPS server I'll take a look.

The error messages states that libcurl is trying to read a certificates file. We would need to study libcurl to see if it "has" to be read from a file or can we specify a blob of data that is a pre-loaded set of certificates? Failing that, we would have to create a Virtual File System that maps "/etc" to some file system where we could place the data that would allow a read. It is possible that the default implementation of libcurl is performing file I/O ... while this is legal on ESP-IDF, it becomes our responsibility to provide a file system mapping such that the file I/O calls succeed.

[loboris]

@gregstewart90

Have a look at viewtopic.php?f=18&t=1833, maybe you could find it interesting.

[gregstewart90]

Kolban, I asked about the issue on the libcurl email forum, and I was told I could use a new flag "CURLOPT_SSL_CTX_FUNCTION". I found this example online on how to use it. It is only available in the latest release of the library (7.54.0). I've updated the libcurl library and I'm on 7.54.1-DEV. I also update the lib/curl_config.h file to the latest you put on github two days ago. The library compiles, but the code does not. Below I have the function that does not compile.

Code: Select all

static CURLcode sslctx_function(CURL *curl, void *sslctx, void *parm)
{
	X509_STORE *store;
	X509 *cert=NULL;
	BIO *bio;
	char *mypem = /* example CA cert PEM - shortened */
			"-----BEGIN CERTIFICATE-----\n"
			"MIIHPTCCBSWgAwIBAgIBADANBgkqhkiG9w0BAQQFADB5MRAwDgYDVQQKEwdSb290\n"
			"IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNB\n"
			"IENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRA\n"
			"Y2FjZXJ0Lm9yZzAeFw0wMzAzMzAxMjI5NDlaFw0zMzAzMjkxMjI5NDlaMHkxEDAO\n"
			"GCSNe9FINSkYQKyTYOGWhlC0elnYjyELn8+CkcY7v2vcB5G5l1YjqrZslMZIBjzk\n"
			"zk6q5PYvCdxTby78dOs6Y5nCpqyJvKeyRKANihDjbPIky/qbn3BHLt4Ui9SyIAmW\n"
			"omTxJBzcoTWcFbLUvFUufQb1nA5V9FrWk9p2rSVzTMVD\n"
			"-----END CERTIFICATE-----\n";
	/* get a BIO */
	bio=BIO_new_mem_buf(mypem, -1);
	/* use it to read the PEM formatted certificate from memory into an
	 * X509 structure that SSL can use
	 */
	PEM_read_bio_X509(bio, &cert, 0, NULL);
	if(cert == NULL)
		printf("PEM_read_bio_X509 failed...\n");

	/* get a pointer to the X509 certificate store (which may be empty) */
	store=SSL_CTX_get_cert_store((SSL_CTX *)sslctx);

	/* add our certificate to this store */
	if(X509_STORE_add_cert(store, cert)==0)
		printf("error adding certificate\n");

	/* decrease reference counts */
	X509_free(cert);
	BIO_free(bio);

	/* all set to go */
	return CURLE_OK;
}

The errors are the following

Code: Select all

main/network/curl.c:59:2: error: unknown type name 'X509_STORE'
  X509_STORE *store;
main/network/curl.c:60:2: error: unknown type name 'X509'
  X509 *cert=NULL;
main/network/curl.c:61:2: error: unknown type name 'BIO'
  BIO *bio;
main/network/curl.c:73:6: error: implicit declaration of function 'BIO_new_mem_buf' [-Werror=implicit-function-declaration]
  bio=BIO_new_mem_buf(mypem, -1);
main/network/curl.c:73:5: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
  bio=BIO_new_mem_buf(mypem, -1);
main/network/curl.c:77:2: error: implicit declaration of function 'PEM_read_bio_X509' [-Werror=implicit-function-declaration]
  PEM_read_bio_X509(bio, &cert, 0, NULL);
main/network/curl.c:82:8: error: implicit declaration of function 'SSL_CTX_get_cert_store' [-Werror=implicit-function-declaration]
  store=SSL_CTX_get_cert_store((SSL_CTX *)sslctx);
  main/network/curl.c:82:32: error: 'SSL_CTX' undeclared (first use in this function)
  store=SSL_CTX_get_cert_store((SSL_CTX *)sslctx);
  main/network/curl.c:82:41: error: expected expression before ')' token
  store=SSL_CTX_get_cert_store((SSL_CTX *)sslctx);
  main/network/curl.c:85:5: error: implicit declaration of function 'X509_STORE_add_cert' [-Werror=implicit-function-declaration]
  if(X509_STORE_add_cert(store, cert)==0)
  main/network/curl.c:89:2: error: implicit declaration of function 'X509_free' [-Werror=implicit-function-declaration]
  X509_free(cert);
main/network/curl.c:90:2: error: implicit declaration of function 'BIO_free' [-Werror=implicit-function-declaration]
  BIO_free(bio);
  
  

I'm not sure if this is a libcurl issue or an esp32 issue.

Also, when you compile the library for the first time, do you get lots of warning? I get tons similar to the one below. I also always have to manually make the build/curl/lib/vtls and build/curl/lib/vauth directories for it to finish compiling.
components/mbedtls/port/include/mbedtls/esp_config.h:785:0: warning: "MBEDTLS_FS_IO" redefined
#define MBEDTLS_FS_IO

[kolban]

The issue you are describing doesn't appear to be either an ESP32 or Libcurl issue. It seems you are referencing OpenSSL data types ... I'd look to see what (if anything) you are doing with OpenSSL. I'm a big fan of lifting code that others have written ... but I hesitate to do that if, after reading it, I don't understand it nor can understand it after studying it hard.

[gregstewart90]

I'm still stuck on this problem. What I have found is that I do not need the cert to make it work. I can connect to other servers and get the same warning about not having the cert in the /etc/ssl/certs. The issue arrises in the next part

Code: Select all

mbedTLS: Connecting to 192.168.1.20:443

The system sits there indefinitely and never connects. I enabled a more verbose mbedTLS and I got:

Code: Select all

* mbedTLS: Connecting to 192.168.1.20:443
* mbedTLS: Set min SSL version to TLS 1.0
W (6432) mbedtls: ssl_tls.c:4425 x509_verify_cert() returned -9984 (-0x2700)

When connecting to other servers, I get

Code: Select all

I (181900) mbedtls: ssl_tls.c:6306 => handshake

The code works if I run it from my mac and connect to the server. It works on the esp32 if I connect to a different server requiring https. It only doesn't work when I run it from the esp32 and try to connect to this server. The libcurl community said it sounds like a mbedtls issue. They suggested I connect to the server using mbedtls and see if I can get that to work. I'm not sure how to do that. I would like to expose this server to the internet for others to try to connect to it, but I am behind a firewall that doesn't allow me to open ports to the outside.

[eugenev13]

Hi Neil,

I enjoyed your cURL - Eclipse integration video. Many thanks for taking the time to make it. I liked the presentation style and the visuals.

Unfortunately, I could not integrate lib cURL into Eclipse properly.

When I follow the video and #include <curl/curl.h> in the main project file, compiler succeeds. But the moment I start using curls API calls, I begin getting errors...
If I create a brand new project and try to link it to curl, I keep getting an error curl/curl.h - No such file or directory.

I am also confused about the files to be moved into the installation directory of cURL. On the video you clearly show that three files are to be moved. But on the today's 'esp32-snippets -> curl -> build_files' directory there are only two files. Do I need the third file to complete the integration?