Tutorial : Sending file to a remote HTTPS server with the ESP32 using the Arduino core.

[Lucasj]

The aim of this tutorial is to demonstrate how to transfer multiple files from a SD Card to a remote server using https with ESP32 and the Arduino core.
The files can be pretty big, so for this reason we do not want to use the dynamic memory, instead we will cut the files in multiple smaller parts.
For this tutorial we need a server with https connection and a certificate. For security reasons, it is important not to use the auto-certificate.
This is a high-level description of what the code does:
-first we list all the files present on the SD card and send their names to the server.
-the server check which files are already there and then send back a list of the missing files.
-then the ESP32 sends the missing files to the server. The files are cut into chunks a 1,000 chars.
-Two different requests can be used to either create a new file on the server or to append to an existing file.
That it! the idea was to write code that would scale to a large number of files on the SD card and to large files. I also wanted to have requests that were not too big (to not overload the server if several clients try to access the server at the same time) and not too small to limit to number to request that are sent to the server.
Here is the code, hope this is useful.


[Codebox]//Global declaration:

//Connect and upload files
#include <WiFi.h> // This library is for the https connection.

#include <WiFiClientSecure.h> //This library is for the wifi connection.

#include "SD_MMC.h" // This library is for use SD card.


//************************HTTPS Send csv files****************
WiFiClientSecure client_secu;
const char* server_name = " www.mywebsite.com "; // Server URL
const char* web_page_post = " https://www.mywebsite.com/tmp_try_upload/";
const char* host = "mywebsite.com";

/* You need to store the https certificate as a const char*. You should here use the certificate of the server you want to use to transfer the files to. */
const char* ca_cert = \
"-----BEGIN CERTIFICATE----- \n" \
"MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/\n" \
"MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\n" \
"DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow\n" \
"SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT\n" \
"GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC\n" \
"AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF\n" \
"q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8\n" \
"SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0\n" \
"Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA\n" \
"a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj\n" \
"/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T\n" \
"AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG\n" \
"CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv\n" \
"bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k\n" \
"c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw\n" \
"VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC\n" \
"ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz\n" \
"MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu\n" \
"Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF\n" \
"AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo\n" \
"uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/\n" \
"wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu\n" \
"X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG\n" \
"PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6\n" \
"KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==\n" \
"-----END CERTIFICATE----- \n";

/*In the Setup() we initial the https connection and the Sd_card. The SD CARd is
connected to the ESP32 using the MMC connection.
We then start the wifi connection and we call the sub to send the files. */
void setup()
{
Serial.begin(115200);

client_secu.setCACert(ca_cert);

if (!SD_MMC.begin()) {
Serial.println("Card reader mount failed");
return;
}

Serial.println("Connection...");
connection_access_point();

Serial.println("Send files...");
send_all_csv_file();

}


void send_all_csv_file() { //***Full code for verifying and uploading every root file.


char list_files[1010] = ""; // is where we put the file names for verification and sending it to the server.
char name[50]; // is create the initial request to the server
char name2[50]; // is to read the answer from the server
int i, j, n;
int nb_files;
bool end = false;
// To send all the files in the root of SD card we need to do two Files variable:
File dir;
File file;

// We open the directory with:
dir = SD_MMC.open("/");
Serial.println("Begin send files...");
file = dir.openNextFile();
while (!end) {
file = dir.openNextFile(); //We take the files one by one:
if (!file) { //if file is null, we finished the read directory
//We don’t break for fish the check and send the last files.
end = true;
name[0] = '\0';
}

if (end || strlen(file.name()) < 50) {//we verify the name of file can put in variable “name”
if (!end) strcpy(name, file.name());
/***We push the names of files in the array***/
if (strlen(list_files) + strlen(name) < 1000) {
sprintf(list_files, "%s%s;", list_files, name);
}
/**************************************************/

/***If array is full***/
else {
if (end && list_files[0] == '\0') {
//If list_files is clear and the “end” is true, we finish the while.
break;
}

// we call file_in_server and get list of files are not in server.
sprintf(list_files, "%s", file_in_server(list_files));

nb_files = 0;
//we get the number of file return
for (i = 0; i < 1000 && list_files != '\0'; i++) {
if (list_files == ';') nb_files++;
}
//we send one by one the files
for (n = 0, i = 0, j = 0; i < 1000 && list_files != '\0'; i++) {
//While we don’t have all name of file put character by character name files in “name2”
if (list_files != ';') {
name2[j] = list_files;
j++;
}
else {
//if we finish read the name we call send_csv_files_to_server
Serial.println("Send" + String(n) + "/" + nb_files + " : " + String(name2));
name2[j] = '\0';
send_csv_files_to_server(name2);
n++;
j = 0;
}
}
list_files[0] = '\0';
sprintf(list_files, "%s%s;", list_files, name);
if (end) break;

}
}
else {
Serial.println("Error : Name file" + String(entry.name()) + " > 50");
}
file.close();
}

dir.close();
Serial.println("End send files");
}



/*For checking if a file is on the server, we take the names of files and we create a block of 1,000 characters. Then we ask the server to check whether those files are present already. The server answers with the names of files that are not present on the server. */


char* file_in_server(char list_files[1010]) {
char buf[100]; //get connection error

int verif = 0; //verification for begin push character in list_files array
char request[1020]; //array for request to the server
sprintf(request, "exist=%s", list_files); //prepare the post request

int i;
/*For this we need to indicate the server name. The second parameter is the port of the server. The default port is 443 (Https port). */

if (!client_secu.connect(server_name, 443)) {
/*If we want to see if there were any error during the connection process, one can use:*/

client_secu.lastError(buf, 100);

if (strcmp(buf, "UNKNOWN ERROR CODE (0050)") == 0) {
return file_in_server(request);
}
else {

Serial.print("Server connection fail : ");
Serial.println(buf);
}
}
else {
/* We use the post method to send the data. An example of the request can be: */

client_secu.println("POST " + String(web_page_post) + " HTTP/1.1");
client_secu.println("Host: " + String(host));
client_secu.println("User-Agent: Arduino/1.0");
client_secu.println("Connection: close");
client_secu.println("Content-Type: application/x-www-form-urlencoded");
client_secu.print("Content-Length: ");
client_secu.println(strlen(request));
client_secu.println("");
client_secu.println(request);

/* Then, we wait for the answer.*/

while (!client_secu.available()) {
delay(50);
}
/* And then we get an answer which begins with “R” and ends with “##;”:*/

i = 0;

list_files[0] = '\0';
while (client_secu.available()) {
char c = client_secu.read();
if (c == 'R') {
verif = 1;
}
else if (c == '#') {
c = client_secu.read();
if (c == '#') {
c = client_secu.read();
if (c == ';') {

list_files = '\0';
break;
}
else {
list_files = '#';
i++;
list_files = '#';
i++;
list_files = c;
i++;
}
}
else {
list_files = '#';
i++;
list_files[i] = c;
i++;
}
}
else if (verif == 1) {
list_files[i] = c;
i++;
}

}

}
//Serial.println(txtLine1);
client_secu.stop();
return list_files;

}

/*The goal of this part is to send a file and never allocate the memory dynamically. The idea is to create a smalls requests with 1000 characters.*/

void send_csv_files_to_server(char tmp_name[50]) {
char name[50];
char path[50];
int i;
int type_request;
char request[1000];
int default_dim;
File file;

//define path and name
if (tmp_name[0] == '/') {
strcpy(path, tmp_name);
for (i = 1; path[i] != '\0'; i++) {
name[i - 1] = path[i];
}
name[i - 1] = '\0';
}
else {
strcpy(name, tmp_name);
sprintf(path, "/%s", name);
}

//verify if file exits in SD card

if (!SD_MMC.exists(path)) {
Serial.println("ERROR: The required file does not exist.");
return;
}
file = SD_MMC.open(path);

//if file is not open
if (!file) {
Serial.println("Failed to open file for reading");
return;
}

//Verify the Wifi connection
for (i = 0; WiFi.status() != WL_CONNECTED && i < 20; i++) {
delay(100);
}
if (i == 20) {
Serial.println("Wifi connection fail");
}


/*We prepare the request. In this request we have:
-Id
-password
- type of request:
-new (file)
-add (file)
-name of file
- the data (max 1000 character)
The idea is to read the file and when the array data is full to send the request. After the first request we change the type of to request to “ADD” in order to signal to the server that this is not a new file.*/

sprintf(request, "i=%s&p=%s&Sta=", "LJ", "L1989tes"); //id and pass server
type_request = strlen(request); //save place where request type
sprintf(request, "%sNew&Nam=%s&t=", request, name); //Type request, name file
default_dim = strlen(request); //size of begin request

/while it’s not end of file/
while (file.available()) {

/***************Connection to server******************/
if (!client_secu.connect(server_name, 443)) {
char buf[100];
client_secu.lastError(buf, 100);
if (strcmp(buf, "UNKNOWN ERROR CODE (0050)") == 0) {
while (!client_secu.connect(server_name, 443)) { Serial.print("!"); } //Server time error, it’s just necessary to retry.
}
else {
Serial.println("Server connection fail");
}
}
/*****************************************************/
else {
//push 1000 character, in request
for (i = default_dim; file.available() && i < 999; i++) {
request[i] = file.read();
}
request[i] = '\0';

//Send request to server
client_secu.println("POST " + String(web_page_post) + " HTTP/1.1");
client_secu.println("Host: " + String(host));
client_secu.println("User-Agent: Arduino/1.0");
client_secu.println("Connection: close");
client_secu.println("Content-Type: application/x-www-form-urlencoded");
client_secu.print("Content-Length: ");
client_secu.println(strlen(request));
client_secu.println("");
client_secu.println(request);

//Wait return from server
while (!client_secu.available()) {
delay(50);
}
/* if data is available then receive and print to Terminal */
while (client_secu.available()) {
char c = client_secu.read();
}
if (i == 999) {
// if I == 999 we prepare the next request with type request = ADD
request[type_request + 1] = 'A';
request[type_request + 2] = 'D';
request[type_request + 3] = 'D';
}
/* if the server disconnected, stop the client */
client_secu.stop();

}
}

}
[/Codebox]
Code running on the PHP server

I chose to use php for the server. The code is simple, we have a first part to verify the files in the server and the second part to write files.
Here I chose to not present the system for the authentication with hash and databased to simplify the presentation.
In this example we chose to accept just the csv files. It is important to not accept every file for the security of your server.

The first part:

To create the answer, if the file is not in the server, we push the name of this file in the result and we print after trying every file.
To understand where the answer starts and where it finishes, we use “R” in the beginning and “##;” in the end.
[Codebox]// If in the Post method we have the “exist” parameter, we explode the value of this parameter.
if (!empty($_POST['exist'])) {
$files = explode(';', $_POST['exist']); // we explode character array when we have ‘;’
$result = "R"; // the begin of answer
foreach($files as $key = > $value) {

$tmp = explode('/', $value); // cut ‘/’ because it’s not valid for my system files.

if (count($tmp) == 2) {
if (!file_exists($tmp[1])) { // Try if the file exist
$result = $result.''.$value.';'; // If not exists we push the result for the answer.
}
}
}
$final_result = $result."##;"; // the end of answer
echo $final_result;
}[/Codebox]

The second part

In this second part, we simply check the id and the password. It is not a good way of doing it from a security standpoint. We do it as is to simplify the code, but one should instead use a database to the hashed password so that they are not hardcoded.
After we read the request from ESP32 to see whether it is a new file or if we add data to an existing file. We get the name and verify if it is a csv file.
[Codebox]if (!empty($_POST['i']) && !empty($_POST['p'])) { //if in post request we have ‘I’ and ‘p’.
if ($_POST['i'] == "LJ" && $_POST['p'] == "L1989tes") {//check id and password
if (!empty($_POST['Sta'])) { // if in post request we have ‘Sta’
if (!empty($_POST['Nam'])) { // if in post request we have ‘Nam’
$name_f = $_POST['Nam'];
$test_csv = explode('.', $name_f);
$test_format = "false";
if (count($test_csv) == 2) {// check if it is a csv file
if ($test_csv[1] == "csv")$test_format = "true";
}
}
if ($test_format) {
if ($_POST['Sta'] == "New") {
file_put_contents($name_f, ''); // if Sta = new
}
if ($_POST['Sta'] == "ADD") {
// if Sta = add
file_put_contents($name_f, '', FILE_APPEND);
}
if (!empty($_POST['t'])) {
//we get data in the file
$texte = file_get_contents($name_f);

//we add the new data
$texte = $texte.$_POST['t'];

//And we push in the file
file_put_contents($name_f, $texte);

}
}
}
}
[/Codebox]