[SOLVED] Help with sending .txt file over HTTP formatted as a .json file

[zydizz]

Hi everyone,
Hello! This is my first time in this forum. I am developing a device that takes measurements from a microphone and stores these measurements into a txt file, logging each second. The end goal is then sending the contents of this .txt file in a .JSON object over HTTP to a server to make datalogging and analysing easy. A minute of calculation data creates a file about 20kb.<br/>


The json object will look like this:

Code: [Select all] [Expand/Collapse]

1. {
2.     "DeviceId": "123456",
3.     "Data": "THE MULTI-LINE DATA READ FROM THE SD CARD AS A VARIABLE"
4. }

GeSHi © Codebox Plus Extension

The steps in my head was like such;

• Read the file into a character array with the size of the file

• Format said array into a string like json

• Send it over http

But the main issue occurs at the second part, I am not able to format the array without any heap overflows. Will address my previous ideas after code snippet. What may be the ideal form of sending the data in json form? Using streams and just the WifiClient?
(Libraries used: WiFi.h, HTTPClient.h, ArduinoJson.h, FS.h, SD.h)

Code: [Select all] [Expand/Collapse]

1. void readPost(fs::FS &fs, const char * path, const char * httpUrl) {
2.   Serial.printf("Reading file: %s\n", path);
3.  
4.   File file = fs.open(path);
5.   if(!file) {
6.     Serial.println("Failed to open file for reading");
7.     return;
8.   }
9.  
10.   const size_t fLen = file.size();
11.   char * fileBuffer = new char[fLen + 1];
12.  
13.   while(file.available()){
14.     fileBuffer[file.position() - 1] = file.read();
15.   }
16.   file.close();
17.  
18.   char * fileData = new char[fLen + 33];
19.   sprintf(fileData, "{\"DeviceId\":\"%s\",\"Data\":\"%s\"}", deviceId, fileBuffer);
20.   delete [] fileBuffer;
21.  
22.   if ((WiFi.status() == WL_CONNECTED)) { //Check the current connection status
23.     HTTPClient http;
24.     http.begin(String(httpUrl)); //Specify the URL and certificate
25.     http.addHeader("Content-Type", "application/json");
26.     int httpResponseCode = http.POST((uint8_t *)fileData, sizeof(fileData));
27.     Serial.print("HTTP Response code: ");
28.     Serial.println(httpResponseCode);
29.     http.end(); //Free the resources
30.   }
31.   delete [] fileData;
32. }

GeSHi © Codebox Plus Extension

[lbernstone]

I don't think the HTTPClient will do what you want. Rather than shoehorning that library to do something out of scope, just use WiFiClientSecure. Read in 4096 bytes at a time from the file, and write it out to the socket as you parse that. This will give you full control over what the output looks like.

[zydizz]

just use WiFiClientSecure. Read in 4096 bytes at a time from the file, and write it out to the socket as you parse that. This will give you full control over what the output looks like.

Thanks, I thought so. Can you provide an example how should i do that? Will newline operators be problematic or will they stil stay in the https request?

Is my current "read to a array then send" method problematic? Should I send the data in 4096 byte partitions as I read? Because I want the whole data in a single json formatted https request.

[lbernstone]

Sorry, I am not going to do your homework for you, mostly because I think your http flow is not correct. The more compliant way to send data like this to a server would be as an argument to the request string. That means your URL would be something like "https://server.example.com/postdata?json={data1:.....}". This is much easier to construct, will work just fine with HTTPClient, and is guaranteed to be accepted by any server.

If you don't have control of the backend, then you will have to fit to what it expects. You need to use a tool like curl/wireshark or postman to generate a transaction that works properly with your server before you try to build it on the esp32. Once you know what the request should look like, it should be fairly easy to use the WiFiClientSecure example (https://github.com/espressif/arduino-es ... Secure.ino) as a template to generate the necessary text. As far as newlines go, it is standard http. A single CR is a line break, two CRs is an EOM marker. WiFiClient::write posts just the string, WiFiClient::print puts a CR at the end.

[zydizz]

Thank you so much for your detailed answer.

Sorry, I am not going to do your homework for you, mostly because I think your http flow is not correct.

Sorry if i made it sound that way, that was not my intention. Since I'm fairly new to http (literally a week) I'm kind of lost between many different methods. And I'm still trying to grasp the concepts of the http request headers, body, etc...

If you don't have control of the backend, then you will have to fit to what it expects. You need to use a tool like curl/wireshark or postman to generate a transaction that works properly with your server before you try to build it on the esp32.

The server expects a string in the form of a serialized json file as such. I will try generating a transaction in postman, thanks.

Code: [Select all] [Expand/Collapse]

1.  
2. {"DeviceId": "123456", "Data": "DATA FROM THE SD CARD"}

GeSHi © Codebox Plus Extension

I am now trying to work with WifiClient.h and at least I got a response from the server that my request was invalid. Good so far. Will a 20kb array be too big for client.print() to send in a single line? Because I cannot see any errors/etc. on the server AND code side.

[lbernstone]

WiFiClient::print will only be limited by your available memory. That said, as your initial issue was about memory consumption, why not keep it as low as possible. json doesn't care about whitespace (you can use \n if you really need it for readability), and you can let the IP stack do the buffering for you, so just printf directly to the client object (and the socket underneath) as you parse the file. Read in 4096 bytes (the size of a "disk" sector), and have the client.printf the string in a for loop. That way you will never be using much more than that 4k, and can keep everything in stack space, and you minimize the processing.

[zydizz]

@lbernstone,

Before trying to implement the sequential read I tried to port my code over to WifiClient.h but after 3 days of trying everything there still is no content in the body of the request. Can you see where my issue lies (im highly suspecting the \n's and \r's)?

(The first part hasn't changed at all, just the part after "WiFiClient client")

Code: [Select all] [Expand/Collapse]

1.   WiFiClient client;
2.   Serial.println("\nStarting connection to server...");
3.   if (!client.connect(httpHost, 80))
4.     Serial.println("Connection failed!");
5.   else {
6.     Serial.println("Connected to server!");
7.     client.println();
8.     client.println();
9.     client.println("POST " + String(httpPath) + " HTTP/1.1");
10.  
11.     // Make a HTTP request
12.     client.println("Host: " + String(httpHost));
13.     client.println("Connection: close");
14.     client.println("Content-Type: application/json");
15.     client.println();
16.  
17.     client.print("{\"DeviceId\":\"");
18.     client.print(deviceId);
19.     client.print("\",\"Data\":\"");
20.     client.print("lorem ipsum");
21.     client.print("\"}");
22.     client.print("\r\n");
23.  
24.     while (client.connected()) {
25.       String line = client.readStringUntil('\n');
26.       if (line == "\r") {
27.         Serial.println("headers received");
28.         break;
29.       }
30.     }
31.  
32.     // if there are incoming bytes available
33.     // from the server, read them and print them:
34.     while (client.available()) {
35.       char c = client.read();
36.       Serial.write(c);
37.     }
38.     Serial.println();
39.   }
40.   client.stop();
41.   delete [] fileBuffer;
42. }

GeSHi © Codebox Plus Extension

Esp32 serial readout:

esp.png (21.58 KiB) Viewed 4418 times

Request recieved by Postman's mock server (internally created post requests are successfully transferred):

postman.png (23.9 KiB) Viewed 4418 times

[zydizz]

ISSUE IS SOLVED, the request just needed a Content-Length header haha. A 1 week brain melting session finally resulted in something.

Thanks a lot for your advices @lbernstone, hope you have a nice day.