ESP32 to SD-Card Datalogging Problem

[hudlbergaJoe]

Hi dear community,

i found no similar solved problem so i decided to create this new topic.
My intention is to log data from sensors (temperature, pressure, humidity, etc.) with a ESP32 to a micro-SD-Card.

I use the Adafruit HUZZAH32 (https://www.adafruit.com/product/3405) and a breakout board for the sd card (https://www.adafruit.com/product/254) with the Arduino IDE.

At the moment i am not able to log data for a longer time respectively log a lot of data.
The latest code compiles without problems and communicates a String via SPI to the sd-card and writes it into a .csv. It is just a modified code that is used in other tutorials (for example: https://iotdesignpro.com/projects/loggi ... sing-esp32).

Code: Select all

#include "FS.h"
#include "SD.h"
#include "SPI.h"

float Temperature;
float Humidity;
String formattedDate;
String dayStamp;
String timeStamp;
String dataMessage;

unsigned int count = 0;
#define SD_CS 33

void setup() {
  Serial.begin(115200);

  pinMode(13, INPUT_PULLUP);	//manuell switch

  SD.begin(SD_CS);  
 
  uint8_t cardType = SD.cardType();

  File file = SD.open("/data.csv");
  if(!file) {
    writeFile(SD, "/data.csv", "Date; Time; Temperature; Humidity \r\n");
  }
  file.close();
}

void loop() {
    if(digitalRead(13) == HIGH){
    count = count + 1;
    Humidity = 44.2;
    Temperature = 24.2;
    logSDCard();
	}
}


void logSDCard() {
  dataMessage = String(count) + ";" + String(Temperature) + ";" + String(Humidity);
  for(int i=0;i <= 20; i++){
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity);
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity)
  dataMessage = dataMessage + ";" + String(Temperature) + ";" + String(Humidity) + "\r\n";
  }
  Serial.print("Save data: ");
  Serial.println(dataMessage);
  appendFile(SD, "/data.csv", dataMessage.c_str());
}

// Write to the SD card (DON'T MODIFY THIS FUNCTION)
void writeFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Writing file: %s\n", path);
  File file = fs.open(path, FILE_WRITE);
  if(!file) {
    Serial.println("Failed to open file for writing");
    return;
  }
  if(file.print(message)) {
    Serial.println("File written");
  } else {
    Serial.println("Write failed");
  }
  file.close();
}
// Append data to the SD card (DON'T MODIFY THIS FUNCTION)
void appendFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Appending to file: %s\n", path);
  File file = fs.open(path, FILE_APPEND);
  if(!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if(file.print(message)) {
    Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

PIN #13 is connected with a switch, so i can start and end the data transfer manually.

After 60 seconds there are 471 KB written, after additional 20 seconds the file is 642 KB in size, after the next 20 seconds 816 KB, and after 20 seconds more it has just 110 KB (old data has been overwriten).

Differently, when i let the script run for 80 seconds without interruption the data on the sd card gets corrupted and i cant even read it on the computer.

But no matter what i do, i am not able to log data for a longer time (lower frequency saving) or log a lot of data for a short time (i've never been able to reach 1 MB). There is no other device connected to the ESP32, just the sd breakoutboard.

Does anyone know such a problem or has some ideas how to solve this?
Many thanks for your time.

[Radu79]

Did you try a different SD card? I had a SD that was very, very slow on the ESP32 (though it worked normally on the PC).

[hudlbergaJoe]

I tried two cards: one with 16GB and Class 10, one with 8 GB and Class 4. Same problem.

[Radu79]

Then you have two solutions:
1. Cache the data in RAM, and write it in larger batches (this is also good for the life of the SD card). This doesn't work if you use deep sleep though.
2. If you need sleep, then just write each line in a different file rather than append it at the end of one file.

[Radu79]

Also keep in mind that SD cards are NOT SSDs. If you keep writing small bits of data fast they will die on you. Their wear leveling is not as good as the one on SSDs, and they are made cheap. That's why SD cards are a fraction of the cost of a similar size SSD.