ESP32 Forum

Posted: **Thu Sep 24, 2020 8:10 am**

Hello, I'm using the ESP32 to log some Sensordata with relatively high throughput (3*16bit at 500Hz), but i have the problem, that the SD Card Speed is not constant over the whole time and I lose some data every now and then. The fluctuation occurs every 5-10 minutes and holds for a few seconds. during that time, the writing speed is extremely low.
The card is connected via VSPI (the sensor is on HSPI) and it is a 16Gb SanDisk Ultra. Cables are the shortest jumper cables i had. Devboard is an ESP32-WROOM-32 from AZDelivery.

What could I do to prevent the problem?

example sketch that shows that also showed the problem:

Code: [Select all] [Expand/Collapse]

#include "MPU9250.h"
#include "SPI.h"
 
#include "FS.h"
#include "SD.h"
 
#include "timer.h"
 
#define FILE_READ       "r"
#define FILE_WRITE      "w"
#define FILE_APPEND     "a"
 
SPIClass SPI_SD(VSPI);
const int CS_SD = 5;
 
int tsm = 16;
int t = 0;
int ts;
 
Timer timer_measure;
File file;
 
String datnam = "/temp.txt";
 
void measure(){
  file = SD.open(datnam,FILE_APPEND);
  ts = millis()-t;
  t = millis();
  if (ts>3*tsm){
    Serial.print(t);
    Serial.print("    ");
    Serial.println(ts);
  }
  for(int i;i<85;i++){
    String line = String(t) + "\t"
                  + String(12345) + "\t"
                  + String(12345) + "\t"
                  + String(12345);
    file.println(line);
  }
  file.close();
}
 
void setup(){
  Serial.begin(115200);
  Serial.println("Setup SD...");
  if(!SD.begin(CS_SD,SPI_SD)){
    Serial.println("Card Mount Failed");
    delay(1000);
    ESP.restart();
  }
  file = SD.open(datnam,FILE_WRITE);
  file.println("millis\ta\tb\tc");
  file.close();
  
  timer_measure.setInterval(tsm);
  timer_measure.setCallback(measure);
  timer_measure.start();
  t = millis();
}
 
void loop(){
  timer_measure.update();
}

GeSHi © Codebox Plus Extension

Posted: **Thu Sep 24, 2020 9:27 am**

That's SD cards for ya; they're actually pretty complicated internally and the thing may need to do some sort of active wear leveling every now and then, slowing everything down. Using a different card may give you different results.

Posted: **Fri Sep 25, 2020 4:28 pm**

How are you writting to the card in product? FATfs I assume?
You might do better going RAW. RAW will give you better power fail immunity & better write speed.
The FAT index table also get updated when you write and so essentially a data write becomes both a data write and a table write.
The table write must involve an erase which will be even slower.
Ok the driver probably caches etc but RAW will be faster.

Even RAW would might not be fast enough. The 'correct' approach is to cache and/or buffer data e.g. using the producer/consumer pattern. Essentially the data collector is one task and the file writer is another task and they share a buffer/queue of readings. Then you will have the buffering space to deal with the SD card's sleepy time.

The other thing to consider is that you are logging >10MB/hr. That gives you around 60 days of disk space. This throughput can start to wear the card. If you are happy with replacing your card then that's all ok but otherwise you should check/change your card's technology. With wear amplification you might only have a couple of years use. Maybe less. Increase your logging rate and less. Most consummer cards won't tell you their type (you can guess, go cheap) and are only rated for 'normal' use - occasional photos etc.

Posted: **Fri Sep 25, 2020 4:43 pm**

Thanks for the deeper explanation.
In the full scetch I use a MPU9250 for Acceleration Data, it has a FIFO Buffer of 512 Byte, I could try to put another buffer inbetween. But i need the high sample rate of 500Hz for the whole time of the measurement.
How would i write in RAW? It is FAT formatted, could I just reformat it on Windows/Linux to RAW Filesystem and would be done?
Yeah, I know that that is a lot of writing. But it doesn't matter since it would only write once/twice a day for 10-30min with that high data rate.
Do you have tips where i could get a better sd card, that would be able to hold that high writing speed for a longer time?

Posted: **Sun Sep 27, 2020 11:07 pm**

In the full scetch I use a MPU9250 for Acceleration Data, it has a FIFO Buffer of 512 Byte, I could try to put another buffer inbetween. But i need the high sample rate of 500Hz for the whole time of the measurement.

The MPU9250 gives you 120mS buffering best (maybe not even an SD erase)? But of course you don't want to max the device buffer because you might then loose data.
Anyway you have an ESP32 & a 32KB IRAM buffer should be cool. Thats like 10 seconds? Not even the most lazy arse SD will keep you waiting that long! The buffer does not slow you, its just like Netflix, it allows other stuff to happen whilst you wait for the SD card. You watch the movie (write to disk at worst) 10 seconds old. Not a problem.

RAW has issues i.e. cannot easily be read by another device. If you hook a FTP server to your SD & use ESP Wifi (or wired) then its a thought. If you use an S2 and have USB then also a thought. FTP should not take you long but of course not as simple as remove SD plug into PC. It depends on how you value power fail reliability really.

A ok/decent card (pSLC) in 16Gb range is gonna cost you around $40. So a burner may be best for you if you have volume? Anyway Digikey, Farnell etc. Be careful, I have seen some parametric searches showing better tech but are not. Always check the datasheet. Also I used to buy cards for evaluation from Amazon etc (lead time really). Never had two cards of the same brand with the same wrapper or card label colour that way. Just sayin there are some jokers out there

ESP32 Forum

SD Card Speed going down sometimes

SD Card Speed going down sometimes

Re: SD Card Speed going down sometimes

Re: SD Card Speed going down sometimes

Re: SD Card Speed going down sometimes

Re: SD Card Speed going down sometimes