Tearing my hair out trying to reliably write logfiles to SDCard

[ruffle]

I'm trying to put together a CAN Bus + other sensors logger for a racing motorcycle with an ESP-32 (ESP-IDF v3.2-dev-209-gbe81d2c1-dirty)

While all the Canbus stuff is working very well, recording data on the integral SD Card is causing me a right headache.

Mounting the card works in SPI mode works (it's integral to the module; a Lolin D32 Pro), until the filesystem gets corrupted; see later, but I'm going round in circles trying to find a reliable way of writing data to it compounded by the fact that I have no control over the power to the ESP32 being cut; ignition switch turned off or the rider hitting a tyre wall! .

If I use fopen and fprintf, the file on the card doesn't seem to get updated (from a filesystem perspective) until I close it. Doing a stat() on the filename while it's open always shows that the file is zero length until it's closed (I'm guessing this is a 'feature' of FAT?). Even doing an fflush() after every fprintf or using setbuf(fp,0) doesn't have any effect.

Low level open and write suffer from the same problem although I'd hoped that O_SYNC would help:

Code: Select all

logfile_fd = open(logfile_name, O_WRONLY | O_CREAT | O_TRUNC | O_SYNC);

If I close and reopen the file every N packets (fopen or open), the file does grow and has an increasing st_size but doing this slows things down and means I lose the last N packets on power loss (the packets are coming in at ~1500 per second and I'm also wanting to add some suspension logging data at ~200Hz so slowing things down isn't good).

If I choose a value of N that I can live with then I still have the problem of the filesystem getting corrupted and refusing to mount when the ESP32 next boots. The ESP comes up with:

Code: Select all

Initializing SDCard...
I (2391) gpio: GPIO[4]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0 
E (2441) sdmmc_cmd: sdmmc_card_init: send_if_cond (1) returned 0x108
I (2441) gpio: GPIO[23]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
I (2441) gpio: GPIO[19]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
I (2451) gpio: GPIO[18]| InputEn: 0| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0 
E (2461) App: Failed to initialize the card (ESP_ERR_INVALID_RESPONSE). Make sure SD card lines have pull-up resistors in place.

Sometimes it's a 0x107 error.

Once the card has got into this state, I have to remove it, stick in my Linux laptop and run a dosfsck on the partition. It normally finds the Free cluster summary is wrong and fixes it. I can then use the card in the ESP32 module again. Until I do this, the whole thing gets stuck at the above point with the need for physical intervention.

I would really like to retain the concept of a filesystem with files of data because I'm presenting and downloading them with a web server.

I surely can't be the only person trying to use an ESP32 as a data stream logger so can anyone help point me in the right direction?

Thanks in advance.

[gunar.kroeger]

Even doing an fflush() after every fprintf or using setbuf(fp,0) doesn't have any effect.

instead of fflush() you should use fsync(). This should have the same behaviour as fclose() without actually closing the file.

[vonnieda]

Even doing an fflush() after every fprintf or using setbuf(fp,0) doesn't have any effect.

instead of fflush() you should use fsync(). This should have the same behaviour as fclose() without actually closing the file.

Just adding on: I'm not sure if it's the same for every FS, but for mine, I've found I actually have to use both:

fflush(file);
fsync(fileno(file));

Jason

[ruffle]

Thanks for that. I'd tried sync() but that doesn't seem to exist and not looked further for fsync.

Unfortunately I've hit a bigger issue now as it seems that one can't use an SD Card and a display on the Lolin D32 Pro (opened new thread).