fwrite fails to write binary data at moderate speeds

[ANF_MCG]

Hello all,

I'm running into this issue and I'm wondering if I'm missing something very obvious.

I'm using a u-blox W101 dev kit with SD Card slot, and using an 8GB card.

The dev kit receives data through the serial port at a rate of 33 bytes and 200 Hz (6.6KBytes per second). That data must be written into a binary file as it arrives.

To do so, the serial data is copied into a Streambuffer (part of the code below):

Code: Select all

switch (uart_event.type) 
             {
                //Event of UART receving data
                /*We'd better handler data event fast, there would be much more data events than
                other types of events. If we take too much time on data event, the queue might
                be full.*/
                case UART_DATA:
                    //Read UART buffer
                    uart_read_bytes(UART_NUM_1, (void *)uart_rx_data, uart_event.size, portMAX_DELAY);
                    // Write to streambuffer of the SD Card
                    bytes_written = xStreamBufferSend(h_sd_streambuffer, (const void*)uart_rx_data, uart_event.size, 1);
                    if(bytes_written != uart_event.size)
                    {
                        ESP_LOGE(tag, "Could not write %u uart bytes to buffer, written: %lu", uart_event.size, bytes_written);
                    }
                    break;

The above code works.

The SD Card is formatted first (temporary test), then a file with a random name is created, and when the data transmission begins it should start storing to file.

Code: Select all

    // Format SD Card
    ret = esp_vfs_fat_sdcard_format(mount_point, card);
    if (ret != ESP_OK) {
        ESP_LOGE(tag, "Failed to format FATFS (%s)", esp_err_to_name(ret));
    }
    else{
        ESP_LOGI(tag, "FATFS Formatted correctly");
    }

    // Create a new file
    time(&now);
    timeinfo = gmtime(&now);
    sprintf(timeinfo_str, "%02d%02d%02dT%02d%02d%02d", timeinfo->tm_year, timeinfo->tm_mon+1, timeinfo->tm_mday, timeinfo->tm_hour, timeinfo->tm_min, timeinfo->tm_sec);
    ESP_LOGD(tag, "Current time: %s", timeinfo_str);
    filenumber = esp_random();
    sprintf(filename, "%s/%lX.bin", MOUNT_POINT, filenumber);
    ESP_LOGD(tag, "File Name: %s", filename);
    file_ptr = fopen(filename, "w+b");
    if(NULL == file_ptr)
    {
        ESP_LOGE(tag, "Error opening file!");
    }

The SD Card task receives this data and tries to put it into the SD Card:

Code: Select all

static void sd_mmc_task(void *params)
{
    while(1)
    {
        sd_streambuffer_rxd = xStreamBufferReceive(h_sd_streambuffer, (void*)data_block, 512, portMAX_DELAY);
        if(sd_streambuffer_rxd != fwrite(data_block, 1, sd_streambuffer_rxd, file_ptr))
        {
            ESP_LOGE(tag, "Error writing to file, counts differ!");
        }
        bytes_written += sd_streambuffer_rxd;
        total_bytes_written += sd_streambuffer_rxd;
        if(bytes_written > 20000)
        {
            ESP_LOGD(tag, "Bytes written: %u", total_bytes_written);
            bytes_written = 0;
            if(0 != fflush(file_ptr))
            {
                ESP_LOGE(tag, "Error flushing file");
            }
        }
    }
}

This seems to work, I get the following debug messages:

D (1477) sdmmc_periph: slot=1 host_div=10 card_div=20 freq=400kHz (max 400kHz)
D (1487) sdmmc_periph: slot=1 width=1
D (1487) sdmmc_req: process_command_response: error 0x107 (status=00000100)
D (1517) sdmmc_sd: SDHC/SDXC card
D (1517) sdmmc_req: process_command_response: error 0x107 (status=00000100)
D (1517) sdmmc_io: sdmmc_init_io: io_send_op_cond (1) returned 0x107; not IO card
D (1807) sdmmc_common: host_ocr=0x40ff8000 card_ocr=0xc0ff8000
D (1807) sdmmc_common: sdmmc_card_init: host_ocr=40ff8000, card_ocr=c0ff8000
D (1807) sdmmc_init: sdmmc_card_init: card type is SD
D (1817) sdmmc_common: sdmmc_init_host_bus_width: using 4-bit bus
D (1817) sdmmc_periph: slot=1 width=4
D (1817) sdmmc_periph: slot=1 host_div=8 card_div=0 freq=20000kHz (max 20000kHz)
D (1827) vfs_fat_sdmmc: using pdrv=0
I (1837) sd_mmc.c: Filesystem mounted
Name:
Type: SDHC/SDXC
Speed: 20.00 MHz (limit: 20.00 MHz)
Size: 7431MB
CSD: ver=2, sector_size=512, capacity=15218688 read_bl_len=9
SSR: bus_width=4
I (1847) vfs_fat_sdmmc: Formatting card, allocation unit size=16384
D (1857) sdmmc_sd: sdmmc_sd_get_erase_timeout_ms: erase timeout 418 s (erasing 3415008 kB, ES=8, ET=4, EO=1, AU=4096 kB)
I (3707) sd_mmc.c: FATFS Formatted correctly
D (3717) sd_mmc.c: Current time: 700101T000003
D (3717) sd_mmc.c: File Name: /sdcard/3B0FEE8.bin
D (3717) sd_mmc.c: SD_MMC task created
D (15247) sd_mmc.c: Bytes written: 20114
D (18627) sd_mmc.c: Bytes written: 40211
D (21987) sd_mmc.c: Bytes written: 60308
D (25397) sd_mmc.c: Bytes written: 80372
D (28817) sd_mmc.c: Bytes written: 100502
D (32237) sd_mmc.c: Bytes written: 120599
D (35627) sd_mmc.c: Bytes written: 140696

However, when I put the SD card on my windows machine, the file size is 0 bytes:

Screenshot 2023-11-10 094042.png (6.21 KiB) Viewed 1060 times

The settings on sdkconfig.defaults related to FATFS are:

Code: Select all

#
# FAT FS
#
CONFIG_FATFS_MAX_LFN=32
CONFIG_FATFS_API_ENCODING_ANSI_OEM=y
CONFIG_FATFS_VFS_FSTAT_BLKSIZE=512
CONFIG_FATFS_VOLUME_COUNT=1
CONFIG_FATFS_LFN_HEAP=y
CONFIG_FATFS_SECTOR_512=y

I also tried with block size 4096 and get the same results...

Any ideas or suggestions are welcome, the original example from ESP32 worked well, but that was text based.

Cheers,

Alberto

[MicroController]

You'll want to fsync(file_ptr) or fclose(file_ptr) at some point to get the open file into a consistent state on the SD card and have its metadata updated before shutting down the ESP and/or removing the SD card.