SD Card formatting causes corruption, bending of space and time

sb_espressif
Posts: 28
Joined: Fri Dec 08, 2023 3:04 am

SD Card formatting causes corruption, bending of space and time

Postby sb_espressif » Mon Feb 05, 2024 12:44 am

I know, what a ridiculous topic name.

I'm so flabbergasted by what I'm experiencing with trying to interact with an SD card and in esp32-s3 that I'm not even sure how to fully articulate it. I'm using esp-idf 5.1.2. I'm trying to get the sd_card example running. I have at least a dozen microSD cards I'm trying - Sandisk, MicroCenter, a bunch of others in various sizes. I have 4 different microSD breakout boards. Every permutation of these causes the example code to fail:

Code: Select all

E (30361) vfs_fat_sdmmc: sdmmc_card_init failed (0x107).
E (30361)   SB - sd-read-write: Failed to initialize the card (ESP_ERR_TIMEOUT). 
Commenting out swaths of the code and sprinkling log statements everywhere, I think I have pinpointed my issue to the formatting of the card - when this operation happens, the error is thrown, and the card becomes corrupted. I have to re-format it in my laptop to bring the card back to a usable state, but this example code very, very consistently behaves like this.

The thing that flabbergasts me the most, however: I've put log statements every few lines - when I bypass the format step, I see all of the log statements run. BUT, if I include the format line, NONE of the log statements print anything! The code almost immediately seems to jump over itself straight to this error. The only way I can get the thing to run how I might expect is by periodically inserting large delays in the code (even then, the format causes corruption, but I can at least see it making its way through the code as I expect).

I just can't imagine why this might be happening - is there something in the ESP that can cause code to run non-linearly?

Code: Select all

void app_main(void)
{
 
    esp_err_t error_status;

    sdmmc_card_t *card;
    const char mount_point[] = MOUNT_POINT;

    // Options for mounting the filesystem.
    esp_vfs_fat_sdmmc_mount_config_t mount_config = {
        .format_if_mount_failed = true,     // format SD card if it fails to mount
        .max_files = 5,                     // max number of concurrent open files
        .allocation_unit_size = 16*1024     // not sure I all the way understand this, but
                                            // it has to do with the case when mounting 
                                            // fails and formatting is done. TO DO:
                                            // learn about allocation unit size? 
    };

    ESP_LOGI(TAG, "Initializing SD card");
    ESP_LOGI(TAG, "Using SPI peripheral");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    // It appears that here we configure the "host", which is the microcontroller itself.
    // Among the things we want to configure is the frequency at which the host
    // communicates with the SD card (the 'peripheral). 
    //
    // From Espressif:
    // By default, SD card frequency is initialized to SDMMC_FREQ_DEFAULT (20MHz)
    // For setting a specific frequency, use host.max_freq_khz (range 400kHz - 20MHz for SDSPI)
    // Example: for fixed frequency of 10MHz, use host.max_freq_khz = 10000;
    sdmmc_host_t host = SDSPI_HOST_DEFAULT();
    host.max_freq_khz = SDMMC_FREQ_DEFAULT;
    //host.max_freq_khz = SDMMC_FREQ_PROBING;

    spi_bus_config_t bus_cfg = {
        .mosi_io_num = PIN_NUM_DI,
        .miso_io_num = PIN_NUM_DO,
        .sclk_io_num = PIN_NUM_CLK,
        // .max_transfer_sz = 4000,// Maximum transfer size.
        // .quadwp_io_num = -1,    // This is for Write Protect, which we apparently aren't reading/using
        // .quadhd_io_num = -1,    // This is for a Hold signal. Apparently when communicating with an 
        //                         // SD card over SPI, we may want to interrupt a given transaction with
        //                         // the card without necessarily aborting communication with it
        //                         // entirely. In such a case, we can issue a Hold signal, which allows
        //                         // us to "unplug" other lines to then handle other tasks, then re-plug
        //                         // them back into the SD card and resume communication. Interesting. 
        
    };

    // Try to initialize the bus (communication path) to the SD card.
    error_status = spi_bus_initialize(  host.slot, 
                                        &bus_cfg, 
                                        SDSPI_DEFAULT_DMA);

    if (error_status != ESP_OK) {
        ESP_LOGE(TAG, "Failed to initialize bus.");
        return;
    }
    ESP_LOGI(TAG, "Bus initialized. Pausing....");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    // This initializes the slot without card detect (CD) and write protect (WP) signals.
    // Modify slot_config.gpio_cd and slot_config.gpio_wp if your board has these signals.
    sdspi_device_config_t slot_config = SDSPI_DEVICE_CONFIG_DEFAULT();
    slot_config.gpio_cs = PIN_NUM_CS;
    slot_config.host_id = host.slot;

    // ---------------------------------
    // Attempt to mount the filesystem. 
    // Something to note here is the bit about needing pullup resistors on all the
    // SD card pins. For my test, I used 10k pullup resistors (a resistor connecting the
    // pin to power) for all of the data pins, and this seems to make things work. Without
    // those, I got the error message about needing to use pullup resistors below. 
    ESP_LOGI(TAG, "Mounting filesystem (pausing...)");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    error_status = esp_vfs_fat_sdspi_mount( mount_point, 
                                            &host, 
                                            &slot_config, 
                                            &mount_config, 
                                            &card);

    if (error_status != ESP_OK) {
        if (error_status == ESP_FAIL) {
            ESP_LOGE(TAG, "Failed to mount filesystem. "
                     "If you want the card to be formatted, set the CONFIG_EXAMPLE_FORMAT_IF_MOUNT_FAILED menuconfig option.");
        } else {
            // ESP_LOGE(TAG, "Failed to initialize the card (%s). "
            //          "Make sure SD card lines have pull-up resistors in place.", esp_err_to_name(error_status));

            ESP_LOGE(TAG, "Failed to initialize the card (%s). ", esp_err_to_name(error_status));
        }

        return;
    }
    ESP_LOGI(TAG, "Filesystem mounted. Pausing...");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    // Card has been initialized, print its properties
    sdmmc_card_print_info(stdout, card);

    

    // ---------------------------------
    // Write a file to the card. 
    //
    // From ESP-IDF docs: Use POSIX and C standard library functions to work with files.
    //
    // POSIX seems sort of like Java for Linux in a way; it's a collection of standards
    // for linux-like machines for things like file IO and other operations, to allow us
    // to generally maintain cross-compatibility among different OSes.

    ESP_LOGI(TAG, "Writing file. Pausing...");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    // First, we will create a file name
    const char *file_hello = MOUNT_POINT"/hello.txt";
    
    // This array will hole a string (line) of text we'll write to the file
    char data[EXAMPLE_MAX_CHAR_SIZE];

    // This is apparently how we do the python equvalent of "string1 + string2"
    // in C; snprintf is a printf but instead of printing to a terminal
    // it prints to the given character array (string).
    snprintf(data, EXAMPLE_MAX_CHAR_SIZE, "%s %s!\n", "Hello ", card->cid.name);
    // Note that at the time I made this, cid.name was returning an empty string. 
    // Not sure why this might be, possibly a shortcoming of my SD card? It behaves
    // fine otherwise though. 
    


    // Try to actually write the file.
    error_status = s_example_write_file(file_hello, data);
    if (error_status != ESP_OK) {
        return;
    }

    ESP_LOGI(TAG, "File written. Pausing...");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 

    // ---------------------------------
    // Rename the file we just created.
    const char *file_foo = MOUNT_POINT"/foo.txt";

    // Check if this file file exists before renaming
    struct stat st;
    if (stat(file_foo, &st) == 0) {
        // Delete it if it exists
        unlink(file_foo);
    }

    // Rename original file
    ESP_LOGI(TAG, "Renaming file %s to %s (pausing...)", file_hello, file_foo);
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 
    if (rename(file_hello, file_foo) != 0) {
        ESP_LOGE(TAG, "Rename failed");
        return;
    }

    // ---------------------------------
    // Try to read the renamed file.
    ESP_LOGI(TAG, "Reading file. Pausing...");
    vTaskDelay(pdMS_TO_TICKS(10000)); // This is a macro. 
    error_status = s_example_read_file(file_foo);
    if (error_status != ESP_OK) {
        return;
    }

    // ---------------------------------
    // Now we're going to format the disk, erasing
    // our test file in the process. 
    ESP_LOGI(TAG, "Trying to format card...(pausing)");

    // feed the watchdog
    vTaskDelay(pdMS_TO_TICKS(1000)); // This is a macro. 

    //////////////////////////////////////////////////////
    // DISASTER STRIKES!!!!!!!!!!!
    error_status = esp_vfs_fat_sdcard_format(mount_point, card);
    ///////////////////////////////////////////////////////

    ESP_LOGI(TAG, "Format Done. (pausing)");
    // feed the watchdog
    vTaskDelay(pdMS_TO_TICKS(1000)); // This is a macro. 

   ...// the rest of the example continues

}
wtf.jpeg
help me obi wan
wtf.jpeg (368.92 KiB) Viewed 1301 times

ESP_adokitkat
Posts: 50
Joined: Thu Jun 22, 2023 12:50 pm

Re: SD Card formatting causes corruption, bending of space and time

Postby ESP_adokitkat » Mon Feb 12, 2024 12:16 am

Hello.

A probable explanation for what you're describing is buffering of print functions. If you call `fflush()` function before the format function, it should flush the buffer and print. You can also use `setvbuf()` function to disable buffering on STDOUT if you want to.
I guess you adding `vTaskDelay()` makes the buffer flush automatically.

The current format function is kind of problematic in a sense it doesn't "yield" (the formatting is done via external FATFS library) and therefore WDT - watchdog timer can trigger, resetting the ESP mid-format and corrupting the filesystem. You can increase WTD timeout for tasks or disable it entirely (not recommended). Increasing allocation unit size speeds up the formatting a lot but also sets minimal size of a written block in the filesystem (with 16kB allocation unit size even if your real file size was 1kB, it takes up 16kB of space - if it has 17kB, it takes up 32kB etc.) - since you already have it set to 16kB it depends on the average file size stored there if you would want increase it further though. Speed of the format is also dependent of the SD card size.

Right now you're using the example to test your ESP with SD cards and that's why you're using the format as well - are you going to format SD cards by your ESP32 regularly? In what environment do you intend to run your project? FAT filesystem is prone to corruption when stopped (e.g. power cut-off) mid-operation (we are developing a journaling component which will help with this but at current time it is not publicly available). If you can upgrade to IDF v5.2+, we have an example there which uses LittleFS filesystem designed for embedded devices (unfortunately backport of the official example is not yet merged to v5.1, but it should be possible to use `esp_littlefs` even in v5.1) which should be more fail-safe - a downside is it is currently limited to only 2GB in partition size.

Also I see you're using SDSPI instead of SDMMC example - is there any reason? If you can, I would advise using SDMMC protocol as it is native to SD cards (SPI is like a fallback mode) and it should be faster.

If you have any more problems, please also share whole code which replicates your problem and also your log output.
Thanks you.

Who is online

Users browsing this forum: No registered users and 92 guests