Flash encryption and FAT filesystem corruption

nick_senva
Posts: 1
Joined: Tue Oct 11, 2022 9:33 pm

Flash encryption and FAT filesystem corruption

Postby nick_senva » Fri Dec 02, 2022 12:07 am

Our team's project is using the FatFs file system demonstrated in the wear_levelling example project to embed three text files up to 4KB each. After enabling flash encryption on the FAT data partition, we have noticed that the second or third file will periodically become corrupted entirely for small strings, or corrupted in the second half of larger files. This occurs every time after the flash partition has been erased and reformatted, and is also occasionally seen after deleting and replacing files. The corrupted values appear to be consistent for a given file (i.e the string '"written using ESP-IDF v4.4.2-2-gc59a104546" will always become "�(p�`���b�,(�`q:�`�k" when corrupted).

This behavior was recreated on the wear_levelling example with a few modifications on an ESP32-PICO-MINI-02 devkit with flash encryption and secure boot v2 enabled. Further experimentation showed that the issue would disappear when the FAT partition size was at least 0x84000 bytes large when WL_SECTOR_SIZE was set to 4096 in the sdkconfig, or 0xB1000 bytes for a sector size of 512.

It is noted in the flash encryption documentation that the NVS storage library is not compatible with flash encryption, but we could not find direct references to the FatFs library. Is flash encryption compatible with the FAT filesystem? And if so, are there additional minimum partition size requirements beyond those described in the FatFs documentation to ensure proper operation?

Modified wear_levelling example code:

Code: Select all

void app_main(void)
{
    ESP_LOGI(TAG, "Mounting FAT filesystem");
    // To mount device we need name of device partition, define base_path
    // and allow format partition in case if it is new one and was not formated before
    const esp_vfs_fat_mount_config_t mount_config = {
            .max_files = 4,
            .format_if_mount_failed = true,
            .allocation_unit_size = CONFIG_WL_SECTOR_SIZE
    };
    esp_err_t err = esp_vfs_fat_spiflash_mount(base_path, "storage", &mount_config, &s_wl_handle);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to mount FATFS (%s)", esp_err_to_name(err));
        return;
    }
    ESP_LOGI(TAG, "Opening file");
    FILE *f = fopen("/spiflash/hello.txt", "wb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for writing");
        return;
    }
    fprintf(f, "written using ESP-IDF %s\n", esp_get_idf_version());
    fclose(f);
    ESP_LOGI(TAG, "File written");

    // Open file for reading
    ESP_LOGI(TAG, "Reading file");
    f = fopen("/spiflash/hello.txt", "rb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for reading");
        return;
    }
    char line[128];
    fgets(line, sizeof(line), f);
    fclose(f);
    // strip newline
    char *pos = strchr(line, '\n');
    if (pos) {
        *pos = '\0';
    }
    ESP_LOGI(TAG, "Read from file: '%s'", line);


    //File 2
    ESP_LOGI(TAG, "Opening file");
    f = fopen("/spiflash/hello2.txt", "wb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for writing");
        return;
    }
	fprintf(f, "written using ESP-IDF %s\n", esp_get_idf_version());
    fclose(f);
    ESP_LOGI(TAG, "File written");

    // Open file for reading
    ESP_LOGI(TAG, "Reading file");
    f = fopen("/spiflash/hello2.txt", "rb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for reading");
        return;
    }
    fgets(line, sizeof(line), f);
    fclose(f);
    // strip newline
    pos = strchr(line, '\n');
    if (pos) {
        *pos = '\0';
    }
    ESP_LOGI(TAG, "Read from file: '%s'", line);

    //File 3
    ESP_LOGI(TAG, "Opening file");
    f = fopen("/spiflash/hello3.txt", "wb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for writing");
        return;
    }
    fprintf(f, "written using ESP-IDF %s\n", esp_get_idf_version());
    fclose(f);
    ESP_LOGI(TAG, "File written");

    // Open file for reading
    ESP_LOGI(TAG, "Reading file");
    f = fopen("/spiflash/hello3.txt", "rb");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for reading");
        return;
    }
    fgets(line, sizeof(line), f);
    fclose(f);
    // strip newline
    pos = strchr(line, '\n');
    if (pos) {
        *pos = '\0';
    }
    ESP_LOGI(TAG, "Read from file: '%s'", line);

    // Unmount FATFS
    ESP_LOGI(TAG, "Unmounting FAT filesystem");
    ESP_ERROR_CHECK( esp_vfs_fat_spiflash_unmount(base_path, s_wl_handle));

    ESP_LOGI(TAG, "Done");
}
Monitor output of bug occuring:

Code: Select all

I (0) cpu_start: App cpu up.
I (179) cpu_start: Pro cpu start user code
I (179) cpu_start: cpu freq: 160000000
I (179) cpu_start: Application information:
I (184) cpu_start: Project name:     wear_levelling_example
I (190) cpu_start: App version:      1
I (195) cpu_start: Compile time:     Dec  1 2022 14:40:40
I (201) cpu_start: ELF file SHA256:  bea26e30432862ed...
I (207) cpu_start: ESP-IDF:          v4.4.2-2-gc59a104546
I (213) heap_init: Initializing. RAM available for dynamic allocation:
I (220) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (226) heap_init: At 3FFB2C58 len 0002D3A8 (180 KiB): DRAM
I (232) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (239) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (245) heap_init: At 4008B6D4 len 0001492C (82 KiB): IRAM
I (252) spi_flash: detected chip: generic
I (256) spi_flash: flash io: dio
W (260) flash_encrypt: Flash encryption mode is DEVELOPMENT (not secure)
I (269) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (278) example: Mounting FAT filesystem
W (288) vfs_fat_spiflash: f_mount failed (13)
I (298) vfs_fat_spiflash: Formatting FATFS partition, allocation unit size=512
I (1998) vfs_fat_spiflash: Mounting again
I (1998) example: Opening file
I (2798) example: File written
I (2798) example: Reading file
I (2808) example: Read from file: 'written using ESP-IDF v4.4.2-2-gc59a104546'
I (2808) example: Opening file
I (3628) example: File written
I (3638) example: Reading file
I (3638) example: Read from file: 'written using ESP-IDF v4.4.2-2-gc59a104546'
I (3648) example: Opening file
I (4478) example: File written
I (4488) example: Reading file
I (4498) example: Read from file: '�(p�`���b�,(�`q:�`�k'
I (4498) example: Unmounting FAT filesystem
I (4638) example: Done

derricksenva
Posts: 21
Joined: Tue Aug 02, 2022 8:19 pm

Re: Flash encryption and FAT filesystem corruption

Postby derricksenva » Tue Dec 06, 2022 4:16 pm

Just following up on this thread to get an immediate response as to whether or not flash encryption is officially supported with the FAT file system components. If the answer is no, we'd like to start looking into an alternative as soon as possible. If yes, we'll continue to wait for a more informative response. Thanks.

pacucha42
Posts: 31
Joined: Fri Mar 29, 2019 12:56 pm

Re: Flash encryption and FAT filesystem corruption

Postby pacucha42 » Wed Dec 07, 2022 7:42 am

Hi @nick_senva and @derricksenva,
generally, there is no problem using FATFS on encrypted partitions (unlike SPIFFS). However, the minimum size of such a partition is 128 sectors (+4 sectors for the Wear-Leveling, if deployed) - in your case it would be 0x80000 with 4kB sector size. Not sure why you needed additional 4 sectors.

I will try your example and come back with more details as soon as possible. In case you haven't yet checked our flash_encryption example: https://github.com/espressif/esp-idf/tr ... encryption

Thank you for your report

pacucha42
Posts: 31
Joined: Fri Mar 29, 2019 12:56 pm

Re: Flash encryption and FAT filesystem corruption

Postby pacucha42 » Wed Dec 07, 2022 11:14 am

pacucha42 wrote:
Wed Dec 07, 2022 7:42 am
Not sure why you needed additional 4 sectors.
Wrong calculation, sorry. Your size is actually correct: (128 + 4) * 0x1000 = 0x84000, ie the minimum partition size for FATFS+WL

derricksenva
Posts: 21
Joined: Tue Aug 02, 2022 8:19 pm

Re: Flash encryption and FAT filesystem corruption

Postby derricksenva » Mon Dec 12, 2022 8:28 pm

pacucha42 wrote:
Wed Dec 07, 2022 7:42 am
Hi @nick_senva and @derricksenva,
generally, there is no problem using FATFS on encrypted partitions (unlike SPIFFS). However, the minimum size of such a partition is 128 sectors (+4 sectors for the Wear-Leveling, if deployed) - in your case it would be 0x80000 with 4kB sector size. Not sure why you needed additional 4 sectors.

I will try your example and come back with more details as soon as possible. In case you haven't yet checked our flash_encryption example: https://github.com/espressif/esp-idf/tr ... encryption

Thank you for your report
Thanks for looking into this. On a semi-related note, our target sector size is 512 bytes, but the minimum partition size with wear leveling didn't seem to fit that requirement. Instead of 512 bytes * (128 + 4 sectors) = 67,584 bytes, the application fails with partition sizes all the way up to 84 kB, but started working at 88 kB (I kept increasing the partition size by 4 kB at a time). The failures all resulted in the log messages below. Any ideas? Is there some other requirement (alignment, etc.) that we're missing?

Code: Select all

W (978) vfs_fat_spiflash: f_mount failed (13)
I (978) vfs_fat_spiflash: Formatting FATFS partition, allocation unit size=512
E (988) vfs_fat_spiflash: f_mkfs failed (14)
E (988) app: failed to mount (ESP_FAIL)
For reference, this is how the partition table was defined.

Code: Select all

otadata,    data,   ota,        0x0000F000, 0x00002000, encrypted,
phy_init,   data,   phy,        0x00011000, 0x00001000, encrypted,
nvs_key,    data,   nvs_keys,   0x00023000, 0x00001000, encrypted,
nvs,        data,   nvs,        0x00024000, 0x00080000,
app_files,  data,   fat,        0x000B0000, 84k,        encrypted,
ota_0,      app,    ota_0,      0x00100000, 0x00200000, encrypted,
ota_1,      app,    ota_1,      0x00300000, 0x00200000, encrypted,

pacucha42
Posts: 31
Joined: Fri Mar 29, 2019 12:56 pm

Re: Flash encryption and FAT filesystem corruption

Postby pacucha42 » Tue Dec 20, 2022 11:22 am

Hi again,
you need to enable Performance mode for the Wear levelling sector storage (idf.py menuconfig: Component config > Wear Levelling > Sector store mode), then the FAT sectors equal the wear-levelled ones. By default, this mode is set to Safety for 512B sectors, which implies different "mapping" of the WL sectors to the real SPI Flash sectors (fixed size 4096B) => FatFS formatting fails due to seemingly insufficient volume space.

The difference between Performance and Safety modes is as follows:

- In Performance mode a data will be stored to the RAM and then
stored back to the flash. Compared to the Safety mode, this operation is
faster, but if power will be lost when erase sector operation is in
progress, then the data from complete flash device sector will be lost.

- In Safety mode data from complete flash device sector will be read from
flash, modified, and then stored back to flash.
Compared to the Performance mode, this operation is slower, but if
power is lost during erase sector operation, then the data from full
flash device sector will not be lost.

Freshly tested, all works in my case. If you keep having troubles with the setup, please let me know.

derricksenva
Posts: 21
Joined: Tue Aug 02, 2022 8:19 pm

Re: Flash encryption and FAT filesystem corruption

Postby derricksenva » Tue Dec 20, 2022 4:52 pm

pacucha42 wrote:
Tue Dec 20, 2022 11:22 am
Hi again,
you need to enable Performance mode for the Wear levelling sector storage (idf.py menuconfig: Component config > Wear Levelling > Sector store mode), then the FAT sectors equal the wear-levelled ones. By default, this mode is set to Safety for 512B sectors, which implies different "mapping" of the WL sectors to the real SPI Flash sectors (fixed size 4096B) => FatFS formatting fails due to seemingly insufficient volume space.

The difference between Performance and Safety modes is as follows:

- In Performance mode a data will be stored to the RAM and then
stored back to the flash. Compared to the Safety mode, this operation is
faster, but if power will be lost when erase sector operation is in
progress, then the data from complete flash device sector will be lost.

- In Safety mode data from complete flash device sector will be read from
flash, modified, and then stored back to flash.
Compared to the Performance mode, this operation is slower, but if
power is lost during erase sector operation, then the data from full
flash device sector will not be lost.

Freshly tested, all works in my case. If you keep having troubles with the setup, please let me know.
1. Thank you for looking into that. Does this solve the issue from the original post about file corruption (we're mostly concerned with this) or just the minimum partition size?

2. Assuming this solves the corruption issue, safety mode is preferred for our application. It's possible we can work around it as long as only the current file being being written/modified can get corrupted on power loss. Is this true, or is it possible to corrupt other files since a flash sector (4096 bytes) could possibly contain FAT sectors (512 bytes) from other files and thus corrupt them?

3. Also, does this mean safety mode is not intended to support 512-byte sector sizes? Is this an error in the documentation or a bug in the firmware? The documentation seems to imply that this combination should work. From docs.espressif.com/projects/esp-idf/en/ ... 20leveling:
2022-12-20 08_43_45-Wear Levelling API - ESP32 - — ESP-IDF Programming Guide v5.0 documentation.png
2022-12-20 08_43_45-Wear Levelling API - ESP32 - — ESP-IDF Programming Guide v5.0 documentation.png (37.01 KiB) Viewed 2952 times

Who is online

Users browsing this forum: Google [Bot], jialin.yang, Majestic-12 [Bot] and 130 guests