can't fail to open file

[kernic]

I am currently using the fatfs library to allocate a contiguous file for my project. The problem is that I think I found a bug in the fatfs library, for example I want to first try opening a file with FA_OPEN_EXISTING, and if the file doesn't exist and the function return fail, I'd like to fall back to creating it with FA_CREATE_NEW.

The problem is that, and I have no idea why, when I put FA_CREATE_NEW as a parameter in the second call to f_open for when OPEN_EXISTING fails, it creates the file directly in the first call to f_open and completely ignore the FA_OPEN_EXISTING.

please, can someone try this out to see what I mean? I think I'm going mad it's been 3 hours debugging this and I can't understand why it is made this way.

All I want is open a file if it exists(FA_OPEN_EXISTING), if so execute a conditional block, and if it doesn't execute another conditional block where I try to create it(FA_CREATE_NEW).

if I use FA_CREATE_NEW only once(anywhere in my code) FA_OPEN_EXISTING doesn't work and acts as FA_CREATE_NEW in the first call

I am currently using the esp idf 5.0.2, this looks like a bug to me, if someone could confirm what I think..

Code: Select all

// Attempt to open the file
    fr = f_open(&file, FILE_PATH, FA_OPEN_EXISTING | FA_READ | FA_WRITE);
    if (fr == FR_OK) {
        printf("File opened with success\n");
        printf("Impossible to bypass this path even when the file doesn't exist yet when the second call contains FA_CREATE_NEW \n");
    }
    else if (fr == FR_NO_FILE) {
        printf("Can't open file, creating it\n");
        fr = f_open(&file, FILE_PATH, FA_CREATE_NEW | FA_READ | FA_WRITE);
        if (fr == FR_OK) {
            printf("File created successfully.\n");
            f_expand(&file, 1000000, 1);
        } else {
            printf("Failed creating file\n");
        }
    } else {
        printf("Failed to open file with error: %d\n", fr);
    }

[MicroController]

After a successful FA_CREATE_NEW, you do delete the file before the next test with FA_OPEN_EXISTING, right?

[kernic]

well, for example I definitely tried to change the filename, like if it was "filetext.txt" I rename it "filetex2.txt".
normally I would expect that opening the same file twice execute the 1st code block(FA_CREATE_NEW creates a new file), then the second open would execute the 2nd code block(FA_OPEN_EXISTING creates a new file)

I do not delete the file.

By the way I also tried with the esp idf 5.3 and I get the exact same behavior

[ESP_adokitkat]

Hello. I am not sure why is it happening, but can you try something like this? I am using `access` to see if the file exists.

Code: [Select all] [Expand/Collapse]

1. #include <unistd.h>
2.  
3. //...
4.  
5. FIL* file = (FIL*) ff_memalloc(sizeof(FIL));
6. if (file == NULL) { ;/* handle NULL */ }
7. memset(file, 0, sizeof(FIL));
8. FRESULT fr = 0;
9.  
10. if (access(FILE_PATH, FR_OK) == 0) {
11.     // file exists
12.     fr = f_open(file, FILE_PATH, FA_WRITE | FA_READ);
13. } else {
14.     // file doesn't exist
15.     fr = f_open(file, FILE_PATH, FA_CREATE_NEW | FA_WRITE);
16.     if (fr == FR_OK) {
17.         fr = f_expand(file, 1000000, 1);
18.         if (fr != FR_OK) { ;/* handle it */}
19.     } else { ;/* handle it */}
20.     f_close(file);
21.     fr = f_open(file, FILE_PATH, FA_WRITE | FA_READ);
22.     //...
23. }
24.  
25. //...
26.  
27. free(file);
28.  
29.  

GeSHi © Codebox Plus Extension

[ESP_adokitkat]

well, for example I definitely tried to change the filename, like if it was "filetext.txt" I rename it "filetex2.txt".
normally I would expect that opening the same file twice execute the 1st code block(FA_CREATE_NEW creates a new file), then the second open would execute the 2nd code block(FA_OPEN_EXISTING creates a new file)

I do not delete the file.

By the way I also tried with the esp idf 5.3 and I get the exact same behavior

Oh, have you enabled long file namew (LFN)? The default for FATFS is 8.3 format (short) file names (SFN), whcih means maximum 8 characters in file name, then a period . and then maximum 3 characters in the extension. If you haven't enabled LFN, then the filename gets truncated and mangled automatically by FATFS. http://elm-chan.org/docs/fat_e.html#name_conversion

[kernic]

hahaha, I love your solution where you check if the file exists, I came to a similar conclusion, except my solution doesn't work unless I do a weird conditional check with the file and then do a wait call for at least 3 seconds, sorry if its a bit convoluted with the function:

Code: Select all

FRESULT checkFileExists(const char* path) {
    FILINFO fno;
    FRESULT fr;

    fr = f_stat(path, &fno);

    if (fr == FR_OK) {
        // File exists
        return FR_OK;
    } else if (fr == FR_NO_FILE) {
        // File does not exist
        return FR_NO_FILE;
    } else {
        // An error occurred
        return fr;
    }
}

    FRESULT fileExists = checkFileExists(FILE_PATH);

    if(fileExists == FR_OK) {
    }
    else if(fileExists == FR_NO_FILE) {
    }
    else {
    }

    vTaskDelay(3000 / portTICK_PERIOD_MS);

    f_open(&file, FILE_PATH, FA_OPEN_ALWAYS | FA_READ | FA_WRITE);

    if(fileExists == FR_OK) {
        printf("File opened with success\n");
    }
    else if(fileExists == FR_NO_FILE) {
        printf("init file for the first time\n");

        f_expand(&file, 10000, 1);
    }
    else {
        printf("something happened with the file\n");
    }

as for the names, indeed I can only use 8 characters by default, which I thought was weird but I didn't give much attention to that.. And when I try to put FF_USE_LFN to 2 or 3 it doesn't compile and gives a bunch of error, and option 1 says it can't run in a safe environment when I try to compile. here's some of the generated error logs when FF_USE_LFN=2 :

Code: Select all

C:/Users/Nicolas/esp/v5.3/esp-idf/components/mbedtls/mbedtls/library/x509_crt.c:1612:21: error: implicit declaration of function 'readdir' [-Werror=implicit-function-declaration]
 1612 |     while ((entry = readdir(dir)) != NULL) {
      |                     ^~~~~~~             ^
C:/Users/Nicolas/esp/v5.3/esp-idf/components/mbedtls/mbedtls/library/x509_crt.c:1614:56: error: invalid use of undefined type 'struct dirent'
 1614 |                                    "%s/%s", path, entry->d_name);
      |                                                        ^~
C:/Users/Nicolas/esp/v5.3/esp-idf/components/mbedtls/mbedtls/library/x509_crt.c:1650:5: error: implicit declaration of function 'closedir' [-Werror=implicit-function-declaration]
 1650 |     closedir(dir);

Thanks a lot for all your help!

[kernic]

Hello. I am not sure why is it happening, but can you try something like this? I am using `access` to see if the file exists.

Code: [Select all] [Expand/Collapse]

1. #include <unistd.h>
2.  
3. //...
4.  
5. FIL* file = (FIL*) ff_memalloc(sizeof(FIL));
6. if (file == NULL) { ;/* handle NULL */ }
7. memset(file, 0, sizeof(FIL));
8. FRESULT fr = 0;
9.  
10. if (access(FILE_PATH, FR_OK) == 0) {
11.     // file exists
12.     fr = f_open(file, FILE_PATH, FA_WRITE | FA_READ);
13. } else {
14.     // file doesn't exist
15.     fr = f_open(file, FILE_PATH, FA_CREATE_NEW | FA_WRITE);
16.     if (fr == FR_OK) {
17.         fr = f_expand(file, 1000000, 1);
18.         if (fr != FR_OK) { ;/* handle it */}
19.     } else { ;/* handle it */}
20.     f_close(file);
21.     fr = f_open(file, FILE_PATH, FA_WRITE | FA_READ);
22.     //...
23. }
24.  
25. //...
26.  
27. free(file);
28.  
29.  

GeSHi © Codebox Plus Extension

I just tried your given code with the esp idf 5.3 (I was hoping yours wouldn't need a 3 second delay like mine to work) and unfortunately your solution doesn't seem to work, I remplaced the comments with printf and it always says "file doesn't exist" each time, even the first time.

I'd really like to find a fix for this, as I really need a contiguous file for my small database project. (and also I need to know when a file is newly created)
Thanks again for reading

[kernic]

Ok, I just noticed a very strange behavior. Even if my code is behaving as 'expected' (provided you delay 3 seconds and execute the weird conditional logic), the file doesn't exist when I insert my sd card in my laptop.. Even if the esp32 is able to read it back again when I mount back the sd card in it..

I hope this will help diagnosing my issue.. Thanks

[kernic]

So this is my last consecutive post(since I don't want to overwhelm the chat with consecutive posts) but I found this very simple snippet:

Code: Select all

    f_open(&file, FILE_PATH, FA_OPEN_ALWAYS | FA_READ | FA_WRITE);

    if(f_size(&file) != 0) {
        printf("File opened with success\n");
    }
    else {
        printf("File opened for the first time\n");
        f_expand(&file, 10000, 1);
    }

    f_close(&file);

This snippet will actually work, but only once (if after creating the 1st file you change the FILE_PATH, afterward if you directly create a 2nd file with the esp32 it won't appear on windows, also when you create the 2nd file for the first time it erroneously thinks that f_size(&file) != 0.. and it produces the same bug again)

Also I implemented the whole thing with the Standard C Library I/O that has a FILE instead of a FIL and makes use of fopen() instead of f_open(), same behavior. I can provide the code for the FILE version if you want. I think at least the Standard C Library I/O version should be working, normally, even if the fatfs version isn't because of some misconfiguration on the fatfs part.

Thanks again..

[ESP_adokitkat]

Ok, I just noticed a very strange behavior. Even if my code is behaving as 'expected' (provided you delay 3 seconds and execute the weird conditional logic), the file doesn't exist when I insert my sd card in my laptop.. Even if the esp32 is able to read it back again when I mount back the sd card in it..

I hope this will help diagnosing my issue.. Thanks

I would try to reformat the sd card, maybe the filesystem got a bit corrupted?

Or this may be related to cached information not being flushed, as in FATFS it is only done when f_close() or f_sync() is called (f_sync doesn't close the file descriptor). You can try it instead of waiting 3s. http://elm-chan.org/fsw/ff/doc/sync.html

If you updated to v5.3, can you try `esp_vfs_fat_create_contiguous_file` and `esp_vfs_fat_test_contiguous_file` functions? Here you can see the implementation https://github.com/espressif/esp-idf/bl ... at.c#L1309 and here tests for it https://github.com/espressif/esp-idf/bl ... on.c#L1162

Are you familiar with `idf.py` script and/or its `menuconfig` option? I am sorry but I don't use the VS code extension so I am not sure how to set it there. In menuconfig you can find `(Top) → Component config → FAT Filesystem support` and there are settings for LFN and also "Enable automatic f_sync" (but I would not enable it in your case if you need performance, just call f_sync yourself when you need it). For performance uplift I would also set " Default block size" there to 4092 as well as enable "Enable fast seek algorithm when using lseek function through VFS FAT" (this only works on files opened in read-only mode).

If you don't know/can't use idf.py menuconfig then you can set all of these in Kconfig file named "sdkconfig.defaults" you can put in the root directory of your project (then then do `idf.py fullclean` or just remove the `build` folder and `sdkconfig` and `sdkconfig.old` files before recompilation).

Code: [Select all] [Expand/Collapse]

1. CONFIG_FATFS_LFN_HEAP=y
2. CONFIG_FATFS_SECTOR_4096=y
3. CONFIG_FATFS_USE_FASTSEEK=y
4. CONFIG_FATFS_FAST_SEEK_BUFFER_SIZE=64
5. CONFIG_FATFS_VFS_FSTAT_BLKSIZE=4092

GeSHi © Codebox Plus Extension