POSIX file operations implemented on ESP32

[zliudr]

Question: if one task does something such as rename files in a directory (move them out of the directory), and another task does something else such as creating and writing files to the same folder, will there be any conflict? I am talking about the directory needs to be opened by the routine that renames/moves files (deletes entries from directory) and also by the routine that creates and writes files (add entries to directory).

I'm sure if it is Linux, the answer is NO conflicts but since this is an ESP32 (or RTOS) implementation, I am hoping for a NO too.

[PeterR]

e.g. https://stackoverflow.com/questions/424 ... ltaneously
So no, not all concurrent file access are safe. Even if declared as safe then I would take a 'I don't want to debug a potential RT issue' and use a C++11 mutex and sleep at night. Simple, quick, no more than a few 10's of nS performance hit etc.
A FLASH writes takes uS if not mS so your mutex use is trivial.

[zliudr]

PeterR,

Thank you for your help! I just read about mutex (seen the term but never investigated). It seems that I should write a function that accesses the file directory and have mutex in it. Then different tasks shall call the same function when they want access to file directories, such as creating a file, moving a file.

If such thing as mutex already exists, why would a file operation library not do it internally to rid developers from having to do the same at application dev. level? Just curious. I've used MS DOS in the distant past. There is a place where DOS stores a status that says it's running. I think it's in case an interrupt handler also calls DOS when the main routine is interrupted while calling DOS. It should be implemented below the app. dev. level in my opinion.

[PeterR]

When would the filesystem release/set the mutex. On fopen()/fclose() or each fwrite()/fread()?
What happens if you fseek() and then the file is renamed/deleted?

Lots of documentation for the filesystem man. Guess he might just say that you should marshal access to the file system, (his) life is too short.

I cannot rename/delete a file that is open on Windows 10 BTW

Fundamentally a file is a shared resource and so I would always recommend marshelling access (because that works and otherwise I need to understand so much).

& yes wrap you're logging function, log.write(log *log) & log.tidy() can lock the open/close/delete of files, initiate tidying etc and launch and synchronise as many threads as needed to get the job done.

[zliudr]

Sorry I just got back to implementing mutex. Is it wise to just have one mutex for file system access or have multiple, one for each file that could be written/read by different tasks? Maybe the former for simplicity and memory saving?

In case of the single mutex for file system, I can just insert mutexTake() right before any file operation and mutexGive() right after. I'm not sure if file content is buffered beyond fclose() and hope not.

[PeterR]

I don't know the ESP specifics.
I started a responce with a long discussion on the 'C' standards.
Its best sumarised with 'don't, I cannot find a statement on multi threading write/rename' (no threads in C).

So sequence. The hardware will only support one erase/write at the same time so you will not loose much by doing so.
EDIT: One semaphore for the entire file system.

[zliudr]

So far so good. Well, collisions are rare events anyway. I am using a mutex to lock the whole sd card file system. There are only two competing tasks: main one that saves scan code (data) to a file, and some credentials after receiving them (all at random time) and another task that reads the scan code (data) from the file and erase those that it processes successfully. This has worked out nicely. I have a persistence queue of jobs between rebooting and am not worried about conflicts between the two tasks.

For some reason, access to sd card takes order of tens of milliseconds, understandable, if fat or file structures aren't buffered like on a real PC. But, access to spiffs takes seconds for say erasing/renaming files.

My method of updating a file is:

Open data file to read out the first (to be discarded)
Open a temp file
Rread all subsequent lines from data file and write to the temp file.
Close both files.
Erase data file.
Rename temp file as data file.

It takes like 4 seconds on spiffs. Same operations took 40ms on sd cards. I'm using SPI mode on sd cards. Does this sound normal?
Thanks.

[PeterR]

Keep in mind that you are testing on one unit. After 100 hrs you see no problem, great. You then start with 10,000 units. Thats a 100x (per hour) MTBF problem. I would not be quick to ignore the maths.
SPIFFS did/does sucks IMHO. See my other posts. There is/was something wrong in the SPIFFS version I used such that if you increases partition size (maybe file count) then SPIFF performance would degrade exponentially.
There are a few other posts with people worrying about HTTPD performance with SPIFFS. I say use lwip rom fs & did so never debugged SPIFFS. No good for you so why not go raw? I am not sure about SPIFFS but most file systems will not be power fail tolerant. Go raw and design in power fail imunity. Typically you have to pay for a transaction based FS.
ESP NVS suggests good power fail imunity but is clear that its not for big data.
EDIT: (1) Used SPI raw from ESP and no problem. (2) My problems were read only SPIFFS from ESP flash. Keep in mind that any ESP flash write (including SPIFFS) may cause execution delays. Cache fetch is disabled whilst Flash writes are in progress & erase/write takes mS++. A file system may be your worse case as write needs to update both file and file table.

[zliudr]

Thanks Peter. I'll read up on transaction-based file systems. In order to get results of 10K units, I need to get say 100 units out into the world so they can run and occasionally fail. So currently I'm not bothered with the fact that my device will fail under a lot of bad-case scenarios. It needs to be tested and improved.

[PeterR]

Np.
Your use case copies a to b and then deletes a. One might suggest 'why bother' but I think you intend more to happen.
'Independent' and 'compentent' tests are hard to deliver. Test only goes so far. Do the design maths.
Transaction based systems tend to be expensive. FAT is inherently not safe, you risk loosing the entire file system. IMHO raw with a checksum works well. In the old days you would write a 'file' block chain then update a double buffered page table (with checksum). Provided 'file' access is marshelled you can see how that works.
EDIT: Assumes wear leveling at hardware level. Old, old school you would also maitain a 'valid' table and mark blocks on write failure.