Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

[rfleming]

Hi All,

I have been running into "bugs" in the FATFS implmentation of the driver. This is directly caused from the way I write to the driver using it in ways that it shouldn't be used. My project requires me to use 1MB of storage and I would prefer not to use FATFS as I have ran into issues with it on roughly 10% of all my units.

Hopefully without redirecting this thread title too much, let me identify my issues so far with the FATFS emulation. Note, this could be bug related, but is quite difficult to fix on my end I believe as there is too many layers between my code and the silicon.
I am happy to send my code directly to the esp-team if required. As this a commerical project I would prefer not to upload it to a public forum.

----
I am afraid I have a few units in field and this just happens after "sometime" (a few weeks of continuous operation) on roughly 10% of ours units. So I cannot quite identify what is the initial trigger to get the units into this state. My guess is that for some reason I have somehow corrupted a file (no idea how) then when I try to write it, I can't so the system creates a new file and continues to write on that one instead.
Specifically what happens is I have a unit that has the below partition table. I am writing to 2 files in the storage section where 1 is a configuration file that only needs to be ~40 bytes and the other is for measurement logging. I am writing to the file in a circular buffer style where it initially fills up the entire partition (with the one file) minus the configuration file. Then once full, starts to overwrite the oldest data at the top of the file. This was my psuedo wear levelling idea to maximise the storage that the esp has.
_Note: If there is an easier way to write to flash directly that would be the prefered option and would likely solve all my issues._

Code: Select all

# Espressif ESP32 Partition Table
# Name, 	Type, 	SubType, 	Offset, 	Size, 		Flags
nvs,      	data,	nvs,     	0x9000,  	0x4000,
otadata,	data,	ota,		0xd000,	0x2000,
phy_init, 	data,	phy,     	0xf000,  	0x1000,
ota_0,	app,		ota_0,	0x10000,	0x190000,
ota_1,	app,		ota_1, 	,		0x190000,
storage,	data,	fat,		,		0xD0000,

After a while I noticed that randomly that data was missing. Doing a directly listing on my unit provided some insite into this issue:

Code: Select all

"ÿÿÿÿÿÿÿÿ.ÿÿÿ", size -2
"ÿÿÿÿÿÿÿÿ.ÿÿÿ", size -2
...repeat ÿÿÿÿÿÿÿÿ.ÿÿÿ 256 times...
"CONFIG.LOG", size 0
"PARTICLE.LOG", size 0

Looking at the structure of the file name, the ÿ is really just 0xFF converted as ASCII and the xxxxxxxx.xxx is the same file name length and structure as PARTICLE.LOG and still contains the dot for file extension, it is safe to assume this is the root of the issue.

First looking into code, obviously built around the c standard, the readdir reads iterates through files, though I found no way to do a stat() on the file that is returned because it requires a file name, that is identical for every entry.

Code: Select all

void Hal_Storage_FatFs::Debug_ListFilesInDir(const char *dir_name) {
	DIR *dir;
	struct dirent *ent;
	if ((dir = opendir(dir_name)) != NULL) {
		/* print all the files and directories within directory */
		while ((ent = readdir(dir)) != NULL) {
			struct stat stat_buf;
			memset(&stat_buf, 0, sizeof(stat_buf));
			long len = 0;
			char path[128];
			strcpy(path, dir_name);
			strcat(path, "/");
			strcat(path, ent->d_name);

			if (stat(path, &stat_buf) == 0) {
				len = stat_buf.st_size;
			}
			else {
				len = -2;  //error. app would likely give 0 or -1, so just do this instead.
			}
			LogD("\"%s\", size %ld", ent->d_name, len);
		}
		closedir(dir);
	}
	else {
		/* could not open directory */
		LogE("FATFS Error couldn't open dir");
	}
}

I dont quite know what happens internally either in my code or in the driver to cause this.

I open the file itself by doing:

Code: Select all

FILE *Hal_Storage_FatFs::OpenFile_ForModifying(const char *file_name) {
	if (access(file_name, F_OK) == -1) {
		//create new file
		LogD("Creating new file.");
		FILE *fileTmp = fopen(file_name, "wb");
		if (fileTmp != NULL)
			fclose(fileTmp);
	}

	//now actually access the file.
	char write_flags[] = "rb+";

	FILE *file = fopen(file_name, write_flags);

	if (file == NULL) {
		LogE("Couldn't open file for modifying '%s'", file_name == NULL ? "NULL" : file_name);
	}
	return file;
}

and call it by doing:

Code: Select all

#define FILE_NAME_DATALOG "/data/particle.log"
this->fileHandle = fatfs->OpenFile_ForModifying(FILE_NAME_DATALOG);

so the file name itself should NEVER change to a 0xFF crazy mess. Nor do I ever delete individual files.

So at the end of the day, it would be a lot simpler to write directly to flash if possible

[ESP_Sprite]

From what I recall, FatFS is only robust to power fails in some configurations. The docs should have more information about this.

It's possible to directly access partitions. You can use the SPI flash api to either do that or write to the entire flash. (Just in case: do make sure you're aware of NOR flash properties wrt wear-out, sector and erase sector sizes etc. Flash without translation layers is quirky if you're used to 'standard' block devices.)

[rfleming]

Thanks for the info Sprite. Our devices have battery backup and spend most of their life asleep. So while it is possible that a few of these units may have these issues triggered from power cycles or similar, its more likely a cause from something else.

I'll have a crack and writing directly to to the SPI flash then. Everytime I saw this in passing previously I always thought it was only for external ICs. I am typically used to NAND flash from the STM32F7 series, AVR series and SAMD21 series. So the ESP may have some quirks I'll have to get used to.

[ESP_Sprite]

Technically, the flash in an ESP32 module is an 'external IC' as in the flash is not on the die itself, so in that aspect it's more-or-less the same as you're used to. The difference is that the ESP32 has some advanced caching that transparently can allow memory access to the chip, so it's not advisable to roll your own SPI access driver for the flash chip, but as long as you use the partition API I pointed out, you should be able to apply your experience without issues.

[rfleming]

Technically, the flash in an ESP32 module is an 'external IC' as in the flash is not on the die itself, so in that aspect it's more-or-less the same as you're used to. The difference is that the ESP32 has some advanced caching that transparently can allow memory access to the chip, so it's not advisable to roll your own SPI access driver for the flash chip, but as long as you use the partition API I pointed out, you should be able to apply your experience without issues.

Perfect!

Since I am using the internal spi flash, can I also utilise the same SPI bus used internally?

That is, currently there are 3 SPI buses:
#define SPI_HOST SPI1_HOST
#define HSPI_HOST SPI2_HOST
#define VSPI_HOST SPI3_HOST

As the first SPI_HOST is typically used for internal flash, it previously was not recommended to use this for external use. So an external IC etc. Though since I am using the internal flash now, is it fine to force this SPI channel to be used for this instead?

Edit: I have a free SPI. Though I dont have any free IOs. I couldn't find any example of using the internal Flash directly. The best I found was in "components/spi_flash/test/test_read_write.c" or
https://github.com/espressif/esp-idf/bl ... ad_write.c

Here a good example that just assumes everything is setup (a lot better for me)

Code: Select all

static void IRAM_ATTR test_read(int src_off, int dst_off, int len)

They simply disable/enable cache when writing and no special considerations are required when reading from any memory location. In terms of the partition itself, as its just a memory location, there is no explicit reason to "mount" it as its not really a file system, just a continuous memory allocation.

Anyway, going to have a crack at this hope it works

[rfleming]

Even better...

examples/storage/partition_api/partition_ops

Here it provides an example how to write/read/erase to a partition. I only a little concerned about how the underlining write operation works though. I'll put this into a new thread as this once has now gone a little out of scope.

[simonjo]

Hi there,

I am recently facing the same issues, with 4 files forming a circular logfile. the active logfile has it's name changed to unreadable chars at random times. This with FATFS+WL. (https://www.esp32.com/viewtopic.php?f=13&t=21091)

After carefully inpecting code to make sure there are no 2 different tasks trying to access the file or any other culprits, I am now trying to spread the file writing more in time to see if this solves the issue. My app writes metering results to the logfile every minute, but different sources are logged with 70ms spacing, which may be too fast!? Just guessing.

At a certain point my app is unable to open the logfile (because it's name is scrambled) and creates a new logfile, thereby loosing all data. So exactly what is described here.

Anyway, while debugging this, I saw the massive erase/write sequences on the wl_flash layer, this is gigantic overhead and wear especially when writing small packets of data (8 bytes/sample).

So now I started to build a FifoPartition class that uses it's own ESP partition and does direct read/write from/to it. I divide the partition in blocks of 16Kb, each block has a signature and a unique monotonic id. When starting to use a block it is erased completely, and the signature+id is written to it. After that bytes that need to be saved are written to the block at the next offset until the block is full.

This way you only write bytes on top of erased space, as opposed to the fat driver which will read the block, erase it and rewrite it with the extra data for each write to the file.

When mounting the partition, each block is scanned (i.e. 64 blocks in a 1MB partition) to determine which block is the start block (lowest id), which is the end block (highest id) and were to start writing.

[rfleming]

Interesting to note...

1. My original file name of "particle.log" eventually turned into "\xff\xff\xff\xff\xff\xff\xff\xff.\xff\xff\xff" which matches the original file name length and even the '.' for the file extension. For whatever reason it only replaced the original alphanumeric characters rather than the file extension dot.

Also after operating my device for over a month, storing data every minute, I ended up with roughly 30 files named like the above. And 1 new file with the target name.

In the end writing directly to the partition has solved all my issues.

[simonjo]

What algo did you use to write directly to the partition, do you maintain begin/end pointers?

regards, Jo

[rfleming]

No algo, if there is one, its pretty custom...

At boot I do a scan through my entire data partition. Each frame that is stored uses binary framing to ensure I know the start and end to each frame. Then within the frames they have a:
- version (to know how to handle the data)
- frame type (to know how to handle the data)
- length (for frame integrity)
- message number (32bit)
- crc (for frame integrity)

my message number when starting from 0 effectively means its almost impossible for the lifetime of the device that it will every overflow from 0xFFFF FFFF -> 0. Thus I can use it to know when I start up. Finally if it were to overflow, I look for the largest gap between the data to know where the start and end were. If the data were to go over the 0xFFF... to 0 bounds, this is still not as big as a gap from '500' to '1,500,000'. Thus I know that is where my start/end pointers are.

Finally when erasing over new boundaries I could suggest something like this vebatim psuedo code

Code: Select all

uint32_t location = 125000;
uint32_t write_size = 500;
if ((write_size % CONFIG_WL_SECTOR_SIZE) > 0) {
	uint32_t new_page_start = location + CONFIG_WL_SECTOR_SIZE;
	new_page_start = new_page_start/CONFIG_WL_SECTOR_SIZE;
	new_page_start = new_page_start*CONFIG_WL_SECTOR_SIZE;
	esp_partition_erase(some_partition, new_page_start, CONFIG_WL_SECTOR_SIZE);
}

//now we can write to those 2 pages.
esp_partition_write(some_partition, location, data, write_size);