Store static file in flash

[mjmorrison]

Hi, thanks for the quick reply.

It comes from right here: https://github.com/whitecatboard/Lua-RT ... ffs.h#L228

I think this was a build environment error on my end! Will continue getting spiffs working.

Ran into another question, posting here for completeness sake:

Is there a distinction between "mounting" and "registering" the filesystem? I see the spiffs_vfs snippet references registering the filesystem at a "mount point" but the spiffs.h library mentions mounting and not registering.

[kolban]

In the Unix file system, there is only one hierarchical tree starting at "/". Within that directory tree space, we could "cross into" different file system implementations by "mounting" different file systems at different paths in the directory tree. When we traverse into those paths, we switch into distinct file systems. The Unix command is "mount" to achieve this. The place where the file system is "grafted" onto the single directory tree is termed a "mount point". In the VFS architecture, one appears to use an API called "esp_vfs_register" to specify a "registration point" in the file system where the transition from one VFS to another takes place. I consider this a "mount point" but I suspect that is just loose language.

[mjmorrison]

RE: Register v. Mount

For reference: spiffs_mount appears in the Lua RTOS port here https://github.com/whitecatboard/Lua-RT ... ffs.h#L382

and esp_vfs_register appears in the esp-idf here: https://github.com/espressif/esp-idf/bl ... vfs.h#L131

I don't think they are explicitly related. I'm looking for some clarity about when each of these pieces should happen.
Intuitively I think the order should be Configure > Mount > Register > Enjoy, but not sure as the addition of a driver interface complicates things.


RE: The problem at hand

I'm following along this "Integrate SPIFFS" guide https://github.com/pellepl/spiffs/wiki/ ... ing-spiffs. Stuck at the point where you have to provide the read, write, erase handles to SPIFFS.

Lookin' at setting up the spiffs_config structure like so:

Code: Select all

	config.hal_read_f = *low_spiffs_read;
	config.hal_write_f = *low_spiffs_write;
	config.hal_erase_f = *low_spiffs_erase;

Giving the config structure the proper handles found here: https://github.com/whitecatboard/Lua-RT ... p_spiffs.h

It looks like the compiler is finding the functions correctly, but I'm getting some errors:

main/./../components/spiffs/esp_spiffs.c: In function 's32_t esp32_spi_flash_read(u32_t, u32_t, u8_t*)':
main/./../components/spiffs/esp_spiffs.c:59:26: error: invalid conversion from 'void*' to 'u8_t* {aka unsigned char*}' [-fpermissive]
buff = malloc(asize + 4);
^
main/./../components/spiffs/esp_spiffs.c: In function 's32_t esp32_spi_flash_write(u32_t, u32_t, const u8_t*)':
main/./../components/spiffs/esp_spiffs.c:103:26: error: invalid conversion from 'void*' to 'u8_t* {aka unsigned char*}' [-fpermissive]
buff = malloc(asize + 4);

Do you think this is an issue with the Lua RTOS port? I remember you said that you had this code working at one point.

[mithrandiir42]

I had a similar issue at first. I had to change the definition of those functions and drop the 'const' in order for it to compile.

int esp32_spi_flash_read(unsigned int addr, unsigned int size, unsigned char *dst);
int esp32_spi_flash_write(unsigned int addr, unsigned int size, unsigned char *src);
int esp32_spi_flash_erase(unsigned int addr, unsigned int size);

However, I still don't have things working yet so it's very possible that I just masked the issue and it didn't actually get me any closer:)


My current issue is that when I open a file, write some text, and read it back, I don't get anything back and it seems as if the buffer pointer passed to fread(...) doesn't make it to vfs_read(...) intact.

Here's some debug output:

(4500) test_memory_vfs(): Mounting SPIFFS...
page index byte len: 256
object lookup pages: 1
page pages per block: 16
page header length: 5
object header index entries: 105
object index entries: 124
available file descriptors: 3
free blocks: 64
(4521) test_memory_vfs(): result=0
(4526) spiffs_registerVFS: esp_vfs_register: OK (0)
(4532) test_memory_vfs(): vfs registered
(4537) vfs_open: >> open path=/myfile, flags=0x601, accessMode=0x1B6
free_obj_id: BITM min:0001 max:0201
create: found free page @ 0001 bix:0 entry:0
open: fd 1 is obj id 8001
(4553) test_memory_vfs(): Opened file '/data/myfile'. file=0x3FFB58C4
(4561) vfs_fstat: >> fstat fd=1, st=0x3FFB7DA0
(4566) test_memory_vfs(): Wrote 6 bytes to file from buffer at 0x3FFB7EDF
(4573) vfs_write: >> write fd=1, data=0x3FFB8C44, size=6
(4580) vfs_close: >> close fd=1
append: 6 bytes @ offs 0 of size -1
append: 8001 load objixhdr page 0001:0000
append: 8001 store new data page, 0002:0000 offset:0, len 6, written 0
append: 8001 wrote page 0002 to objix_hdr entry 00 in mem
append: 8001 store fresh objix_hdr page, 0001:0000, written 6
callback: setting fd 1:8001 objix_hdr_pix to 0001, size:6
callback: setting fd 1:8001 span:0000 objix_pix to 0001
(4619) test_memory_vfs(): fclose=0
(4623) vfs_open: >> open path=/myfile, flags=0x0, accessMode=0x1B6
open: fd 1 is obj id 8001
(4632) test_memory_vfs(): Attempting to read 6 bytes from file 0x3FFB58C4 into buffer at 0x3FFB8C44.
(4642) vfs_fstat: >> fstat fd=1, st=0x3FFB7DE0
(4648) vfs_read: >> read ctx=3FFB1E90, fd=1, dst=0x3FFB8C8C, size=128
(4655) test_memory_vfs(): Read 0 bytes from file 0x3FFB58C4 into buffer at 0x3FFB8C44: '',
(4664) vfs_close: >> close fd=1

Also, here's my repo on github with this code in case it's helpful.
https://github.com/jwlusby/esp32_testing

[ESP_igrr]

C standard library does some buffering for you, it reads into a buffer stored in the FILE structure, and then copies data into the buffer you have passed to fread. Now why the data doesn't get read, I cannot say. Can you please add some debugging to see how many bytes SPIFFS_read gives you?

[mithrandiir42]

Ah that makes sense about the buffer pointers then. Thanks for the suggestion. I'll do that when I get back to my desk.

I do know that if I read/write files directly with the SPIFFS api, things work as expected. Things stop working when I go through the VFS functions. So it is likely going to turn out to be something simple.

[mithrandiir42]

I figured out my issue. After adding some debug print statements, I found that SPIFFS_read was failing with SPIFFS_ERR_NOT_READABLE and that got me looking at the flags used when opening the file. I was using "r" (read only) initially. I tried "r+" and that worked.

That made me realize there was a bug in the code for vfs_open(...) where since the O_RDONLY was defined as 0, the if statement checking that flag was always resulting in false.

Code: Select all

    
	if (flags & O_RDONLY) {
		spiffsFlags |= SPIFFS_RDONLY;
	}

So the whole function after fixing that is now:

Code: Select all

static int vfs_open(void *ctx, char *path, int flags, int accessMode) {
	static char tag[] = "vfs_open";
	ESP_LOGI(tag, ">> open path=%s, flags=0x%x, accessMode=0x%X", path, flags, accessMode);
	logFlags(flags);
	spiffs *fs = (spiffs *)ctx;
	int spiffsFlags = 0;
    int rw = (flags & 3);

    if (rw == O_RDONLY || rw == O_RDWR) {
    	spiffsFlags |= SPIFFS_RDONLY;
    }

    if (rw == O_WRONLY || rw == O_RDWR) {
    	spiffsFlags |= SPIFFS_WRONLY;
    }

    if (flags & O_CREAT) {
    	spiffsFlags |= SPIFFS_CREAT;
    }

    if (flags & O_TRUNC) {
    	spiffsFlags |= SPIFFS_TRUNC;
    }

    if (flags & O_APPEND) {
    	spiffsFlags |= SPIFFS_APPEND;
    }
	int rc = SPIFFS_open(fs, path, spiffsFlags, accessMode);
	return rc;
} // vfs_open

[mjmorrison]

Hey all,

I'm trying to set up some read/ write functions for the SPIFFS vfs example that yall have been working on. Here https://github.com/jwlusby/esp32_testing.

I've tried rolling back the arduino repo to the previous commit referenced in github, but when I try to compile, it throws some compilation errors about some SPI related library.

I've done some hacking around on my own and arrived here, with these functions being called from the main loop.

Code: Select all

#include <stdio.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"

#include "Arduino.h"

#include "esp_wifi.h"
#include "esp_system.h"
#include "esp_event.h"
#include "esp_event_loop.h"

#include "esp_log.h"

#include "nvs_flash.h"
#include "driver/gpio.h"

#include "esp_spi_flash.h"
#include "esp_partition.h"
#include "spiffs.h"
#include "esp_spiffs.h"
#include "spiffs_vfs.h"


//0x10000 + (1024*512) + (1024*512) + (1024*512) to hex
#define NVS_PARTITION_START     0x9000
#define NVS_PARTITION_SIZE      0x6000
#define NVS_PHY_START           0xF000
#define NVS_PHY_SIZE            0x1000
#define FACTORY_PARTITION_START 0x10000
#define FACTORY_PARTITION_SIZE  1024*512

#define OTA1_PARTITION_START  FACTORY_PARTITION_START + FACTORY_PARTITION_SIZE
#define OTA1_PARTITION_SIZE   FACTORY_PARTITION_SIZE
#define OTA2_PARTITION_START  OTA1_PARTITION_START + OTA1_PARTITION_SIZE
#define OTA2_PARTITION_SIZE   FACTORY_PARTITION_SIZE
#define DATA_PARTITION_START  OTA2_PARTITION_START + OTA2_PARTITION_SIZE
#define DATA_PARTITION_SIZE   1024*256

#define SECTOR_SIZE             4096
#define PAGE_SIZE               256
#define MEMORY_SIZE             DATA_PARTITION_SIZE
#define MAX_CONCURRENT_FDS      4

static uint8_t spiffs_work_buf[PAGE_SIZE * 2];
static uint8_t spiffs_fds[32 * MAX_CONCURRENT_FDS];
static uint8_t spiffs_cache_buf[(PAGE_SIZE + 32) * 4];
static spiffs fs;
static spiffs_config cfg;

static uint32_t g_wbuf[SECTOR_SIZE / 4]; // is this necessary?
static uint32_t g_rbuf[SECTOR_SIZE / 4];


bool storage_initialized = false;

int read_pos;
int write_pos;

const char* filename = "/data/customer1";

void configure_spiffs(){
	// configure the filesystem
	cfg.phys_size = MEMORY_SIZE;
	cfg.phys_addr = DATA_PARTITION_START;
	cfg.phys_erase_block = SECTOR_SIZE;
	cfg.log_block_size = SECTOR_SIZE;
	cfg.log_page_size = PAGE_SIZE;
	cfg.hal_read_f = esp32_spi_flash_read;
	cfg.hal_write_f = esp32_spi_flash_write;
	cfg.hal_erase_f = esp32_spi_flash_erase;
}

//Test file read/write using VFS component & spiffs (fopen/fread/fwrite)
void mount_format_spiffs()
{
	char tag[] = "test_memory_vfs()";
	int ret;

	configure_spiffs();

	// mount the filesystem
	ESP_LOGI(tag, "Mounting SPIFFS");
	ret = SPIFFS_mount(&fs, &cfg, spiffs_work_buf, spiffs_fds, sizeof(spiffs_fds), spiffs_cache_buf, sizeof(spiffs_cache_buf), NULL);
	if (ret != SPIFFS_OK )
	{
		ESP_LOGI(tag, "Mounting SPIFFS failed with result=%i", ret);
		
		ESP_LOGI(tag, "Unmounting SPIFFS...");
		SPIFFS_unmount(&fs);

		ESP_LOGI(tag, "Formatting SPIFFS...");
		ret = SPIFFS_format(&fs);
		ESP_LOGI(tag, "   result=%i", ret);

		ESP_LOGI(tag, "Remounting SPIFFS...");
		ret = SPIFFS_mount(&fs, &cfg, spiffs_work_buf, spiffs_fds, sizeof(spiffs_fds), spiffs_cache_buf, sizeof(spiffs_cache_buf), NULL);
		ESP_LOGI(tag, "   result=%i", ret);
	}
	else if (ret == SPIFFS_OK)
	{
		ESP_LOGI(tag, "SPIFFS mounted");
	}

	spiffs_registerVFS("/data", &fs);
	ESP_LOGI(tag, "vfs registered");

	storage_initialized = true;
}

void test_file_read( const char* filename )
{
	char tag[] = "read_spiffs";
	int nbytes;
	FILE* file = fopen( filename, "r" );

	if(file <= NULL)
	{
		ESP_LOGE(tag, "Failed to open file!");
		return;
	}
	else
	{
		// create pointer to empty buf size 64, is 64 arbitrary
		char *buf;
		buf = (char*)malloc(64);
		memset(buf, 0, 64);

		// ESP_LOGI(tag, "Attempting to read 6 bytes from file 0x%X into buffer at 0x%X.", (uint32_t)file, (uint32_t)buf);
		nbytes = fread(buf,1,6,file);
		ESP_LOGI(tag, "Read %i bytes from file 0x%X into buffer at 0x%X: '%s', ", nbytes, (uint32_t)file, (uint32_t)buf, buf);

		// cleanup
		fclose(file);
		free(buf);
	}
}

void test_file_write( const char* filename, void* data )
{
	char tag[] = "test_file_write";
	int ret;
	FILE* file = fopen("data/customer1", "r");

	// move file head to end of the file
	// ret = fseek(file, write_pos, SEEK_END);
	// ESP_LOGI(tag, "fseek=%i", ret);

	if (file <= NULL)
	{
		ESP_LOGE(tag, "failed to open file '%s'. result=0x%X",filename, (uint32_t)file);
		return;
	}
	else
	{
		ESP_LOGI(tag, "Opened file '%s'. file=0x%X", filename, (uint32_t)file);

		int nbytes = fwrite( data, sizeof( struct reading ), 1, file);
		write_pos = write_pos + nbytes;

		if(nbytes==-1)
		{
			ESP_LOGE(tag, "Failed to write to file. result=%i", nbytes);
		}

		else
		{
			ESP_LOGI(tag, "Wrote %i bytes to file from buffer at 0x%X", nbytes, (uint32_t)data);

			ret = fclose(file);
			ESP_LOGI(tag, "fclose=%i", ret);
		}
	}
}

When I try to open the file, im getting "failed to open file." I'll keep truckin for a couple more days on this but any advice would be much appreciated!

[SpenZerX]

Hello all,

i included SPIFFS in my app.
Inspired by some sources:
https://github.com/jwlusby/esp32_testing

TEST 1:
spiffs format - OK
new file create - Ok
read of that file - Ok

TEST 2:
spiffs image creation by makefile with files - ok
spiffs image upload by makefile - ok
no spiffs format this time
new file 1 create - Ok
read file 2 from created and uploaded spiffs - NOT OK -> Open OK (OK, file is there!), but always reads 0 bytes ...
read of that file 1 - ok

So the problem is:
- i can not read from files that are initially included in the uploaded image.
- file open passed ok (only if requested file is there -> so i think the image is there)
- file read ALWAYS 0 BYTES,

Any advise, hint...?

Used settings:
cfg.phys_size = 1024*512;
cfg.phys_erase_block = 4096;
cfg.log_block_size = 4096;
cfg.log_page_size = 256;

mkspiffs -c filesystem -b 4096-p 256 -s 524288

[aaquilina]

Have you managed to proceed further with your program by any chance ? I'm able to write and read to my raw spiffs implementation too but I cannot find the file I uploaded to the flash using the esptoool.py.

Hello all,

i included SPIFFS in my app.
Inspired by some sources:
https://github.com/jwlusby/esp32_testing

TEST 1:
spiffs format - OK
new file create - Ok
read of that file - Ok

TEST 2:
spiffs image creation by makefile with files - ok
spiffs image upload by makefile - ok
no spiffs format this time
new file 1 create - Ok
read file 2 from created and uploaded spiffs - NOT OK -> Open OK (OK, file is there!), but always reads 0 bytes ...
read of that file 1 - ok

So the problem is:
- i can not read from files that are initially included in the uploaded image.
- file open passed ok (only if requested file is there -> so i think the image is there)
- file read ALWAYS 0 BYTES,

Any advise, hint...?

Used settings:
cfg.phys_size = 1024*512;
cfg.phys_erase_block = 4096;
cfg.log_block_size = 4096;
cfg.log_page_size = 256;

mkspiffs -c filesystem -b 4096-p 256 -s 524288