ESP-IDF SPIFFS can't find files on ESP32?

[nockieboy]

I'm building a test http server application using the ESP-IDF on my ESP32 and learning C whilst I'm doing it. I've sorted out the http server part, now I'm trying to move on to getting SPIFFS working so I can serve html, css and js files from the ESP32.

SPIFFS is working, as best I can tell - I've used the example SPIFFS code to add the functionality to my project and it has formatted the file system on the ESP32 and initialises just fine, according to the serial console.

I've now tried to upload some files (just .htm, .css and .js files) which I've done as follows:

• Relevant files were placed in a folder (.data)
  I've used mkspiffs to build an image of the file folder:

  Code: Select all

  mkspiffs -c data -b 4096 -p 256 -s 0x100000 spiffs.bin

This has produced a spiffs.bin file, which I have then uploaded to the ESP32 at offset 0xF0000 (according to the custom partition.csv, which is identical to the one in the SPIFFS example project). I have verified the contents of spiffs.bin with a hex editor and it contains the three files it is supposed to.

Code: Select all

python esptool.py --chip esp32 --port COM14 --baud 115200 write_flash -z 0xF0000 spiffs.bin

This seems to work - the script completes without any errors.

The ESP32 is reset and the following code executes - it should say that the files exist, but it says it can't find them!

Code: Select all

void initialise_SPIFFS()
{
    ESP_LOGI(TAG, "Initializing SPIFFS");
    esp_vfs_spiffs_conf_t conf = {
      .base_path = "/spiffs",
      .partition_label = NULL,
      .max_files = 5,
      .format_if_mount_failed = true
    };

    // Use settings defined above to initialize and mount SPIFFS filesystem.
    // Note: esp_vfs_spiffs_register is an all-in-one convenience function.
    esp_err_t ret = esp_vfs_spiffs_register(&conf);

    if (ret != ESP_OK) {
        if (ret == ESP_FAIL) {
            ESP_LOGE(TAG, "Failed to mount or format filesystem");
        } else if (ret == ESP_ERR_NOT_FOUND) {
            ESP_LOGE(TAG, "Failed to find SPIFFS partition");
        } else {
            ESP_LOGE(TAG, "Failed to initialize SPIFFS (%s)", esp_err_to_name(ret));
        }
        return;
    }

    ret = esp_spiffs_info(NULL, &total, &used);
    if (ret != ESP_OK) {
        ESP_LOGE(TAG, "Failed to get SPIFFS partition information (%s)", esp_err_to_name(ret));
    } else {
        // SPIFFS set up a-ok
        SPIFFS_OK = true;
        // Check if files exist
        struct stat st;
        if (stat("/spiffs/dashboard.css", &st) == 0) {
            ESP_LOGI(TAG, "Found dashboard.css!");
        } else {
            ESP_LOGI(TAG, "Can't find dashboard.css!");
        }
        if (stat("/spiffs/index.htm", &st) == 0) {
            ESP_LOGI(TAG, "Found index.htm!");
        } else {
            ESP_LOGI(TAG, "Can't find index.htm!");
        }
        if (stat("/spiffs/scripts.js", &st) == 0) {
            ESP_LOGI(TAG, "Found scripts.js!");
        } else {
            ESP_LOGI(TAG, "Can't find scripts.js!");
        }
    }

}

What am I doing wrong?

[mikemoy]

Did you ever find a solution to this ?

[nockieboy]

No, not yet.

[mikemoy]

I got this working last night. There is allot the docs assume you already know

I am still honing in values but this should get you started as well.

#1) create a "partitions.csv" file in your project folder. This is what I have in there.

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, spiffs, 0x210000 , 1M, 

#2) in menuconfig select Serial flasher config ---> Flash size, and set it for 4MB
(NOTE: I was uploading a large site so you may not need to do this step)

#3) in menuconfig select Partition Table ---> Partition Table (Custom partition table CSV) ---> and choose "Custom partition table CSV" and press select, this should bring you back to Partition Table (Custom partition table CSV). Now scroll one down to "Custom partition CSV file" and press enter, then enter the filename "partitions.csv" and choose ok.

#4) make your project again

#5) Now we need to get the files into it. I believe you already know this step so I used this (change C:/z to your folder where your files are at)

Code: Select all

./mkspiffs.exe -c C:/z -b 4096 -p 256 -s 0x100000 spiffs.bin

Then to upload it:

Code: Select all

python $IDF_PATH/components/esptool_py/esptool/esptool.py --chip esp32 --port COM8 --baud 921600 write_flash -z 0x210000 spiffs.bin

NOTE: the 0x210000 in this line should be the same as in yoru .vsc file.

#6) now flash your board and it should be good now.

FYI, I wanted to list the directory and found this example which does a great job at it.
https://github.com/loboris/ESP32_spiffs ... stSpiffs.c

Now I am off to see how i can stream these file in the HTTP server, and whole other nut to crack.

[nockieboy]

I got this working last night. There is allot the docs assume you already know

I'm finding that a lot with those docs...

I am still honing in values but this should get you started as well.

#1) create a "partitions.csv" file in your project folder. This is what I have in there.

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, spiffs, 0x210000 , 1M, 

Yes, I've been using the partitions.csv file from the ESP-IDF SPIFFS examples in the github repo - seems there's a bit of information
missing from it - the size parameter for the storage partition! That was the issue I was having and why my ESP32 wasn't seeing any
files, because the partitions file wasn't set up properly. Still, you live and learn...

FYI, I wanted to list the directory and found this example which does a great job at it.
https://github.com/loboris/ESP32_spiffs ... stSpiffs.c

Now I am off to see how i can stream these file in the HTTP server, and whole other nut to crack.

Lovely, thanks for the help and the link - will check that out! I'm just glad the ESP32 is detecting the files now. As with you, off to sort out serving files to the browser now!

[mikemoy]

Well, i spoke too soon. Though it lists the files and directories just fine. I can open a file without error, but when i try fread, i get an error.
Also, I found out that you have to (enable) long filename support:
→ Component config → FAT Filesystem support → Long filename support
Then you can select Max long filename length (default is 255)
From this post here. https://esp32.com/viewtopic.php?f=2&t=2651
Sadly this did not help yet.

List of Directory [/spiffs/]
-----------------------------------
T Size Date/Time Name
-----------------------------------
f 121155 03/01/1970 06:36 bootstrap.min.css
f 37045 22/02/1972 19:16 bootstrap.min.js
f 318 18/10/1972 17:45 favicon.ico
f 6486 21/10/1972 18:34 index.html
f 86927 02/12/1972 11:50 jquery-3.3.1.min.js
f 378 14/06/1974 12:43 on-off-switch-onload.js
f 1774 18/06/1974 07:44 on-off-switch.css
f 8128 30/06/1974 11:01 on-off-switch.js
f 320 21/08/1974 19:08 test.txt
-----------------------------------
262531 in 9 file(s)
-----------------------------------
SPIFFS: free 672 KB of 934 KB
-----------------------------------

[nockieboy]

Haven't gotten round to working on serving up files yet, but don't know if this forum post would be of any help?

viewtopic.php?t=4830#p20838

[mikemoy]

Thanks, I have seen that. For some reason even though its printing the directory, I cannot open files yet for reading.

[nockieboy]

You can't open the files, or can't send them to the client?

I've got it all up and running now. If you can't open files, check to make sure the buffer you'd set up was big enough (it was only 1K?
I've modified that to set the buffer to size of the file being read).

Here's my code:

Code: Select all

char *readFile(char *fname)
{
    int size, res;
    char *buf;
    struct stat st;

    if (stat(fname, &st) == 0) {
        size = st.st_size;
    } else {
        ESP_LOGE(TAG, " Error - cannot find file!");
        return NULL;
    }

    ESP_LOGI(TAG, " Opening file %s, size: %d", fname, size);

    FILE *fd = fopen(fname, "rb");
    if (fd == NULL) {
        ESP_LOGE(TAG, " Error opening file");
        return NULL;
    }
    
    if (size == 0) {
        ESP_LOGE(TAG, " Error - file size zero");
        return NULL;
    }

    buf = calloc(size, 1); // File buffer
    if (buf == NULL) {
        ESP_LOGE(TAG, " Error allocating read buffer");
        free(buf);
        fclose(fd);
        return NULL;
    }

    res = size;
    res = fread(buf, 1, size, fd);
    if (res <= 0) {
        ESP_LOGE(TAG, " Error reading from file");
    }
    else {
        ESP_LOGI(TAG, " %d bytes read", res);
        buf[res] = '\0';
    }

    return((char*)buf);

    res = fclose(fd);
    if (res) {
        ESP_LOGE(TAG, " Error closing file");
    }
} 

And for the request handler:

Code: Select all

esp_err_t index_get_handler(httpd_req_t *req)
{
    /* Send response with custom headers and body set as the
     * string passed in user context*/
    // Get the file to return
    const char* resp_str = (const char*) readFile("/spiffs/index.htm");
    // Set the response headers
    httpd_resp_set_hdr(req, "status", "200");
    httpd_resp_set_hdr(req, "content-type", "text/html; charset=UTF-8");
    httpd_resp_set_hdr(req, "server", "ESP32-10103");
    // Return the response
    httpd_resp_send(req, resp_str, strlen(resp_str));
    return ESP_OK;
}

EDIT: That said, I'm getting heap errors if I try to reload the page, so my (very) basic knowledge of C is causing issues - probably because the file buffer isn't getting emptied for each file and/or I'm doing something stupid with pointers or something?

[mikemoy]

You can't open the files

Nope, i get an error. I found that you can embed files into your project. I though this to be a better solution than to upload files, as that would get old if you had to program many devices for production.
I started following down this rabbit hole here but now ran into a issue with adding source directories

https://docs.espressif.com/projects/esp ... irectories

I want to break away from Arduino, but the docs for IDF are frustrating at times.