sppiffs using up a lot of space

[akbarhash]

I want to write upto 8000 records into spiffs when there is no wifi and send the data when it connects to wifi.
I have allocated 2 MB to SPIFSS, each record will be max of 256 bytes, it should support around 8192 records according to my calculations.
I am creating a new file for each record like spiffs/1, spiffs/2, spiffs/3 etc... then when it connects i am sending the data from 1, 2, 3 and so on.
The problem is the spiffs files are using up much more space than what is required.
For example a 24 52byte files are using 12048 bytes. That comes out to 502 bytes for each file.
I am only able to write around 4000 files to spiffs folder each containing ~52 bytes before the space runs out.
Is there some mistake I am making? Also if there is a better way to do it?
I am using this to save the GPS co-ordinates while I move around and when I come home it should upload the data to the server.
I am using the standard 4MB ESP32 device.

[WiFive]

What is your config SPIFFS_PAGE_SIZE?

[akbarhash]

Currently using default settings
SPIFFS_PAGE_SIZE = 256
SPIFFS_OBJ_NAME_LEN = 32
SPIFFS_MAX_PARTITIONS = 3
SPIFFS_CACHE enabled
SPIFFS_CACHE_WR enabled

[WiFive]

Hence, if a file is created being one byte big, it will occupy one page for index and one page for data - it will occupy 2 x size of a logical page on flash.

https://github.com/pellepl/spiffs/blob/ ... C#L58-L123

[bobtidey]

Why not use 1 file for all the records to maximise space efficiency?

Decide on your maximum record size.
Block in file is Length (4 bytes) + max size.
Seek to record position (n x (4 + max Size)
Read or write length + data

[akbarhash]

Why not use 1 file for all the records to maximise space efficiency?

Decide on your maximum record size.
Block in file is Length (4 bytes) + max size.
Seek to record position (n x (4 + max Size)
Read or write length + data

Will try this method as well.

Tried to put sixty records in a file and it worked ok. I am able to get upto six thousand records of 256 bytes each in 2 MB memory. I think there is a maximum file size for spiffs after 100 records the write to file gets slow still investigating.

[akbarhash]

Please try this code.
main.c:

Code: Select all

#include <stdio.h>
#include <string.h>
#include <sys/unistd.h>
#include <sys/stat.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_err.h"
#include "esp_log.h"
#include "esp_spiffs.h"
#include "nvs.h"
#include "nvs_flash.h"


static char *payload = "$Lorem ipsum dolor sit amet, consectetur adipiscing elit, sedsequat interdum varius sit amet mattis vulputate. Ac turpis egestas sed tempus urna et pharetra pharetra. At auctor urna nunc id. Proin nibh nisl condimentum id venenatis a condimentum vitae.*";

static const char *NVSTAG = "nvs";
static const char *SPIFFSTAG = "spiffs";
#define STORAGE_NAMESPACE "storage"

/* Save total message count value in NVS
   Return an error if anything goes wrong
   during this process.
 */
esp_err_t save_total_count(uint16_t total_count)
{
	nvs_handle my_handle;
	esp_err_t err;

	// Open
	err = nvs_open(STORAGE_NAMESPACE, NVS_READWRITE, &my_handle);
	if (err != ESP_OK)
		return err;

	// Write value
	err = nvs_set_u16(my_handle, "total_count", total_count);
	if (err != ESP_OK)
		return err;
	// Commit written value.
	// After setting any values, nvs_commit() must be called to ensure changes are written
	// to flash storage. Implementations may write to storage at other times,
	// but this is not guaranteed.
	err = nvs_commit(my_handle);
	if (err != ESP_OK)
		return err;

	// Close
	nvs_close(my_handle);
	return ESP_OK;
}

/* Get total message count value in NVS
   Return an error if anything goes wrong
   during this process.
 */
esp_err_t get_total_count(uint16_t* total_count)
{
	nvs_handle my_handle;
	esp_err_t err;

	// Open
	err = nvs_open(STORAGE_NAMESPACE, NVS_READWRITE, &my_handle);
	if (err != ESP_OK)
		return err;

	// Read value
	err = nvs_get_u16(my_handle, "total_count", total_count);
    if (err != ESP_OK && err != ESP_ERR_NVS_NOT_FOUND) return err;

	// Close
	nvs_close(my_handle);
	return ESP_OK;
}

esp_err_t writetofile(char* textbuffer, uint16_t len, uint16_t index)
{
    FILE* f = NULL;
    ESP_LOGI(SPIFFSTAG, "Writing to file at index %d", index);
    f = fopen("/spiffs/hello.txt", "a");
    if (f == NULL) {
        ESP_LOGE(SPIFFSTAG, "Failed to open file for writing");
        return ESP_FAIL;
    }       
    
    char* tp = textbuffer;
    while(*tp != '\0') {
        fputc(*tp++,f);
    }
    for(int i = len; i < 256; i++)
        fputc(32,f);

    fclose(f);
    return ESP_OK;
}

esp_err_t readfromfile(char* textbuffer, uint16_t index)
{
    FILE* f = NULL;

    ESP_LOGI(SPIFFSTAG, "Reading from file at index %d", index);
    f = fopen("/spiffs/hello.txt", "r");
    if (f == NULL) {
        ESP_LOGE(SPIFFSTAG, "Failed to open file for reading");
        return ESP_FAIL;
    }       
    fseek(f, 256*index, SEEK_SET);
    //Read the File One Line At A Time
    fgets(textbuffer, 256, f);
    // strip newline
    char* pos = strchr(textbuffer, '\n');
    if (pos) {
        *(pos+1) = '\0';
    }
    ESP_LOGI(SPIFFSTAG, "Read from file: '%s'", textbuffer);
    fclose(f);
    return ESP_OK;
}

void app_main(void)
{
    ESP_LOGI(SPIFFSTAG, "Initializing SPIFFS");
    
    esp_vfs_spiffs_conf_t conf = {
      .base_path = "/spiffs",
      .partition_label = NULL,
      .max_files = 5,
      .format_if_mount_failed = true
    };
    
    // Use settings defined above to initialize and mount SPIFFS filesystem.
    // Note: esp_vfs_spiffs_register is an all-in-one convenience function.
    esp_err_t ret = esp_vfs_spiffs_register(&conf);

    if (ret != ESP_OK) {
        if (ret == ESP_FAIL) {
            ESP_LOGE(SPIFFSTAG, "Failed to mount or format filesystem");
        } else if (ret == ESP_ERR_NOT_FOUND) {
            ESP_LOGE(SPIFFSTAG, "Failed to find SPIFFS partition");
        } else {
            ESP_LOGE(SPIFFSTAG, "Failed to initialize SPIFFS (%s)", esp_err_to_name(ret));
        }
        return;
    }

    size_t total = 0, used = 0;
    ret = esp_spiffs_info(NULL, &total, &used);
    if (ret != ESP_OK) {
        ESP_LOGE(SPIFFSTAG, "Failed to get SPIFFS partition information (%s)", esp_err_to_name(ret));
    } else {
        ESP_LOGI(SPIFFSTAG, "Partition size: total: %d, used: %d", total, used);
    }

    //nvs initilization
    uint16_t totalindex = 0;
    esp_err_t err = nvs_flash_init(); 
    if (err != ESP_OK){
	ESP_ERROR_CHECK(nvs_flash_erase());
	ESP_ERROR_CHECK(nvs_flash_init());
    }
    if (get_total_count(&totalindex)!= ESP_OK) {
        ESP_LOGE(NVSTAG, "Failed to read Total Count from NVS");
    }
    ESP_LOGI(NVSTAG, "Total index is %d", totalindex);
    
    char rx_buffer[256];
    uint8_t linetoread = 200;
    if(totalindex>linetoread)
        readfromfile(rx_buffer, linetoread);

    while(writetofile(payload, strlen(payload), totalindex) == ESP_OK)
        save_total_count(++totalindex);

    // All done, unmount partition and disable SPIFFS
    esp_vfs_spiffs_unregister(NULL);
    ESP_LOGI(SPIFFSTAG, "SPIFFS unmounted");
    for (int i = 10; i >= 0; i--) {
        printf("Restarting in %d seconds...\n", i);
        vTaskDelay(1000 / portTICK_PERIOD_MS);
    }
    printf("Restarting now.\n");
    fflush(stdout);
    esp_restart();
}

partitions.csv:

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, spiffs,  ,        2M,

Complete Logs: Warning Heavy:

Code: Select all

I (30) boot: ESP-IDF v4.0-dev-1095-g16014079f 2nd stage bootloader
I (31) boot: compile time 11:42:02
I (39) boot: Enabling RNG early entropy source...
I (39) boot: SPI Speed      : 40MHz
I (41) boot: SPI Mode       : DIO
I (45) boot: SPI Flash Size : 4MB
I (49) boot: Partition Table:
I (52) boot: ## Label            Usage          Type ST Offset   Length
I (60) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (67) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (75) boot:  2 factory          factory app      00 00 00010000 00100000
I (82) boot:  3 storage          Unknown data     01 82 00110000 00200000
I (90) boot: End of partition table
I (94) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x07fd8 ( 32728) map
I (114) esp_image: segment 1: paddr=0x00018000 vaddr=0x3ffb0000 size=0x02018 (  8216) load
I (118) esp_image: segment 2: paddr=0x0001a020 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /home/akbarhash/esp/esp-idf/components/freertos/xtensa_vectors.S:1778

I (123) esp_image: segment 3: paddr=0x0001a428 vaddr=0x40080400 size=0x05be8 ( 23528) load
I (141) esp_image: segment 4: paddr=0x00020018 vaddr=0x400d0018 size=0x1d648 (120392) map
0x400d0018: _stext at ??:?

I (184) esp_image: segment 5: paddr=0x0003d668 vaddr=0x40085fe8 size=0x038c0 ( 14528) load
0x40085fe8: disableAllWdts at /home/akbarhash/esp/esp-idf/components/esp32/panic.c:428
 (inlined by) panicHandler at /home/akbarhash/esp/esp-idf/components/esp32/panic.c:311

I (197) boot: Loaded app from partition at offset 0x10000
I (197) boot: Disabling RNG early entropy source...
I (197) cpu_start: Pro cpu up.
I (201) cpu_start: Application information:
I (206) cpu_start: Project name:     spiffs
I (211) cpu_start: App version:      1
I (215) cpu_start: Compile time:     Sep 13 2019 11:42:05
I (221) cpu_start: ELF file SHA256:  5f99404fe50a7358...
I (227) cpu_start: ESP-IDF:          v4.0-dev-1095-g16014079f
I (234) cpu_start: Starting app cpu, entry point is 0x400810e4
0x400810e4: call_start_cpu1 at /home/akbarhash/esp/esp-idf/components/esp32/cpu_start.c:279

I (0) cpu_start: App cpu up.
I (244) heap_init: Initializing. RAM available for dynamic allocation:
I (251) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (257) heap_init: At 3FFB3078 len 0002CF88 (179 KiB): DRAM
I (263) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (270) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (276) heap_init: At 400898A8 len 00016758 (89 KiB): IRAM
I (282) cpu_start: Pro cpu start user code
I (300) spi_flash: detected chip: generic
I (301) spi_flash: flash io: dio
I (301) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (309) spiffs: Initializing SPIFFS
W (319) SPIFFS: mount failed, -10025. formatting...
E (5349) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (5349) task_wdt:  - IDLE0 (CPU 0)
E (5349) task_wdt: Tasks currently running:
E (5349) task_wdt: CPU 0: main
E (5349) task_wdt: CPU 1: IDLE1
E (10319) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (10349) task_wdt:  - IDLE0 (CPU 0)
E (10349) task_wdt: Tasks currently running:
E (10349) task_wdt: CPU 0: main
E (10349) task_wdt: CPU 1: IDLE1
E (15349) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (15379) task_wdt:  - IDLE0 (CPU 0)
E (15379) task_wdt: Tasks currently running:
E (15379) task_wdt: CPU 0: main
E (15379) task_wdt: CPU 1: IDLE1
E (20379) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (20379) task_wdt:  - IDLE0 (CPU 0)
E (20379) task_wdt: Tasks currently running:
E (20379) task_wdt: CPU 0: main
E (20379) task_wdt: CPU 1: IDLE1
I (20589) spiffs: Partition size: total: 1920401, used: 0
I (20609) nvs: Total index is 0
I (20609) spiffs: Writing to file at index 0
I (20819) spiffs: Writing to file at index 1
I (20869) spiffs: Writing to file at index 2
I (20909) spiffs: Writing to file at index 3
I (20959) spiffs: Writing to file at index 4
I (20999) spiffs: Writing to file at index 5
I (21049) spiffs: Writing to file at index 6
I (21099) spiffs: Writing to file at index 7
I (21139) spiffs: Writing to file at index 8
I (21189) spiffs: Writing to file at index 9
I (21239) spiffs: Writing to file at index 10
I (21279) spiffs: Writing to file at index 11
I (21329) spiffs: Writing to file at index 12
I (21369) spiffs: Writing to file at index 13
I (21419) spiffs: Writing to file at index 14
I (21469) spiffs: Writing to file at index 15
I (21509) spiffs: Writing to file at index 16
I (21559) spiffs: Writing to file at index 17
I (21609) spiffs: Writing to file at index 18
I (21649) spiffs: Writing to file at index 19
I (21699) spiffs: Writing to file at index 20
I (21739) spiffs: Writing to file at index 21
I (21789) spiffs: Writing to file at index 22
I (21839) spiffs: Writing to file at index 23
I (21879) spiffs: Writing to file at index 24
I (21929) spiffs: Writing to file at index 25
I (21979) spiffs: Writing to file at index 26
I (22019) spiffs: Writing to file at index 27
I (22069) spiffs: Writing to file at index 28
I (22109) spiffs: Writing to file at index 29
I (22159) spiffs: Writing to file at index 30
I (22209) spiffs: Writing to file at index 31
I (22249) spiffs: Writing to file at index 32
I (22299) spiffs: Writing to file at index 33
I (22349) spiffs: Writing to file at index 34
I (22389) spiffs: Writing to file at index 35
I (22439) spiffs: Writing to file at index 36
I (22479) spiffs: Writing to file at index 37
I (22529) spiffs: Writing to file at index 38
I (22579) spiffs: Writing to file at index 39
I (22619) spiffs: Writing to file at index 40
I (22669) spiffs: Writing to file at index 41
I (22719) spiffs: Writing to file at index 42
I (22759) spiffs: Writing to file at index 43
I (22809) spiffs: Writing to file at index 44
I (22859) spiffs: Writing to file at index 45
I (22899) spiffs: Writing to file at index 46
I (22949) spiffs: Writing to file at index 47
I (22989) spiffs: Writing to file at index 48
I (23039) spiffs: Writing to file at index 49
I (23089) spiffs: Writing to file at index 50
I (23139) spiffs: Writing to file at index 51
I (23179) spiffs: Writing to file at index 52
I (23229) spiffs: Writing to file at index 53
I (23279) spiffs: Writing to file at index 54
I (23319) spiffs: Writing to file at index 55
I (23369) spiffs: Writing to file at index 56
I (23409) spiffs: Writing to file at index 57
I (23459) spiffs: Writing to file at index 58
I (23509) spiffs: Writing to file at index 59
I (23549) spiffs: Writing to file at index 60
I (23599) spiffs: Writing to file at index 61
I (23649) spiffs: Writing to file at index 62
I (23689) spiffs: Writing to file at index 63
I (23739) spiffs: Writing to file at index 64
I (23779) spiffs: Writing to file at index 65
I (23829) spiffs: Writing to file at index 66
I (23879) spiffs: Writing to file at index 67
I (23919) spiffs: Writing to file at index 68
I (23969) spiffs: Writing to file at index 69
I (24009) spiffs: Writing to file at index 70
I (24059) spiffs: Writing to file at index 71
I (24109) spiffs: Writing to file at index 72
I (24149) spiffs: Writing to file at index 73
I (24199) spiffs: Writing to file at index 74
I (24239) spiffs: Writing to file at index 75
I (24289) spiffs: Writing to file at index 76
I (24339) spiffs: Writing to file at index 77
I (24379) spiffs: Writing to file at index 78
I (24429) spiffs: Writing to file at index 79
I (24479) spiffs: Writing to file at index 80
I (24519) spiffs: Writing to file at index 81
I (24569) spiffs: Writing to file at index 82
I (24609) spiffs: Writing to file at index 83
I (24659) spiffs: Writing to file at index 84
I (24709) spiffs: Writing to file at index 85
I (24749) spiffs: Writing to file at index 86
I (24799) spiffs: Writing to file at index 87
I (24849) spiffs: Writing to file at index 88
I (24889) spiffs: Writing to file at index 89
I (24939) spiffs: Writing to file at index 90
I (24989) spiffs: Writing to file at index 91
I (25029) spiffs: Writing to file at index 92
I (25079) spiffs: Writing to file at index 93
I (25119) spiffs: Writing to file at index 94
I (25169) spiffs: Writing to file at index 95
I (25219) spiffs: Writing to file at index 96
I (25259) spiffs: Writing to file at index 97
I (25309) spiffs: Writing to file at index 98
I (25359) spiffs: Writing to file at index 99
truncated
I (146349) spiffs: Writing to file at index 1000
I (146489) spiffs: Writing to file at index 1001
I (146629) spiffs: Writing to file at index 1002
I (146759) spiffs: Writing to file at index 1003
I (146899) spiffs: Writing to file at index 1004
I (147029) spiffs: Writing to file at index 1005
I (147169) spiffs: Writing to file at index 1006
I (147299) spiffs: Writing to file at index 1007
I (147479) spiffs: Writing to file at index 1008
I (147609) spiffs: Writing to file at index 1009
I (147749) spiffs: Writing to file at index 1010
truncated
I (288459) spiffs: Writing to file at index 2000
I (288759) spiffs: Writing to file at index 2001
I (288879) spiffs: Writing to file at index 2002
I (289179) spiffs: Writing to file at index 2003
I (289299) spiffs: Writing to file at index 2004
I (289419) spiffs: Writing to file at index 2005
I (289719) spiffs: Writing to file at index 2006
I (289839) spiffs: Writing to file at index 2007
I (289959) spiffs: Writing to file at index 2008
I (290279) spiffs: Writing to file at index 2009
I (290389) spiffs: Writing to file at index 2010
truncated
I (489789) spiffs: Writing to file at index 3000
I (489919) spiffs: Writing to file at index 3001
I (490049) spiffs: Writing to file at index 3002
I (490349) spiffs: Writing to file at index 3003
I (490479) spiffs: Writing to file at index 3004
I (490789) spiffs: Writing to file at index 3005
I (490919) spiffs: Writing to file at index 3006
I (491049) spiffs: Writing to file at index 3007
I (491359) spiffs: Writing to file at index 3008
I (491489) spiffs: Writing to file at index 3009
I (491799) spiffs: Writing to file at index 3010
truncated
I (759499) spiffs: Writing to file at index 4000
I (759929) spiffs: Writing to file at index 4001
I (760069) spiffs: Writing to file at index 4002
I (760509) spiffs: Writing to file at index 4003
I (760969) spiffs: Writing to file at index 4004
I (761499) spiffs: Writing to file at index 4005
I (761959) spiffs: Writing to file at index 4006
I (762409) spiffs: Writing to file at index 4007
I (762859) spiffs: Writing to file at index 4008
I (762959) spiffs: Writing to file at index 4009
I (763409) spiffs: Writing to file at index 4010
truncated
I (1773529) spiffs: Writing to file at index 5000
I (1777959) spiffs: Writing to file at index 5001
I (1780999) spiffs: Writing to file at index 5002
I (1783359) spiffs: Writing to file at index 5003
I (1789689) spiffs: Writing to file at index 5004
I (1796559) spiffs: Writing to file at index 5005
I (1800039) spiffs: Writing to file at index 5006
I (1806619) spiffs: Writing to file at index 5007
I (1809839) spiffs: Writing to file at index 5008
I (1815049) spiffs: Writing to file at index 5009
I (1818869) spiffs: Writing to file at index 5010
I (1826499) spiffs: Writing to file at index 5011
I (1831629) spiffs: Writing to file at index 5012
I (1835539) spiffs: Writing to file at index 5013
I (1840499) spiffs: Writing to file at index 5014
I (1847049) spiffs: Writing to file at index 5015
I (1853019) spiffs: Writing to file at index 5016
I (1856339) spiffs: Writing to file at index 5017
I (1862179) spiffs: Writing to file at index 5018
I (1869239) spiffs: Writing to file at index 5019
I (1874399) spiffs: Writing to file at index 5020
I (1879029) spiffs: Writing to file at index 5021
I (1887829) spiffs: Writing to file at index 5022
I (1892019) spiffs: Writing to file at index 5023
I (1897929) spiffs: Writing to file at index 5024
I (1903599) spiffs: Writing to file at index 5025
I (1908059) spiffs: Writing to file at index 5026
I (1912939) spiffs: Writing to file at index 5027
I (1917389) spiffs: Writing to file at index 5028
I (1925759) spiffs: Writing to file at index 5029
I (1931749) spiffs: Writing to file at index 5030
I (1935409) spiffs: Writing to file at index 5031
I (1943139) spiffs: Writing to file at index 5032
I (1946959) spiffs: Writing to file at index 5033
I (1954529) spiffs: Writing to file at index 5034
I (1959759) spiffs: Writing to file at index 5035
I (1962819) spiffs: Writing to file at index 5036
I (1969079) spiffs: Writing to file at index 5037
I (1973949) spiffs: Writing to file at index 5038
I (1982129) spiffs: Writing to file at index 5039
I (1986499) spiffs: Writing to file at index 5040
I (1994589) spiffs: Writing to file at index 5041
I (1999519) spiffs: Writing to file at index 5042
I (2003529) spiffs: Writing to file at index 5043
I (2009169) spiffs: Writing to file at index 5044
I (2012139) spiffs: Writing to file at index 5045
I (2017949) spiffs: Writing to file at index 5046
I (2021609) spiffs: Writing to file at index 5047
I (2028109) spiffs: Writing to file at index 5048
I (2033329) spiffs: Writing to file at index 5049
I (2039239) spiffs: Writing to file at index 5050
I (2046079) spiffs: Writing to file at index 5051
I (2053979) spiffs: Writing to file at index 5052
I (2063409) spiffs: Writing to file at index 5053
I (2072789) spiffs: Writing to file at index 5054
I (2081299) spiffs: Writing to file at index 5055
I (2086669) spiffs: Writing to file at index 5056
I (2092419) spiffs: Writing to file at index 5057
I (2094609) spiffs: Writing to file at index 5058
I (2101159) spiffs: Writing to file at index 5059
I (2106129) spiffs: Writing to file at index 5060
I (2109349) spiffs: Writing to file at index 5061
I (2115609) spiffs: Writing to file at index 5062
I (2119369) spiffs: Writing to file at index 5063
I (2124599) spiffs: Writing to file at index 5064
I (2130109) spiffs: Writing to file at index 5065
I (2135749) spiffs: Writing to file at index 5066
I (2138799) spiffs: Writing to file at index 5067
I (2140119) spiffs: Writing to file at index 5068
I (2147069) spiffs: Writing to file at index 5069
I (2149509) spiffs: Writing to file at index 5070
I (2155779) spiffs: Writing to file at index 5071
I (2164129) spiffs: Writing to file at index 5072
I (2169609) spiffs: Writing to file at index 5073
I (2176329) spiffs: Writing to file at index 5074
I (2182769) spiffs: Writing to file at index 5075
I (2190169) spiffs: Writing to file at index 5076
I (2198099) spiffs: Writing to file at index 5077
I (2203299) spiffs: Writing to file at index 5078
I (2211809) spiffs: Writing to file at index 5079
I (2217949) spiffs: Writing to file at index 5080
I (2220979) spiffs: Writing to file at index 5081
I (2227409) spiffs: Writing to file at index 5082
I (2233499) spiffs: Writing to file at index 5083
I (2236979) spiffs: Writing to file at index 5084
I (2244609) spiffs: Writing to file at index 5085
I (2248109) spiffs: Writing to file at index 5086
I (2253659) spiffs: Writing to file at index 5087
I (2259049) spiffs: Writing to file at index 5088
I (2267149) spiffs: Writing to file at index 5089
I (2275199) spiffs: Writing to file at index 5090
I (2279189) spiffs: Writing to file at index 5091
I (2283229) spiffs: Writing to file at index 5092
I (2288089) spiffs: Writing to file at index 5093
I (2294129) spiffs: Writing to file at index 5094
I (2300589) spiffs: Writing to file at index 5095
I (2307729) spiffs: Writing to file at index 5096
I (2310019) spiffs: Writing to file at index 5097
I (2316169) spiffs: Writing to file at index 5098
I (2320939) spiffs: Writing to file at index 5099
I (2326949) spiffs: Writing to file at index 5100
I (2334439) spiffs: Writing to file at index 5101
I (2340889) spiffs: Writing to file at index 5102
I (2346719) spiffs: Writing to file at index 5103
I (2353849) spiffs: Writing to file at index 5104
I (2356479) spiffs: Writing to file at index 5105
I (2364229) spiffs: Writing to file at index 5106
I (2372689) spiffs: Writing to file at index 5107
I (2380939) spiffs: Writing to file at index 5108
I (2389809) spiffs: Writing to file at index 5109
I (2395899) spiffs: Writing to file at index 5110
I (2402759) spiffs: Writing to file at index 5111
I (2409889) spiffs: Writing to file at index 5112
I (2414339) spiffs: Writing to file at index 5113
I (2418949) spiffs: Writing to file at index 5114
I (2423469) spiffs: Writing to file at index 5115
I (2429879) spiffs: Writing to file at index 5116
I (2431039) spiffs: Writing to file at index 5117
I (2435809) spiffs: Writing to file at index 5118
I (2441539) spiffs: Writing to file at index 5119
I (2445979) spiffs: Writing to file at index 5120
I (2454429) spiffs: Writing to file at index 5121
I (2462139) spiffs: Writing to file at index 5122
I (2465449) spiffs: Writing to file at index 5123
I (2469359) spiffs: Writing to file at index 5124
I (2473949) spiffs: Writing to file at index 5125
I (2478569) spiffs: Writing to file at index 5126
I (2481179) spiffs: Writing to file at index 5127
I (2483099) spiffs: Writing to file at index 5128
I (2492349) spiffs: Writing to file at index 5129
I (2497719) spiffs: Writing to file at index 5130
I (2505289) spiffs: Writing to file at index 5131
I (2512969) spiffs: Writing to file at index 5132
I (2521539) spiffs: Writing to file at index 5133
I (2530159) spiffs: Writing to file at index 5134
I (2538899) spiffs: Writing to file at index 5135
I (2547359) spiffs: Writing to file at index 5136
I (2555689) spiffs: Writing to file at index 5137
I (2562099) spiffs: Writing to file at index 5138
I (2566619) spiffs: Writing to file at index 5139
I (2570889) spiffs: Writing to file at index 5140
I (2576649) spiffs: Writing to file at index 5141
I (2584849) spiffs: Writing to file at index 5142
I (2593269) spiffs: Writing to file at index 5143
I (2598539) spiffs: Writing to file at index 5144
I (2603309) spiffs: Writing to file at index 5145
I (2607979) spiffs: Writing to file at index 5146
I (2616579) spiffs: Writing to file at index 5147
I (2624649) spiffs: Writing to file at index 5148
I (2632279) spiffs: Writing to file at index 5149
I (2640439) spiffs: Writing to file at index 5150
I (2644159) spiffs: Writing to file at index 5151
I (2652319) spiffs: Writing to file at index 5152
I (2658909) spiffs: Writing to file at index 5153
I (2667339) spiffs: Writing to file at index 5154
I (2671909) spiffs: Writing to file at index 5155
I (2679549) spiffs: Writing to file at index 5156
I (2682839) spiffs: Writing to file at index 5157
I (2686839) spiffs: Writing to file at index 5158
I (2693709) spiffs: Writing to file at index 5159
I (2697109) spiffs: Writing to file at index 5160
I (2702579) spiffs: Writing to file at index 5161
I (2705629) spiffs: Writing to file at index 5162
I (2714509) spiffs: Writing to file at index 5163
I (2721189) spiffs: Writing to file at index 5164
I (2726859) spiffs: Writing to file at index 5165
I (2732129) spiffs: Writing to file at index 5166
I (2736419) spiffs: Writing to file at index 5167
I (2743309) spiffs: Writing to file at index 5168
I (2751569) spiffs: Writing to file at index 5169
I (2755819) spiffs: Writing to file at index 5170
I (2758189) spiffs: Writing to file at index 5171
I (2761329) spiffs: Writing to file at index 5172
I (2763599) spiffs: Writing to file at index 5173
I (2769969) spiffs: Writing to file at index 5174
I (2775869) spiffs: Writing to file at index 5175
I (2781529) spiffs: Writing to file at index 5176
I (2789569) spiffs: Writing to file at index 5177
I (2796469) spiffs: Writing to file at index 5178
I (2805309) spiffs: Writing to file at index 5179
I (2813459) spiffs: Writing to file at index 5180
I (2821809) spiffs: Writing to file at index 5181
I (2826899) spiffs: Writing to file at index 5182
I (2829939) spiffs: Writing to file at index 5183
I (2832819) spiffs: Writing to file at index 5184
I (2840499) spiffs: Writing to file at index 5185
I (2848089) spiffs: Writing to file at index 5186
I (2855539) spiffs: Writing to file at index 5187
I (2860909) spiffs: Writing to file at index 5188
I (2868389) spiffs: Writing to file at index 5189
I (2874269) spiffs: Writing to file at index 5190
I (2878149) spiffs: Writing to file at index 5191
I (2884589) spiffs: Writing to file at index 5192
I (2893029) spiffs: Writing to file at index 5193
I (2901639) spiffs: Writing to file at index 5194
I (2910719) spiffs: Writing to file at index 5195
I (2919329) spiffs: Writing to file at index 5196
I (2928079) spiffs: Writing to file at index 5197
I (2936889) spiffs: Writing to file at index 5198
I (2945779) spiffs: Writing to file at index 5199
I (2954569) spiffs: Writing to file at index 5200
I (2962989) spiffs: Writing to file at index 5201
I (2971329) spiffs: Writing to file at index 5202
I (2979249) spiffs: Writing to file at index 5203
I (2986639) spiffs: Writing to file at index 5204
I (2994909) spiffs: Writing to file at index 5205
I (3001049) spiffs: Writing to file at index 5206
I (3006609) spiffs: Writing to file at index 5207
I (3011039) spiffs: Writing to file at index 5208
I (3017309) spiffs: Writing to file at index 5209
I (3025769) spiffs: Writing to file at index 5210
I (3030719) spiffs: Writing to file at index 5211
I (3036229) spiffs: Writing to file at index 5212
I (3043759) spiffs: Writing to file at index 5213
I (3048629) spiffs: Writing to file at index 5214
I (3053319) spiffs: Writing to file at index 5215
I (3056289) spiffs: Writing to file at index 5216
I (3064119) spiffs: Writing to file at index 5217
I (3071859) spiffs: Writing to file at index 5218
I (3079989) spiffs: Writing to file at index 5219
I (3087319) spiffs: Writing to file at index 5220
I (3094949) spiffs: Writing to file at index 5221
I (3098069) spiffs: Writing to file at index 5222
I (3100939) spiffs: Writing to file at index 5223
I (3103909) spiffs: Writing to file at index 5224
I (3107979) spiffs: Writing to file at index 5225
I (3111709) spiffs: Writing to file at index 5226
I (3114149) spiffs: Writing to file at index 5227
I (3120409) spiffs: Writing to file at index 5228
I (3129599) spiffs: Writing to file at index 5229
I (3138359) spiffs: Writing to file at index 5230
I (3146849) spiffs: Writing to file at index 5231
I (3153139) spiffs: Writing to file at index 5232
I (3155669) spiffs: Writing to file at index 5233
I (3164429) spiffs: Writing to file at index 5234
I (3173039) spiffs: Writing to file at index 5235
I (3181669) spiffs: Writing to file at index 5236
I (3186599) spiffs: Writing to file at index 5237
I (3192169) spiffs: Writing to file at index 5238
I (3200499) spiffs: Writing to file at index 5239
I (3208609) spiffs: Writing to file at index 5240
I (3216459) spiffs: Writing to file at index 5241
I (3225019) spiffs: Writing to file at index 5242
I (3233599) spiffs: Writing to file at index 5243
I (3242299) spiffs: Writing to file at index 5244
I (3251209) spiffs: Writing to file at index 5245
I (3260149) spiffs: Writing to file at index 5246
I (3264739) spiffs: Writing to file at index 5247
I (3272829) spiffs: Writing to file at index 5248
I (3280429) spiffs: Writing to file at index 5249
I (3288119) spiffs: Writing to file at index 5250
I (3296259) spiffs: Writing to file at index 5251
I (3304529) spiffs: Writing to file at index 5252
I (3312429) spiffs: Writing to file at index 5253
I (3320809) spiffs: Writing to file at index 5254
I (3329159) spiffs: Writing to file at index 5255
I (3337629) spiffs: Writing to file at index 5256
I (3346499) spiffs: Writing to file at index 5257
I (3355619) spiffs: Writing to file at index 5258
I (3364669) spiffs: Writing to file at index 5259
I (3374099) spiffs: Writing to file at index 5260
I (3379829) spiffs: Writing to file at index 5261
I (3388569) spiffs: Writing to file at index 5262
I (3389939) spiffs: Writing to file at index 5263
I (3395219) spiffs: Writing to file at index 5264
I (3400159) spiffs: Writing to file at index 5265
I (3406389) spiffs: Writing to file at index 5266
I (3414769) spiffs: Writing to file at index 5267
I (3424209) spiffs: Writing to file at index 5268
I (3427029) spiffs: Writing to file at index 5269
I (3435259) spiffs: Writing to file at index 5270
I (3443179) spiffs: Writing to file at index 5271
I (3446059) spiffs: Writing to file at index 5272
E (3446059) spiffs: Failed to open file for writing
I (3446059) spiffs: SPIFFS unmounted
Restarting in 10 seconds...
Restarting in 9 seconds...
Restarting in 8 seconds...
Restarting in 7 seconds...
Restarting in 6 seconds...
Restarting in 5 seconds...
Restarting in 4 seconds...
Restarting in 3 seconds...
Restarting in 2 seconds...
Restarting in 1 seconds...
Restarting in 0 seconds...
Restarting now.
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:6468
load:0x40078000,len:11652
ho 0 tail 12 room 4
load:0x40080400,len:6632
entry 0x40080768
I (31) boot: ESP-IDF v4.0-dev-1095-g16014079f 2nd stage bootloader
I (31) boot: compile time 11:42:02
I (39) boot: Enabling RNG early entropy source...
I (40) boot: SPI Speed      : 40MHz
I (41) boot: SPI Mode       : DIO
I (45) boot: SPI Flash Size : 4MB
I (49) boot: Partition Table:
I (53) boot: ## Label            Usage          Type ST Offset   Length
I (60) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (68) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (75) boot:  2 factory          factory app      00 00 00010000 00100000
I (82) boot:  3 storage          Unknown data     01 82 00110000 00200000
I (90) boot: End of partition table
I (94) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x07fd8 ( 32728) map
I (115) esp_image: segment 1: paddr=0x00018000 vaddr=0x3ffb0000 size=0x02018 (  8216) load
I (118) esp_image: segment 2: paddr=0x0001a020 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /home/akbarhash/esp/esp-idf/components/freertos/xtensa_vectors.S:1778

I (123) esp_image: segment 3: paddr=0x0001a428 vaddr=0x40080400 size=0x05be8 ( 23528) load
I (142) esp_image: segment 4: paddr=0x00020018 vaddr=0x400d0018 size=0x1d648 (120392) map
0x400d0018: _stext at ??:?

I (185) esp_image: segment 5: paddr=0x0003d668 vaddr=0x40085fe8 size=0x038c0 ( 14528) load
0x40085fe8: disableAllWdts at /home/akbarhash/esp/esp-idf/components/esp32/panic.c:428
 (inlined by) panicHandler at /home/akbarhash/esp/esp-idf/components/esp32/panic.c:311

I (197) boot: Loaded app from partition at offset 0x10000
I (197) boot: Disabling RNG early entropy source...
I (198) cpu_start: Pro cpu up.
I (201) cpu_start: Application information:
I (206) cpu_start: Project name:     spiffs
I (211) cpu_start: App version:      1
I (215) cpu_start: Compile time:     Sep 13 2019 11:42:05
I (221) cpu_start: ELF file SHA256:  5f99404fe50a7358...
I (227) cpu_start: ESP-IDF:          v4.0-dev-1095-g16014079f
I (234) cpu_start: Starting app cpu, entry point is 0x400810e4
0x400810e4: call_start_cpu1 at /home/akbarhash/esp/esp-idf/components/esp32/cpu_start.c:279

I (225) cpu_start: App cpu up.
I (244) heap_init: Initializing. RAM available for dynamic allocation:
I (251) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (257) heap_init: At 3FFB3078 len 0002CF88 (179 KiB): DRAM
I (264) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (270) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (276) heap_init: At 400898A8 len 00016758 (89 KiB): IRAM
I (283) cpu_start: Pro cpu start user code
I (301) spi_flash: detected chip: generic
I (301) spi_flash: flash io: dio
I (302) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (310) spiffs: Initializing SPIFFS
I (510) spiffs: Partition size: total: 1920401, used: 1358663
I (520) nvs: Total index is 5272
I (520) spiffs: Reading from file at index 200
I (610) spiffs: Read from file: '$Lorem ipsum dolor sit amet, consectetur adipiscing elit, sedsequat interdum varius sit amet mattis vulputate. Ac turpis egestas sed tempus urna et pharetra pharetra. At auctor urna nunc id. Proin nibh nisl condimentum id venenatis a condimentum vitae.*  '
I (620) spiffs: Writing to file at index 5272
I (5760) spiffs: Writing to file at index 5273
I (7380) spiffs: Writing to file at index 5274
W (7540) SPIFFS: Failed to update mtime (-10001)
I (10000) spiffs: Writing to file at index 5275
I (17440) spiffs: Writing to file at index 5276
I (21800) spiffs: Writing to file at index 5277
I (29750) spiffs: Writing to file at index 5278
I (38060) spiffs: Writing to file at index 5279
I (46760) spiffs: Writing to file at index 5280
I (51490) spiffs: Writing to file at index 5281
I (53230) spiffs: Writing to file at index 5282
E (53230) spiffs: Failed to open file for writing
I (53230) spiffs: SPIFFS unmounted
Restarting in 10 seconds...
Restarting in 9 seconds...
Restarting in 8 seconds...
Restarting in 7 seconds...
Restarting in 6 seconds...
Restarting in 5 seconds...
Restarting in 4 seconds...
Restarting in 3 seconds...
Restarting in 2 seconds...
Restarting in 1 seconds...
Restarting in 0 seconds...
Restarting now.

Doesn't work for more than ~5200 records of 256 bytes each.
Default settings for spiffs. Also note time taken to write one record and used space.

[WiFive]

https://github.com/pellepl/spiffs/issue ... t-94305664

[bobtidey]

There does seem to be an issue with SPIFFS slowing down appending onto large files and ultimately failing.

So maybe the thing to do is to have a series of files each capable of holding say 256 records. Indexing would still be pretty straightforward as the high byte of the index would select the file and the low byte the position within the file.

That way you would get the benefit of efficient storage and still keep the speed and reliability up.

[akbarhash]

There does seem to be an issue with SPIFFS slowing down appending onto large files and ultimately failing.
So maybe the thing to do is to have a series of files each capable of holding say 256 records. Indexing would still be pretty straightforward as the high byte of the index would select the file and the low byte the position within the file.
That way you would get the benefit of efficient storage and still keep the speed and reliability up.

https://github.com/pellepl/spiffs/issue ... t-94305664

Changing to 256 records per file with multiple smaller files doesn't do much. There seems to be a hard limit as discussed in the github thread linked by WiFive where manually writing to a 2 MB partition shows

Code: Select all

I (20589) spiffs: Partition size: total: 1920401, used: 0

when formatted and this

Code: Select all

I (510) spiffs: Partition size: total: 1920401, used: 1358663

when it shows that roughly 30% is still free, but infact it is already full and the rest of the place has been occupied by the metadata.

Thanks for the help.

I will leave this here for someone trying something similar. To get good spiffs performance settle with using around half the storage spiffs has been allocated, or look into different options. For example if you plan to use 1MB then allocate 2MB. After 1MB of data has been written the performance seems to degrade. Source: timestamp log attached in my previous comment.

Currently testing writing directly to flash as suggested in this thread viewtopic.php?f=2&t=5070.