Reading in large files from spiffs

[lildenty]

Hi Guys.

I'm trying to read in a large file from spiffs so i can use it as an output to a webserver.

My code seems to work fine on small files ie anything under approx 1000 bytes

but soon as its over my esp32 panics and im trying to work out what ive missed out or not thought about.

Code: Select all

static void readTest(char *fname)
{
    FILE* file;
    char *file_buf;
    size_t bytes_read;
    unsigned int file_size = 0;
    char cont_len[8];

    file = fopen(fname, "r");
    if(file != NULL) {
        fseek(file, 0, SEEK_END);
        file_size = ftell(file);
        fseek(file, 0, SEEK_SET);
        sprintf(cont_len, "%d", file_size);
        printf("File size: %d Bytes\n", file_size);

        printf("content-length %s\n", cont_len);

        file_buf = (char *)malloc(file_size);
        if (file_buf == NULL) {
		    printf("Failed to allocate memory\n");
		  
	    }
        bytes_read = fread(file_buf, file_size, 1, file);
        std::string l_output(file_buf,file_size);
        l_output = mspUtils::trim(l_output);
        printf("output:\n");
        printf("%s\n", l_output.c_str());
        printf("#####\n");

        fclose(file);
    } else {
        printf("Failed to open file for sending\n");
    }
}

when i run this on a file this is the output i get

#########################
test reading js file
#########################
File size: 43569 Bytes
content-length 43569

abort() was called at PC 0x40166df7 on core 0

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x40088c20:0x3ffb1dd0 0x40088e6d:0x3ffb1df0 0x40166df7:0x3ffb1e10 0x40166e3e:0x3ffb1e30 0x40167677:0x3ffb1e50 0x4016789a:0x3ffb1e70 0x40173ccd:0x3ffb1e90 0x400d29ef:0x3ffb1eb0 0x400d3a11:0x3ffb1ee0 0x400d42c6:0x3ffb1f20 0x4010679b:0x3ffb2400 0x4008da39:0x3ffb2420

Rebooting...
ets Jun 8 2016 00:22:57

any help or point into the right direction would be great

[lildenty]

This might not be related to reading in files after all.

It seems to be related to the size of a std::string.

as when i come to resize the string.

Code: Select all

size = 43569; 
a_Content->resize(size); 

it dies on the resize part

Code: Select all

Program received signal 
SIGTRAP, Trace/breakpoint trap.
Info : Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x4019141E 
Info : Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x4019141E 

has std::string got a max size i'm not aware off ?

[lildenty]

im not sure if the solution is in

https://esp32.com/viewtopic.php?t=1542

but checking it out

[ESP_Sprite]

If anything, suggest you decode the backtrace, or use the esp.py monitor command to monitor serial output. There's a lot of info in the serial dump you posted, and we can't use any of it because we don't know what your binary looks like. Having the backtrace helps a lot there.

[lildenty]

I've managed to break it down further.

Using this method below. I can read in the file without issue.

Code: Select all

std::streampos size;
  char * memblock;

  std::ifstream file (a_Filename, std::ios::in|std::ios::binary|std::ios::ate);
  if (file.is_open())
  {
    size = file.tellg();
    memblock = new char [size];
    file.seekg (0, std::ios::beg);
    file.read (memblock, size);
    file.close();
    printf("Test Start\n");
    printf(memblock);
    printf("Test End\n");
    printf("STD string size : %d\n",a_Content->max_size());
    a_Content->assign(memblock,size);

    delete[] memblock;
  }

this will print the whole file to the console fine. However soon as i call a_Content->assign(memblock,size); the esp32 panics.
a_Content being std::String

im guessing theres a size limition or something relating to std:string implementation which i dont know about. tho call max_size() kind of indicates it should fit.

[lildenty]

I can cause the panic by calling

std::string l_test1(43569,'x');
std::string l_test2;
l_test2.resize(43569);

it seems i'm not able to have both these variables, only one

guessing memory issue. but i dont understand why ?

I also found out that if i read in the file to a char* then pass it to a stringstream. it truncates it to 8192 (8 x 1024) size. without crashing ? is there a limit in size i can change ?

[lildenty]

think im hitting the limitations of the esp32.

I tried a number of different ways to read this file in.

the latest one. I read it in by line. and store it in a vector<string>
however it only auto resizes to 1024 size and then it truncates the rest of the data.(better than it panics and dies)

when I reserve/set the size of the vector to be what I want 11000 it just dies and panics. it doesnt tell me I cant exceed any number?

any suggestions to fix this?

Code: Select all

std::vector<std::string> l_test;
l_test.reserve(11000);

it dies doing the last line