Store and read static pages in flash

[nmtrivedi]

Hi,

I am working on ESP32_Core_board_v2. I am developing an application in which I need to develop http web-server which presents static web pages when requested by http client. I followed discussion at http://www.esp32.com/viewtopic.php?t=676 and tried but didn't succeed. Need help regarding what I am missing.

I made SPIFFS image using https://github.com/igrr/mkspiffs tool. That has 3 static pages. The output of ./mkspiffs -l command on the image is:

Code: Select all

9774	/resource/image.png
5251	/resource/index.html
2650	/resource/ajax_script.js

I flashed this image using esptool at 0x130000 address.

The vfs structure is assigned values as below:

Code: Select all

vfs.flags = ESP_VFS_FLAG_DEFAULT;
vfs.write = esp_vfs_write;
vfs.read = esp_vfs_read;
vfs.open = esp_vfs_open;

I registered vfs path using

Code: Select all

esp_vfs_register("/resource", &vfs, NULL);

which returned 0.

Now, I am trying to open index.html file like shown below:

Code: Select all

open("/resource/index.html", O_RDONLY);

I have also tried using fopen to open the file. When I call this function, Guru Meditation Error of type LoadProhibited occurres and the board reboots. Advice on what I am doing wrong?

Thanks!

[ESP_igrr]

You should set open/read/write members of vfs structure to the implementations specific to your filesystem. esp_vfs_xxx functions are used by the upper layer (newlib) to talk to VFS. What _you_ need to provide in vfs structure are functions which would allow VFS to talk to the actual filesystem driver.

You will likely have to write these yourself, or adopt implementations from another project which uses VFS along with SPIFFS (mongoose and Lua-RTOS come to mind as the examples).

[dspau1]

You could do it all inline in the one html file - css and js in their respective head tags, and images can be inlined using base64 encoding:

<img alt="Embedded Image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIA..." />

Base64 will of course increase the file size of the images a bit, so gzip it all to save flash space, just remember to set the "Content-Encoding" response header to "gzip".

This strategy will have performance benefits (lower latency/less server requests) and also simplify your web server code. You don't even have to use the file system if you embed the gziped data as a C byte array in your source code.

The primary consideration is whether this strategy will fit into your existing workflow.

[ESP_Angus]

This strategy will have performance benefits (lower latency/less server requests) and also simplify your web server code. You don't even have to use the file system if you embed the gziped data as a C byte array in your source code.

If you go this way then it's possible to have the build system embed the file as a byte array, rather than manually pre-processing it:
http://esp-idf.readthedocs.io/en/latest ... inary-data

(However I think SPIFFS is a good approach, as per Ivan's suggestions.)

[blavoie]

I've been wanting to try this, but I'm not sure which parameters to use with creating the flash image. I'm also wondering the command line to write it to flash. Would you be able to provide the command line details for creating the image and writing it to flash?

Thanks

[mjmorrison]

This is something I have been wondering as well.

Mkspiffs: https://github.com/igrr/mkspiffs builds and unpacks spiffs image, but I am not sure this is necessary for my use case.

At one point I had the filesystem test by @jwlusby running. Something has broken in the meantime... but it was working without using the mkspiffs tool.

For some context I'm trying to save a stream of sensor data at runtime to be persistent across reboots.

[nmtrivedi]

You could do it all inline in the one html file - css and js in their respective head tags, and images can be inlined using base64 encoding:

<img alt="Embedded Image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIA..." />

Base64 will of course increase the file size of the images a bit, so gzip it all to save flash space, just remember to set the "Content-Encoding" response header to "gzip".

This strategy will have performance benefits (lower latency/less server requests) and also simplify your web server code. You don't even have to use the file system if you embed the gziped data as a C byte array in your source code.

The primary consideration is whether this strategy will fit into your existing workflow.

Thanks @dspau1. This solved my problem.

[mzincali]

Sorry to reopen an old thread. I have an HTML block that is about 7,000bytes. I have it declared as a `const char[]` in my response function to can HTTP GET. When I start to add more functionality to this block (in the form of more JS or more HTML, i get a stack overflow. If I move the html block into the global scope (outside of my function), the stack overflow goes away. I assume that is because the latter case has the HTML stored in my function's stack and the latter case the HTML is stored in my global DRAM space, which doesn't have the limitations of a fixed size stack.

Having this HTML in the global space means that it will be taking up that much space from my DRAM, all the time. Since this HTML is used very very rarely, I hate allocating so much DRAM to it. My original idea was that as a `const char[]`it was going to be kept in the flash with the rest of the application, and only copied into the String that I read it into, at runtime, if and when that function was utilized. And after the function exited, the store would be freed. That's not what seems to be happening, since I am getting a stack overflow.

I've tried to use F() around my string, and also PROGMEM, but neither of them seemed to work. I am now leaning towards learning about SPIFFS, but the problem there might be that OTA updates get more complex as I would have to write the HTML file to the SPIFFS after the OTA as a separate network download.

Anyone have ideas that would help? I really expect there to be a way to say, "don't load the string from the application in flash into the stack space. Let it be accessed directly from flash and be copied into heap DRAM."

(PROGMEM seems to be a no-op on ESP32:

#define PROGMEM
#define PGM_P const char *
#define PGM_VOID_P const void *
#define PSTR(s) (s)
#define _SFR_BYTE(n) (n)

#define pgm_read_byte(addr) (*(const unsigned char *)(addr))
…

#define memcmp_P memcmp
#define memccpy_P memccpy
…

[ESP_igrr]

Does this array need to be writable? If not, you can declare it as "static const char[]". This will result in the array placed into constant data region (rodata) in Flash. You can then work with this variable directly without copying the contents of the array onto the stack.

If you do need it to be writable, then you have to manually allocate space in heap (via "malloc") and then copy the original const array into the newly allocated heap region (via memcpy or strlcpy). Finally, release the heap memory using "free".

[ESP_Sprite]

Note that the architecture of the ESP32 chips is different from the Atmel ones in Arduino; the ESP32 uses an unified address space for both flash and RAM. As such, something marked as '[static] const char' doesn't need to be copied to RAM if you only use it in a read-only fashion. (It's also why macros like PROGMEM and F() don't really do anything, even when they're available; there's no real need for them.)