Data as a resource in flash, esp32-s3

Slinisa
Posts: 32
Joined: Sat Oct 07, 2023 8:21 am

Data as a resource in flash, esp32-s3

Postby Slinisa » Wed Dec 06, 2023 8:13 am

At the moment I'm storing the data as a int array, e.g.:
uint32_t data[16000]={
0x0C0C0C0C, 0x1415140C, 0x0C0C0C0C ...

and later use it in code:
uint32_t *ptr=data;

My question is if it is possible to store somehow binary data in flash as a resource and then fetch it as needed? The way I use it now is cumbersome and it takes a long time to compile the sketch when the data exceeds dozens of KB (few minutes on my computer), since all the data is embedded within a sketch as text. Besides it occupies RAM.
The original data is in binary format anyways, so at the moment I use a script that I wrote that reads binary data and saves it as a text file which I then copy/paste into a sketch which then gets over 200 kB for only one set of data. It's a nightmare to compile.

Edit: I forgot to mention that I'd rather skip using filesystem.

lbernstone
Posts: 829
Joined: Mon Jul 22, 2019 3:20 pm

Re: Data as a resource in flash, esp32-s3

Postby lbernstone » Wed Dec 06, 2023 8:37 am

Absolutely. If you include an array as a const in esp32, it will be memory mapped so that it is read directly from ROM inline in your code. You can use `xxd -i` to convert a binary file into a C-style and then include it in a header. You can script this to include a whole directory or structure (eg. a web framework) and the server.on() functions to deliver those assets.
Here's an example where I am storing an image file as a constant for delivery in web pages.

Slinisa
Posts: 32
Joined: Sat Oct 07, 2023 8:21 am

Re: Data as a resource in flash, esp32-s3

Postby Slinisa » Wed Dec 06, 2023 12:46 pm

At first I thought that this will not reduce compile time, only put the data into flash instead of RAM.
But, for some reason, it also reduces compile time significantly. I still don't understand why it's the case, since it also needs to compile a huge text file full of data.
Anyway, I put the data into separate file, even doubled it (just to see the difference) and compilation still takes under a minute.
Thanks for the suggestion, it works perfectly.

lbernstone
Posts: 829
Joined: Mon Jul 22, 2019 3:20 pm

Re: Data as a resource in flash, esp32-s3

Postby lbernstone » Wed Dec 06, 2023 4:24 pm

It's a win-win. You also get the advantage of DMA transfers if you are moving the data to a device that can accept that.

Who is online

Users browsing this forum: No registered users and 135 guests