[Q] user/custom space addr room on the SPI Flash for data

[rudi ;-)]

[no hurry]

hi guys

sry for this,
but i am not sure for this, have we or have we not.
edit: read just in time
edit: and the doc
Once a full user API is in place for partition access, these limitations will be resolved and you'll be able to use the partition mechanism fully for storing data in flash.


is there a small info about the "new" spi flash organice on the ESP32, where we can store small data on the spi flash?
i try just in time to store small pic data and other binary for the web srv.

for the esp31 have take in jan 2016

Code: Select all

#define APKNAME 	"ESP32AP.apk"	         // name of Android APK
#define APKADDR 	0x12000			// ADDR of Android APK on SPI Flash
#define APKSIZE 	138412			// size of Android APK  ( 138.412 kb )

#define DONEPIC 	"done.jpg"               // name of picture
#define DONEADDR	0x35000                   // ADDR of picture on SPI Flash
#define DONESIZE	36569                      // size of picture in Bytes

this was ok for ESP31

now i have done the same on ESP32 without Files on Flash,
but now i try to build in two files on System SPI Flash

recall me on this for ESP32
bootloader : 0x 1000
partitions_singleapp bin: 0x 4000
app bin : 0x 10000

we can use the space after partitions_singleapp.bin and app.bin ?
are there stored data, keys?

my second question is,
for data transfer over wifi, how much can we setup the highest size of unsigned char VAR for this?
my doing is just in time to split each file bigger then 1024 and send 1024 packets.

thank you

best wishes
rudi

[ESP_Sprite]

Our plan for that is that eventually, you can do something like:

Code: Select all

	int partitionType=0x12, partitionSubtype=0x34;
	size_t size=partition_size(partitionType, partitionSubtype);
	char *partData=partition_mmap(partitionType, partitionSubtype, 0, size);
	for (int x=0; x<size; x++) printf("Byte %d of the partition is %hhx", x, partData[x]);

but we're not there yet.

[jmattsson]

I'm already using the partition table to establish a SPIFFS partition in NodeMCU; You might find that code a useful reference. It's in the NodeMCU repo, on the WIP esp32 branch. Once there's an official Espressif API, we'll probably start using that. Wasn't going to wait for that though

[ESP_Sprite]

Sure. I see you also have code to add partitions. Be advised we at Espressif meant the partitioning scheme to be static, as in: it will describe the flash layout during the lifetime of a device and messing with the table is only something that happens in very specific circumstances. Not sure what will happen when you rewrite the table on the fly. Is there any specific reason why you have that code?

[jmattsson]

@ESP_Sprite Yes, there is. NodeMCU is designed to handle being flashed to different sizes of flash chips and Do The Right Thing automatically. As such we don't at firmware build-time know what the size of the file system will be (or necessarily even where it will be). On the ESP8266 we use a combination of linker symbols, flash probing and heuristics to determine where we can safely place the file system. That approach doesn't fly on the ESP32 since there are unused pages within the individual partitions, and with the different packing format of the app in flash using linker symbols to find the end of the application becomes rather iffy (tried that on the 31B). Besides, clearly the clean approach is to have an actual designated partition for the file system. That really only left me with the option of dynamically adding the filesystem partition on first boot, which is the first time we know what size flash we're dealing with. Of course, if a user has explicitly entered a partition for the file system in a custom csv and build using that, then NodeMCU honors that and doesn't mess with the partitions.

TL;DR: We're being user/newbie-friendly.

In general I agree completely - don't mess with the partition table! You might've noticed that there's no code for modifying existing entries...

[ESP_Sprite]

Hmm, interesting point. I'll see if I can take it along in the development process of the partition API.

[rudi ;-)]

ah...interesting

txs guys

noted btw
we have an update 2016.09.28 in technical documents

ESP32 Technical Reference Manual targets application developers.
The manual provides detailed and complete information on how to use the ESP32 memory and peripherals.


visit the ESP32 resource

best wishes
rudi

[rudi ;-)]

hi guys

ok - pic/data requests runs like a dream ( so fast ! ) from flash.

one question is start on the thinking about the addresses:

we have
Single factory app, no OTA
0x01000 for the bootloader
0x04000 for the partion table
0x10000 for the user app
0xEOFAPP+1000hh for user data/resource

what is on 0x00000 ?
or what comes to 0x00000 ?

can we combine the files to an user "onefile app" with on top addressed things and other user can flash as combine file on 0x00000 without problems?

have tested this here, and it seams, that the thing goes in perfection. but not sure, is there a reserved space on "users" flash, that is then override ( "DeviceMasterKey", "other keys"..) in space of 0x00000 - 0x01000 (second bootloader on spi flash). so i ask.

thank you!
best wishes
rudi

[ESP_Sprite]

The flash layout is only constrained by a few things, mostly the ROM and the bootloader.

The ROM has these constraints:
The first 4K of flash is reserved for secure-boot purposes. The very first program in flash (normally the 2nd bootloader) should start at 0x1000 and be in the app format the ROM bootloader can load.

The 2nd bootloader has these additional constraints:
At flash address 0x4000, there should be a partition table. This partition table should define at least one app which can be loaded and booted.

Furthermore, libraries in ESP-IDF can have constraints (e.g. NVS), but they mostly come down to needing a partition that is described in the partition table, but can be located anywhere in flash.

[ESP_igrr]

Hi @jmattsson,
We've been discussing how we can implement mutable partition table, while making sure we don't accidentally corrupt it (due to sudden power off, for example). Trying to balance some requirements like bootloader complexity, reliability, and so on, we came up with the following.

- In addition to the main partition table, we allow applications to define extended partition table.
- Extended partition table occupies one sector anywhere in flash memory and has the same format as the main partition table.
- Extended partition table is listed as a partition of some special type in the main partition table.
- Second stage bootloader does not read or parse extended partition table, so all bootable applications should be present in the main partition table.
- Partition API layer in ESP-IDF parses extended partition table, and populates internal list of partitions with items found therein.
- Partition API has a function to append items to the extended partition table.
- No effort is made to make sure that partition table is not corrupted due to sudden power off. If the partition table is corrupted (which is detected by mismatch of MD5 entry value), ESP-IDF will ignore its contents. It is then up to application to detect such a condition and recover by adding partitions again.
- Just like the main partition table, extended partition table is encrypted.

Does this proposal look usable to you?