Is there a way to use SPI external flash on ESP32/ ESP32S3 by accessing it only through SPI and not using address space

[djixon]

Is there a way to "disable" the SPI external flash (integrated inside the module) so it doesn't map into address space of ESP?
It would mean ESP-IDF and RTOS (during chip power up) skip any kind of its initialization, (or testing for its presence) and also skip its mapping into physical ESP address space, so at runtime, it becomes invisible and not accessible for cache mechanism, both DMA and ABI, the same way, as if it is physically not presented in the module?

Would user be able to still access it at the runtime by using direct SPI routines and low level flash commands (and not through the address space of ESP)? For example, in the case of low level flash's reading commands (like reading of flash status register, flash ID or data from flash) result fetched from the SPI, would stored into user provided block in RAM.

The main purpose of such approach would be to use SPI external flash, like a runtime read/write data storage immune to power loss events and preserving data in it. That way, using of any robust FileSystems (FAT, NVS or any other) is avoided. Those require partitions management and file-like approach (where some of those even do not have re-writing capabilities over already occupied data) and most of those are extremely sensitive on power loss during writing operations, leaving possibly corrupted files, which can be an issue if your application depends on it, to be able, for example, to continue some process automatically, once power is restored.

[ESP_Sprite]

Your question is very unclear. Could you describe what such a setup would look like, in practice, how would it work? What would the ESP boot from, what would you use instead of partitions/fat/nvs/...?

[MicroController]

most of those are extremely sensitive on power loss during writing operations

Wondering how an external SPI flash chip would solve that issue in a way internal flash can't.

Options for your consideration:
1. using the NVS-API
2. using LittleFS
3. using the Partition API
4. using an external non-volatile memory for data storage, having only the application and read-only data in internal flash
4.1 using an SD card as the external memory.
4.2 using an external SPI flash chip

[djixon]

@MicroController: Internal flash (inside raw ESP chip) will contain the code. However 4MB external SPI flash (already integrated in the module ESP32 WROOM E32) I want to use freely (not through address space but calling low level flash commands). Double-buffering technique can be used to provide data consistency in the case of power-loss during write operations.

@ESP-Sprite:
Emulating RAM-like access (read/write) to external flash just by utilizing the low level SPI routines using flash commands and real addresses from flash space chip. (For example, for 4MB flash GD25LQ32C, organization of internal memory structure is 64 blocks, each containing 16 sectors, where each sector contains 16 pages each of 256 bytes. So an address (in hexadecimal notation) 0xQQSPVV represents an address of a byte in flash memory in block defined by nibbles QQ, in sector defined by nibble S, in page defined by nibble P, at address within page (0..255) represented by last two nibbles VV. For example address 0x123456 is the address of byte in block 0x12 (18 decimal) in sector 3 in page 4 at address 0x56 (86 decimal). So just by knowing an address, all things (block, sector, page and address within page) are known. For that chip, the smallest erasable area is the sector (contains 16 pages), but smallest programmable area is the page (contains 256 bytes). Such a library, should provide the functions for RAM-like access to flash. For an example WriteByte(addr, value). Such an operation for a single byte writing at some address (doesn't matter if something is already written at that address) is achieved like this:
- take the whole sector (it is smallest erasable area on that chip) which contains that address and copy into RAM (using SPI routines only)
- write that byte into RAM at proper address within that copied sector
- erase complete sector in flash
- program whole sector by coping content from whole sector in RAM back into the flash (using SPI routines only)

This is just an illustrative example of a concept. Sure it would be too slow (and inefficient) to do all that per byte basis. Those libraries (of other chip manufacturers) handle that problem like this:
In flash_init routine, whole flash chip is read and a list of pages which are fully empty and ready to be written is created. Any write to flash must be initiated by a BeginWrite() function which does nothing special, except some allocations in RAM where the list of writing (address, value) pairs will be stored. BeginWrite() function is then followed by an arbitrary many function calls WriteFlash(addr, byte). Still nothing happens to real flash. Those WriteFlash function call, just update the list in RAM which contains all (address, value) pairs which have to be written in flash. Once CommitWrite() function is called, the following happens: Global writing_in_progress flag is set. Then, whole list of (address, value) pairs is processed. The output of that processing is the list of all pages which have to be updated. Then, sectors with pages which are "dirty" which require whole updating to be performed through the RAM, are re-written by reloading them from flash into ram, then overwritten by new bytes, and stored back into flash, sector by sector, single sector at a time. Once that whole process is completed, the rest of pages which were already ready_to_be_written and not belonging to any of previously written sectors are written after that. If page was ready for a writing but belongs to sector which required updating through RAM (due to some other dirty byte from some other page belonging to that sector) was included in that sector once it has been copied in RAM so it will be updated through that sector update phase. After whole writing is completed, writing_in_progress flag is reset.

A new BeginWrite() can occur after CommitWrite() even if it has not completed yet (while writing_in_progress is still set), allowing a user to add a new address,value pairs to a new list. However, another CommitWrite() can not be called until previous CommitWrite() is completed. So call to CommitWrite() is possible only if writing_in_progress flag is cleared.
Any two calls to BeginWrite() (without CommitWrite() in between) will make second BeginWrite() discards any list of address,value pairs formed by the first BeginWrite(). (if any) So calling BeginWirite() will discard all address,value pairs of any previous BeginWrite if CommitWrite() has never be called.

Sure, all physical writes to flash respect the state of flags BP0-BP4 and CMP in flash status register and handled by hardware in flash chip itself, because any attempt of writing/erasing a protected block/sector will be discarded at hardware level. However, function WriteFlash(address, value) (which only populates the list of address,val pairs), can recognize if passed address fits in any protected area by reading those BP flags in flash status register and throw an exception or error result before CommitWrite() is called etc. etc.

Benefits of such approach in comparison to most of FileSystem approach are following:
- it is extremally small library usually initialized to be called just from one Core (in multi core systems)
- FileSystems formed in flash (require partitioning) are usually without optimized re-write_over_existed_data capabilities an leave garbage. However even if it is supported, deleting files leaves fragmented space requiring much robust mechanisms for re-using of that space etc.
- FileSystems are extremely sensitive to power_loss during writing operations. It leaves usually corrupted data which might be an issue if your application depends on it, for proper operation (for example to automatically continue running process once power is restored)

New series of microchip controllers have such a library already implemented. Users just access the flash integrated in the chip just the same as described above, by simple calls to ReadByte and WriteByte functions. All the complex behavior behind is hidden from user and handled by library functions automatically, so users do not care if their blob they want to write is overlapping among the pages in flash, or does sector needs erasing first etc.

However, to work safe on ESP32, there should be an option to EXCLUDE external flash even if it is presented so MMU can't see it and not map it into internal address space. That way it will be invisible to both cores directly through address space, however users would be able to access it trough a functions of that small library.

[MicroController]

There are some misconceptions underway.
The ESP modules come with only one flash memory. There are ESP variants with flash integrated into the chip's package, and there are variants without internal flash. The latter require an external flash chip (included on the modules), the former come without external flash. In both variants the flash is connected via SPI to the core and can be used as SPI flash in addition to parts of it being address-mapped to run the application.

You can use the partition or SPI flash API to roll your own journaling filesystem in the flash, or you just use NVS or LittleFS or SPIFFS to handle that for you. Depends on your use case, e.g. the amount of data to store.

[djixon]

So there is no combination of both internal (in esp chip flash) + external (in ESP module flash) models?