Using SD as (very slow) memory mapped space

[stantheman286]

Hi all,

I'd like to process some large chunks of data on the ESP32-S3, but of course running malloc() on several MB of data is well past the limit of the onboard RAM. We do have an SD card connected, so is there a way to map the SD card such that we could use it for memory space? I understand with the access times and flakiness of SD cards this is probably a horrible idea, but I wanted to try and figure out a way to process this data on the ESP if we can. Much appreciated!

Cheers,
Matt

[ESP_Sprite]

No, sorry, the ESP doesn't have an MMU that allows for mapping internal memory to a large chunk of address space, which is what you'd want for memory mapping the SD card. You could theoretically try this with the internal flash (which can be memory mapped) but with the size constraints on that, it's probably easier to get an ESP32S3 with 8MiB of PSRAM and use that.

[stantheman286]

Thanks ESP_Sprite, I appreciate it. Yeah, that was what I expected. We're considering some of the -S3 variants with the onboard PSRAM (ours is the MINI :-/)

On the internal flash, we do have a good chunk of our 8MB free. How would we go about memory mapping that?

Cheers,
Matt

[ESP_Sprite]

You can create a partition and then use esp_partition_mmap() for this. Note that this is a read-only mapping; to write you would still need to call the esp_partition_erase/write functions.

[stantheman286]

Awesome, I'll give that a shot. That will at least give us a few MB of space to work with and test out. Thanks!
Matt

[stantheman286]

Ok, so I've gotten basics of initializing and writing the flash paritition based on the paritition_mmap example (https://github.com/espressif/esp-idf/tr ... ition_mmap).

On the readback, if I generate a void* pointer from esp_partition_mmap and cast it to a string, once I have written data to that pointer using esp_partition_write is it possible read the string directly or do I need to use esp_partition_read?

For example, something like this:

Code: Select all

...
const void *mmap_ptr;
esp_partition_mmap_handle_t mmap_handle;
const esp_partition_t *mmap_partition;
char *input_string = "test";

mmap_partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, "storage");
assert(mmap_partition != NULL);
ESP_ERROR_CHECK(esp_partition_erase_range(mmap_partition, 0, mmap_partition->size));

// Map the partition to data memory
ESP_ERROR_CHECK(esp_partition_mmap(mmap_partition, 0, mmap_partition->size, ESP_PARTITION_MMAP_DATA, (const void**)&mmap_ptr, &mmap_handle));

// Write out the test string to flash
ESP_ERROR_CHECK(esp_partition_write(mmap_partition, 0, input_string, sizeof(input_string));

// Test readback
char *out_string = (char*)mmap_ptr;
ESP_LOGI(TAG, "Test string -> %s", out_string);

Or perhaps I need to write parition data before mapping as the partition_mmap example shows?

Matt

[ESP_Sprite]

The partition write function should clear the cache, so your memory mapped data should update immediately.

[stantheman286]

After doing some more resting, I found one of my pointers wasn't allocated correctly. Now the writes to the partition and reading through accessing the variable work great so looks like I'm all set!

Cheers,
Matt