Page 1 of 1
Using SD as (very slow) memory mapped space
Posted: Tue Feb 28, 2023 11:38 pm
by stantheman286
Hi all,
I'd like to process some large chunks of data on the ESP32-S3, but of course running malloc() on several MB of data is well past the limit of the onboard RAM. We do have an SD card connected, so is there a way to map the SD card such that we could use it for memory space? I understand with the access times and flakiness of SD cards this is probably a horrible idea, but I wanted to try and figure out a way to process this data on the ESP if we can. Much appreciated!
Cheers,
Matt
Re: Using SD as (very slow) memory mapped space
Posted: Wed Mar 01, 2023 12:34 am
by ESP_Sprite
No, sorry, the ESP doesn't have an MMU that allows for mapping internal memory to a large chunk of address space, which is what you'd want for memory mapping the SD card. You could theoretically try this with the internal flash (which can be memory mapped) but with the size constraints on that, it's probably easier to get an ESP32S3 with 8MiB of PSRAM and use that.
Re: Using SD as (very slow) memory mapped space
Posted: Wed Mar 01, 2023 12:39 am
by stantheman286
Thanks ESP_Sprite, I appreciate it. Yeah, that was what I expected. We're considering some of the -S3 variants with the onboard PSRAM (ours is the MINI :-/)
On the internal flash, we do have a good chunk of our 8MB free. How would we go about memory mapping that?
Cheers,
Matt
Re: Using SD as (very slow) memory mapped space
Posted: Wed Mar 01, 2023 12:10 pm
by ESP_Sprite
You can create a partition and then use esp_partition_mmap() for this. Note that this is a read-only mapping; to write you would still need to call the esp_partition_erase/write functions.
Re: Using SD as (very slow) memory mapped space
Posted: Wed Mar 01, 2023 5:27 pm
by stantheman286
Awesome, I'll give that a shot. That will at least give us a few MB of space to work with and test out. Thanks!
Matt
Re: Using SD as (very slow) memory mapped space
Posted: Wed Mar 01, 2023 7:20 pm
by stantheman286
Ok, so I've gotten basics of initializing and writing the flash paritition based on the paritition_mmap example (
https://github.com/espressif/esp-idf/tr ... ition_mmap).
On the readback, if I generate a void* pointer from
esp_partition_mmap and cast it to a string, once I have written data to that pointer using
esp_partition_write is it possible read the string directly or do I need to use
esp_partition_read?
For example, something like this:
Code: Select all
...
const void *mmap_ptr;
esp_partition_mmap_handle_t mmap_handle;
const esp_partition_t *mmap_partition;
char *input_string = "test";
mmap_partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, "storage");
assert(mmap_partition != NULL);
ESP_ERROR_CHECK(esp_partition_erase_range(mmap_partition, 0, mmap_partition->size));
// Map the partition to data memory
ESP_ERROR_CHECK(esp_partition_mmap(mmap_partition, 0, mmap_partition->size, ESP_PARTITION_MMAP_DATA, (const void**)&mmap_ptr, &mmap_handle));
// Write out the test string to flash
ESP_ERROR_CHECK(esp_partition_write(mmap_partition, 0, input_string, sizeof(input_string));
// Test readback
char *out_string = (char*)mmap_ptr;
ESP_LOGI(TAG, "Test string -> %s", out_string);
Or perhaps I need to write parition data before mapping as the partition_mmap example shows?
Matt
Re: Using SD as (very slow) memory mapped space
Posted: Thu Mar 02, 2023 12:50 am
by ESP_Sprite
The partition write function should clear the cache, so your memory mapped data should update immediately.
Re: Using SD as (very slow) memory mapped space
Posted: Thu Mar 02, 2023 3:23 am
by stantheman286
After doing some more resting, I found one of my pointers wasn't allocated correctly. Now the writes to the partition and reading through accessing the variable work great so looks like I'm all set!
Cheers,
Matt