Partition Tables, Heap and Flash Size
Partition Tables, Heap and Flash Size
Anyone able to shed a little light on the interaction between these factors. If I have an application on a 4MB ESP32 device. I've added heaps of tasks and I find that during OTA update, it's failing because of an overflow. Will using a larger flash size jut "fix" this problem. I.e. without making any changes to the partition table. Or do I have to increase the partition table size too, to inform the devie that more memory is available?
My perhaps naive assumption was that the partition table earmarks memory that must be reserved, but any memory over and above that is forwarded to the heap by the compiler to be used dynamically where ever it is needed. I'm struggling to peice together this picture. A little help please!
My perhaps naive assumption was that the partition table earmarks memory that must be reserved, but any memory over and above that is forwarded to the heap by the compiler to be used dynamically where ever it is needed. I'm struggling to peice together this picture. A little help please!
Re: Partition Tables, Heap and Flash Size
Partition table does have to be coordinated with flash size.
Heap is completely separate from flash. The way to increase heap is move some things out of iram using menuconfig options or add psram.
You can also shut down other tasks and free their memory before starting ota.
Heap is completely separate from flash. The way to increase heap is move some things out of iram using menuconfig options or add psram.
You can also shut down other tasks and free their memory before starting ota.
Re: Partition Tables, Heap and Flash Size
So, if I understand you correctly - I move from a 4MB ESP32 to a 16MB. I should increase my OTA0 and OTA1 partition by a factor of just over 4. And any memory not declared in the partition table is not usable by the code. So, if I leave the partitin table the same, I might as well not have upgraded the device? Is there a way to check what the maximum size available is, or is this just a pencil and paper exercise based on the total size of the other partitions, and the known overall size?
Makes sense to close down other tasks before starting OTA. I think the OTA uses HTTPS security inside another task it starts. Therefore, you need quite a lot of spare memory in the heap or else you get all sorts of mbedtls failures due to memory allocation failing. However, if I increase the memory to 16MB, and make the changes to the partition, these memory allocation failures should be a thing of the past. Thought I should ask before buying a new bunch of devices and building up new boards.
Why does moving things out of IRAM specifically free up heap. Is heap only allocated from free IRAM, not the DRAM or the dual purpose memory area? This is a fairly complicated with all of this multiplied by the additional dimension of tasks with their own allocated heaps. Is there any comprehensive documentation that you're aware of for this. I've been googling, but it's difficult to recognise the right resource, when you don't really understand the subject matter well enough yet.
Makes sense to close down other tasks before starting OTA. I think the OTA uses HTTPS security inside another task it starts. Therefore, you need quite a lot of spare memory in the heap or else you get all sorts of mbedtls failures due to memory allocation failing. However, if I increase the memory to 16MB, and make the changes to the partition, these memory allocation failures should be a thing of the past. Thought I should ask before buying a new bunch of devices and building up new boards.
Why does moving things out of IRAM specifically free up heap. Is heap only allocated from free IRAM, not the DRAM or the dual purpose memory area? This is a fairly complicated with all of this multiplied by the additional dimension of tasks with their own allocated heaps. Is there any comprehensive documentation that you're aware of for this. I've been googling, but it's difficult to recognise the right resource, when you don't really understand the subject matter well enough yet.
Re: Partition Tables, Heap and Flash Size
You're still thinking partitions and heap are related but they are not.
https://docs.espressif.com/projects/esp ... ables.html
https://docs.espressif.com/projects/esp ... types.html
https://docs.espressif.com/projects/esp ... ables.html
https://docs.espressif.com/projects/esp ... types.html
Re: Partition Tables, Heap and Flash Size
Hmmm. I've read your links and they make perfect sense in isolation. But I still can't join the dots.
If I print get my app size for my 4MB device, it looks like this. My OTA partition is 1600KB, so this is pretty full.
The DRAM is 124 kB. The IRAM is 131 kB. Is it the case that this is fixed for all ESP32-WROOM-32e devices regardless of flash size?
Looking at the spec for the devices, they all say: 448kB ROM, 536kB SRAM. No clue what that corresponds to. At this point, I must confess I am completely confused.
Is there perhaps some sort of trade off between Flash and RAM... So, if I use a massive amount of flash, the full amount of RAM will become available. But in resource constrained designs like this, I end up loading up the Flash, so there is very little space left for the RAM? in that way, the Flash and the RAM (and hence the heap) are very much inter-related.
**extra info**
I thought it would be helpful to put int he partition info too for completeness:
So, it looks like I'm leaving quite a bit of memory on the table! If I have 4 MB to use up, I could actually increase the size of the OTA0 and OTA1 partitions to 2000K. I wonder, based on my completely speculative logic above, would that restore my entire allocation of RAM?
On second thought, that makes no sense at all. We're going to have to go back to basics on this one. What's the story on ESP32 memory? A - Z !
If I print get my app size for my 4MB device, it looks like this. My OTA partition is 1600KB, so this is pretty full.
Code: Select all
Total sizes:
DRAM .data size: 14560 bytes
DRAM .bss size: 87664 bytes
Used static DRAM: 102224 bytes ( 22356 available, 82.1% used)
Used static IRAM: 123643 bytes ( 7429 available, 94.3% used)
Flash code: 1119071 bytes
Flash rodata: 282548 bytes
Total image size:~1539822 bytes (.bin may be padded larger)
Looking at the spec for the devices, they all say: 448kB ROM, 536kB SRAM. No clue what that corresponds to. At this point, I must confess I am completely confused.
Is there perhaps some sort of trade off between Flash and RAM... So, if I use a massive amount of flash, the full amount of RAM will become available. But in resource constrained designs like this, I end up loading up the Flash, so there is very little space left for the RAM? in that way, the Flash and the RAM (and hence the heap) are very much inter-related.
**extra info**
I thought it would be helpful to put int he partition info too for completeness:
Code: Select all
# Name Type SubType Offset Size
# Note: Firmware partition offset needs to be 64K aligned initial 36K (9 sectors) are reserved for bootloader and partition table
nvs data nvs 0x6000
otadata data ota 0x2000
phy_init data phy 0x1000
ota_0 app ota_0 0x20000 1600K
ota_1 app ota_1 1600K
fctry data nvs 0x340000 0x6000
On second thought, that makes no sense at all. We're going to have to go back to basics on this one. What's the story on ESP32 memory? A - Z !
Re: Partition Tables, Heap and Flash Size
The only tradeoff between flash and ram is how much code you put in iram vs keeping it in flash. You are correct that the ram size is the same for all ESP32-WROOM-32e devices regardless of flash size. The static iram + dram shown doesn't add up to the spec because of restrictions and reserved regions. Some of it will be added to the heap at boot time.
Re: Partition Tables, Heap and Flash Size
This post is the best I've found on ESP32 memory usage.
https://medium.com/the-esp-journal/esp3 ... 9444d89387
upon reading that, my only question is why do I apparently have about 122 kB of DRAM available? I'm not enabling Tracing. I do use Bluetooth, so that could account for about 54 kB lost. I subtract 24 kB for ROM, Fast & Slow RTC.
https://medium.com/the-esp-journal/esp3 ... 9444d89387
upon reading that, my only question is why do I apparently have about 122 kB of DRAM available? I'm not enabling Tracing. I do use Bluetooth, so that could account for about 54 kB lost. I subtract 24 kB for ROM, Fast & Slow RTC.
Re: Partition Tables, Heap and Flash Size
There is 176kb dram available at compile time because after that it runs into a reserved region and the linker can't work around it.
Re: Partition Tables, Heap and Flash Size
So to sum up. We want to leave as much memory as possible for heap allocation. Simple things to do are :
- Return memory to the heap after BLE execution is no longer required.
- Make sure enough Flash is available so the IRAM isn't inadvertently used. Question: is any direction required by the compiler on this point. How do you tell the compiler to prefer Flash over IRAM?
- Don't use heap debugging / tracing unless you actually need it.
- Single Core mode will give you at least a 32 kB heap bonus, salvaged from the unused CPU1 cache.
- Only use IRAM_ATTR & DRAM_ATTR when absolutely necessary (Does this answer my own question above on priority for Flash over IRAM etc.)
- Be stingy when creating tasks, not to over allocate heap. Maybe use high water mark functions to see how much head room you have.
- Any note on returning memory to the heap. I'm thinking abck to the FreeRTOS heap allocation mode. We aren't using static allocation or anything are we? That would certainly be worth knowing about.
Re: Partition Tables, Heap and Flash Size
Yes flash is preferred but there are options to put code in iram for performance reasons that are configured in menuconfig.
Who is online
Users browsing this forum: No registered users and 266 guests