Partition Tables, Heap and Flash Size

[monkey]

Anyone able to shed a little light on the interaction between these factors. If I have an application on a 4MB ESP32 device. I've added heaps of tasks and I find that during OTA update, it's failing because of an overflow. Will using a larger flash size jut "fix" this problem. I.e. without making any changes to the partition table. Or do I have to increase the partition table size too, to inform the devie that more memory is available?

My perhaps naive assumption was that the partition table earmarks memory that must be reserved, but any memory over and above that is forwarded to the heap by the compiler to be used dynamically where ever it is needed. I'm struggling to peice together this picture. A little help please!

[WiFive]

Partition table does have to be coordinated with flash size.
Heap is completely separate from flash. The way to increase heap is move some things out of iram using menuconfig options or add psram.
You can also shut down other tasks and free their memory before starting ota.

[monkey]

So, if I understand you correctly - I move from a 4MB ESP32 to a 16MB. I should increase my OTA0 and OTA1 partition by a factor of just over 4. And any memory not declared in the partition table is not usable by the code. So, if I leave the partitin table the same, I might as well not have upgraded the device? Is there a way to check what the maximum size available is, or is this just a pencil and paper exercise based on the total size of the other partitions, and the known overall size?

Makes sense to close down other tasks before starting OTA. I think the OTA uses HTTPS security inside another task it starts. Therefore, you need quite a lot of spare memory in the heap or else you get all sorts of mbedtls failures due to memory allocation failing. However, if I increase the memory to 16MB, and make the changes to the partition, these memory allocation failures should be a thing of the past. Thought I should ask before buying a new bunch of devices and building up new boards.

Why does moving things out of IRAM specifically free up heap. Is heap only allocated from free IRAM, not the DRAM or the dual purpose memory area? This is a fairly complicated with all of this multiplied by the additional dimension of tasks with their own allocated heaps. Is there any comprehensive documentation that you're aware of for this. I've been googling, but it's difficult to recognise the right resource, when you don't really understand the subject matter well enough yet.

[WiFive]

You're still thinking partitions and heap are related but they are not.

https://docs.espressif.com/projects/esp ... ables.html
https://docs.espressif.com/projects/esp ... types.html

[monkey]

Hmmm. I've read your links and they make perfect sense in isolation. But I still can't join the dots.

If I print get my app size for my 4MB device, it looks like this. My OTA partition is 1600KB, so this is pretty full.

Code: Select all

Total sizes:
 DRAM .data size:   14560 bytes
 DRAM .bss  size:   87664 bytes
Used static DRAM:  102224 bytes (  22356 available, 82.1% used)
Used static IRAM:  123643 bytes (   7429 available, 94.3% used)
      Flash code: 1119071 bytes
    Flash rodata:  282548 bytes
Total image size:~1539822 bytes (.bin may be padded larger)

The DRAM is 124 kB. The IRAM is 131 kB. Is it the case that this is fixed for all ESP32-WROOM-32e devices regardless of flash size?

Looking at the spec for the devices, they all say: 448kB ROM, 536kB SRAM. No clue what that corresponds to. At this point, I must confess I am completely confused.

Is there perhaps some sort of trade off between Flash and RAM... So, if I use a massive amount of flash, the full amount of RAM will become available. But in resource constrained designs like this, I end up loading up the Flash, so there is very little space left for the RAM? in that way, the Flash and the RAM (and hence the heap) are very much inter-related.

**extra info**
I thought it would be helpful to put int he partition info too for completeness:

Code: Select all

# Name	   Type	 SubType	 Offset	  Size
# Note: Firmware partition offset needs to be 64K aligned	 initial 36K (9 sectors) are reserved for bootloader and partition table			
nvs	      data	 nvs	     	          0x6000
otadata	  data	 ota	     	          0x2000
phy_init	 data	 phy	     	          0x1000
ota_0	    app	  ota_0	   0x20000	   1600K
ota_1	    app	  ota_1	   	          1600K
fctry	    data	 nvs	     0x340000	  0x6000

So, it looks like I'm leaving quite a bit of memory on the table! If I have 4 MB to use up, I could actually increase the size of the OTA0 and OTA1 partitions to 2000K. I wonder, based on my completely speculative logic above, would that restore my entire allocation of RAM?

On second thought, that makes no sense at all. We're going to have to go back to basics on this one. What's the story on ESP32 memory? A - Z !

[WiFive]

The only tradeoff between flash and ram is how much code you put in iram vs keeping it in flash. You are correct that the ram size is the same for all ESP32-WROOM-32e devices regardless of flash size. The static iram + dram shown doesn't add up to the spec because of restrictions and reserved regions. Some of it will be added to the heap at boot time.

[monkey]

This post is the best I've found on ESP32 memory usage.

https://medium.com/the-esp-journal/esp3 ... 9444d89387

upon reading that, my only question is why do I apparently have about 122 kB of DRAM available? I'm not enabling Tracing. I do use Bluetooth, so that could account for about 54 kB lost. I subtract 24 kB for ROM, Fast & Slow RTC.

[WiFive]

There is 176kb dram available at compile time because after that it runs into a reserved region and the linker can't work around it.

[monkey]

So to sum up. We want to leave as much memory as possible for heap allocation. Simple things to do are :

• Return memory to the heap after BLE execution is no longer required.
• Make sure enough Flash is available so the IRAM isn't inadvertently used. Question: is any direction required by the compiler on this point. How do you tell the compiler to prefer Flash over IRAM?
• Don't use heap debugging / tracing unless you actually need it.
• Single Core mode will give you at least a 32 kB heap bonus, salvaged from the unused CPU1 cache.
• Only use IRAM_ATTR & DRAM_ATTR when absolutely necessary (Does this answer my own question above on priority for Flash over IRAM etc.)
• Be stingy when creating tasks, not to over allocate heap. Maybe use high water mark functions to see how much head room you have.
• Any note on returning memory to the heap. I'm thinking abck to the FreeRTOS heap allocation mode. We aren't using static allocation or anything are we? That would certainly be worth knowing about.

There's some good discussion to be had around these points. Keen for more thoughts.

[WiFive]

Yes flash is preferred but there are options to put code in iram for performance reasons that are configured in menuconfig.