Any way to override MBEDTLS_PLATFORM_MEMORY define?

[vonnieda]

Is there any way, or any planned way, to enable the MBEDTLS_PLATFORM_MEMORY option defined in $IDF_PATH/components/mbedtls/port/include/mbedtls/esp_config.h?

This define enables mbedtls_platform_set_calloc_free() which allows you to replace the default calloc/free calls with your own. In particular, this makes it possible to have mbedtls allocate from SPI RAM, instead of internal memory.

I'd like to be able to do this without hacking my IDF installation. So, can it be done?

For background on why this is important, see: https://github.com/espressif/esp-idf/issues/2184

Thanks,
Jason

[ESP_Angus]

Hi vonnieda,

There's no way to do this at the moment, but if you copy the "mbedtls" component into your project's components directory, you can modify it there without modifying the rest of ESP-IDF.

Reading the issue on GitHub (WiFi allocations failing as all DMA-capable RAM is exhausted), I have a couple of other suggestions:

You mentioned that you got good results by modifying SPIRAM_MALLOC_ALWAYSINTERNAL. This is one of two recommended "knobs" for tuning SPIRAM allocations. It defaults to 16KB (which means 16KB allocations try from internal memory first), and each TLS session allocates 2x 16KB buffers. So setting this to a smaller value like 4KB may give good results. I noticed you set it to 256, which will also work but you will lose more performance this way from more smaller buffers in (slower) SPIRAM.

The other "knob" you can adjust is SPIRAM_MALLOC_RESERVE_INTERNAL, which reserves a pool of memory for DMA-required use only. This pool defaults to 32KB but you can set it larger to prevent the WiFi stack from running out of RAM. This may allow more complete use of the remaining internal DRAM, which will give you better performance overall.

A final thing you can do is change the WiFi TX buffer type to "static", and reduce the maximum number of dynamic RX buffers. This means that the WiFi stack will allocate more of its memory at initialize time and hold onto it, so it's less likely to end up caught out by running out of DMA memory.

Finally (and most importantly), you mention in the GitHub issue that you think TLS connections may not be releasing memory when they are closed. You should rule this out before doing any more tuning - because if you have a memory leak then no amount of treating the symptoms will address the cause. You can read about diagnosing memory leaks in the docs here:
https://docs.espressif.com/projects/esp ... mory-leaks

Angus

[vonnieda]

Hi vonnieda,

There's no way to do this at the moment, but if you copy the "mbedtls" component into your project's components directory, you can modify it there without modifying the rest of ESP-IDF.

Reading the issue on GitHub (WiFi allocations failing as all DMA-capable RAM is exhausted), I have a couple of other suggestions:

You mentioned that you got good results by modifying SPIRAM_MALLOC_ALWAYSINTERNAL. This is one of two recommended "knobs" for tuning SPIRAM allocations. It defaults to 16KB (which means 16KB allocations try from internal memory first), and each TLS session allocates 2x 16KB buffers. So setting this to a smaller value like 4KB may give good results. I noticed you set it to 256, which will also work but you will lose more performance this way from more smaller buffers in (slower) SPIRAM.

The other "knob" you can adjust is SPIRAM_MALLOC_RESERVE_INTERNAL, which reserves a pool of memory for DMA-required use only. This pool defaults to 32KB but you can set it larger to prevent the WiFi stack from running out of RAM. This may allow more complete use of the remaining internal DRAM, which will give you better performance overall.

A final thing you can do is change the WiFi TX buffer type to "static", and reduce the maximum number of dynamic RX buffers. This means that the WiFi stack will allocate more of its memory at initialize time and hold onto it, so it's less likely to end up caught out by running out of DMA memory.

Finally (and most importantly), you mention in the GitHub issue that you think TLS connections may not be releasing memory when they are closed. You should rule this out before doing any more tuning - because if you have a memory leak then no amount of treating the symptoms will address the cause. You can read about diagnosing memory leaks in the docs here:
https://docs.espressif.com/projects/esp ... mory-leaks

Angus

Thanks for the detailed response Angus!

I'll probably go ahead and move mbedtls into my project and swap the allocator, just so I can have some insight into it's allocations and control it better.

I normally run SPIRAM_MALLOC_ALWAYSINTERNAL=2048 - the 256 was just a test to see if I could get mbedtls to stop allocating too much DMA capable RAM.

Thank you for the reminder about SPIRAM_MALLOC_RESERVE_INTERNAL. This is probably the source of my issue. I had set this to 0 at some point months ago while experimenting and had forgotten about it. I'm low on internal memory due to many FreeRTOS stacks, but I'll see if I can get this back up to 32k.

Regarding the memory leak - I was suggesting there might be a leak in the mbedtls code when one of it's allocations fails, but I think it's just that the system can't recover once "W (6502) wifi: m f null" happens. What I've noticed is that mbedtls will grab a bunch of memory during a connect and then release it once the connection is made, but if I get the "W (6502) wifi: m f null" the memory is never released.

In any case, I'll do some heap debugging and see if I've got any leaks during failed connections. Never hurts to double check

Thanks!

Jason

[vonnieda]

Quick follow up on this: I went ahead and pulled mbedtls into my components directory, changed the defines as I mentioned earlier and am using my own allocator and this seems to be the best way to go for me now.

As Angus said, mbedtls allocates 2 16k buffers per session, along with quite a few other smaller allocations. Looks like > 40k of allocations. And since it uses calloc, much of that comes from internal memory.

I have pretty limited internal memory because of FreeRTOS stacks, so pushing all those allocations into SPI RAM works best for me.

Thanks,
Jason

[vonnieda]

I went ahead and did a pull request to add a Kconfig option for this. Seems like a trivial change and no downside to making it available.

https://github.com/espressif/esp-idf/pull/2237

Thanks,
Jason

[ESP_Angus]

Hi Jason,

Glad you found a solution with a copy of the mbedTLS component.

I am curious whether you tried either of the other options suggested in my reply above. As I suggested there, forcing all mbedTLS memory into SPIRAM does solve your immediate problem but it almost certainly reduces performance and under-utlises internal memory, compared to using the other settings mentioned in the post (which are designed to work around the problem you were seeing, apologies that they're not better documented as such.)


Angus

[vonnieda]

Hi Jason,

Glad you found a solution with a copy of the mbedTLS component.

I am curious whether you tried either of the other options suggested in my reply above. As I suggested there, forcing all mbedTLS memory into SPIRAM does solve your immediate problem but it almost certainly reduces performance and under-utlises internal memory, compared to using the other settings mentioned in the post (which are designed to work around the problem you were seeing, apologies that they're not better documented as such.)


Angus

Hi Angus,

Yes, I've tried all of the options you mentioned in various forms. On Friday I did quite a bit of testing to try to see if I could get things in a state where multiple TLS connections would not wedge WiFi and the only way I could get there was to push TLS into SPI RAM. Not sure if you missed my other replies in this thread, but I went into some detail previously.

I have very limited internal RAM. I'm using WiFi, BLE, and quite a few FreeRTOS stacks. My firmware generally runs around 40k free total internal RAM, and < 16k free DMA ram after boot

I use 6 dynamic WiFi buffers, with the default of 6 and 6 statics. I use SPIRAM_MALLOC_ALWAYSINTERNAL at 2048, and I have SPIRAM_MALLOC_RESERVE_INTERNAL set to 0. SPIRAM_MALLOC_RESERVE_INTERNAL is not particularly useful because FreeRTOS tasks are allocated from this area, so it ends up being a wash whether it's on or not. I've also tried it at 32k and it didn't really make a difference in the amount of free internal RAM after I'm booted up.

I could probably reduce the dynamic WiFi buffers down to 1, but that would only save me another 8k which doesn't really buy me much additional breathing room.

So, with about 40k free, if I happen to need to do two TLS connections at the same time (MQTT reconnect and a HTTPS API call) I run out of memory and WiFi dies.

I am not as concerned with performance as I am stability. I can deal with a slightly slower TLS, although so far in testing the difference seems negligible, but I can't accept a TLS connection potentially wedging the WiFi / TCP stack so bad that a reboot is required.

Thanks,
Jason

[ESP_Angus]

Hi jason,

I'm sorry, you're correct that I only noticed your second reply not your first one. Appreciate the detailed explanation(s) of what you've tried and what is working.

I use 6 dynamic WiFi buffers, with the default of 6 and 6 statics. I use SPIRAM_MALLOC_ALWAYSINTERNAL at 2048, and I have SPIRAM_MALLOC_RESERVE_INTERNAL set to 0. SPIRAM_MALLOC_RESERVE_INTERNAL is not particularly useful because FreeRTOS tasks are allocated from this area, so it ends up being a wash whether it's on or not. I've also tried it at 32k and it didn't really make a difference in the amount of free internal RAM after I'm booted up.

There is an important reason for this, which is that we don't support calling ROM code from tasks whose stacks are in external memory. Never calling ROM code can be hard to do (a lot of libc functions are in ROM, for example), so the only use case we recommend at all is to enable CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY and specifically heap_caps_malloc() certain tasks' stack buffers from SPIRAM, and then use the FreeRTOS "static" task allocation API to use that buffer as the task stack. This way you can control which task(s) are safe to run from external RAM.

With SPIRAM_MALLOC_RESERVE_INTERNAL==0, task stacks will be allocated from anywhere - so given you're running at close to the RAM limit this probably means you have some stacks allocated in SPIRAM already. You may want to look into this.

EDIT: This is incorrect, tasks always allocated from internal RAM using xTaskCreate()

Depending on how much RAM you're using for stacks, my recommendation would be to set SPIRAM_MALLOC_RESERVE_INTERNAL to (total size needed for all task stacks) plus 32KB (or more) for WiFi buffers.

I am not as concerned with performance as I am stability. I can deal with a slightly slower TLS, although so far in testing the difference seems negligible, but I can't accept a TLS connection potentially wedging the WiFi / TCP stack so bad that a reboot is required.

I see. Given you're this pressed for RAM then I can see how this may be the only option (although I suggest trying setting SPIRAM_MALLOC_RESERVE_INTERNAL as shown above, as this will fix the WiFi buffer problem and guarantee all task stacks stay in internal RAM).

BTW, once the next mbedTLS version is released, it will be possible to independently reduce the TX-side TLS buffer size without breaking TLS spec compliance. So this should allow you to save something like 8-12KB per TLS connection (the exact number depends on the cipher schemes in use.)

[vonnieda]

Hi jason,

There is an important reason for this, which is that we don't support calling ROM code from tasks whose stacks are in external memory. Never calling ROM code can be hard to do (a lot of libc functions are in ROM, for example), so the only use case we recommend at all is to enable CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY and specifically heap_caps_malloc() certain tasks' stack buffers from SPIRAM, and then use the FreeRTOS "static" task allocation API to use that buffer as the task stack. This way you can control which task(s) are safe to run from external RAM.

With SPIRAM_MALLOC_RESERVE_INTERNAL==0, task stacks will be allocated from anywhere - so given you're running at close to the RAM limit this probably means you have some stacks allocated in SPIRAM already. You may want to look into this.

Depending on how much RAM you're using for stacks, my recommendation would be to set SPIRAM_MALLOC_RESERVE_INTERNAL to (total size needed for all task stacks) plus 32KB (or more) for WiFi buffers.

Hi Angus,

I don't have CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY enabled, so unless I misunderstand, none of my stacks should be in SPI RAM. The help in menuconfig says:

" for this reason xTaskCreate and friends always allocate stack in internal memory and xTaskCreateStatic will check if the memory passed to it is in internal memory"

I don't use xTaskCreateStatic, I only use xTaskCreate, so to my understanding all my stacks should be in internal RAM.
[/quote]

BTW, once the next mbedTLS version is released, it will be possible to independently reduce the TX-side TLS buffer size without breaking TLS spec compliance. So this should allow you to save something like 8-12KB per TLS connection (the exact number depends on the cipher schemes in use.)

That's good news! I can definitely use it

Thanks,
Jason

[ESP_Angus]

I don't have CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY enabled, so unless I misunderstand, none of my stacks should be in SPI RAM. The help in menuconfig says:

" for this reason xTaskCreate and friends always allocate stack in internal memory and xTaskCreateStatic will check if the memory passed to it is in internal memory"

I don't use xTaskCreateStatic, I only use xTaskCreate, so to my understanding all my stacks should be in internal RAM.

You're entirely correct, I mis-remembered how task stack allocation works in my earlier post (have edited it).

I still think it would be worth setting SPIRAM_MALLOC_RESERVE_INTERNAL to the value to the size of all stacks, plus headroom for buffers, and seeing if the memory situation resolves itself.

I'll also take another look at the MBEDTLS_PLATFORM_MEMORY PR, ASAP.