Any way to override MBEDTLS_PLATFORM_MEMORY define?

[vonnieda]

I don't have CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY enabled, so unless I misunderstand, none of my stacks should be in SPI RAM. The help in menuconfig says:

" for this reason xTaskCreate and friends always allocate stack in internal memory and xTaskCreateStatic will check if the memory passed to it is in internal memory"

I don't use xTaskCreateStatic, I only use xTaskCreate, so to my understanding all my stacks should be in internal RAM.

You're entirely correct, I mis-remembered how task stack allocation works in my earlier post (have edited it).

I still think it would be worth setting SPIRAM_MALLOC_RESERVE_INTERNAL to the value to the size of all stacks, plus headroom for buffers, and seeing if the memory situation resolves itself.

I'll also take another look at the MBEDTLS_PLATFORM_MEMORY PR, ASAP.

Hi Angus,

Are you suggesting to set SPIRAM_MALLOC_RESERVE_INTERNAL to the total of stack tasks + wifi static and dynamic buffers + mbedtls buffers? If so, is 1.6k per wifi buffer correct? And 32k per active TLS connection?

Thanks,
Jason

[vonnieda]

Went ahead and tried that, and the CPU won't boot:

Code: Select all

I (605) cpu_start: Pro cpu up.
I (609) cpu_start: Single core mode
I (613) heap_init: Initializing. RAM available for dynamic allocation:
I (620) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (626) heap_init: At 3FFCEF20 len 000110E0 (68 KiB): DRAM
I (632) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (638) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (645) heap_init: At 40096DE8 len 00009218 (36 KiB): IRAM
I (651) cpu_start: Pro cpu start user code
I (656) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (664) spiram: Reserving pool of 118K of internal memory for DMA/internal allocations
E (672) cpu_start: Could not reserve internal/DMA pool!
abort() was called at PC 0x400811ae on core 0
0x400811ae: start_cpu0_default at /Users/jason/esp/esp-idf/components/esp32/./cpu_start.c:265 (discriminator 1)


Backtrace: 0x400902e4:0x3ffe3b90 0x4009040b:0x3ffe3bb0 0x400811ae:0x3ffe3bd0 0x400813c7:0x3ffe3c00 0x40078c6a:0x3ffe3c40 0x40078d1d:0x3ffe3c70 0x40078f8e:0x3ffe3cb0 0x400790d6:0x3ffe3e70 0x40007c31:0x3ffe3eb0 0x4000073d:0x3ffe3f20
0x400902e4: invoke_abort at /Users/jason/esp/esp-idf/components/esp32/./panic.c:572

0x4009040b: abort at /Users/jason/esp/esp-idf/components/esp32/./panic.c:572

0x400811ae: start_cpu0_default at /Users/jason/esp/esp-idf/components/esp32/./cpu_start.c:265 (discriminator 1)

0x400813c7: call_start_cpu0 at /Users/jason/esp/esp-idf/components/esp32/./cpu_start.c:203 (discriminator 1)


E (703) esp_core_dump: Core dump flash config is corrupted! CRC=0xffffffff instead of 0x0
Rebooting...

I suspect there is no single internal memory block big enough. I set the value to 120832. I calculated this as:
- 89k for stacks
- 6 + 6 static buffers + 6 dynamic buffers = 18 buffers * 1.6k = 28.8k
- total: ~118k * 1024 = 120832

I tried some other increasingly smaller numbers until I got down to 64k which did allow the chip to boot, but quickly resulted in the same wifi "null" error, as soon as I tried to connect the TLS. Here is what that looks like:

Code: Select all

I (617) heap_init: Initializing. RAM available for dynamic allocation:
I (624) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (630) heap_init: At 3FFCEF20 len 000110E0 (68 KiB): DRAM
I (636) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (643) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (649) heap_init: At 40096DE4 len 0000921C (36 KiB): IRAM
I (655) cpu_start: Pro cpu start user code
I (660) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (668) spiram: Reserving pool of 64K of internal memory for DMA/internal allocations
I (23) cpu_start: Starting scheduler on PRO CPU.
I (950) wifi: wifi firmware version: de47fad
I (950) wifi: config NVS flash: disabled
I (950) wifi: config nano formating: disabled
I (950) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (960) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (970) wifi: Init dynamic tx buffer num: 32
I (970) wifi: Init data frame dynamic rx buffer num: 6
I (980) wifi: Init management frame dynamic rx buffer num: 6
I (990) wifi: Init static tx buffer num: 6
I (990) wifi: wifi driver task: 3ffe828c, prio:23, stack:3584
I (990) wifi: Init static rx buffer num: 6
I (1000) wifi: Init dynamic rx buffer num: 6
I (1000) wifi: wifi power manager task: 0x3ffeb67c prio: 21 stack: 2560
I (1040) phy: phy_version: 3662, 711a97c, May  9 2018, 14:29:06, 0, 0
I (1040) wifi: mode : sta (30:ae:a4:5f:99:24)
I (1040) wifi: sleep enable
I (1040) wifi: type: 1
S (1050)  xx_wifi.c:331  (0x0000) WiFi connecting to "xx".
I (1170) BTDM_INIT: BT controller compile version [44e9e66]

I (1170) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (1410)  xx_mqtt.c:212  (0x0000) WiFi not connected, waiting for it.
| 8BIT  | 32BIT | 32BIT - 8BIT | INTERNAL | SPIRAM |  DMA  | Heap Total | Used  | Free  | SOA Used | Free  |  HW   |
|=======|=======|==============|==========|========|=======|============|=======|=======|==========|=======|=======|
|1705480|1742848|         37368|     68220| 1674628|  30852|     4325684|2625416|1700268|         1|  32767|  32592|
I (2440) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1
I (3200) wifi: state: init -> auth (b0)
I (3210) wifi: state: auth -> assoc (0)
|1697548|1734916|         37368|     64852| 1670064|  27484|     4325684|2625416|1700268|         1|  32767|  32592|
I (3220) wifi: state: assoc -> run (10)
I (3230) wifi: connected with xxLabs5G, channel 11
S (3240)  xx_wifi.c:181  (0x0000) SYSTEM_EVENT_STA_CONNECTED ssid "xxLabs5G", bssid 80:2a:a8:03:0d:5b, channel 11, authmode WIFI_AUTH_WPA2_PSK.
|1701664|1739032|         37368|     64584| 1674448|  27216|     4325684|2625416|1700268|         1|  32767|  32592|
I (5150) event: sta ip: 192.168.10.141, mask: 255.255.255.0, gw: 192.168.10.1
S (5150)  xx_wifi.c:200  (0x0000) SYSTEM_EVENT_STA_GOT_IP IP 192.168.10.141, GW 192.168.10.1, Netmask 255.255.255.0.
I (5170)  xx_mqtt.c:115  (0x0000) Connecting to MQTT at xx.io:8883, use_tls = 1
|1689892|1727260|         37368|     55144| 1672116|  17776|     4325684|2625416|1700268|         1|  32767|  32592|
I (6220) wifi: pm start, type:1

|1603440|1640808|         37368|     42396| 1598412|   5028|     4325684|2625416|1700268|         1|  32767|  32592|
W (6460) wifi: alloc eb len=24 type=3 fail

W (6460) wifi: m f null

|1598952|1636320|         37368|     42352| 1593968|   4984|     4325684|2625416|1700268|         1|  32767|  32592|
|1583452|1620756|         37368|     42488| 1578060|   5120|     4325684|2625416|1700268|         1|  32767|  32592|
|1592304|1629672|         37368|     42504| 1587168|   5136|     4325684|2625416|1700268|         1|  32767|  32592|
|1590492|1627860|         37368|     42504| 1585356|   5136|     4325684|2625416|1700268|         1|  32767|  32592|
|1588616|1625984|         37368|     42448| 1583536|   5080|     4325684|2625416|1700268|         1|  32767|  32592|
|1589264|1626632|         37368|     42472| 1584160|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589344|1626712|         37368|     42472| 1584240|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589344|1626712|         37368|     42472| 1584240|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589344|1626712|         37368|     42472| 1584240|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
E (15800) xx_MQTT_ESP32: mbedtls_ssl_handshake client: -0x7880 - SSL - The peer notified us that the connection is going to be closed
E (15810)  xx_mqtt.c:137  (0xffff8780) xx_mqtt_initialize: NetworkConnectTLS failed
E (15840)  xx_mqtt.c:223  (0xffff8780) MQTT connect failed
|1588696|1626064|         37368|     42448| 1583616|   5080|     4325684|2625416|1700268|         1|  32767|  32592|
|1589344|1626712|         37368|     42472| 1584240|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589344|1626712|         37368|     42472| 1584240|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589268|1626636|         37368|     42472| 1584164|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589268|1626636|         37368|     42472| 1584164|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
I (20840)  xx_mqtt.c:115  (0x0000) Connecting to MQTT at xx.io:8883, use_tls = 1
|1588616|1625984|         37368|     42448| 1583536|   5080|     4325684|2625416|1700268|         1|  32767|  32592|
|1589268|1626636|         37368|     42472| 1584164|   5104|     4325684|2625416|1700268|         1|  32767|  32592|
|1589268|1626636|         37368|     42472| 1584164|   5104|     4325684|2625416|1700268|         1|  32767|  32592|

So, in this case, I had about 64k of internal, and 27k of DMA memory free. As soon as I attempted to open a TLS connection to my MQTT server it went down to 42k internal and 5k DMA and WiFi bombed with the "null" message.

So, I think the bottom line here is that unless I can get like 64k or more of DMA memory free, this won't work for me. Moving TLS into SPI solves the problem entirely, as far as I can tell, with very little performance hit.

Thanks,
Jason

[ESP_Angus]

OK, thanks for trying this out. I think you're right that the heap is too fragmented for a single large block of this kind.

Will think about what we can do to solve this problem generically in the future, but the TLS-in-SPI solution does seem like the best one for now. Thanks again for bearing with me on this.

[ESP_Angus]

An update, the MBEDTLS_PLATFORM_MEMORY PR is queued for merging.

There is also a change in internal review for SPIRAM_MALLOC_RESERVE_INTERNAL, to allow this pool to be allocated from multiple non-contiguous regions if needed, to make up the complete size. It also increases the maximum reservation size to 256KB, and allocates the pool a little later in the boot process so that the startup stack region (which is not available as heap in early boot) can be used as well.

[vonnieda]

An update, the MBEDTLS_PLATFORM_MEMORY PR is queued for merging.

There is also a change in internal review for SPIRAM_MALLOC_RESERVE_INTERNAL, to allow this pool to be allocated from multiple non-contiguous regions if needed, to make up the complete size. It also increases the maximum reservation size to 256KB, and allocates the pool a little later in the boot process so that the startup stack region (which is not available as heap in early boot) can be used as well.

That's great news, thank you for your help!