rom/miniz heap space requirements

lerebel103@gmail.com
Posts: 9
Joined: Mon Jul 08, 2019 2:34 am

rom/miniz heap space requirements

Postby lerebel103@gmail.com » Mon Jul 08, 2019 3:22 am

Hi all,

Can rom/miniz be used on a reference ESP32 (520KB ram), which offers just under 300KB of usable heap in user space? When running the following snippet as a test on esp-idf v3.2.2:
  1. #include <stdio.h>
  2. #include <rom/miniz.h>
  3. #include <lwip/opt.h>
  4.  
  5. void app_main() {
  6.     printf("Free heap memory: %d\n", esp_get_free_heap_size());
  7.     printf("   8-BIT blocks available: %d\n", heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));
  8.     printf("   32-BIT blocks available: %d\n", heap_caps_get_largest_free_block(MALLOC_CAP_32BIT));
  9.  
  10.     printf("Memory needed for a tdelf compressor: %d\n", (int) sizeof(tdefl_compressor));
  11.     tdefl_compressor* comp = (tdefl_compressor*)malloc(sizeof(tdefl_compressor));
  12.     if (comp == NULL) {
  13.         printf("Failed to allocate compressor!!!!!\n");
  14.     } else {
  15.         printf("Free heap memory: %d", esp_get_free_heap_size());
  16.         printf("Compressor allocated.\n");
  17.     }
  18. }
I can see:
  1. Free heap memory: 296484
  2.    8-BIT blocks available: 126412
  3.    32-BIT blocks available: 126412
  4. Memory needed for a tdelf compressor: 167744
  5. Failed to allocate compressor!!!!!

It looks as thought about 167KB of contiguous memory is needed to allocate tdefl_compressor, which fails. Many others have successfully used miniz on ESP32, what am I missing please?

Thanks!
Last edited by lerebel103@gmail.com on Mon Jul 08, 2019 5:28 am, edited 1 time in total.

ESP_Angus
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: rom/miniz heap space requirements

Postby ESP_Angus » Mon Jul 08, 2019 5:17 am

I think the others may be decompressing, which doesn't need as much contiguous RAM.

The only options I can think of are:

- Recompile miniz from source and decouple some of these buffers (so that the tdefl buffer members are pointers to other data structures which are allocated independently). May need some other changes in the miniz code if code like sizeof(buffer member) is ever used.

- Enable PSRAM. The structure will fit easily in 4MB of external RAM.

Sorry I can't offer a better solution.

lerebel103@gmail.com
Posts: 9
Joined: Mon Jul 08, 2019 2:34 am

Re: rom/miniz heap space requirements

Postby lerebel103@gmail.com » Mon Jul 08, 2019 5:24 am

Nah, that's great thanks for confirming this @ESP_Angus, I could see many success stories out there and wasn't sure if I had missed something embarrassingly simple. Sure thing, I can see that these could be allocated separately, great tip:
  1.   mz_uint8 m_dict[TDEFL_LZ_DICT_SIZE + TDEFL_MAX_MATCH_LEN - 1];
  2.   mz_uint16 m_huff_count[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS];
  3.   mz_uint16 m_huff_codes[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS];
  4.   mz_uint8 m_huff_code_sizes[TDEFL_MAX_HUFF_TABLES][TDEFL_MAX_HUFF_SYMBOLS];
  5.   mz_uint8 m_lz_code_buf[TDEFL_LZ_CODE_BUF_SIZE];
  6.   mz_uint16 m_next[TDEFL_LZ_DICT_SIZE];
  7.   mz_uint16 m_hash[TDEFL_LZ_HASH_SIZE];
  8.   mz_uint8 m_output_buf[TDEFL_OUT_BUF_SIZE];
Cheers.

Who is online

Users browsing this forum: No registered users and 123 guests