miniz library in ROM - usable for application development?

[ChrisHolza]

With

Code: Select all

esp-idf/components/esp32/include/rom/miniz.h

it looks like there is a Compression/Decompression library like zlib contained in ROM.
Are there any plans to make it usable from inside the application, like it is today with nano-printf?
Is it usable today? Of course the linker is unable to resolve the symbols contained in the header.

Thanks.

[ESP_Angus]

Hi Chris,

Yes, you should be able to use the miniz library from ROM from esp-idf applications. AFAIK this library doesn't hold any internal static state, so it should work from inside esp-idf without reservation. (For code which doesn't use any hardware features, the only reason why ROM code can't be used from esp-idf applications is static state and/or non-reentrant-ness.)

However, the version of miniz in ROM is stripped back from the full library's features. As shown in the top of the header - the archive, zlib & zip APIs are not compiled in. Only the lower level "tinfl" and "tdefl" functions are present in ROM, which provide low-level implementations of inflate/deflate that can be used with gzipped data. The miniz source repository has some examples of using these APIs to gzip/gunzip data. This API is also used in the esptool flasher stub.

Any code which compiles against these APIs (ie compiles against functions where the declarations in the header aren't #ifdef-ed out) should link, using the symbol addresses in components/esp32/ld/esp32.rom.ld. If you're getting linker errors (rather than compiler errors) then this is a bug, please let us know.

If you want to add the higher level APIs, which are wrappers around the lower level tinfl/tdefl API, then it should be fairly straightforward to make a miniz component that adds these APIs but uses the ROM functions for the heavy lifting (essentially, add the entire miniz library and then remove the implementations which are already in ROM, making sure no common structures have a different size due to enabled/disabled features.) Let us know if you encounter any problems doing that.

Angus

[kolban]

Howdy,
Just purely for my curiosity, do you have a specific need for the code to be executed/used from ROM as opposed to linking with an application level library that would provide the compression/uncompression functions you might be looking for?

[ESP_Sprite]

In theory, it can be quicker (less pressure on the flash cache) and uses less flash memory. It's in ROM anyway, why not use it?

[ChrisHolza]

Thanks guys,

this indeed works!

The steps are:

1. - add miniz.c to project, I used the latest version 9.1.15 which is the same as the rom/miniz.h header.

• - add #include "rom/miniz.h" on top of miniz.c to make use of the pre-configured header for esp32. This #define's out the standard miniz.c fake-header.

• - in miniz.c, comment out all functions contained in ESP32's ROM, namely

Code: Select all

mz_adler32
mz_crc32
mz_free
tdefl_compress
tdefl_compress_buffer
tdefl_compress_mem_to_mem
tdefl_compress_mem_to_output
tdefl_get_adler32
tdefl_get_prev_return_status
tdefl_init
tdefl_write_image_to_png_file_in_memory
tdefl_write_image_to_png_file_in_memory_ex
tinfl_decompress
tinfl_decompress_mem_to_callback
tinfl_decompress_mem_to_mem

The linking goes fine and you should be able to use miniz as expected.

For decompressing a 8,5 kb standard gzip file, miniz with ROM routines takes about 8 ms to complete decompression, while a full zlib implementation takes about 10 ms. (Well, milliseconds are quite unprecise to measure when all you have is a 1 ms tick. This was measured in "debug" configuration.)

I found miniz to be rather greedy regarding malloc'd heap. For a usual inflateInit2(), a total of 43784 bytes is allocated, which is more than I can spend on my project. Most of it (32 k) is used for miniz "LZ dictionary" (#define TINFL_LZ_DICT_SIZE 32768). I'm not an expert on compression algorithms, so I just tuned this down (needs to be a power of 2) and found my decompressed data corrupted.
Zlib works in nice little chunks of about 7 kb, which is more suitable for my and possibly any ESP32 project.
Is there any way to optimize miniz and let it work on statically allocated input/output without a huge heap buffer? I have found none at first try.

Anyway, thanks a lot Sprite & Angus for the quick response and Espressif for creating such an inspiring product. I love it!

[ESP_Angus]

Hi Chris,

Glad you got it working, and thanks for posting the steps for anyone who follows. If you feel like sending a Pull Request to add a ROM-backed "miniz" component to esp-idf, then we'd almost certainly accept it (although no pressure to do so.)

I found miniz to be rather greedy regarding malloc'd heap. For a usual inflateInit2(), a total of 43784 bytes is allocated, which is more than I can spend on my project. Most of it (32 k) is used for miniz "LZ dictionary" (#define TINFL_LZ_DICT_SIZE 32768). I'm not an expert on compression algorithms, so I just tuned this down (needs to be a power of 2) and found my decompressed data corrupted.

I'm not an expert on DEFLATE/zlib either, but I think the only guarantee zlib makes about the dictionary (as opposed to the window size) is that it will be less than 32KB. So depending on what deflated your file, the dictionary may be up to this size. If you use a tool which allows you to limit the dictionary size (ie compress with miniz, or there must be other tools which let you do this) then it might work. Although if the source file is only 8.5KB, it seems odd for the dictionary to be so big!

(I was also wondering if the old TINFL_LZ_DICT_SIZE may be compiled into the ROM functions somewhere already, but this doesn't seem like it gets used that way. To be extra sure you could try compiling a ROM-free version of miniz and see if it behaves differently.)

[ChrisHolza]

Hi Angus,

for my project I tend do stick with the zlib because of the dynamically lower heap requirements. The TINFL_LZ_DICT_SIZE buffer seems to be allocated in any case, as a part of the higher-level miniz features.

I'd love to contribute to your SDK anyway and started a fork. When throwing the miniz.c into the esp-idf/components/esp32 path instead of compiling it as part of my "main" project files its behavior changes, and inflate() hangs in a loop. So I have to evaluate why, which is not my number 1 prio at the moment.

Some general questions for contributing:
- As a library file, should it be refactored to meet your coding style?
- Would components/esp32 be the appropriate path to place it?

Thanks,
Chris

[WiFive]

Maybe should go in components/miniz

[kolban]

In a project I am working upon I needed to decompress (inflate) a ZLIB stream and thought that using the ROM based tinfl_decompress() would be great. Right now I'm getting an exception so thought I would look deeper at existing usage. I found that the esptool flasher stub appears to use this function:

https://github.com/espressif/esptool/bl ... ash.c#L250

and I have modeled my code on this example. However, I seem to see something strange. The flasher stub seems to compile itself with the full miniz library ... see:

https://github.com/espressif/esptool/bl ... ub/miniz.c

and that is the meat of this question. Looking at flasher stub, it appears to limit itself to tinfl_decompress() which is apparently supplied in ROM but yet when I look at the only example of usage I can find, it appears to use a local/duplicate copy of the code.

[ESP_Angus]

Hi Neil,

The flasher stub seems to compile itself with the full miniz library ...

A flasher stub is built for both ESP8266 and ESP32. ESP8266 has no miniz in ROM, so it's compiled into the the ESP8266 version of the stub.

You can see this line in the Makefile which adds miniz.c for ESP8266 builds:
https://github.com/espressif/esptool/bl ... kefile#L53

(The net result is that the stub binary file for ESP8266 is slightly larger, so it takes a fraction longer to load into IRAM when esptool is setting up. I don't think this really has any significant effect on flashing performance, though. The effect of ROM vs Compiled would probably be more pronounced in a running app where code is constantly being loaded via flash cache.)

Now you mention it, though, it does look like both ESP8266 & ESP32 build with the included miniz.h (but ESP32 links to ROM). So it is possible there's a bug in the ESP-IDF copy of the miniz.h header, although I don't think so as this functionality has previously been used successfully...