Support for file compression

[AjitSingh]

Is there any way to decompress zip/tar file which contain multiple file? I know there is something called zlib, but i can't find the any example for unzip the zip file of multiple file. Or is there any other better way to do that?

[Craige Hales]

https://en.wikipedia.org/wiki/Info-ZIP (pointer to zip-file code, there are others)
https://en.wikipedia.org/wiki/Gzip (pointer to gzip code, there might be others)
https://en.wikipedia.org/wiki/Tar_(computing) (what is a TAR file)
https://en.wikipedia.org/wiki/List_of_archive_formats (list of formats)
https://superuser.com/questions/146754/ ... zip-matter (discussion zip vs tar.gz)

TAR files are often gzipped, or compressed with another utility.
A gzipped tar file is NOT the same as a ZIP file. gz is the extension for gzip.

Linux users often use xxx.tar.gz files rather than xxx.zip files, probably because the Linux command line tools make it easy to pipe the tar command output into the gzip command.

The directory you think is in a ZIP file is actually just a list of names that might or might not have \ or / characters, and a offset into the ZIP file where the compressed data begins. That directory is appended to the end of the ZIP file. A zip utility might present it as a hierarchical list, but it is really a simple list.

Gzip files have no directory and hold only one file. That file (inside the gzip) might be a TAR file, which can have multiple files. gz is a minimal wrapper around a compressed blob. If the filename ends with tar.gz, then the compressed blob is probably a TAR file.

[ESP_Sprite]

FWIW, there is a lightweight CPIO extraction library in the ROMs of all ESPs with USB-OTG support: API is here; it's used in DFU firmware download mode (an ESP DFU firmware binary actually is a CPIO archive). If there's more interest, I can see if we can also publish the source to it.

[AjitSingh]

I'm using esp-idf for esp32. i have to download the a archive file(zip/tar.gz)from server and uncompressed it. That archive files will contain multiple files. which i need to extract. how can i do that. is there any example which i can refer.

[Craige Hales]

1) figure out if you are looking at a zip file or a tar.gz file. They are NOT the same.
2) if it is a tar.gz file, google turned this up: https://github.com/tobozo/ESP32-targz . I have not looked at it, but it might be a step in the right direction.

[Zeni241]

Have a look at this link:
https://github.com/richgel999/miniz

Miniz is a lossless, high performance data compression library in a single source file that implements the zlib (RFC 1950) and Deflate (RFC 1951) compressed data format specification standards. It supports the most commonly used functions exported by the zlib library, but is a completely independent implementation so zlib's licensing requirements do not apply. Miniz also contains simple to use functions for writing .PNG format image files and reading/writing/appending .ZIP format archives. Miniz's compression speed has been tuned to be comparable to zlib's, and it also has a specialized real-time compressor function designed to compare well against fastlz/minilzo.

[shriya_dhar]

I am facing a similar situation when attempting to decompress data compressed using the `miniz` library on the ESP32 platform. My objective is to compress data using miniz and store it on a text file and later uncompressing this text file but using python instead.

As per the documentation and information available, the `miniz` library is described as a "lossless, high-performance data compression library" that implements the zlib (RFC 1950) and Deflate (RFC 1951) compressed data format specification standards. However, I am encountering difficulties when attempting to decompress the compressed data using Python.

Specifically, when using the Python `zlib` module, which should be compatible with the zlib and Deflate formats, I am receiving an "unknown compression method" error. I have verified that the file and data are correct, and they were compressed using the `miniz` library on the ESP32. Despite this, I am unable to successfully decompress the data. I am need of a miniz-compatible wrapper function that works with uncompressing of data in Python or a format that compresses the data using my ESP32 but decompressing it using python.

I have reviewed the code and tried different approaches, but I have not been able to resolve this issue. I would greatly appreciate any help or any insights you can provide regarding how to properly decompress data that has been compressed with the `miniz` library using Python.

Thank you.

[nopnop2002]

I'm using zlib instead of miniz.

Try this:
https://github.com/nopnop2002/esp-idf-zlib

[DrMickeyLauer]

@nopnop2000: Thanks for publishing your findings. I wonder how zlib can have better compression ratios than brotli. In other tests, brotli always wins hands down. Which compression levels are you comparing with? Do you have any idea about the RAM footprints of zlib, brotli, and miniz?

[nopnop2002]

@DrMickeyLauer

>I wonder how zlib can have better compression ratios than brotli.

If you don't consider memory limitations, you're right.
You can try out several compression levels.
brotli will always win.

>Which compression levels are you comparing with

brotli
quality = 0;
If you specify anything other than this, malloc will fail.


zlib
level = Z_DEFAULT_COMPRESSION;

---


Footprint for zlib

zlib's memory footprint can also be specified fairly precisely.
It is larger for compression than for decompression, and the exact requirements depend on how the library was compiled.

The memory requirements for compression depend on two parameters, windowBits and memLevel:

deflate memory usage (bytes) = (1 << (windowBits+2)) + (1 << (memLevel+9)) + 6 KB

For the default values of 15 and 8, respectively, this is 268 KB, where the approximately 6 KB is for the deflate data structure. Both windowBits and memLevel can be set to lower values at compile time via the MAX_WBITS and MAX_MEM_LEVEL macros, but only at a cost in compression efficiency.

The memory requirements for decompression depend only on windowBits, but this is, in a sense, a harsher limitation: whereas data streams compressed with a smaller window will merely be a bit larger than they would have otherwise, a reduced window size for decompression means that streams compressed with larger windows cannot be decompressed at all. Having said that:

inflate memory usage (bytes) = (1 << windowBits) + 7 KB

Typically, therefore, inflate() requires no more than 40 KB of storage on a 64-bit machine. This includes the 32768-byte sliding window and approximately 7 KB for the inflate data structure.

The MAX_WBITS and MAX_MEM_LEVEL macros can be specified as compile options.