Tar is short for tape archive. It is specifically intended for streaming because the intended use case is to take data and create a stream that can be written to the tape. Adding in compression is composable in exactly the same way.

As I recall, both zip and tar put their dictionaries at the end of the file, because that’s when you have already written them (PDF does the same - write all of the objects, then write the dictionary at the end).

Compressing an uncompressed zip file has the same problems as compressing a tar archive: you need to decompress it entirely to be able to get any file out. This is why zip does per-file compression: the dictionary is uncompressed (but is tiny), each file can be pulled out one at a time and used. This is why zip works nicely as a read-only filesystem (each file can be decompressed on demand as it’s used) but tar.gz does not (you must decompress the whole thing to read one file).

Modern compressors such as zstd have a dictionary mode, where you give it a load of data and it generates something roughly like a Huffman encoding of the common sequences across the entire data set, then you can individually compress files referencing that dictionary, so they’re independent but the common parts are in the dictionary. I’d like to see a ZIP-like format that used this, with a shared dictionary but the ability to decode individual files (and, ideally, individual blocks of files, with something like a 2 MiB granularity) independently.

score 1 · canonical