Created: 2009-03-21
Contributors:
Packages affected: apt, dpkg, apt-*, dpkg-*
See also: SpecTemplate
Summary
lzma (used by 7-zip) is a compression method that provides a high compression ratio and very fast decompression. By switching from gzip to lzma mirrors can save ~25-30% of their bandwidth costs and we could fit ~40-50% more software on CDs and DVDs
Rationale
lzma is faster and smaller than bzip2 and gives much better compression ratios than gzip.
debian-500-i386-CD-1.iso (gzip) : 677857280
ar x and lzma -9 : 471043278
%30.5 savings
With lzma we could fit %44 more software on a CD
Some benchmarks: http://tukaani.org/lzma/benchmarks
Ubuntu has a page on switching over: https://wiki.ubuntu.com/dpkg-lzma
SUSE uses lzma: http://en.opensuse.org/LZMA
Something all these pages miss is that lzma has many options. The higher memory requirements and slower compression times can be rectified with small drops in compression efficiency. The lzma package seems a little old however and is missing dictionary sizing.
Use Cases
- Rob is constantally browsing and installing new software. Because of lzma he can download and install software faster.
- Alice has a computer without a DVD drive. She is able to get a feature-rich desktop from a single install CD because of better compression.
- John maintains a mirror. He can serve more peers because packages downloaded are smaller.
Scope
This covers the debs produced for official mirrors in future packages. Old packages and supplementary mirrors can keep using gzip. gzip will continue to be supported into the foreseeable future.
Design
Please give some feedback on this:
- Compress all .deb files with lzma compression
- use d8m or d16m for 8 or 16 megabyte maximum dictionary sizes on older or emdebed oriented architectures: alpha armel i386 mips mipsel powerpc. d32m for the others: amd64 ia64 sparc s390. Memory usage is dictionary size + 0.05MB. Dictionaries only reach large sizes on large packages.
- because it significantly increases compression times with little shrinkage -9 should only be used for testing and stable. others should use -7
lzma home page: http://www.7-zip.org/sdk.html
Summary
Compress all debs with lzma
Rationale
Make packages smaller and quicker to download
Scope and Use Cases
gzip can/should remain default for debs that are not to be mass-distributed as gzip has much quicker compression speeds.
Implementation Plan
Implementation
Outstanding Issues
Not really an outstanding issue but 7zip has special filters for many binary ISAs that can reportedly increase compression by another 5-10% without a speed hit. There is no current support for this outside of p7zip so it is unlikely to get implemented in the foreseeable future.