headers
Gilles Espinasse | 21 Feb 2012 08:11
Picon
Favicon

Memory requirement for linux kernel compression/decompression

Since some time I am the happy user of xz to compress linux kernel (2.6.27
patched, 2.6.32
and now 3.0).
All tests below were made with xz-5.0.3.
Machine has 1 GB memory, x86 32 bits.

On 2.6.32, only lzma option is available and by default, kernel use lzma -9
and
this require much more memory than needed. As one compiler of my distrib
reported a compilation
breakage on a 512 MB VM during kernel compression, I started hacking
scripts/Makefile.lib, removed -9 and added -vv.
I then played with information displayed during compression to adjust xz
memory requirement.

Using the value from linux kernel (lzma -9), adding -vv show as result
lzma: Filter
chain: --lzma1=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
lzma: 674 MiB of memory is required. The limit is 17592186044416 MiB.
lzma: Decompression will need 65 MiB of memory.
(stdin): 1413.3 KiB / 3437.4 KiB = 0.411, 741 KiB/s, 0:04

Removing -9 show as result
lzma: Filter
chain: --lzma1=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
lzma: 94 MiB of memory is required. The limit is 17592186044416 MiB.
lzma: Decompression will need 9 MiB of memory.
(stdin): 1413.3 KiB / 3437.4 KiB = 0.411, 747 KiB/s, 0:04
So still same size result with less memory requirement.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Lasse Collin | 21 Feb 2012 19:47
Favicon

Re: Memory requirement for linux kernel compression/decompression

On 2012-02-21 Gilles Espinasse wrote:
> On 2.6.32, only lzma option is available and by default, kernel use
> lzma -9 and this require much more memory than needed. As one
> compiler of my distrib reported a compilation breakage on a 512 MB VM
> during kernel compression, I started hacking scripts/Makefile.lib,
> removed -9 and added -vv. I then played with information displayed
> during compression to adjust xz memory requirement.

lzma -9 from LZMA Utils uses 32 MiB dictionary and requires 311 MiB of
memory. xz -9 uses 64 MiB dictionary and requires 674 MiB of memory.
The lzma emulation in xz uses the same presets as xz, so lzma -9 from
XZ Utils needs 674 MiB of memory. So the emulation isn't very good
although by default both XZ Utils and LZMA Utils use 8 MiB dictionary.

Using a dictionary bigger than the uncompressed file is waste of
memory. So if the kernel image is small, switching to a much smaller
dictionary doesn't affect compression ratio.

> Should not a patch be pushed on LKLM to at least remove the -9 part?

I don't know. If -9 is removed, then a kernel bigger than 8 MiB may
compress worse than it does now.

The -9 probably was put there when XZ Utils hadn't taken over LZMA
Utils, so the memory usage was much lower. Using a high setting is fine
from decompression point of view, because in the specific case of
kernel decompression the dictionary size doesn't affect the
decompressor memory usage. So from that point of view it is fine to use
a high setting "just in case".
(Continue reading)

Permalink | Reply | 0 comments |

Gmane