headers
Gilles Espinasse | 5 Mar 2012 08:06
Picon
Favicon

Oddities with --lzma2 options

I played with lzma2 options and find two curious things.

First, start with something simple as reference (everything tested with
xz-5.0.3)
xz -vv  -8e < coreutils-8.15.tar >/dev/null
xz: Filter
chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=273,mf=bt4,depth=5
12
xz: 370 MiB of memory is required. The limit is 17592186044416 MiB.
xz: Decompression will need 33 MiB of memory.
  100 %       4832.6 KiB / 44.2 MiB = 0.107   361 KiB/s       2:05

Then the goal of the game is :
- to remove -8e so the instruction could be applied on different files size
with less memory required when possible
- still achieve the small as possible result (no more than 1% excess than
the maximum compression)

I tried using
xz -vv --lzma2=dict=$(du -sk coreutils-8.15.tar | awk '{ printf "%dKiB", $1
* 3 / 4 }'),nice=273,depth=512 < coreutils-8.15.tar >/dev/null
xz: Filter
chain: --lzma2=dict=33993KiB,lc=3,lp=0,pb=2,mode=normal,nice=273,mf=bt4,dept
h=512
xz: 381 MiB of memory is required. The limit is 17592186044416 MiB.
xz: Decompression will need 34 MiB of memory.
  100 %       4833.4 KiB / 44.2 MiB = 0.107   358 KiB/s       2:06

That way to set lzma2 option only loose 0.8 KiB here and should work with
various size to be compressed.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Lasse Collin | 8 Mar 2012 14:20
Favicon

Re: Oddities with --lzma2 options

On 2012-03-05 Gilles Espinasse wrote:
> I find strange here that with a dictionary size even a bit bigger
> than with bare -8e the compressed file is a bit bigger.

This can sometimes happen, but it shouldn't be the common case. A bigger
dictionary might allow encoding some sections of the file better, but
it can cause the internal state to be less optimal for other sections
of the file. So with bad luck a bigger dictionary gives a tiny bit
bigger output.

> Trying to add -e doesnt change the result (and time to compress) when
> nice=273 depth=512 are set.

If you specify something like

    xz --lzma2 -e

the -e option is ignored. The -e only affects the presets -0 ... -9. If
you want to take -8e as the starting point and then adjust the
dictionary size, use this:

    xz --lzma2=preset=8e,dict=${DICT}KiB

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode
Permalink | Reply | 0 comments |
headers
Mike Melanson | 28 Mar 2012 03:19

The right data for Embedded XZ?

Hi,

I am trying to use Embedded XZ in a project. Is there something special I
should know about how to compress data so that it is suitable for decoding
by Embedded XZ? Is it supposed to work with any valid .xz file? Or is it
supposed to use raw .xz files?

Here is what I have done: Compress a file as:

  xz file

So I have file.xz. I wrote a C program to load that entire file into an
array, call xz_dec_init() and then xz_dec_run(), with all the right xz_buf
parameters filled in. xz_dec_run() always returns XZ_DATA_ERROR, which
xz.h defines as meaning the data is corrupt. I figured that perhaps I'm
not supposed to include the 6-byte magic header (FD 37 7A 58 5A 00) so I
defined xz_buf.in_pos to skip those bytes. This changes the returned error
to XZ_FORMAT_ERROR.

How should I compress the data so that Embedded XZ likes it? Or have I got
the wrong problem?

Thanks,
--
    -Mike Melanson
Permalink | Reply | 8 comments |
headers
g.esp | 28 Mar 2012 08:47
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Mercredi 28 Mars 2012 03:19:09
> Objet: [xz-devel] The right data for Embedded XZ?
> 
> Hi,
> 
> I am trying to use Embedded XZ in a project. Is there something special I
> should know about how to compress data so that it is suitable for decoding
> by Embedded XZ? Is it supposed to work with any valid .xz file? Or is it
> supposed to use raw .xz files?
> 
> Here is what I have done: Compress a file as:
> 
>   xz file
> 
> So I have file.xz. I wrote a C program to load that entire file into an
> array, call xz_dec_init() and then xz_dec_run(), with all the right xz_buf
> parameters filled in. xz_dec_run() always returns XZ_DATA_ERROR, which
> xz.h defines as meaning the data is corrupt. I figured that perhaps I'm
> not supposed to include the 6-byte magic header (FD 37 7A 58 5A 00) so I
> defined xz_buf.in_pos to skip those bytes. This changes the returned error
> to XZ_FORMAT_ERROR.
> 
> How should I compress the data so that Embedded XZ likes it? Or have I got
> the wrong problem?
> 
> Thanks,
(Continue reading)

Permalink | Reply | 7 comments |
headers
Mike Melanson | 28 Mar 2012 09:01

Re: The right data for Embedded XZ?

On 3/27/2012 11:47 PM, g.esp@... wrote:
> I have used xzminidec code unchanged and --check=crc32 has to be used during compression with xz as this is
the onlyt crc supported by xzminidec.
>
> Depending of data to be compressed size, setting dictionary size on compression to the size of
uncompressed data help to minimize memory requirement on decompression.
> To create a floppy image, Lasse advised me to use xz --check=crc32 --lzma2=dict=2MiB,nice=273,depth=512

Thanks for the tips. I'm still having the same difficulty with these 
options, though (xz_dec_run() just returns XZ_DATA_ERROR).

Is xzminidec the same as Embedded XZ that I'm trying to use?

Thanks,
--

-- 
     -Mike Melanson
Permalink | Reply | 6 comments |
headers
g.esp | 28 Mar 2012 17:00
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Mercredi 28 Mars 2012 09:01:24
> Objet: Re: [xz-devel] The right data for Embedded XZ?
> 
> On 3/27/2012 11:47 PM, g.esp@... wrote:
> > I have used xzminidec code unchanged and --check=crc32 has to be
> > used during compression with xz as this is the onlyt crc supported
> > by xzminidec.
> >
> > Depending of data to be compressed size, setting dictionary size on
> > compression to the size of uncompressed data help to minimize
> > memory requirement on decompression.
> > To create a floppy image, Lasse advised me to use xz --check=crc32
> > --lzma2=dict=2MiB,nice=273,depth=512
> 
> Thanks for the tips. I'm still having the same difficulty with these
> options, though (xz_dec_run() just returns XZ_DATA_ERROR).
> 
> Is xzminidec the same as Embedded XZ that I'm trying to use?
> 
> Thanks,
> --
>      -Mike Melanson
> 
> 
Yes, using
cd userspace && make
(Continue reading)

Permalink | Reply | 5 comments |
headers
Mike Melanson | 29 Mar 2012 01:52

Re: The right data for Embedded XZ?

> Yes, using
> cd userspace && make
> should compile xzminidec for you.
>
> I hacked the makefile a bit more to compile against klibc.

Ah, I hadn't seen this sample app before. Unfortunately, building on my
Ubuntu Linux system failed with:

gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86 -DXZ_DEC_POWERPC
-DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
-DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o boottest.o
boottest.c
In file included from ../linux/lib/decompress_unxz.c:235:0,
                 from boottest.c:22:
../linux/lib/xz/xz_dec_lzma2.c: In function ‘xz_dec_lzma2_run’:
/usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining failed in
call to ‘memmove’: redefined extern inline functions are not considered
for inlining
../linux/lib/xz/xz_dec_lzma2.c:884:11: sorry, unimplemented: called from here
make: *** [boottest.o] Error 1

*However!* Studying the source code in that directory demonstrated what
was wrong in my own sample app-- I need to call xz_crc32_init() before the
other functions. I see that's mentioned near the end of xz.h; perhaps it
warrants an earlier mention.

Anyway, I got past that problem. It should be noted that "--check=crc32"
really is necessary for compressed data if Embedded XZ will be chewing on
it. The library returns XZ_OPTIONS_ERROR otherwise, but only on the first
(Continue reading)

Permalink | Reply | 4 comments |
headers
Thorsten Glaser | 29 Mar 2012 15:59
Picon
Gravatar

Re: The right data for Embedded XZ?

Mike Melanson dixit:

>gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86 -DXZ_DEC_POWERPC
     ^^^^^^^^^^
You probably want -std=gnu99 here.

>-DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
>-DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o boottest.o
>boottest.c
>In file included from ../linux/lib/decompress_unxz.c:235:0,
>                 from boottest.c:22:
>../linux/lib/xz/xz_dec_lzma2.c: In function xz_dec_lzma2_run:
>/usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining failed in
>call to memmove: redefined extern inline functions are not considered
>for inlining

Yes well, that will of course break. (Huh. So there i̲s̲ a
legitimate case for that error message. I usually only see
it when compiling mksh with -fwhole-program --combine on
about three differently broken GCC versions. Luckily there
is now LTO which usually works.)

bye,
//mirabilos
--

-- 
> Hi, does anyone sell openbsd stickers by themselves and not packaged
> with other products?
No, the only way I've seen them sold is for $40 with a free OpenBSD CD.
	-- Haroon Khalid and Steve Shockley in gmane.os.openbsd.misc
(Continue reading)

Permalink | Reply | 1 comment |
headers
g.esp | 29 Mar 2012 22:43
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Jeudi 29 Mars 2012 01:52:42
> Objet: Re: [xz-devel] The right data for Embedded XZ?
> 
> > Yes, using
> > cd userspace && make
> > should compile xzminidec for you.
> >
> > I hacked the makefile a bit more to compile against klibc.
> 
> Ah, I hadn't seen this sample app before. Unfortunately, building on
> my
> Ubuntu Linux system failed with:
> 
> gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86
> -DXZ_DEC_POWERPC
> -DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
> -DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o
> boottest.o
> boottest.c
> In file included from ../linux/lib/decompress_unxz.c:235:0,
>                  from boottest.c:22:
> ../linux/lib/xz/xz_dec_lzma2.c: In function ‘xz_dec_lzma2_run’:
> /usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining
> failed in
> call to ‘memmove’: redefined extern inline functions are not
> considered
(Continue reading)

Permalink | Reply | 0 comments |
headers
Lasse Collin | 31 Mar 2012 09:48
Favicon

Re: The right data for Embedded XZ?

On 2012-03-28 Mike Melanson wrote:
> *However!* Studying the source code in that directory demonstrated
> what was wrong in my own sample app-- I need to call xz_crc32_init()
> before the other functions. I see that's mentioned near the end of
> xz.h; perhaps it warrants an earlier mention.

XZ Embedded has been written primarily for the Linux kernel and there
you don't need xz_crc32_init() except in decompress_unxz.c. So in the
Linux context it's better to keep the xz_crc32_init() docs at the end of
xz.h. I'm sorry that you didn't notice this or the existence of
xzminidec earlier. I added a reference to xzminidec.c to README.

> Anyway, I got past that problem. It should be noted that
> "--check=crc32" really is necessary for compressed data if Embedded
> XZ will be chewing on it.

Yes. This is mentioned in linux/Documentation/xz.txt. The existence
of this file is pointed in README.

> The library returns XZ_OPTIONS_ERROR otherwise, but only on the first
> call. If you call xz_dec_run() again, decoding will proceed fine (and
> accurately). I figured this out when I made a mistake in my decode
> loop and didn't terminate on error.

Calling xz_dec_run() again after XZ_OPTIONS_ERROR leads to undefined
behavior in sense that I haven't thought what will happen. liblzma
would keep returning the same error code if one calls lzma_code() again
after an error, but in XZ Embedded I skipped such things to make the
code smaller.
(Continue reading)

Permalink | Reply | 0 comments |

Gmane