ty armour | 24 Jan 21:50 2016

Tutorial suggesitons

I have a theory on data compression, but I need tutorials on data compression in order to implement my theory.

Anyway if you want to, just post tutorials on every and any aspect of data execution and data compression.

I am building my own computers(like my own microchips) and this will help me out greatly to not only have a data compression software, but to also be able to save money and not need nearly as much ram.

Thanks for your time
Rich Prohaska | 3 Nov 17:21 2015

valgrind conditional jump depends on uninitialized value

Valgrind is reporting a uninitialized value problem when using xz library.

$ valgrind ./lzma-uninit-prepare
==21883== Memcheck, a memory error detector
==21883== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==21883== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==21883== Command: ./lzma-uninit-prepare
lzma 5.2.2
1 148
==21883== Conditional jump or move depends on uninitialised value(s)
==21883==    at 0x4E4329D: lz_encoder_prepare (lz_encoder.c:231)
==21883==    by 0x4E43902: lzma_lz_encoder_init (lz_encoder.c:560)
==21883==    by 0x4E39D9B: lzma_raw_coder_init (filter_common.c:286)
==21883==    by 0x4E3B6AC: block_encode_normal (block_buffer_encoder.c:189)
==21883==    by 0x4E3B6AC: block_buffer_encode.part.1
==21883==    by 0x4E3B9A0: block_buffer_encode (block_buffer_encoder.c:322)
==21883==    by 0x4E3B9A0: lzma_block_buffer_encode (block_buffer_encoder.c:323)
==21883==    by 0x4E3CDE7: lzma_stream_buffer_encode
==21883==    by 0x4E3C0C3: lzma_easy_buffer_encode (easy_buffer_encoder.c:25)
==21883==    by 0x400913: main (lzma-uninit-prepare.c:21)

mf->size is read is lz_encoder.c:226 before initialized. mf is
allocated in lz_encoder.c:532, buffer is iniitalized, size is NOT
initialized.  gcc 4.9 apparently compiles lz_encoder.c:231 into code
that uses old_size before the buffer != NULL check occurs.

set mf->size = 0 after allocation in lz_encoder.c:532.

git clone http://git.tukaani.org/xz.git && git checkout v5.2.2
./configure --prefix=$HOME/usr/local/xz && make install
make and run my reproducer.

$ cat Makefile
XZBASE = $(HOME)/usr/local/xz
CPPFLAGS = -I$(XZBASE)/include
CFLAGS = -g -O0 -std=c99
LDFLAGS = -L$(XZBASE)/lib -llzma

lzma-uninit-prepare: lzma-uninit-prepare.c
        $(CC) $(CPPFLAGS) $(CFLAGS) -o $ <at>  $< $(LDFLAGS)

        rm -rf lzma-uninit-prepare

$ cat lzma-uninit-prepare.c
#include <stdio.h>
#include <assert.h>
#include <malloc.h>
#include <lzma.h>

int main(void) {
    printf("lzma %s\n", lzma_version_string());
    size_t src_len = 1;
    uint8_t *src = (uint8_t *) malloc(src_len);
    for (int i = 0; i < src_len; i++)
        src[i] = 0;
    size_t compress_bound = lzma_stream_buffer_bound(src_len);
    printf("%lu %lu\n", src_len, compress_bound);

    size_t dest_len = 1 + compress_bound;
    uint8_t *dest = (uint8_t *) malloc(dest_len);

    size_t compress_size = 1;
    lzma_ret lr = lzma_easy_buffer_encode(2, LZMA_CHECK_NONE, NULL,
src, src_len, dest, &compress_size, dest_len);
    assert(lr == LZMA_OK);
    printf("%lu\n", compress_size);


    return 0;

Ian S. Nelson | 23 Oct 15:58 2015


subscribe xz-devel

Ian S. Nelson
PGP/GPG email preferred.
Public Key: 00D3D983
Fingerprint: 3EFD7B86B888D7E229B69E97576F1B9700D3D983

Lasse Collin | 29 Sep 14:53 2015

XZ Utils 5.2.2

XZ Utils 5.2.2 is available at <http://tukaani.org/xz/>. Here is an 
extract from the NEWS file:

  * Fixed bugs in QNX-specific code.

  * Omitted the use of pipe2() even if it is available to avoid
    portability issues with some old Linux and glibc combinations.

  * Updated German translation.

  * Added project files to build static and shared liblzma (not the
    whole XZ Utils) with Visual Studio 2013 update 2 or later.

  * Documented that threaded decompression hasn't been implemented
    yet. A 5.2.0 NEWS entry describing multi-threading support had
    incorrectly said "decompression" when it should have said

The 5.0.x branch is no longer maintained. As of 2015-09-29 there are no
known major bugs and it is fine to keep using 5.0.8 for now if you
don't want to upgrade to 5.2.x. The API and ABI in 5.2.x are backward
compatible with 5.0.x so it should be straightforward to upgrade from
5.0.x to 5.2.x.


Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Adam Walling | 22 May 20:10 2015

minimal liblzma MSVC2013 solution and project

This is a minimal solution and two project files which allow liblzma to be built with MSVC2013. I did not attempt to include any tools or tests, or address any warnings. However, it is functional, so I would like to share this as an alternative to the 'Fairly Complete' solution / project.

This does not require any changes to the code itself, just adding the files within the /windows subdirectory is enough.

liblzma.vcxproj builds the static library
liblzma_dll.vcxproj builds a dll

Both projects have x86 and x86-64 platform configurations, as well as Debug, Release, and ReleaseMT configuration -- MT is the compiler switch to link to the CRT statically, so it will not have any other DLL dependencies.

I just put the files up on github alone rather than fork the entire repository:

I created these a while ago but put things on hold waiting for the other solution and projects to materialize, but it has been long enough now I figure I might as well share.



- Adam D. Walling
Hochhaus, Andy | 11 May 09:13 2015

xz 5.2.1 multi-threaded decompress problems (mutli-threaded compress works file)


I am using xz 5.2.1 (compiled from source) on Debian Jessie.

$ xz --version
xz (XZ Utils) 5.2.1
liblzma 5.2.1

When I run parallel compress I see all cores utilized (checked with htop).

xz -T 0 2275.dat

However, when I attempt to run parallel decompress I only see a single
core utilized.

xz -d -T 0 2275.dat.xz

The file I'm working with is sufficiently large (2.3G compressed ; 46G
uncompressed) that I would expect significant benefit from
decompression parallelism.

I noticed that a few other users in the phoronix forum [1] were seeing
similar behavior. Am I missing something obvious?


[1] http://www.phoronix.com/forums/forum/software/general-linux-open-source/47329-xz-5-2-adds-new-multi-threaded-options

Lasse Collin | 26 Feb 18:12 2015

XZ Utils 5.2.1

XZ Utils 5.2.1 is available at <http://tukaani.org/xz/>. Here is an 
extract from the NEWS file:

  * Fixed a compression-ratio regression in fast mode of LZMA1 and
    LZMA2. The bug is present in 5.1.4beta and 5.2.0 releases.

  * Fixed a portability problem in xz that affected at least OpenBSD.

  * Fixed xzdiff to be compatible with FreeBSD's mktemp which differs
    from most other mktemp implementations.

  * Changed CPU core count detection to use cpuset_getaffinity() on


Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Gabi Davar | 23 Jan 12:19 2015

Fairly Complete MSVC 2013 Solution/Project


I've created vs2013 based solution for lzma DLL, lzma static and most
utils. It based on the works of Garen & Mārtiņš Možeiko.
It can be found at https://github.com/mindw/xz vs2013_520 branch.

The cmake branch has the build system ported to cmake (only for VS).
It enabled me to build xz for Python 2.7 use. (Intel C++ Compiler
targeted at VS2008).

Any interest?


Dusty Mabe | 5 Jan 05:21 2015

Possible bug in xz/lzma compression


I am noticing an error when compressing/decompressing using xz on Fedora 20. I 
was initially using xz-5.1.2-8alpha.fc20.x86_64 but then upgraded to 
xz-5.1.2-12alpha.fc20.x86_64 to see if that would fix things, but it didn't seem
to help.

I have a file at [1] that I am compressing/decompressing using xz. I first uncompress
it so that it is a plain text file and then compress it with xz. I then decompress it
back to plain text and compare with the original. I keep getting differences.. For example:

[dustymabe <at> localhost traces]$ cat /tmp/diffs 
--- /mnt/SEQ_LBM.t.gz.uncompressed      2015-01-04 17:14:33.178364955 -0500
+++ /mnt/SEQ_LBM.t.xz.uncompressed      2015-01-04 20:32:20.951230115 -0500
 <at>  <at>  -125953555,7 +125953555,7  <at>  <at> 
 r 0x7f0422a8
 w 0x7a7a0218
 r 0x6d480000
-r 0x7f0422a8
+p 0x7f0422a8
 w 0x7a7a00e0
 r 0x6d480008
 r 0x7f0422a8

Can someone please download the file at [1] (~194M) and run a similar test on it? 
The sums on the file are as follows for the gz compressed and uncompressed forms:

[dustymabe <at> localhost traces]$ sha256sum SEQ_LBM.t.gz /mnt/SEQ_LBM.t.gz.uncompressed 
017b3456df8e658e2e50ff3f33ca66ce11eb91b59abfd6c3d259502b53951608  SEQ_LBM.t.gz
ca07c3604b104631bf377cac62537a40bd79be1c3971a23e7605843d5c588e72  /mnt/SEQ_LBM.t.gz.uncompressed

Please confirm if you see the behavior or not or possibly point me to a bug/commit
where this issue was already fixed.


[1] - https://www.dropbox.com/s/mebey21eynzt1lq/SEQ_LBM.t.gz?dl=0
Lasse Collin | 21 Dec 22:08 2014

XZ Utils 5.0.8 and 5.2.0

XZ Utils 5.0.8 and 5.2.0 are available at <http://tukaani.org/xz/>.
Here is an extract from the NEWS file:

5.0.8 (2014-12-21)

    * Fixed an old bug in xzgrep that affected OpenBSD and probably
      a few other operating systems too.

    * Updated French and German translations.

    * Added support for detecting the amount of RAM on AmigaOS/AROS.

    * Minor build system updates.

5.2.0 (2014-12-21)

    Since 5.1.4beta:

    * All fixes from 5.0.8

    * liblzma: Fixed lzma_stream_encoder_mt_memusage() when a preset
      was used.

    * xzdiff: If mktemp isn't installed, mkdir will be used as
      a fallback to create a temporary directory. Installing mktemp
      is still recommended.

    * Updated French, German, Italian, Polish, and Vietnamese

    Summary of fixes and new features added in the 5.1.x development

    * liblzma:

        - Added support for multi-threaded compression. See the
          lzma_mt structure, lzma_stream_encoder_mt(), and
          lzma_stream_encoder_mt_memusage() in <lzma/container.h>,
          lzma_get_progress() in <lzma/base.h>, and lzma_cputhreads()
          in <lzma/hardware.h> for details.

        - Made the uses of lzma_allocator const correct.

        - Added lzma_block_uncomp_encode() to create uncompressed
          .xz Blocks using LZMA2 uncompressed chunks.

        - Added support for LZMA_IGNORE_CHECK.

        - A few speed optimizations were made.

        - Added support for symbol versioning. It is enabled by default
          on GNU/Linux, other GNU-based systems, and FreeBSD.

        - liblzma (not the whole XZ Utils) should now be buildable
          with MSVC 2013 update 2 or later using windows/config.h.

    * xz:

        - Fixed a race condition in the signal handling. It was
          possible that e.g. the first SIGINT didn't make xz exit
          if reading or writing blocked and one had bad luck. The fix
          is non-trivial, so as of writing it is unknown if it will be
          backported to the v5.0 branch.

        - Multi-threaded decompression can be enabled with the
          --threads (-T) option.

        - New command line options in xz: --single-stream,
          --block-size=SIZE, --block-list=SIZES,
          --flush-timeout=TIMEOUT, and --ignore-check.

        - xz -lvv now shows the minimum xz version that is required to
          decompress the file. Currently it is 5.0.0 for all supported
          .xz files except files with empty LZMA2 streams require 5.0.2.

    * xzdiff and xzgrep now support .lzo files if lzop is installed.
      The .tzo suffix is also recognized as a shorthand for .tar.lzo.


Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Kevin Wilson | 9 Nov 19:24 2014

How to compress a folder with xz ?

Hi, all,

The forum webpage give an error. So I hope you will forgive me
for sending to xz-devel, in the hope that someone may advice.

According to the man page of xz, it seems that
xz folderName or
xz folderName/
is enough.
However, when running:
xz /etc/
I get;
xz: /etc/: Is a directory, skipping

I saw that it is possible with the tar command, with soem command options,
but isn't it possible with xz ?