Tom Trauth | 6 Nov 14:17 2011
Picon
Picon

Is the xz format stable?

Hi!  I am trying to submit a patch to an open source project to add xz support to it, but before accepting it the maintainer wants me to get a promise from the xz developers that the xz format is now stable and will have no backwardly incompatible modifications in the future.  As far as I can tell it has been stable for over two years, and the fact that GNU and kernel.org are now using it extensively seems like proof enough of its stability to me, and I argued that point.  But he apparently had a bad experience with the lzma format changing its format several times and therefore does not trust xz.

 

So, can you give me a promise that the xz is stable and that no backwardly incompatible modifications will be made to its format so that I can tell the maintainer of the project that upstream has promised the xz is stable and it is therefore safe to accept my patch?  Thanks.

 

In case you are curious, my patch lets the program handle patched files compressed with xz (it already handles patches compressed with gzip and bzip2).

 

Regards,

 

Tom

Lasse Collin | 6 Nov 15:11 2011

Re: Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> I am trying to submit a patch to an open source project to add xz
> support to it, but before accepting it the maintainer wants me to get
> a promise from the xz developers that the xz format is now stable and
> will have no backwardly incompatible modifications in the future.

It is stable in sense that new tools will always be able to decompress
old .xz files that have been created with a stable release of XZ Utils.
It is possible and even somewhat likely that new features will be added
in the future which old programs won't support.

Compare to the .zip format. It has got support for new compression
methods and other features over the years, including LZMA support.
When maximum portability is needed, people stick to the Deflate
algorithm which all non-ancient .zip implementations support.

> But he
> apparently had a bad experience with the lzma format changing its
> format several times and therefore does not trust xz.

The old .lzma format hasn't changed since it was introduced in LZMA
SDK and also used by LZMA Utils. There were development versions of
the .xz format that used also the .lzma suffix, but no one has claimed
that those alpha versions would be stable. If someone has thought the
development versions were stable, it has been a major misunderstanding.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Tom Trauth | 6 Nov 15:15 2011
Picon
Picon

RE: Is the xz format stable?

Thanks.

-----Original Message-----
From: owner-xz-devel@...
[mailto:owner-xz-devel@...] On
Behalf Of Lasse Collin
Sent: Sunday, November 06, 2011 9:11 AM
To: xz-devel@...
Subject: Re: [xz-devel] Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> I am trying to submit a patch to an open source project to add xz
> support to it, but before accepting it the maintainer wants me to get
> a promise from the xz developers that the xz format is now stable and
> will have no backwardly incompatible modifications in the future.

It is stable in sense that new tools will always be able to decompress
old .xz files that have been created with a stable release of XZ Utils.
It is possible and even somewhat likely that new features will be added
in the future which old programs won't support.

Compare to the .zip format. It has got support for new compression
methods and other features over the years, including LZMA support.
When maximum portability is needed, people stick to the Deflate
algorithm which all non-ancient .zip implementations support.

> But he
> apparently had a bad experience with the lzma format changing its
> format several times and therefore does not trust xz.

The old .lzma format hasn't changed since it was introduced in LZMA
SDK and also used by LZMA Utils. There were development versions of
the .xz format that used also the .lzma suffix, but no one has claimed
that those alpha versions would be stable. If someone has thought the
development versions were stable, it has been a major misunderstanding.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Tom Trauth | 6 Nov 16:50 2011
Picon
Picon

RE: Is the xz format stable?

Lasse,

The maintainer is happy that no backwardly-incompatible changes will be
made, but is still a bit concerned that older versions of xz utils may not
work with newer versions of the xz format.  Given an xz file, is there a way
to determine which version of the xz format it uses.  Something like:

xz-get-version foo.xz --> foo.xz uses XZ format version 1.0.4

Thanks.

Regards,

Tom

-----Original Message-----
From: owner-xz-devel@...
[mailto:owner-xz-devel@...] On
Behalf Of Lasse Collin
Sent: Sunday, November 06, 2011 9:11 AM
To: xz-devel@...
Subject: Re: [xz-devel] Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> I am trying to submit a patch to an open source project to add xz
> support to it, but before accepting it the maintainer wants me to get
> a promise from the xz developers that the xz format is now stable and
> will have no backwardly incompatible modifications in the future.

It is stable in sense that new tools will always be able to decompress
old .xz files that have been created with a stable release of XZ Utils.
It is possible and even somewhat likely that new features will be added
in the future which old programs won't support.

Compare to the .zip format. It has got support for new compression
methods and other features over the years, including LZMA support.
When maximum portability is needed, people stick to the Deflate
algorithm which all non-ancient .zip implementations support.

> But he
> apparently had a bad experience with the lzma format changing its
> format several times and therefore does not trust xz.

The old .lzma format hasn't changed since it was introduced in LZMA
SDK and also used by LZMA Utils. There were development versions of
the .xz format that used also the .lzma suffix, but no one has claimed
that those alpha versions would be stable. If someone has thought the
development versions were stable, it has been a major misunderstanding.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Jonathan Nieder | 6 Nov 17:58 2011
Picon

Re: Is the xz format stable?

Hi Tom,

Tom Trauth wrote:

> Given an xz file, is there a way
> to determine which version of the xz format it uses.  Something like:
>
> xz-get-version foo.xz --> foo.xz uses XZ format version 1.0.4

Can you say more about the use case?  Would "xz --test" work?

The xz file format does not have a version number field --- instead,
there are reserved fields in various places.  This is convenient for
implementation: multiple new features can be developed in parallel
without knowing in advance which is going to be adopted first.  It is
also convenient for interoperability: a decoder supporting one feature
and not another does not need to lie via its interpretation of a
version number field.

Hope that helps,
Jonathan

Tom Trauth | 6 Nov 18:12 2011
Picon
Picon

RE: Is the xz format stable?

Jonathan,

The use case is as follows.  I have xz-utils 5.0.3 and an xz file that was
created with a future xz-utils 5.0.4 that uses a new feature that is not
recognized by xz-utils 5.0.3, so the xz utility in xz-utils 5.0.3 cannot
uncompress it.  Is there a way for xz to check whether the xz file that it
is trying uncompress is compatible with it, or that it uses a new feature
that would require an upgrade to a newer version of xz-utils in order to
uncompress it?

Basically the maintainer wants to make sure that there is way to ensure that
the version of xz used in his project can check to make sure that any xz
file that it tries to uncompress is compatible with it.

Thanks.

Regards,

Tom

-----Original Message-----
From: owner-xz-devel@...
[mailto:owner-xz-devel@...] On
Behalf Of Jonathan Nieder
Sent: Sunday, November 06, 2011 11:58 AM
To: xz-devel@...
Subject: Re: [xz-devel] Is the xz format stable?

Hi Tom,

Tom Trauth wrote:

> Given an xz file, is there a way
> to determine which version of the xz format it uses.  Something like:
>
> xz-get-version foo.xz --> foo.xz uses XZ format version 1.0.4

Can you say more about the use case?  Would "xz --test" work?

The xz file format does not have a version number field --- instead,
there are reserved fields in various places.  This is convenient for
implementation: multiple new features can be developed in parallel
without knowing in advance which is going to be adopted first.  It is
also convenient for interoperability: a decoder supporting one feature
and not another does not need to lie via its interpretation of a
version number field.

Hope that helps,
Jonathan

Lasse Collin | 6 Nov 18:31 2011

Re: Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> Given an xz file,
> is there a way to determine which version of the xz format it uses.
> Something like:
> 
> xz-get-version foo.xz --> foo.xz uses XZ format version 1.0.4

Right now there is no way to get a version number of the format.

I could make xz -lvv show the oldest XZ Utils version that will
decompress the file. It can only work for files that are supported by
the xz tool, so it's not possible to make an old xz tool to display how
much newer xz is required for a given file; the old tool could only tell
that it doesn't support it. I don't know if this could be good enough
for you.

To understand the reason for the above, it's good to understand how
incompatible additions may happen:

(1) A new filter/method ID may be added into the official .xz format
    specification. Old tools will show that there is an unsupported
    filter ID and cannot decompress such files (will display an error).

(2) Third-party developers may use custom filter IDs which aren't in
    the official specification and aren't supported by XZ Utils. If
    they don't deviate from the .xz specification in any other way,
    this is OK. Old tools cannot distinguish this situation from (1).

(3) A new .xz format specification may add new features to the
    container format. The old tools will detect such files as
    unsupported (they won't claim them to be corrupt). With old tools,
    the difference to (1) and (2) is that the old tools won't be able
    to list even the filter IDs.

If incompatible additions are made, the xz tool won't use them by
default. Maybe they might become a default after several years have
passed and old xz versions aren't common anymore. But it won't be done
easily, because it would make people angry if the default settings
created files that many wouldn't be able to decompress without extra
work.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Tom Trauth | 6 Nov 18:59 2011
Picon
Picon

RE: Is the xz format stable?

Lasse,

This is what the maintainer wants, in his own words:
We need a way to verify that a specific  named version of xz, which might be
older than the version someone has  installed, will be able to uncompress a
particular file.

His main fear is that someone will create an xz file with a version of xz
which is newer than the version of xz that his project supports, and then
his project will not be able to read that file.  However, your last
paragraph implies xz, including future versions, will create a file that is
unreadable by current versions of xz unless special parameters are used,
because to do otherwise would anger a lot of people.  Is that true?  If so,
I think it will help allay his fears.

Regards,

Tom

-----Original Message-----
From: owner-xz-devel@...
[mailto:owner-xz-devel@...] On
Behalf Of Lasse Collin
Sent: Sunday, November 06, 2011 12:32 PM
To: xz-devel@...
Subject: Re: [xz-devel] Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> Given an xz file,
> is there a way to determine which version of the xz format it uses.
> Something like:
> 
> xz-get-version foo.xz --> foo.xz uses XZ format version 1.0.4

Right now there is no way to get a version number of the format.

I could make xz -lvv show the oldest XZ Utils version that will
decompress the file. It can only work for files that are supported by
the xz tool, so it's not possible to make an old xz tool to display how
much newer xz is required for a given file; the old tool could only tell
that it doesn't support it. I don't know if this could be good enough
for you.

To understand the reason for the above, it's good to understand how
incompatible additions may happen:

(1) A new filter/method ID may be added into the official .xz format
    specification. Old tools will show that there is an unsupported
    filter ID and cannot decompress such files (will display an error).

(2) Third-party developers may use custom filter IDs which aren't in
    the official specification and aren't supported by XZ Utils. If
    they don't deviate from the .xz specification in any other way,
    this is OK. Old tools cannot distinguish this situation from (1).

(3) A new .xz format specification may add new features to the
    container format. The old tools will detect such files as
    unsupported (they won't claim them to be corrupt). With old tools,
    the difference to (1) and (2) is that the old tools won't be able
    to list even the filter IDs.

If incompatible additions are made, the xz tool won't use them by
default. Maybe they might become a default after several years have
passed and old xz versions aren't common anymore. But it won't be done
easily, because it would make people angry if the default settings
created files that many wouldn't be able to decompress without extra
work.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Lasse Collin | 6 Nov 19:41 2011

Re: Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> This is what the maintainer wants, in his own words:
> We need a way to verify that a specific  named version of xz, which
> might be older than the version someone has  installed, will be able
> to uncompress a particular file.

When compressing with a future version of xz, don't enable incompatible
features. If you already have a .xz file and want to find out if an old
xz will support it too, there's no simple way right now, but I think I
will make xz -lvv show the minimum required XZ Utils version.

> His main fear is that someone will create an xz file with a version
> of xz which is newer than the version of xz that his project
> supports, and then his project will not be able to read that file.

Such a situation will probably be possible in the future if someone
enables an incompatible feature when compressing.

A comparable situation is technically possible even now because it is
possible to have a .xz file that requires 4 GiB of memory to
decompress, which is too much for many systems. No one creates such
files in practice though. :-)

If one wants to be extra safe, one could define what features and
memory usage are allowed to guarantee compatibility. This is already
required when creating files for XZ Embedded, which doesn't support
all .xz features. XZ Embedded is used e.g. in Linux as an option to
compress the kernel and initramfs and for Squashfs images.

> However, your last paragraph implies xz, including future versions,
> will create a file that is unreadable by current versions of xz
> unless special parameters are used, because to do otherwise would
> anger a lot of people.  Is that true?  If so, I think it will help
> allay his fears.

Assuming that you meant "will not create", yes, it is true with one
possible exception: If a new very nice but incompatible feature is added
in 2012, I might consider making that a default in 2018-2020. At point
the old versions should have pretty much vanished. Even then it will
be possible to create files that are compatible with XZ Utils 5.0.0 by
using extra options.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode

Tom Trauth | 6 Nov 20:20 2011
Picon
Picon

RE: Is the xz format stable?

Thanks.

-----Original Message-----
From: owner-xz-devel@...
[mailto:owner-xz-devel@...] On
Behalf Of Lasse Collin
Sent: Sunday, November 06, 2011 1:42 PM
To: xz-devel@...
Subject: Re: [xz-devel] Is the xz format stable?

On 2011-11-06 Tom Trauth wrote:
> This is what the maintainer wants, in his own words:
> We need a way to verify that a specific  named version of xz, which
> might be older than the version someone has  installed, will be able
> to uncompress a particular file.

When compressing with a future version of xz, don't enable incompatible
features. If you already have a .xz file and want to find out if an old
xz will support it too, there's no simple way right now, but I think I
will make xz -lvv show the minimum required XZ Utils version.

> His main fear is that someone will create an xz file with a version
> of xz which is newer than the version of xz that his project
> supports, and then his project will not be able to read that file.

Such a situation will probably be possible in the future if someone
enables an incompatible feature when compressing.

A comparable situation is technically possible even now because it is
possible to have a .xz file that requires 4 GiB of memory to
decompress, which is too much for many systems. No one creates such
files in practice though. :-)

If one wants to be extra safe, one could define what features and
memory usage are allowed to guarantee compatibility. This is already
required when creating files for XZ Embedded, which doesn't support
all .xz features. XZ Embedded is used e.g. in Linux as an option to
compress the kernel and initramfs and for Squashfs images.

> However, your last paragraph implies xz, including future versions,
> will create a file that is unreadable by current versions of xz
> unless special parameters are used, because to do otherwise would
> anger a lot of people.  Is that true?  If so, I think it will help
> allay his fears.

Assuming that you meant "will not create", yes, it is true with one
possible exception: If a new very nice but incompatible feature is added
in 2012, I might consider making that a default in 2018-2020. At point
the old versions should have pretty much vanished. Even then it will
be possible to create files that are compatible with XZ Utils 5.0.0 by
using extra options.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode


Gmane