http://www.lessfs.com

Hi,

I saw your post on Tux3,
It might be nice to know that a high performance GPL data dedup
filesystem already exists. It offers good performance and compression
See : http://www.lessfs.com or http://sourceforge.net/projects/lessfs/
for more information.

Mark.

> Hi,
>
> The prototype for data deduplication for tux3 in userspace is finally complete. The implementation
closely follows the design
> doc posted on the mailing list earlier. It has been developed with the 25th Jan 2009 snapshot as the base.
The complete Tux3 
> code with data deduplication in userspace can be downloaded/ cloned from
http://bitbucket.org/kushal/tux3_dedup/ . 
> The code uses the openssl SHA1 implementation and requires openssl libs to be installed.
> The dleaf part of the tux3graph output after copying the same file twice with and without deduplication
are also attached along with the patch. 
> Keeping in mind archival storage as the main application of deduplication edits and deletes have not been
implemented yet. 
>
>   

Mailing List Archive dead?

Hello everybody,

the advertised link
  http://mailman.tux3.org/pipermail/tux3/
(from http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3)
doesn't work anymore.

Was that intented?

Regards,

Phil

Re: Mailing List Archive dead?

On Friday 10 April 2009, Philipp Marek wrote:
> http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3

No, it was collateral damage migrating the server to a new disk.  Fixed
now.

Regards,

Daniel

Formal description of versioned pointers

Hi guys,

We are students of Eindhoven University of Technology. As part of the Linux kernel course we were asked to
look at the tux3 filesystem. We decided to look at the versioned pointers method in particular. Please
find attached the resulting report.

Regards,

Martin Harwig
Mohammed El-Kebir

_______________________________________________
Tux3 mailing list
Tux3 <at> tux3.org
http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3

Deduplication?

Potential end user here, hope you don't mind the intrusion. 

I've read a bit here in the past on deduplication coming to Tux3.  I'm
thinking of this in terms of block-based deduplication, a la what we
have on WAFL with NetApp.  Will Tux3 have something along these lines?
Does it already?

We love dedupe for its application in virtualization environment (for
storing VMDK files specifically, which are great candicates for
deduplication).  However, I know of no other filesystem other than WAFL
that has this functionality, and others like ZFS don't seem to be in a
hurry to add it for whatever reason.  Maybe they don't see the value as
anything more than a niche thing more for backup administrators?

Thanks!
Ray

Re: Deduplication?

On Thursday 16 April 2009, Ray Van Dolson wrote:
> Potential end user here, hope you don't mind the intrusion. 
> 
> I've read a bit here in the past on deduplication coming to Tux3.  I'm
> thinking of this in terms of block-based deduplication, a la what we
> have on WAFL with NetApp.  Will Tux3 have something along these lines?
> Does it already?
> 
> We love dedupe for its application in virtualization environment (for
> storing VMDK files specifically, which are great candicates for
> deduplication).  However, I know of no other filesystem other than WAFL
> that has this functionality, and others like ZFS don't seem to be in a
> hurry to add it for whatever reason.  Maybe they don't see the value as
> anything more than a niche thing more for backup administrators?
> 
> Thanks!
> Ray

Our Pune Institute volunteers put in the work to demonstrate the
effectiveness of deduplication, and to get a respectable level of
functionality working.  This pretty much decides the question for me.
Block level deduplication will be supported by Tux3.

There are competing approaches to deduplication that should also be
investigated.  Deduplication can be implemented at three different
layers in the storage stack:

  1) Block level
  2) Filesystem level
  3) Stacked filesystem level

Re: Deduplication?

> On Thursday 16 April 2009, Ray Van Dolson wrote:
> > Potential end user here, hope you don't mind the intrusion. 
> >
> > I've read a bit here in the past on deduplication coming to Tux3.  I'm
> > thinking of this in terms of block-based deduplication, a la what we
> > have on WAFL with NetApp.  Will Tux3 have something along these lines?
> > Does it already?

Yes. As Daniel said, we have implemented block level deduplication in Tux3 
Fuse implementation. The code uses a btree based index and what we call 
'buckets' for faster search. The code with this implementation and collision 
handling is at www.bitbucket.org/cdkamat/tux3_dedup . You can try it .. not 
tested extensively :)

> >
> > We love dedupe for its application in virtualization environment (for
> > storing VMDK files specifically, which are great candicates for
> > deduplication).  However, I know of no other filesystem other than WAFL
> > that has this functionality, and others like ZFS don't seem to be in a
> > hurry to add it for whatever reason.  Maybe they don't see the value as
> > anything more than a niche thing more for backup administrators?
> >
> > Thanks!
> > Ray
>
> Our Pune Institute volunteers put in the work to demonstrate the
> effectiveness of deduplication, and to get a respectable level of
> functionality working.  This pretty much decides the question for me.
> Block level deduplication will be supported by Tux3.
>

Re: Formal description of versioned pointers

On Saturday 11 April 2009, Harwig, M.P. wrote:
> Hi guys,
> 
> We are students of Eindhoven University of Technology. As part of the Linux kernel course we were asked to
look at the tux3 filesystem. We decided to look at the versioned pointers method in particular. Please
find attached the resulting report.
> 
> Regards,
> 
> Martin Harwig
> Mohammed El-Kebir

Hi,

I have been through it lightly, but not deep enough to debate whether
the delete algorithm is O(versions) or O(entries) yet.  Soon.

One thing: Phillips as in Daniel Phillips is spelled with two l's.

Regards,

Daniel

Current Activities?

Hi Daniel,

I want to apologize upfront if I sound like one of those "when will it
be done" questions that are best left unasked with most open source
projects.  Actually, I'm just really curious about what's been going
on lately.  I am somewhat embarrassed to admit that I became
fascinated with tux3's development since the first time it was
featured in an article on kerneltrap.  I've greatly enjoyed reading
your design notes and dialog with the btrfs team.  I had no naive
hopes that tux3 would get integrated into the linux kernel
immediately, but I've been extremely surprised at the loss of public
progress notes from you after the initial review request back and
forth died off.  In fact it seems like there's only been 7 postings
from you in the month since the last kernel merge related thread.

I've really missed the public view into tux3's progress.  If it's not
too presumptuous, how are you? How's tux3 coming along?  What part of
the kernel port is taking the bulk of the work?  What new and
interesting challenges have you been wrestling with?

Thanks!
Steve

Re: Current Activities?

On Wednesday 29 April 2009, t3right.thebashar <at> xoxy.net wrote:
> Hi Daniel,
> 
> I want to apologize upfront if I sound like one of those "when will it
> be done" questions that are best left unasked with most open source
> projects.  Actually, I'm just really curious about what's been going
> on lately.  I am somewhat embarrassed to admit that I became
> fascinated with tux3's development since the first time it was
> featured in an article on kerneltrap.  I've greatly enjoyed reading
> your design notes and dialog with the btrfs team.  I had no naive
> hopes that tux3 would get integrated into the linux kernel
> immediately, but I've been extremely surprised at the loss of public
> progress notes from you after the initial review request back and
> forth died off.  In fact it seems like there's only been 7 postings
> from you in the month since the last kernel merge related thread.
> 
> I've really missed the public view into tux3's progress.  If it's not
> too presumptuous, how are you? How's tux3 coming along?  What part of
> the kernel port is taking the bulk of the work?  What new and
> interesting challenges have you been wrestling with?

Actually, I have been busy with real life for the last couple
of months.  An interesting challenge is how I can keep up the
previous development without any corporate support.  Zero.  Zip.
Nada.  There has been exactly zero support from the usual
suspects, who all have their own good reasons no doubt, which
do not necessarily have much to do with the good of the Linux
kernel or the Linux community.

That challenge is being addressed.  Addressing it takes time and