We need a Bittorrent style hash file

To properly do multisource we are going to need to have a Bittorrent
style hash file.

The way things are set up now, one small glitch in a file and you have
to wait until the final hash check to find out your file is corrupted.

I suggest that each 1MB section of a file have a simple 16 bit CRC
generated. When asking for info on a file, a list of all the CRCs are
sent back so the downloader can check each 1MB section as it comes in.

If one of the 1MB sections is found to be bad, the client can retry from
that source again and/or switch to a different source to obtain a
correct section.

Later on there could be a verified list of files and their CRC lists put
up on some web site out there to weed out the bad files. Until then
there will always be the possibility that the first source you find for
a file may have a corrupted CRC list and the client would just keep on
trying until you manually pick a new initial source.

If the info file format was kept simple, like the CRC in hex (four
characters) and then a comma, then the next CRC, for a 600MB file the
info file packet would only be about 3K. I suggest a CRC because it's
very fast to generate and check on the fly. A complete hash check is
always done when finished, like we do now.

--

-- 
http://www.fastmail.fm - A no graphics, no pop-ups email service

-------------------------------------------------------

Re: We need a Bittorrent style hash file

that sounds reasonable... 

I know that Jason investigated the tiger-tree hash and
said it was fairly complex... 

this would at least speed things up some.. 

--- Nate <fileman <at> fastmail.fm> wrote:

> To properly do multisource we are going to need to
> have a Bittorrent
> style hash file.
> 
> The way things are set up now, one small glitch in a
> file and you have
> to wait until the final hash check to find out your
> file is corrupted.
> 
> I suggest that each 1MB section of a file have a
> simple 16 bit CRC
> generated. When asking for info on a file, a list of
> all the CRCs are
> sent back so the downloader can check each 1MB
> section as it comes in.
> 
> If one of the 1MB sections is found to be bad, the
> client can retry from
> that source again and/or switch to a different
> source to obtain a
> correct section.

Re: We need a Bittorrent style hash file

of course there is still the issue of rogue clients
altering the CRCs and what not.. but I guess that
would eliminate them as a host if they sent too many
bad CRCs.

--- Nate <fileman <at> fastmail.fm> wrote:

> To properly do multisource we are going to need to
> have a Bittorrent
> style hash file.
> 
> The way things are set up now, one small glitch in a
> file and you have
> to wait until the final hash check to find out your
> file is corrupted.
> 
> I suggest that each 1MB section of a file have a
> simple 16 bit CRC
> generated. When asking for info on a file, a list of
> all the CRCs are
> sent back so the downloader can check each 1MB
> section as it comes in.
> 
> If one of the 1MB sections is found to be bad, the
> client can retry from
> that source again and/or switch to a different
> source to obtain a
> correct section.
> 
> Later on there could be a verified list of files and

Re: We need a Bittorrent style hash file

On Tue, 11 Apr 2006 14:02:53 -0700 (PDT), "Eric" <jrocnuck <at> yahoo.com>
said:
> of course there is still the issue of rogue clients
> altering the CRCs and what not.. but I guess that
> would eliminate them as a host if they sent too many
> bad CRCs.

That's what an "external" catalog/database of all files would be for.
The community would have to get involved in reporting bad file info to
the central database.

This would be incompatible with other formats and would make the catalog
much larger if it also contained torrents. So it may be better to just
copy what bittorrent does, or support a part of it so that it's easy to
use torrent information and/or extract what we need.

And there's no guarantee that the bittorrent format won't change in the
future. I don't see a need to make this too complicated, all you need is
a general check of each section since you do a complete SHA1 at the end.
The bittorrent format is probably not based on our chunk size either, so
it may be way more complicated to be compatible.

It would be possible to create something that would fix corrupted files
if you knew what section was bad so that you don't have to download a
1GB file again.

I am not sure if we need a completely new file info request to do this,
or that we can just add the info to the current info reply if the
request contains some sort of flag.

Speed routing concept

If the speed of each connection is monitored we can use that to make
routing tend to pick the fastest route. (using average over time)

If all nodes did this then a sort of automatic high speed backbone would
form. It would even load balance itself.

The method I am thinking of would be if there are two or more routes
available for a packet, use the fastest one. This shouldn't be hard to
add to the routing code but the speed monitoring may take a bit of work.

As for security concerns, the speed of connections will go up and down
so it should tend to pick the fastest routes, but not always the same
one.

An extra benefit of this would be that high speed nodes could increase
their number of connections a lot more and help prevent any chance of
being "surrounded".

Each connection would need it's own little data set like was put in for
each download. This allows display and use of information for each item.

Anyone else see other security concerns with this?

--

-- 
http://www.fastmail.fm - mmm... Fastmail...

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!

Second ANts client out: Marabunta

http://marabunta.laotracara.com/english.php

--

-- 
Analog-/ISDN-Nutzer sparen mit GMX SmartSurfer bis zu 70%!
Kostenlos downloaden: http://www.gmx.net/de/go/smartsurfer

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: Second ANts client out: Marabunta

thomasasta <at> gmx.net wrote:
> http://marabunta.laotracara.com/english.php

Marabunta is not an ANtsP2P client.

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: Second ANts client out: Marabunta

Re: Second ANts client out: Marabunta

Tsafa <at> aol.com wrote:
> I hope they put up an English section on their website.

http://marabunta.laotracara.com/english.php

-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642

Re: Second ANts client out: Marabunta