Re: Assertion in torrent::on_name_lookup - Doublewebseeding

I agreed with Arvid about no gain from 2 simultaneous connections to the
same IP - all idea of BT is to distribute load across multiple links.

However, it is obvious that webseeds has to be treated differently then an
ordinary BT peer. From distribution perspective, webseed (WS) has at least
two distinguishing features : (a) always available ( as opposite to random
lifespan of ordinary BT peer) (b) "expensive" both in technical and
financial senses. 

I think that WS has two major roles - (a) seed underdeveloped swarm (b) give
newcomer a few initial pieces to be used in tit-for-tat trading. I think
that second role if often overlooked and as result, usually it takes some
time to new peer to come up to speed. WS can be a great help in bootstrap
process.  On the other hand, overuse of the WS will have a negative effect
for the swarm ( even not talking about web owner who need to pick bandwidth
bill)

It seems that bandwidth prioritization ( WS gets only what left from
ordinary peers) can be a good solution that fit both WS roles - WS will be
use only until the rest of the swarm starts feed us with bits).

On topic of WS chunk size - I just paid attention that many webservers are
configured without KeepAlive ( at least it is a default Fedora
configuration). In such configuration only a first request will be honored
by server, making 16k block delivery VERY inefficient ( TCP connection can't
get to its max speed due "slow start" effect). "Full piece" download is
indeed very useful optimization.

Regards
Stas 

Re: Assertion in torrent::on_name_lookup - Doublewebseeding

On Jan 31, 2007, at 4:54 , Stas Khirman wrote:

> I agreed with Arvid about no gain from 2 simultaneous connections  
> to the
> same IP - all idea of BT is to distribute load across multiple links.

So, would you say it would be reasonable behavior to drop the second  
web seed connection if there's already another connection to that IP?

> However, it is obvious that webseeds has to be treated differently  
> then an
> ordinary BT peer. From distribution perspective, webseed (WS) has  
> at least
> two distinguishing features : (a) always available ( as opposite to  
> random
> lifespan of ordinary BT peer) (b) "expensive" both in technical and
> financial senses.

Right.

> I think that WS has two major roles - (a) seed underdeveloped swarm  
> (b) give
> newcomer a few initial pieces to be used in tit-for-tat trading. I  
> think
> that second role if often overlooked and as result, usually it  
> takes some
> time to new peer to come up to speed. WS can be a great help in  
> bootstrap
> process.  On the other hand, overuse of the WS will have a negative  

Re: Assertion in torrent::on_name_lookup -Doublewebseeding

Just to avoid misunderstanding;

1.) Yes, I think it will be ok to drop the second webseed connection if
there is one established. However, second webseed may be used on the second
attempt if first webseed returns error ( if I understand code correctly  ,
it is how it will work now after remove_url_seed() removes first faulted
URL). BTW, IMHO current implementation doesn't support redirects - have to
be an easy fix in the web_peer_connection::on_receive().

2.) Sorry for misunderstanding, but I didn't said that there is any benefit
for web server to close connection - I said that in many default
installations (Fedora Linux for example ) Apache server configured WITHOUT
support for KeepAlive. Please do not ask me why - I also was extremely
surprised, but it is a fact.... As result, in such configurations, server
will always return "connection: close" regardless of client request to keep
connection open. It is easy to enable this feature - just set "KeepAlive On"
in the httpd.conf, but even in this case Apache has restriction on number of
simultaneously kept alive session ( 100 y default). 
Bottom line - I recomend NOT to assume that web seeder will be able to keep
connection alive between block requests. This limitation can be overcomed by
requesting a few sequential blocks in one shot (or whole piece ).

Stas

-----Original Message-----
From: libtorrent-discuss-bounces <at> lists.sourceforge.net
[mailto:libtorrent-discuss-bounces <at> lists.sourceforge.net] On Behalf Of Arvid
Norberg
Sent: Wednesday, January 31, 2007 8:44 PM
To: General discussion about libtorrent

Re: Assertion in torrent::on_name_lookup -Doublewebseeding

On Jan 31, 2007, at 9:28 , Stas Khirman wrote:

> Just to avoid misunderstanding;
>
> 1.) Yes, I think it will be ok to drop the second webseed  
> connection if
> there is one established. However, second webseed may be used on  
> the second
> attempt if first webseed returns error ( if I understand code  
> correctly  ,
> it is how it will work now after remove_url_seed() removes first  
> faulted
> URL).

Right. The current implementation will constantly retry all non- 
connected web seeds. Which also means that any web server that closes  
the connection will be retried immediately. The only case when a web  
seed is removed is when the server returns an error that is  
considered being fatal. I guess a better strategy here would be to  
have an exponential back off.

> BTW, IMHO current implementation doesn't support redirects - have to
> be an easy fix in the web_peer_connection::on_receive().

Right. Redirection is on my todo list.

> 2.) Sorry for misunderstanding, but I didn't said that there is any  
> benefit
> for web server to close connection - I said that in many default

Re: Assertion in torrent::on_name_lookup - Doublewebseeding

>> Yes, I've had the same problem with http web seeds.
>> It makes sense to install multiple web servers as webseeds on a LAN
>> behind a
>> router, because by definition bittorrent web seeding is very hdd
>> intensive
>> so multiple computers sharing the same IP address (with different
>> virtual
>> hosts) can ease the load.
>
> Can you please elaborate on this? I'm not sure I understand what kind
> of set up you mean. What can you gain from having 2 or more
> connections to the same web server (via different virtual hosts)
> downloading the same file, compared to just having one connection to
> that web server?

My mistake - no two machines can have the same ip address, however:

You may have:
storage.mysite.com/ -> pointing to 190.190.190.1
storage2.mysite.com/ -> pointing to 190.190.190.1

storage, storage2 are two different virtual hosts, having two different 
webserver accounts (on the same server).
Having several webseeds (even pointing to the same IP) in the torrent file 
can be helpful, because it is easy to switch the DNS when a new physical 
server must be added.

Since we are on the topic - have you considered a http proxy for the webseed 
?

Re: Assertion in torrent::on_name_lookup - Doublewebseeding

On Feb 1, 2007, at 12:50 , Florin Braghis wrote:

>
>>> Yes, I've had the same problem with http web seeds.
>>> It makes sense to install multiple web servers as webseeds on a LAN
>>> behind a
>>> router, because by definition bittorrent web seeding is very hdd
>>> intensive
>>> so multiple computers sharing the same IP address (with different
>>> virtual
>>> hosts) can ease the load.
>>
>> Can you please elaborate on this? I'm not sure I understand what kind
>> of set up you mean. What can you gain from having 2 or more
>> connections to the same web server (via different virtual hosts)
>> downloading the same file, compared to just having one connection to
>> that web server?
>
> My mistake - no two machines can have the same ip address, however:
>
> You may have:
> storage.mysite.com/ -> pointing to 190.190.190.1
> storage2.mysite.com/ -> pointing to 190.190.190.1
>
> storage, storage2 are two different virtual hosts, having two  
> different
> webserver accounts (on the same server).
> Having several webseeds (even pointing to the same IP) in the  
> torrent file

about the way calculating the "upload_share" which in function "distribute_free_upload"

Hello everybody:

the way calculating the "upload_share" puzzles me several days:

// in function "distribute_free_upload" which in "policy.cpp"
if (total_diff >= 0)
{
     upload_share = std::min(free_upload, total_diff) / num_peers;
}
else
{
     upload_share = (free_upload + total_diff) / num_peers;
}

why "total_diff >= 0" use std::min(free_upload, total_diff),
and why "total_diff < 0 use free_upload + total_diff"?

Any help appreciated. Thanks in advance

herochen
2007-02-01

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________

torrent.cpp invariant check

The torrent invariant check accesses the m_have_pieces vector.
So if the user thread calles function like torrent_handle::pause()/resume() etc
the torrent invariant checked is invoked.
Other libtorrent threads might modify the m_have_pieces vector like
check_fastresume() after adding a torrent. Calling pause() (for example) after
calling add_torrent() can lead to a crash or assert in the torrent invariant check.

This line is the problem:

assert(m_num_pieces	== std::count(m_have_pieces.begin(), m_have_pieces.end(), true));

IMO this is only a debug version problem. Removing the invariant checks in
the particular functions solved the problem for me.

MassaRoddel

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Code review questions

Hi Stas,

On Feb 2, 2007, at 12:41 , Stas Khirman wrote:

> Arvid,
>
> I start an in-depth review of your code ( v 0.11) and already have  
> a few
> questions, comments, etc. Hope you'll find beneficiary to answer on  
> these
> point (some of them may help me to understand code, but some may  
> help you to
> make code even better). Please let me know if you find my messages
> inconvenient or out of line in any way. ( BTW - let me know if you  
> prefer to
> move this discussion into mailing list).

I'm always happy to discuss the code, since I'm always interested in  
improving it. I will cc my reply to the mailing list.

> 1.) peer_connection::init()- make no sense to call it from inside  
> of the
> constructor (at least it's main part) - we didn't yet received peer
> handshake and have no yet knowledge about available pieces.
>
> Removing main part of init() from constructor , may also eliminate  
> use of
> exception as a seed->seed condition handling.

What would be the gain of not using exceptions there? Right now it's  

Re: torrent.cpp invariant check

On Feb 3, 2007, at 5:22 , MooPolice <at> massaroddel.de wrote:

> The torrent invariant check accesses the m_have_pieces vector.
> So if the user thread calles function like torrent_handle::pause()/ 
> resume() etc
> the torrent invariant checked is invoked.
> Other libtorrent threads might modify the m_have_pieces vector like
> check_fastresume() after adding a torrent. Calling pause() (for  
> example) after
> calling add_torrent() can lead to a crash or assert in the torrent  
> invariant check.
>
> This line is the problem:
>
> assert(m_num_pieces	== std::count(m_have_pieces.begin(),  
> m_have_pieces.end(), true));
>
>
> IMO this is only a debug version problem. Removing the invariant  
> checks in
> the particular functions solved the problem for me.

 From what I can tell, I would say the problem is that  
torrent::check_fastresume() is called without the session mutex being  
locked. I've checked in a fix for this now, could you please confirm  
that it fixes the problem?

thanks!

--