headers
fredrik.allansson@gmail.com | 3 Feb 10:26
Picon

Problem with url seeds and proxy

Hi all,

I'm having a problem with an application using libtorrent and url seeds
when behind a HTTP proxy server.

>From what I can gather from debugging, it seems that the proxy server
closes the connection when the last chunk of a file is requested over HTTP
(using the "Range" header). This is detected by the following bit of code
from peer_connection::peer_connection::on_receive_data_nolock:

                        error_code ec;
                        bytes_transferred = try_read(read_sync, ec);
                        if (ec && ec != asio::error::would_block)
                        {

 m_statistics.trancieve_ip_packet(bytes_in_loop,
m_remote.address().is_v6());
                                disconnect(ec);
                                return;
                        }

the call to try_read returns asio::error::eof, and the data is discarded.
libtorrent will request the chunk again, which causes the same thing to
happen again. The download is effectively stalled.

Has anyone had any similar problems, and know a way to get around it?

Regards,
Fredrik
------------------------------------------------------------------------------
(Continue reading)

Permalink | Reply | 2 comments |
headers
arvid | 4 Feb 04:39
Picon
Picon
Favicon

Re: About limiting offered pieces

Quoting ALVARO SAURIN PARRA <saurin <at> tid.es>:
> Hi there,
> 
> I was about to start a project where we have to implement a modified
> bittorrent protocol. The main differences with the standard protocol are:
> 
> - once a peer serves a piece, it never sends (or tries to obtain) it again.
> - there is always a bandwidth limit between any two peers.
> 
> There is also an implementation constraint: it must be done from Python. So
> I've been looking at the sources and it seems the second constraint can be
> implemented with set_peer_upload_limit()/set_peer_download_limit(), but I
> don't know if there is a way for limiting the pieces offered/accepted to/from
> other peers. Could this be done from Python?

Unfortunately there's no feedback of when a specific piece (or in fact block) is
served. And even if there were, using that to somehow remove the piece and set
its priority to 0, there would still be a race condition.

This could be done from a libtorrent plugin, that could simply mark every block
that has been served and reject any further requests that overlap it. Once a
full piece has been sent, it could be removed. However, there is no standard
i-no-longer-have message (there's an extension message lt_dont_have, but not
many clients support that).

libtorrent plugins are not generally available to python though. There is an
attempt at exposing the plugin API to python, but it's unclear if it works.

Please feel free to give that a try, or extend it. I would be happy to apply
patches fixing it.
(Continue reading)

Permalink | Reply | 0 comments |
headers
arvid | 4 Feb 04:45
Picon
Picon
Favicon

Re: Problem with url seeds and proxy

Quoting "fredrik.allansson <at> gmail.com" <fredrik.allansson <at> gmail.com>:

> Hi all,
> 
> I'm having a problem with an application using libtorrent and url seeds
> when behind a HTTP proxy server.
> 
> >From what I can gather from debugging, it seems that the proxy server
> closes the connection when the last chunk of a file is requested over HTTP
> (using the "Range" header). This is detected by the following bit of code
> from peer_connection::peer_connection::on_receive_data_nolock:
> 
>                         error_code ec;
>                         bytes_transferred = try_read(read_sync, ec);
>                         if (ec && ec != asio::error::would_block)
>                         {
> 
>  m_statistics.trancieve_ip_packet(bytes_in_loop,
> m_remote.address().is_v6());
>                                 disconnect(ec);
>                                 return;
>                         }
> 
> the call to try_read returns asio::error::eof, and the data is discarded.
> libtorrent will request the chunk again, which causes the same thing to
> happen again. The download is effectively stalled.
> 
> Has anyone had any similar problems, and know a way to get around it?

Yes, this is a limitation in libtorrent. The granularity that data is reported
(Continue reading)

Permalink | Reply | 1 comment |
headers
Alexander Tretyak | 4 Feb 12:38
Picon

Re: Problem with url seeds and proxy

>This is not trivial with the connection model used in libtorrent. The actual
>connection and setting up the socket happens outside of the connection object.
>
>A more general solution would be to actually report all partial blocks down to
>the core, and make peers that actually send "odd" requests. (odd as in not block
>aligned). That too is not trivial, since the piece picker probably would be the
>data structure to store the partial blocks (which it already does to some
>degree).
>
>Let me know if you have any ideas. Also, feel free to make an attempt at fixing
>it!

Is it possible to use another connection model special for url-seeds?
I have the similar problem too, because of some firewalls closing connections
(when checking http-traffic) right after first response delivery (i.e. like 
without support of "Connection: keep-alive").
I propose to use a separate thread for each url-seed in each torrent, and just 
use blocking network I/O in that threads to download requested blocks file-by-
file. Sorry, I don't know how this solution suit into current design, but if 
libtorrent can correctly handle such cases (may be optionally, via specific 
setting in the session), it would be great!
Or may be, it is possible to make a proxy-thread, that emulate support of keep-
alive connection, or local http-proxy server in separate thread, or something 
like that.

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
(Continue reading)

Permalink | Reply | 0 comments |
headers
Kyle Flavin | 7 Feb 20:56
Picon

Clients announcing as stopped

I'm noticing an issue with the announce strings that I can't explain.
After I start my client and begin adding torrents to my session, I'll see
"event=started" announce URL's going to my tracker.  However, sometimes, I
will also see my clients sending out "event=stopped" in the announce.  This
happens immediately after I begin serving torrents from the started
client.  What would cause a client to send out a "stopped" event
immediately after beginning to serve torrents?  Is there a limit to the
number of torrents that can be served at one time?  I thought "stopped"
events are only sent out when the client shuts down.  I'm using the python
bindings.
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 3 comments |
headers
arvid | 8 Feb 02:43
Picon
Picon
Favicon

Re: Clients announcing as stopped

Quoting Kyle Flavin <kyle.flavin <at> gmail.com>:

> I'm noticing an issue with the announce strings that I can't explain.
> After I start my client and begin adding torrents to my session, I'll see
> "event=started" announce URL's going to my tracker.  However, sometimes, I
> will also see my clients sending out "event=stopped" in the announce.  This
> happens immediately after I begin serving torrents from the started
> client.  What would cause a client to send out a "stopped" event
> immediately after beginning to serve torrents?  Is there a limit to the
> number of torrents that can be served at one time?

Yes, by default that limit is 8 torrents (session_settings::active_downloads).

You can either raise that limit (session::set_settings) or start all your
torrents without the auto_managed flag set (see add_torrent_params).

Only torrents that are auto_managed are subject to that limit.

> I thought "stopped"
> events are only sent out when the client shuts down.  I'm using the python
> bindings.

whenever a torrent is stopped, yes.

--

-- 
Arvid Norberg

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
(Continue reading)

Permalink | Reply | 2 comments |
headers
John Matthew | 8 Feb 03:01

Re: Clients announcing as stopped

Is there a limit to the number of torrents LT can serve?

On Tue, Feb 7, 2012 at 5:43 PM, <arvid <at> cs.umu.se> wrote:

> Quoting Kyle Flavin <kyle.flavin <at> gmail.com>:
>
> > I'm noticing an issue with the announce strings that I can't explain.
> > After I start my client and begin adding torrents to my session, I'll see
> > "event=started" announce URL's going to my tracker.  However, sometimes,
> I
> > will also see my clients sending out "event=stopped" in the announce.
>  This
> > happens immediately after I begin serving torrents from the started
> > client.  What would cause a client to send out a "stopped" event
> > immediately after beginning to serve torrents?  Is there a limit to the
> > number of torrents that can be served at one time?
>
> Yes, by default that limit is 8 torrents
> (session_settings::active_downloads).
>
> You can either raise that limit (session::set_settings) or start all your
> torrents without the auto_managed flag set (see add_torrent_params).
>
> Only torrents that are auto_managed are subject to that limit.
>
> > I thought "stopped"
> > events are only sent out when the client shuts down.  I'm using the
> python
> > bindings.
>
(Continue reading)

Permalink | Reply | 1 comment |
headers
arvid | 8 Feb 03:45
Picon
Picon
Favicon

Re: Clients announcing as stopped

Quoting John Matthew <john <at> compunique.com>:

> Is there a limit to the number of torrents LT can serve?

There's no explicit limit. In the latest stable and trunk there
are a few bottlenecks you may run into when you get into the tens
of thousands of torrents.

I've fixed most of those in the (still experimental) libtorrent_aio branch.

In that branch, the limit you would typically hit is the amount of RAM just
loading the .torrent files will use.

For more info, see my blog post:

  http://www.rasterbar.com/libtorrent_blog/2012/01/seeding-a-million-torrents/

--

-- 
Arvid Norberg

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 0 comments |
headers
zhang peng | 8 Feb 11:54
Picon

Maybe a bug when web_peer_connection to request pad_files

Hi.

I found this maybe one bug :  When one torrent has some padding files
(_____padding_file_0 ...),  and when it retrives data from one url
seed, It sends request like:  GET /test/_____padding_file_10 HTTP/1.1.

But the url seed server doesn't have this file, so it disconnect from
the server. And then it cannot download files completely.

Waiting for your reply, thanks.

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 1 comment |
headers
arvid | 8 Feb 17:05
Picon
Picon
Favicon

Re: Maybe a bug when web_peer_connection to request pad_files

Quoting zhang peng <hrybird <at> gmail.com>:

> Hi.
> 
> I found this maybe one bug :  When one torrent has some padding files
> (_____padding_file_0 ...),  and when it retrives data from one url
> seed, It sends request like:  GET /test/_____padding_file_10 HTTP/1.1.
> 
> But the url seed server doesn't have this file, so it disconnect from
> the server. And then it cannot download files completely.
> 
> Waiting for your reply, thanks.

This has been fixed in trunk, but not released as stable yet.

--

-- 
Arvid Norberg

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 0 comments |

Gmane