Bruno Harbulot | 11 Sep 16:27 2014

Response hanging after headers sent


I've been trying to implement a simple notification system via long polling using Twisted 14.0.0. On the client side, a web page makes Ajax calls to an "update" resource, which will deliver the latest status message known to the application. On the server side, a thread feeds a queue with status events and another one polls this queue and sends the latest value to any pending request, trying to keep the request open for 30 seconds, or until a new status event arrives, whichever happens sooner (this is the long polling).

Occasionally, one of responses will just hang. "request.finish()" is called properly (I've even checked this with "notifyFinish()"), the response headers are sent, but the response body isn't (according to Wireshark), thereby leaving the browser hanging (since it thinks a valid response has started to come through).

I have noticed this behaviour with Twisted 13 and 14, but not with earlier versions. This is also an intermittent problem. I have been able to reproduce it a number of times, but not always, and it doesn't necessarily happen after the same number of requests. It seems to fail more easily on machines with one CPU core, or when the execution is limited to one core, with only one request at a time (for some reason, having concurrent requests from multiple browsers or tabs, or being able to use multiple CPUs on the server seems to make the problem harder to reproduce).

I'm pasting at the end of this message a simple example where the status events are coming from a counter running in a separate thread. When limiting the server process to one CPU, the counter visible on the webpage rarely seems to go much above 2000. I've only tried under Linux. I've tried this on an actual PC (limiting to one core using "taskset -c 0 python"), a VM running in VirtualBox (set to 1 CPU) and even a Raspberry Pi. The Raspberry Pi is where the problem seems to be the easiest to reproduce, but this also happens with the actual PC and virtual machine. This was also only tried on a LAN (considering that, for the purpose of this test case, the notifications are rather frequent). I've tried with the default chunked encoding, or by setting the content-length explicitly, but it didn't make a difference.

Is there a way to force to the response to be flushed, after a call to "request.finish()"? Any other suggestions to fix this would also be appreciated.

Thank you,


Only 2 files are required for this test case: counter.html,
I've run this in a virtual environment, with Python 2.7 (in which I've installed Twisted with `pip install twisted==14.0.0`).
On a multicore machine, to limit the execution to one core, call the server with taskset: "taskset -c 0 python".
Once the server is running, point a browser (tried with Chrome and Firefox) to "http://the.ip.address.example:8880/", you should see a counter increasing (and eventually stopping, which illustrates the problem). The issue seems to happen more often when only 1 client (or tab) is connected to the server.

- counter.html:

<script type="application/javascript" src="//"></script>
<div id="counter" style="font-size: 100pt; font-weight: bold;"></div>
<script type="application/javascript">
// <![CDATA[
$(document).ready(function() {
    var eTag = null;
    var normalRefreshDelay = 50;
    var refreshDelay = normalRefreshDelay;
    var maxRefreshDelay = 5000;
    function refresh() {
        var headers = {};
        if (eTag) {
            headers['If-None-Match'] = eTag;
            url: "/update",
            headers: headers,
            dataType: "json",
            cache: false,
            success: function(data, status, xhr) {
                try {
                    if (data[1] != null) { $("#counter").text("Counter: "+data[1]); }
                    eTag = xhr.getResponseHeader("ETag") || null;
                    refreshDelay = normalRefreshDelay;
                } catch (e) {
                    refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
            error: function(xhr, text, error) {
                refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
            complete: function(xhr, textStatus) {
                setTimeout(refresh, refreshDelay);
// ]]>


# -*- coding: utf-8 -*-

import threading
import time
import Queue
import uuid
import socket
import json

from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
from twisted.web.static import File

q = Queue.Queue()

def fake_counter():
    i = 1
    while True:
        q.put({ 1: str(i) })
        i += 1

producer_thread = threading.Thread(target=fake_counter)
producer_thread.daemon = True

class CounterUpdateResource(Resource):
    def __init__(self, msg_queue):
        self.msg_queue = msg_queue
        self.msg = {}
        self.etag = None
        self.pending_requests = []
        t = threading.Thread(target=self._poll_messages)
        t.daemon = True
    def _poll_messages(self):
        while True:
                self.msg = self.msg_queue.get(True, 30)
                self.etag = '"%s"' % (uuid.uuid4(),)
                print "-> Got new message: %s" % (self.msg,)
            except Queue.Empty:
            json_str = json.dumps(self.msg)
            json_len = len(json_str)
            while True:
                    request = self.pending_requests.pop()
                    self._actual_render(request, json_str, json_len)
                except IndexError:
    def _actual_render(self, request, json_msg, json_len):
            request.setHeader("Content-Type", "application/json")
            request.setHeader("Content-Length", json_len)

    def render_GET(self, request):
        ifNoneMatchHeader = request.getHeader("If-None-Match")
        if ifNoneMatchHeader is None or ifNoneMatchHeader != self.etag:
            if self.etag:
            request.setHeader("Content-Type", "application/json")
            data = json.dumps(self.msg)
            request.setHeader("Content-Length", len(data))
            return data
            return NOT_DONE_YET

root = Resource()
root.putChild("", File("counter.html"))
root.putChild("update", CounterUpdateResource(q))

factory = Site(root)
reactor.listenTCP(8880, factory)

- Sample Wireshark output:

Here is a sample output using Wireshark:

GET /update?_=1410440411047 HTTP/1.1
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "84c25895-de1e-4b50-954b-438be9d8d9b7"
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Content-Type: application/json
Server: TwistedWeb/14.0.0

{"1": "371"}

GET /update?_=1410440411048 HTTP/1.1
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "ddd10400-3530-4cb4-a9cd-0b123032d8af"
Content-Type: application/json
Server: TwistedWeb/14.0.0

(I've inserted a coupe of line returns after "{"1": "371"}" for readability.)
Here, nothing is sent on the wire after the headers of the last response (the "\r\n\r\n" are present).

Twisted-web mailing list
Twisted-web <at>
Sha Hua | 27 Jan 18:58 2014

How to write an HTTP1.1 Proxy using Twisted?

Hi all,

I have recently started to learn and use Twisted. The first task is to write an HTTP proxy.
I found Twisted provides a built-in proxy module described here:

It says the server part inherits http.HTTPChannel, while the client part inherits http.HTTPClient.

However, as I later find out, http.HTTPClient is only a HTTP 1.0 client, which is not enough for my task. I also found there is another class called twisted.web.client.Agent which seems to be more advanced than http.HTTPClient and supports HTTP 1.1.

1. Can twisted.web.client.Agent be used in a Proxy implementation with http.HTTPChannel? Is there any examples?
2. Does http.HTTPChannel support HTTP 1.1?


Twisted-web mailing list
Twisted-web <at>
yoni s | 8 Dec 22:21 2013

Streaming MP3 using twisted matrix

Hi All,

I'm looking for a simple example that streams MP3 over http. 

I know how to Google, and even to use GitHub search but while I found several examples for using python I didn't find anything using twisted.

Any help would be appreciated 

Twisted-web mailing list
Twisted-web <at>
HawkOwl | 8 Nov 19:49 2013

Twisted 13.2.0 Release Announcement

On behalf of Twisted Matrix Laboratories, I am honoured to announce
the release of Twisted 13.2!

The highlights of this release are:

 * Twisted now includes a HostnameEndpoint implementation which uses
IPv4 and IPv6 in parallel, speeding up the connection by using
whichever connects first (the 'Happy Eyeballs'/RFC 6555 algorithm).

 * Improved support for Cancellable Deferreds by kaizhang, our GSoC
student. (#4320, #6532, #6572, #6639)

 * Improved Twisted.Mail documentation by shira, our Outreach Program
for Women intern. (#6649, #6652)

 * twistd now waits for the application to start successfully before
exiting after daemonization. (#823)

 * SSL server endpoint string descriptions now support the
specification of chain certificates. (#6499)

 * Over 70 closed tickets since 13.1.0.

For more information, check the NEWS file (link provided below).

You can find the downloads at <>
(or alternatively <>) .
The NEWS file is also available at

Many thanks to everyone who had a part in this release - the
supporters of the Twisted Software Foundation, the developers who
contributed code as well as documentation, and all the people building
great things with Twisted!

Twisted Regards,
Axel Rau | 5 Nov 13:19 2013

deployment with twistd of simple web application


just doing my 1st dynamic web application with twisted.

I have some working python modules, where the entry point is:
resource = RootWeatherPage()
meteoFactory = Site(resource)
reactor.listenTCP(80, factory, interface='some IP4')
reactor.listenTCP(80, factory, interface='some IP6')

Now I want to use twistd web.
How do I do that?

Thanks Axel
PGP-Key:29E99DD6  ☀ +49 151 2300 9283  ☀ computing  <at>  chaos claudius

Twisted-web mailing list
Twisted-web <at>
Phil Mayers | 16 Sep 18:58 2013

Double tracebacks for t.web & klein

I'm playing with klein for a simple rest API (because I like the 
routing, mainly).

When a method raises an exception, a traceback gets logged twice - once 
by the t.web Request.processingFailed, called from here:

...and once by the deferred garbage collection, as 
Request.processingFailed doesn't eat the deferred:

Obviously this double-traceback thing is hugely irritating. Who is at 
fault here? Is t.web doing the right thing by returning the failure from 
processingFailed, or is klein doing the wrong thing, either by using the 
(undocumented) processingFailed or omitting an errback further down the 

Obviously there's no way for *me* to add an errback - klein generates 
the deferred for me.
D Brian Kimmel | 29 Jul 20:01 2013

Need Help


I have a large home automation project that uses twisted.
I am adding a web interface and need athena for COMET activity.

I have been having a very hard time finding any current documentation to help 
me get the athena part working.
I am looking for brains to pick on getting this portion working bi-

I am also looking for others who may be interested in working on this type of 

github -> DBrianKimmel/PyHouse

Pranav Bhardwaj | 17 Jul 02:58 2013

twisted.web : limiting file size during upload (using python)


I have a script to be able to upload logs to a twisted web server.

I have the following requirements:

1) to be able to limit the size of the upload to 10MB.
2)  to NOT store file in memory during the request.

Does twisted.web provide these functionality?

Any inputs on these would be welcome!

Here is the server part os the script:

            oStream = open(filename, 'wb')

            # handle exception

a test client script:

<!DOCTYPE html>


<form action="https://serverUrl"



<div class="row">

<label for="fileUploadToServer">Select file</label><br />

fname : <input type="file" name="fname" />

<input type="submit" value="submit">




Twisted-web mailing list
Twisted-web <at>
Alexei Colin | 13 Jul 21:15 2013

Add response logging to example


The example processes the request, but not the
response. Yes, one can learn how to do that from the docs and from
stackoverflow, but that is an avoidable time burden. Sometimes this
example is all one needs, if only it were complete with both directions.

Could the example be revised or new one added with something like the
attached? Thank you for the consideration.

Attachment ( text/x-python, 2377 bytes
Twisted-web mailing list
Twisted-web <at>
Christopher Lozinski | 8 Jul 00:06 2013

Re: Nginx vs Twisted Web

 Thank so much to all of you for the excellent discussion on this topic.

I particularly liked Jean Paul from Twisted matrix's careful response.

And Burak Arslan's comment about Nginx supporting SendFile was hugely interesting.

So which one of those two did I choose?  At first I was going to go to Nginx, but then a developer I am cooperating with, who is also doing a Zope 3 server
went with Cherokee, and I followed in his lead.  Why???  Well Cherokee has a web GUI, that hugely simplifies my life, and the guy I am
cooperating with and I can support each other.  Really ease of use trumps all other issues for me.  I just have way way too much complexity that I am managing in life.

You can read more about my decision here.

If people want to continue this discussion, I would love to see a comparison of Twisted Web and Cherokee.  Best to start a new thread for that.

And if there were a web gui for Twisted Web, I would certainly have gone with that, even if it were an early release.  I can imagine one could create resources TTW Through The Web,
and store them in the ZODB.   Eventually I would like to see a tighter integration between Twisted Web and Zope 3.  I am starting with the release of Zope 3.  But all things take time. 

Christopher Lozinski

Twisted-web mailing list
Twisted-web <at>
Tristan Seligmann | 2 Jul 16:25 2013

Re: Nginx vs Twisted Web

On Tue, Jul 2, 2013 at 3:25 PM, Richard Wall <m-lists <at>> wrote:

Another problem is that Twisted doesn't yet support SSL on adopted sockets.

Is there a bug report for this? I would have expected it to Just Work (using twisted.protocols.tls, that is).
mithrandi, i Ainil en-Balandor, a faer Ambar
Twisted-web mailing list
Twisted-web <at>