Ian Schenck | 20 May 19:56 2015
Picon

A twisted wsgi success

I mostly lurk on here, but I wanted to share my recent experience switching my company's wsgi container to twisted wsgi.

We had been using a particular wsgi container implemented in C. As time went on problems creeped up with cython modules, segfaults happened, and threading had to be disabled. Things that worked in development would not work in production. At this point I wanted two things: the development container should be the production container, and the container of choice should provide a python interpreter environment consistent with actual python. These containers written in C which treat your application like a plugin fail to do that across the board.

We no longer have any of the issues we had before (cython, threading, segfaults), and if something were to break in production, it is probably broken in development and won't make it out the door. Development servers run the same twisted wsgi container that runs in production, in fact they are both launched the same (but middlewares and reloaders differ). That consistency is a really big deal. Additionally, deployment and config management dropped a few hundred lines.

The biggest pushback I received was over performance. We benched our real world application (versus, ahem, just returning a string) and found our median response time was consistently 1-2ms higher across various concurrency levels. That's insignificant for us. Under high load, twisted wsgi actually did better with tail latency (99%ile and max). Maybe that slight bump in median latency is because our web workers now speak http instead of a "light weight protocol" akin to scgi, but the benefits of talking http (curl much?) are also worth a millisecond or two.

Thanks for all the hard work, and thanks Glyph for your 2011 Djangocon talk and your tweets leading me to twisted wsgi. We're very happy. If anyone wants details or a peek at some of the wiring, I can probably get approval to share more.
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Benjamin BERTRAND | 23 Apr 22:44 2015
Picon

web UI example for Twisted application

Hello,

I have a Twisted network application that I deployed as a twistd plugin.
I have different configuration files defining several parameters including some Redis pub/sub channels.
Today I use a script to stop and start the plugin with the proper configuration file.
I’d like to make a web UI to be able to change the configuration of the application.

I already saw some people asking about that but I haven’t found any example.
Could anyone point me to some examples of a Twisted application with a web UI?

I have developed some Flask web app, so I think it would be easier for me to start with Klein.
But I’m not against trying Nevow (Twisted docs recommend it for this purpose).

Thanks

Benjamin
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Chris Drane | 13 Apr 20:29 2015
Picon

Client.agent - PartialDownloadError - cannot determine cause

I am new with Twisted so chances are that this is *not* a bug. If that is the case, I would very much appreciate it if you help me understand what I am doing incorrectly.

I have some very simple code to just download a particular page via HTTP. The problem I'm running into is that I get PartialDownloadErrors for some sites, while others load fine. And when I examine the HTTP response, it looks like the page fully downloaded. The traceback and logging information have been very unhelpful. I cannot determine what exactly is causing the error. 

import sys
from twisted.internet.task import react
from twisted.python.log import err, startLogging
from twisted.web.client import Agent, BrowserLikeRedirectAgent, readBody
from twisted.web.http_headers import Headers
from twisted.internet import reactor
from twisted.internet.ssl import ClientContextFactory

def cbBody(r):
    print "Response body:"
    print r

def cbRequest(response):
    print "Received response"
    d = readBody(response)
    d.addCallbacks(cbBody, err)
    return d

def err(e):
    try:
        e.raiseException()
    except Exception, err:
        pass
        # display HTTP response
        # print err.response
    e.printTraceback()
    sys.stderr.write(str(e))

def main(reactor):
    startLogging(sys.stdout)
    agent = BrowserLikeRedirectAgent(Agent(reactor))
    d = agent.request("GET", b"http://www.google.com",
            Headers({'User-Agent': ['Twisted Web Client Example']}),
            None)
    d.addCallbacks(cbRequest, err)
    return d

react(main)

2015-04-13 14:13:11-0400 [-] Log opened.
2015-04-13 14:13:11-0400 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Received response
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Traceback (most recent call last):
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Failure: twisted.web.client.PartialDownloadError: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] [Failure instance: Traceback (failure with no frames): <class 'twisted.web.client.PartialDownloadError'>: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [-] Main loop terminated.

Thanks!

Chris


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Guido Winkelmann | 2 Apr 19:49 2015
Picon

treq requests failing with twisted.web._newclient.ResponseNeverReceived

Hi,

Recently, I have been experiencing some problems in our codebase wth treq 
requests failing with error messages like this:

2015-04-02 19:20:22 ERROR    - pf.agent.http.client - POST 
http://127.0.0.1:5000/api/v1/agents/a113b5a8-9822-413b-b1fd-fce3014956cf has 
failed (uid: 78c7ec2a4b4d):
Traceback (most recent call last):
Failure: twisted.web._newclient.ResponseNeverReceived: 
[<twisted.python.failure.Failure <class 
'twisted.internet.error.ConnectionDone'>>]

Unfortunately, I have not been able to successfully produce a reduced test 
case that will reproduce this behaviour.  It only happens under some weird set 
of circumstances, and I am still not sure exactly which.  It appears to be 
highly sensitive to the order of requests.  Funnily enough, this error does 
not show in the codebase as it is on github right now, but if I reorder some 
of the requests, it will show.  Here is one example that has cost me some time 
and head-scratching:

https://github.com/pyfarm/pyfarm-agent/issues/249

I think that these problems have started after upgrading to Twisted 15.0, but 
I'm not completely certain about this.

Here is one of the methods that's affected by this:

https://github.com/pyfarm/pyfarm-agent/blob/master/pyfarm/agent/service.py#L231

(post_direct() is just a thin wrapper over treq.request that adds some headers 
we need in our codebase.  I have tried just using treq.request directly 
instead, the result was the same.)

Can someone tell me under which circumstances treq.request would produce an 
error like this?

Without actually knowing the source in question, I would guess that something 
in there is, for some some reason, erroneously trying to reuse a TCP 
connection that has already been closed.

Regards,

	Guido W.
甘肃电大 | 11 Mar 14:06 2015
Picon

Auto-Re: List admin required

您发的邮件收到了!
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Phil Mayers | 11 Mar 14:20 2015
Picon

List admin required

Can a list admin please disable this person, they're bouncing on every 
message...

-------- Forwarded Message --------
Subject: [Twisted-web] Auto-Re: Recommendations for Twisted Web 
implementation?
Date: Wed, 11 Mar 2015 05:17:26 +0800
From: 甘肃电大 <wanghe <at> gsrtvu.cn>
Reply-To: Twisted Web World <twisted-web <at> twistedmatrix.com>
To: twisted-web <at> twistedmatrix.com

您发的邮件收到了!
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Benjamin Schollnick | 10 Mar 22:31 2015
Picon

Recommendations for Twisted Web implementation?

Folks,

I’m mostly done with a web server / web gallery application using Twisted Web.  

Now I’ve used a variety of different technologies, including:

  • pybonjour
  • scandir
  • Bootstrap
  • Jquery

But I’m having a little bit of a problem cleaning this up for uploading to github.  

Can anyone give me some better suggestions for resources being shared between different web resource classes?
For example, I am using jinja2 and I have to make it a global to be available to all three web resources classes.

  •     root.putChild("index", IndexPage(ctx))
  •     root.putChild("login", LoginPage(connection))
  •     root.putChild("logout", LogoutPage())
  •     root.putChild("javascript", static.File("javascript", "application/javascript”))

I haven’t been able to come up with a better solution?

A working code example is available from https://github.com/sheffler/login_logic_jinja2.  It’s not my code, but the basic code layout is nigh identical.  I used it as a working model for the user authentication and built my gallery code as an additional child resource.

Worded another way, what’s the best practice for sharing data, code, and logic between children in a twisted web application, without making them global?

If global is the best answer, then I’ll continue with that.  But it just seems like there should be another way?

- Benjamin

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Alec Taylor | 6 Nov 04:59 2014
Picon

Registering error handlers in Klein?

Here's my simple TODO app with basic error handling using your
handle_errors decorator:
https://gist.github.com/SamuelMarks/9d7c796b8a336bb556f3

Unfortunately it is always throwing 500s. Is there some other way I'm
meant to be passing errors? E.g.: in Node.js most of my functions have
the signature (err, res).

Thanks for all assistance

PS: The functions from line 103 to 110 was another attempt of mine to
get exceptions handled

PPS: Issue reported 10 days ago - https://github.com/twisted/klein/issues/58
Glyph | 8 Oct 07:49 2014

Re: Making Secure HTTPS requests, SSL Method, and Certificate Management using Twisted


On Oct 6, 2014, at 7:55 AM, Carl Waldbieser <cwaldbieser <at> gmail.com> wrote:

I have a couple projects I am working on where I would like to make HTTPS requests using Twisted.  I reviewed the articles "Using TLS in Twisted" [1] and "Using the Twisted Web Client" (section "HTTP over SSL") [2].  It seems like various options exist that will allow me to make HTTPS requests using the CA certs bundled on the client OS.

Thank you for diligently studying all of our existing documentation on the subject.  I apologize for its deficiencies.  This API has grown in fits and starts over the last decade or so and it has not ended in the most consistent shape.  Hopefully we can improve this based on your feedback for the next release.

I would like to be able to tell my HTTPS clients to accept specific *internal* CA certificates *in addition* to the certificates provided by the OS.  Initially, I thought this might be possible by passing a custom t.w.c.BrowserLikePolicyForHTTPS to the t.w.c.Agent as its `contextFactory` argument. I wasn't quite sure how to go about doing this, so I got some advice on StackOverflow [3]. With some slight modifications to the solution presented there, I was able to create a custom Trust Root that accepted a list of paths to CA cert files in PEM format that I wanted to add to the client. The custom trust root is passed to the BrowserLikePolicyForHTTPS. The policy is passed to the Agent.

The one hitch is that the IOpenSSLTrustRoot interface upon which my custom trust root is based is located in `twisted.internet._sslverify`[4], which if I understand correctly, is a private module and not supposed to be used as an API.  Is there a *supported* way to specify *additional* CA certs to use during SSL verification when making HTTPS requests using Twisted? If so, what is the recommended method?

Well, the "good" news is, on OS X, if you're using the bundled system OpenSSL, you can't turn this behavior off: <https://hynek.me/articles/apple-openssl-verification-surprises/>.  Although we may eventually ship a mitigation for this vulnerability in Twisted, so probably best not to rely on that long-term :).

The bad news is that, no, we really had two use-cases in mind here; either:

  1. your software is trusting the trust cartel as specified by the operating system vendor and/or the user, and your software is opting out of any trust configuration, or
  2. your software has a specific understanding of its trust root and is specifying it explicitly, because it knows who it expects the peer to be signed by

So while your use-case sort of makes sense to me, it didn't come up in the last round of SSL enhancements and there's no straightforward way to go about this.

However, it should nevertheless be possible to do it without delving into the private API.

If you write your own IOpenSSLClientConnectionCreator, like this:

from twisted.internet.interfaces import IOpenSSLClientConnectionCreator
from twisted.python.components import proxyForInterface

class AddExtraTrustRoots(proxyForInterface(IOpenSSLClientConnectionCreator)):
    def __init__(self, extraTrustRoots, original):
        self._extraTrustRoots = extraTrustRoots
        super(AddExtraTrustRoots, self).__init__(original)


    def clientConnectionForTLS(self, tlsProtocol):
        connection = (super(AddExtraTrustRoots, self)
                      .clientConnectionForTLS(tlsProtocol))
        cert_store = connection.get_context().get_cert_store()
        for cert in self._extraTrustRoots:
            cert_store.add_cert(cert)
        return connection

you can delegate to the pyOpenSSL API for manipulating the trust settings of the connection, although Twisted will not provide a high-level wrapper for this.

You can then use the above wrapper like:

options = AddExtraTrustRoots([OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, "..."), ...], optionsForClientTLS(u"google.com"))

I haven't tested this, let me know if I've overlooked something terrible :).

Another related concept that was not clear to me is how one might specify the SSL method (e.g. SSLv23_METHOD, SSLv3_METHOD, etc.) when making the request. Is there some recommended way to pass options to indicate the SSL method that ought to be used?

The default configuration allows for TLSv1.0, TLSv1.1, and TLSv1.2 (insofar as those versions are supported by your OpenSSL).  Do you really, really, really need to support worse protocol versions than that?

You don't actually want to change the "method", as "method" is a bizarre fiction of the OpenSSL API and is not actually how the protocol works; SSLv23_METHOD is the only "method" that allows protocol version negotiation, so you always want to start there and then set/unset the appropriate OP_NO_* options for the protocol versions you want to exclude.

This is all pretty tedious and gross so if you're sure you want to do it I will describe how in the next reply :).

Any guidance would be appreciated.

I hope that this was sufficient!

-glyph

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Bruno Harbulot | 11 Sep 16:27 2014
Picon

Response hanging after headers sent

Hello,

I've been trying to implement a simple notification system via long polling using Twisted 14.0.0. On the client side, a web page makes Ajax calls to an "update" resource, which will deliver the latest status message known to the application. On the server side, a thread feeds a queue with status events and another one polls this queue and sends the latest value to any pending request, trying to keep the request open for 30 seconds, or until a new status event arrives, whichever happens sooner (this is the long polling).

Occasionally, one of responses will just hang. "request.finish()" is called properly (I've even checked this with "notifyFinish()"), the response headers are sent, but the response body isn't (according to Wireshark), thereby leaving the browser hanging (since it thinks a valid response has started to come through).

I have noticed this behaviour with Twisted 13 and 14, but not with earlier versions. This is also an intermittent problem. I have been able to reproduce it a number of times, but not always, and it doesn't necessarily happen after the same number of requests. It seems to fail more easily on machines with one CPU core, or when the execution is limited to one core, with only one request at a time (for some reason, having concurrent requests from multiple browsers or tabs, or being able to use multiple CPUs on the server seems to make the problem harder to reproduce).

I'm pasting at the end of this message a simple example where the status events are coming from a counter running in a separate thread. When limiting the server process to one CPU, the counter visible on the webpage rarely seems to go much above 2000. I've only tried under Linux. I've tried this on an actual PC (limiting to one core using "taskset -c 0 python test_server.py"), a VM running in VirtualBox (set to 1 CPU) and even a Raspberry Pi. The Raspberry Pi is where the problem seems to be the easiest to reproduce, but this also happens with the actual PC and virtual machine. This was also only tried on a LAN (considering that, for the purpose of this test case, the notifications are rather frequent). I've tried with the default chunked encoding, or by setting the content-length explicitly, but it didn't make a difference.

Is there a way to force to the response to be flushed, after a call to "request.finish()"? Any other suggestions to fix this would also be appreciated.

Thank you,

Bruno.




Only 2 files are required for this test case: counter.html, test_server.py.
I've run this in a virtual environment, with Python 2.7 (in which I've installed Twisted with `pip install twisted==14.0.0`).
On a multicore machine, to limit the execution to one core, call the server with taskset: "taskset -c 0 python test_server.py".
Once the server is running, point a browser (tried with Chrome and Firefox) to "http://the.ip.address.example:8880/", you should see a counter increasing (and eventually stopping, which illustrates the problem). The issue seems to happen more often when only 1 client (or tab) is connected to the server.


- counter.html:

<html>
<head>
<script type="application/javascript" src="//code.jquery.com/jquery-1.11.0.min.js"></script>
</head>
<body>
<div id="counter" style="font-size: 100pt; font-weight: bold;"></div>
<script type="application/javascript">
// <![CDATA[
$(document).ready(function() {
    var eTag = null;
    var normalRefreshDelay = 50;
    var refreshDelay = normalRefreshDelay;
    var maxRefreshDelay = 5000;
    function refresh() {
        var headers = {};
        if (eTag) {
            headers['If-None-Match'] = eTag;
        }
        $.ajax({
            url: "/update",
            headers: headers,
            dataType: "json",
            cache: false,
            success: function(data, status, xhr) {
                try {
                    if (data[1] != null) { $("#counter").text("Counter: "+data[1]); }
                    eTag = xhr.getResponseHeader("ETag") || null;
                    refreshDelay = normalRefreshDelay;
                } catch (e) {
                    refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
                }
            },
            error: function(xhr, text, error) {
                refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
            },
            complete: function(xhr, textStatus) {
                setTimeout(refresh, refreshDelay);
            }
        });
    }
    refresh();
});
// ]]>
</script>
</body>
</html>



- test_server.py


# -*- coding: utf-8 -*-

import threading
import time
import Queue
import uuid
import socket
import json

from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
from twisted.web.static import File

q = Queue.Queue()

def fake_counter():
    i = 1
    while True:
        q.put({ 1: str(i) })
        time.sleep(0.15)
        i += 1

producer_thread = threading.Thread(target=fake_counter)
producer_thread.daemon = True
producer_thread.start()


class CounterUpdateResource(Resource):
    def __init__(self, msg_queue):
        Resource.__init__(self)
        self.msg_queue = msg_queue
        self.msg = {}
        self.etag = None
        self.pending_requests = []
       
        t = threading.Thread(target=self._poll_messages)
        t.daemon = True
        t.start()
       
    def _poll_messages(self):
        while True:
            try:
                self.msg = self.msg_queue.get(True, 30)
                self.etag = '"%s"' % (uuid.uuid4(),)
                print "-> Got new message: %s" % (self.msg,)
            except Queue.Empty:
                pass
            json_str = json.dumps(self.msg)
            json_len = len(json_str)
            while True:
                try:
                    request = self.pending_requests.pop()
                    self._actual_render(request, json_str, json_len)
                except IndexError:
                    break
           
    def _actual_render(self, request, json_msg, json_len):
        try:
            request.setETag(self.etag)
            request.setHeader("Content-Type", "application/json")
            request.setHeader("Content-Length", json_len)
            request.write(json_msg)
            request.finish()
        except:
            pass

    def render_GET(self, request):
        ifNoneMatchHeader = request.getHeader("If-None-Match")
        if ifNoneMatchHeader is None or ifNoneMatchHeader != self.etag:
            if self.etag:
                request.setETag(self.etag)
            request.setHeader("Content-Type", "application/json")
            data = json.dumps(self.msg)
            request.setHeader("Content-Length", len(data))
            return data
        else:
            self.pending_requests.append(request)
            return NOT_DONE_YET

root = Resource()
root.putChild("", File("counter.html"))
root.putChild("update", CounterUpdateResource(q))

factory = Site(root)
reactor.listenTCP(8880, factory)
reactor.run()






- Sample Wireshark output:


Here is a sample output using Wireshark:

GET /update?_=1410440411047 HTTP/1.1
Host: 192.168.0.8:8880
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "84c25895-de1e-4b50-954b-438be9d8d9b7"
Referer: http://192.168.0.8:8880/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Content-Type: application/json
Server: TwistedWeb/14.0.0

{"1": "371"}

GET /update?_=1410440411048 HTTP/1.1
Host: 192.168.0.8:8880
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Referer: http://192.168.0.8:8880/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "ddd10400-3530-4cb4-a9cd-0b123032d8af"
Content-Type: application/json
Server: TwistedWeb/14.0.0



(I've inserted a coupe of line returns after "{"1": "371"}" for readability.)
Here, nothing is sent on the wire after the headers of the last response (the "\r\n\r\n" are present).


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Sha Hua | 27 Jan 18:58 2014
Picon

How to write an HTTP1.1 Proxy using Twisted?

Hi all,

I have recently started to learn and use Twisted. The first task is to write an HTTP proxy.
I found Twisted provides a built-in proxy module described here:


It says the server part inherits http.HTTPChannel, while the client part inherits http.HTTPClient.

However, as I later find out, http.HTTPClient is only a HTTP 1.0 client, which is not enough for my task. I also found there is another class called twisted.web.client.Agent which seems to be more advanced than http.HTTPClient and supports HTTP 1.1.

Questions:
1. Can twisted.web.client.Agent be used in a Proxy implementation with http.HTTPChannel? Is there any examples?
2. Does http.HTTPChannel support HTTP 1.1?

Thanks!

Sha
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

Gmane