Glyph | 8 Oct 07:49 2014

Re: Making Secure HTTPS requests, SSL Method, and Certificate Management using Twisted


On Oct 6, 2014, at 7:55 AM, Carl Waldbieser <cwaldbieser <at> gmail.com> wrote:

I have a couple projects I am working on where I would like to make HTTPS requests using Twisted.  I reviewed the articles "Using TLS in Twisted" [1] and "Using the Twisted Web Client" (section "HTTP over SSL") [2].  It seems like various options exist that will allow me to make HTTPS requests using the CA certs bundled on the client OS.

Thank you for diligently studying all of our existing documentation on the subject.  I apologize for its deficiencies.  This API has grown in fits and starts over the last decade or so and it has not ended in the most consistent shape.  Hopefully we can improve this based on your feedback for the next release.

I would like to be able to tell my HTTPS clients to accept specific *internal* CA certificates *in addition* to the certificates provided by the OS.  Initially, I thought this might be possible by passing a custom t.w.c.BrowserLikePolicyForHTTPS to the t.w.c.Agent as its `contextFactory` argument. I wasn't quite sure how to go about doing this, so I got some advice on StackOverflow [3]. With some slight modifications to the solution presented there, I was able to create a custom Trust Root that accepted a list of paths to CA cert files in PEM format that I wanted to add to the client. The custom trust root is passed to the BrowserLikePolicyForHTTPS. The policy is passed to the Agent.

The one hitch is that the IOpenSSLTrustRoot interface upon which my custom trust root is based is located in `twisted.internet._sslverify`[4], which if I understand correctly, is a private module and not supposed to be used as an API.  Is there a *supported* way to specify *additional* CA certs to use during SSL verification when making HTTPS requests using Twisted? If so, what is the recommended method?

Well, the "good" news is, on OS X, if you're using the bundled system OpenSSL, you can't turn this behavior off: <https://hynek.me/articles/apple-openssl-verification-surprises/>.  Although we may eventually ship a mitigation for this vulnerability in Twisted, so probably best not to rely on that long-term :).

The bad news is that, no, we really had two use-cases in mind here; either:

  1. your software is trusting the trust cartel as specified by the operating system vendor and/or the user, and your software is opting out of any trust configuration, or
  2. your software has a specific understanding of its trust root and is specifying it explicitly, because it knows who it expects the peer to be signed by

So while your use-case sort of makes sense to me, it didn't come up in the last round of SSL enhancements and there's no straightforward way to go about this.

However, it should nevertheless be possible to do it without delving into the private API.

If you write your own IOpenSSLClientConnectionCreator, like this:

from twisted.internet.interfaces import IOpenSSLClientConnectionCreator
from twisted.python.components import proxyForInterface

class AddExtraTrustRoots(proxyForInterface(IOpenSSLClientConnectionCreator)):
    def __init__(self, extraTrustRoots, original):
        self._extraTrustRoots = extraTrustRoots
        super(AddExtraTrustRoots, self).__init__(original)


    def clientConnectionForTLS(self, tlsProtocol):
        connection = (super(AddExtraTrustRoots, self)
                      .clientConnectionForTLS(tlsProtocol))
        cert_store = connection.get_context().get_cert_store()
        for cert in self._extraTrustRoots:
            cert_store.add_cert(cert)
        return connection

you can delegate to the pyOpenSSL API for manipulating the trust settings of the connection, although Twisted will not provide a high-level wrapper for this.

You can then use the above wrapper like:

options = AddExtraTrustRoots([OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, "..."), ...], optionsForClientTLS(u"google.com"))

I haven't tested this, let me know if I've overlooked something terrible :).

Another related concept that was not clear to me is how one might specify the SSL method (e.g. SSLv23_METHOD, SSLv3_METHOD, etc.) when making the request. Is there some recommended way to pass options to indicate the SSL method that ought to be used?

The default configuration allows for TLSv1.0, TLSv1.1, and TLSv1.2 (insofar as those versions are supported by your OpenSSL).  Do you really, really, really need to support worse protocol versions than that?

You don't actually want to change the "method", as "method" is a bizarre fiction of the OpenSSL API and is not actually how the protocol works; SSLv23_METHOD is the only "method" that allows protocol version negotiation, so you always want to start there and then set/unset the appropriate OP_NO_* options for the protocol versions you want to exclude.

This is all pretty tedious and gross so if you're sure you want to do it I will describe how in the next reply :).

Any guidance would be appreciated.

I hope that this was sufficient!

-glyph

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Bruno Harbulot | 11 Sep 16:27 2014
Picon

Response hanging after headers sent

Hello,

I've been trying to implement a simple notification system via long polling using Twisted 14.0.0. On the client side, a web page makes Ajax calls to an "update" resource, which will deliver the latest status message known to the application. On the server side, a thread feeds a queue with status events and another one polls this queue and sends the latest value to any pending request, trying to keep the request open for 30 seconds, or until a new status event arrives, whichever happens sooner (this is the long polling).

Occasionally, one of responses will just hang. "request.finish()" is called properly (I've even checked this with "notifyFinish()"), the response headers are sent, but the response body isn't (according to Wireshark), thereby leaving the browser hanging (since it thinks a valid response has started to come through).

I have noticed this behaviour with Twisted 13 and 14, but not with earlier versions. This is also an intermittent problem. I have been able to reproduce it a number of times, but not always, and it doesn't necessarily happen after the same number of requests. It seems to fail more easily on machines with one CPU core, or when the execution is limited to one core, with only one request at a time (for some reason, having concurrent requests from multiple browsers or tabs, or being able to use multiple CPUs on the server seems to make the problem harder to reproduce).

I'm pasting at the end of this message a simple example where the status events are coming from a counter running in a separate thread. When limiting the server process to one CPU, the counter visible on the webpage rarely seems to go much above 2000. I've only tried under Linux. I've tried this on an actual PC (limiting to one core using "taskset -c 0 python test_server.py"), a VM running in VirtualBox (set to 1 CPU) and even a Raspberry Pi. The Raspberry Pi is where the problem seems to be the easiest to reproduce, but this also happens with the actual PC and virtual machine. This was also only tried on a LAN (considering that, for the purpose of this test case, the notifications are rather frequent). I've tried with the default chunked encoding, or by setting the content-length explicitly, but it didn't make a difference.

Is there a way to force to the response to be flushed, after a call to "request.finish()"? Any other suggestions to fix this would also be appreciated.

Thank you,

Bruno.




Only 2 files are required for this test case: counter.html, test_server.py.
I've run this in a virtual environment, with Python 2.7 (in which I've installed Twisted with `pip install twisted==14.0.0`).
On a multicore machine, to limit the execution to one core, call the server with taskset: "taskset -c 0 python test_server.py".
Once the server is running, point a browser (tried with Chrome and Firefox) to "http://the.ip.address.example:8880/", you should see a counter increasing (and eventually stopping, which illustrates the problem). The issue seems to happen more often when only 1 client (or tab) is connected to the server.


- counter.html:

<html>
<head>
<script type="application/javascript" src="//code.jquery.com/jquery-1.11.0.min.js"></script>
</head>
<body>
<div id="counter" style="font-size: 100pt; font-weight: bold;"></div>
<script type="application/javascript">
// <![CDATA[
$(document).ready(function() {
    var eTag = null;
    var normalRefreshDelay = 50;
    var refreshDelay = normalRefreshDelay;
    var maxRefreshDelay = 5000;
    function refresh() {
        var headers = {};
        if (eTag) {
            headers['If-None-Match'] = eTag;
        }
        $.ajax({
            url: "/update",
            headers: headers,
            dataType: "json",
            cache: false,
            success: function(data, status, xhr) {
                try {
                    if (data[1] != null) { $("#counter").text("Counter: "+data[1]); }
                    eTag = xhr.getResponseHeader("ETag") || null;
                    refreshDelay = normalRefreshDelay;
                } catch (e) {
                    refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
                }
            },
            error: function(xhr, text, error) {
                refreshDelay = Math.min(refreshDelay * 2, maxRefreshDelay);
            },
            complete: function(xhr, textStatus) {
                setTimeout(refresh, refreshDelay);
            }
        });
    }
    refresh();
});
// ]]>
</script>
</body>
</html>



- test_server.py


# -*- coding: utf-8 -*-

import threading
import time
import Queue
import uuid
import socket
import json

from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
from twisted.web.static import File

q = Queue.Queue()

def fake_counter():
    i = 1
    while True:
        q.put({ 1: str(i) })
        time.sleep(0.15)
        i += 1

producer_thread = threading.Thread(target=fake_counter)
producer_thread.daemon = True
producer_thread.start()


class CounterUpdateResource(Resource):
    def __init__(self, msg_queue):
        Resource.__init__(self)
        self.msg_queue = msg_queue
        self.msg = {}
        self.etag = None
        self.pending_requests = []
       
        t = threading.Thread(target=self._poll_messages)
        t.daemon = True
        t.start()
       
    def _poll_messages(self):
        while True:
            try:
                self.msg = self.msg_queue.get(True, 30)
                self.etag = '"%s"' % (uuid.uuid4(),)
                print "-> Got new message: %s" % (self.msg,)
            except Queue.Empty:
                pass
            json_str = json.dumps(self.msg)
            json_len = len(json_str)
            while True:
                try:
                    request = self.pending_requests.pop()
                    self._actual_render(request, json_str, json_len)
                except IndexError:
                    break
           
    def _actual_render(self, request, json_msg, json_len):
        try:
            request.setETag(self.etag)
            request.setHeader("Content-Type", "application/json")
            request.setHeader("Content-Length", json_len)
            request.write(json_msg)
            request.finish()
        except:
            pass

    def render_GET(self, request):
        ifNoneMatchHeader = request.getHeader("If-None-Match")
        if ifNoneMatchHeader is None or ifNoneMatchHeader != self.etag:
            if self.etag:
                request.setETag(self.etag)
            request.setHeader("Content-Type", "application/json")
            data = json.dumps(self.msg)
            request.setHeader("Content-Length", len(data))
            return data
        else:
            self.pending_requests.append(request)
            return NOT_DONE_YET

root = Resource()
root.putChild("", File("counter.html"))
root.putChild("update", CounterUpdateResource(q))

factory = Site(root)
reactor.listenTCP(8880, factory)
reactor.run()






- Sample Wireshark output:


Here is a sample output using Wireshark:

GET /update?_=1410440411047 HTTP/1.1
Host: 192.168.0.8:8880
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "84c25895-de1e-4b50-954b-438be9d8d9b7"
Referer: http://192.168.0.8:8880/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Content-Type: application/json
Server: TwistedWeb/14.0.0

{"1": "371"}

GET /update?_=1410440411048 HTTP/1.1
Host: 192.168.0.8:8880
Connection: keep-alive
Cache-Control: max-age=0
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
If-None-Match: "7736231b-ee3a-4a9f-a93e-ced0994df098"
Referer: http://192.168.0.8:8880/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6

HTTP/1.1 200 OK
Date: Thu, 11 Sep 2014 13:06:23 GMT
Content-Length: 12
ETag: "ddd10400-3530-4cb4-a9cd-0b123032d8af"
Content-Type: application/json
Server: TwistedWeb/14.0.0



(I've inserted a coupe of line returns after "{"1": "371"}" for readability.)
Here, nothing is sent on the wire after the headers of the last response (the "\r\n\r\n" are present).


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Sha Hua | 27 Jan 18:58 2014
Picon

How to write an HTTP1.1 Proxy using Twisted?

Hi all,

I have recently started to learn and use Twisted. The first task is to write an HTTP proxy.
I found Twisted provides a built-in proxy module described here:


It says the server part inherits http.HTTPChannel, while the client part inherits http.HTTPClient.

However, as I later find out, http.HTTPClient is only a HTTP 1.0 client, which is not enough for my task. I also found there is another class called twisted.web.client.Agent which seems to be more advanced than http.HTTPClient and supports HTTP 1.1.

Questions:
1. Can twisted.web.client.Agent be used in a Proxy implementation with http.HTTPChannel? Is there any examples?
2. Does http.HTTPChannel support HTTP 1.1?

Thanks!

Sha
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
yoni s | 8 Dec 22:21 2013
Picon

Streaming MP3 using twisted matrix

Hi All,

I'm looking for a simple example that streams MP3 over http. 

I know how to Google, and even to use GitHub search but while I found several examples for using python I didn't find anything using twisted.

Any help would be appreciated 

Thanks,
Yoni
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
HawkOwl | 8 Nov 19:49 2013
Picon

Twisted 13.2.0 Release Announcement


On behalf of Twisted Matrix Laboratories, I am honoured to announce
the release of Twisted 13.2!

The highlights of this release are:

 * Twisted now includes a HostnameEndpoint implementation which uses
IPv4 and IPv6 in parallel, speeding up the connection by using
whichever connects first (the 'Happy Eyeballs'/RFC 6555 algorithm).
(#4859)

 * Improved support for Cancellable Deferreds by kaizhang, our GSoC
student. (#4320, #6532, #6572, #6639)

 * Improved Twisted.Mail documentation by shira, our Outreach Program
for Women intern. (#6649, #6652)

 * twistd now waits for the application to start successfully before
exiting after daemonization. (#823)

 * SSL server endpoint string descriptions now support the
specification of chain certificates. (#6499)

 * Over 70 closed tickets since 13.1.0.

For more information, check the NEWS file (link provided below).

You can find the downloads at <https://pypi.python.org/pypi/Twisted>
(or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>) .
The NEWS file is also available at
<http://twistedmatrix.com/Releases/Twisted/13.2/NEWS.txt>.

Many thanks to everyone who had a part in this release - the
supporters of the Twisted Software Foundation, the developers who
contributed code as well as documentation, and all the people building
great things with Twisted!

Twisted Regards,
HawkOwl
Axel Rau | 5 Nov 13:19 2013
Picon

deployment with twistd of simple web application

Hi,

just doing my 1st dynamic web application with twisted.

I have some working python modules, where the entry point is:
---
resource = RootWeatherPage()
meteoFactory = Site(resource)
reactor.listenTCP(80, factory, interface='some IP4')
reactor.listenTCP(80, factory, interface='some IP6')
reactor.run()
---

Now I want to use twistd web.
How do I do that?

Thanks Axel
---
PGP-Key:29E99DD6  ☀ +49 151 2300 9283  ☀ computing  <at>  chaos claudius

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Phil Mayers | 16 Sep 18:58 2013
Picon

Double tracebacks for t.web & klein

I'm playing with klein for a simple rest API (because I like the 
routing, mainly).

When a method raises an exception, a traceback gets logged twice - once 
by the t.web Request.processingFailed, called from here:

https://github.com/twisted/klein/blob/master/klein/resource.py#L125

...and once by the deferred garbage collection, as 
Request.processingFailed doesn't eat the deferred:

http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.1.0/twisted/web/server.py#L314

Obviously this double-traceback thing is hugely irritating. Who is at 
fault here? Is t.web doing the right thing by returning the failure from 
processingFailed, or is klein doing the wrong thing, either by using the 
(undocumented) processingFailed or omitting an errback further down the 
chain?

Obviously there's no way for *me* to add an errback - klein generates 
the deferred for me.
D Brian Kimmel | 29 Jul 20:01 2013

Need Help

Howdy,

I have a large home automation project that uses twisted.
I am adding a web interface and need athena for COMET activity.

I have been having a very hard time finding any current documentation to help 
me get the athena part working.
I am looking for brains to pick on getting this portion working bi-
directionally.

I am also looking for others who may be interested in working on this type of 
software.

github -> DBrianKimmel/PyHouse

Thanks
Brian
Pranav Bhardwaj | 17 Jul 02:58 2013
Picon

twisted.web : limiting file size during upload (using python)

Hey

I have a script to be able to upload logs to a twisted web server.

I have the following requirements:

1) to be able to limit the size of the upload to 10MB.
2)  to NOT store file in memory during the request.

Does twisted.web provide these functionality?

Any inputs on these would be welcome!



Here is the server part os the script:

        try:
            oStream = open(filename, 'wb')
            oStream.write(request.args['fname'][0])
            outputStream.close()

        except:
            # handle exception


a test client script:


<!DOCTYPE html>

<html>

<form action="https://serverUrl"

enctype="multipart/form-data"

method="post">

<div class="row">

<label for="fileUploadToServer">Select file</label><br />

fname : <input type="file" name="fname" />

<input type="submit" value="submit">

</div>

</form>

</html>

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Alexei Colin | 13 Jul 21:15 2013

Add response logging to logging-proxy.py example

Hello,

The example logging-proxy.py processes the request, but not the
response. Yes, one can learn how to do that from the docs and from
stackoverflow, but that is an avoidable time burden. Sometimes this
example is all one needs, if only it were complete with both directions.

Could the example be revised or new one added with something like the
attached? Thank you for the consideration.

-alexei
Attachment (logging-proxy.py): text/x-python, 2377 bytes
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Christopher Lozinski | 8 Jul 00:06 2013

Re: Nginx vs Twisted Web

 Thank so much to all of you for the excellent discussion on this topic.

I particularly liked Jean Paul from Twisted matrix's careful response.

And Burak Arslan's comment about Nginx supporting SendFile was hugely interesting.

So which one of those two did I choose?  At first I was going to go to Nginx, but then a developer I am cooperating with, who is also doing a Zope 3 server
went with Cherokee, and I followed in his lead.  Why???  Well Cherokee has a web GUI, that hugely simplifies my life, and the guy I am
cooperating with and I can support each other.  Really ease of use trumps all other issues for me.  I just have way way too much complexity that I am managing in life.

You can read more about my decision here.

http://zopache.com/ApacheNginxCherokeeTwistedWeb

If people want to continue this discussion, I would love to see a comparison of Twisted Web and Cherokee.  Best to start a new thread for that.

And if there were a web gui for Twisted Web, I would certainly have gone with that, even if it were an early release.  I can imagine one could create resources TTW Through The Web,
and store them in the ZODB.   Eventually I would like to see a tighter integration between Twisted Web and Zope 3.  I am starting with the  ZTFY.org release of Zope 3.  But all things take time. 

Regards
Christopher Lozinski


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

Gmane