Amber "Hawkie" Brown | 23 Jul 10:09 2015
Picon

Twisted 15.3.0 Prerelease 1 Announcement

Hi everyone!

Another Twisted release cycle dawns, and I bring you the first prerelease of Twisted 15.3.

This release is characterised by cleanups and porting, with at least ten modules ported to Python 3, and the
removal of Lore, Twisted's use of Hotshot, and various other smaller deprecated classes and functions.
Twisted has also shed its usage of "subprojects" -- there is now only one Twisted (long may it reign).

The tarball and the newsfile summarising all the changes can be downloaded from <http://twistedmatrix.com/Releases/pre/15.3.0pre1/>.

Enjoy, and test! If no issues are found, 15.3 will be released sometime next week.

Twisted Regards,

Amber "Hawkie" Brown
GPG: https://keybase.io/hawkowl

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
vvangelovski | 27 Jun 22:00 2015
Picon

You have a new notification from vvangelovski <at> gmail.com. View?

You have a new notification from

View?
No
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
vvangelovski | 24 Jun 22:34 2015
Picon

You have a new notification from vvangelovski <at> gmail.com. View??

You have a new notification from

View?
No
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
vvangelovski | 21 Jun 23:42 2015
Picon

vvangelovski <at> gmail.com has indicated you're a friend. Accept?

I would like to add you as a friend

Accept
Decline
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Sagar Dixit | 10 Jun 04:01 2015
Picon

How does Twisted Web Multiprocess work ?

Hi,

I am exploring Twisted Web for my RESTful application. My application is stateless and involves storing and retrieving objects based on Object-ID. This application will run on beefy (multicore, lots of memory) machine. However, not all APIs that the application issues to underlying storage are async and hence I cannot fully utilize Deferreds
Which means, there will some blocking calls and hence my primary interest is to use Twisted Web in multiprocessing mode


However, I am not sure if it is the "correct" way of doing things.

Hence I had some questions around it:

1.  Is there an interface (similar to defertoThread) which allows me to execute a blocking call in a separate process ?

2. Does reactor synchronize access of all processes to the shared listen socket ?

3. Is there a sample code I can refer to  where the application is spawning subprocesses to handle HTTP requests ?

Thanks in advance!

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Amber "Hawkie" Brown | 8 Jun 07:57 2015
Picon

Twisted is removing Python 2.6 support

Hi everyone,

As mentioned in http://twistedmatrix.com/pipermail/twisted-python/2015-March/029258.html ,
Twisted is removing Python 2.6 support. As such, I would like to announce that 15.3 will be the last release
with 2.6 support. After this release, the Python 2.6 buildbots will be removed from the 'supported' list
and most likely retired.

This will remove all buildbot coverage of OS X and RHEL/CentOS. As I've been taking point on the buildbots
recently, I'm going to be deploying a new CentOS 7 buildslave which will provide coverage of that
platform. That just leaves OS X uncovered -- so if you would like to donate buildslaves for this, please
send me an email.

Amber "Hawkie" Brown
GPG: https://keybase.io/hawkowl

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Christopher Lozinski | 8 Jun 06:17 2015

Web Server Bandwidth Tracking

I want to release zopache.com as a web development platform, but I worry
that someone will overload the servers.

So I have to track bandwidth for each user account. 

Traditionally this is done by creating log files and processing them,
but that seems exceedingly complex.  Much easier to just have a python
object for each user account, and track the amount of bandwidth used in
the last day, week and month.  Even for 50K users that cannot require
that much RAM. 
Even if the web server goes down, and I loose the data, I do not care. 
I just need to be emailed when limits are exceeded.

Is this a good idea, or should I be using log files?

For years i have been watching Twisted, but never really required it. 
Maybe this is a good excuse to start using Twisted.   Currently I am
using NGinx. 

Is there a library you would recommend to help me with this.  An hour
spent searching the web did not turn up anything.   It can't be that
hard to do.

What do you think?

Regards
Chris
Ian Schenck | 20 May 19:56 2015
Picon

A twisted wsgi success

I mostly lurk on here, but I wanted to share my recent experience switching my company's wsgi container to twisted wsgi.

We had been using a particular wsgi container implemented in C. As time went on problems creeped up with cython modules, segfaults happened, and threading had to be disabled. Things that worked in development would not work in production. At this point I wanted two things: the development container should be the production container, and the container of choice should provide a python interpreter environment consistent with actual python. These containers written in C which treat your application like a plugin fail to do that across the board.

We no longer have any of the issues we had before (cython, threading, segfaults), and if something were to break in production, it is probably broken in development and won't make it out the door. Development servers run the same twisted wsgi container that runs in production, in fact they are both launched the same (but middlewares and reloaders differ). That consistency is a really big deal. Additionally, deployment and config management dropped a few hundred lines.

The biggest pushback I received was over performance. We benched our real world application (versus, ahem, just returning a string) and found our median response time was consistently 1-2ms higher across various concurrency levels. That's insignificant for us. Under high load, twisted wsgi actually did better with tail latency (99%ile and max). Maybe that slight bump in median latency is because our web workers now speak http instead of a "light weight protocol" akin to scgi, but the benefits of talking http (curl much?) are also worth a millisecond or two.

Thanks for all the hard work, and thanks Glyph for your 2011 Djangocon talk and your tweets leading me to twisted wsgi. We're very happy. If anyone wants details or a peek at some of the wiring, I can probably get approval to share more.
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Benjamin BERTRAND | 23 Apr 22:44 2015
Picon

web UI example for Twisted application

Hello,

I have a Twisted network application that I deployed as a twistd plugin.
I have different configuration files defining several parameters including some Redis pub/sub channels.
Today I use a script to stop and start the plugin with the proper configuration file.
I’d like to make a web UI to be able to change the configuration of the application.

I already saw some people asking about that but I haven’t found any example.
Could anyone point me to some examples of a Twisted application with a web UI?

I have developed some Flask web app, so I think it would be easier for me to start with Klein.
But I’m not against trying Nevow (Twisted docs recommend it for this purpose).

Thanks

Benjamin
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Chris Drane | 13 Apr 20:29 2015
Picon

Client.agent - PartialDownloadError - cannot determine cause

I am new with Twisted so chances are that this is *not* a bug. If that is the case, I would very much appreciate it if you help me understand what I am doing incorrectly.

I have some very simple code to just download a particular page via HTTP. The problem I'm running into is that I get PartialDownloadErrors for some sites, while others load fine. And when I examine the HTTP response, it looks like the page fully downloaded. The traceback and logging information have been very unhelpful. I cannot determine what exactly is causing the error. 

import sys
from twisted.internet.task import react
from twisted.python.log import err, startLogging
from twisted.web.client import Agent, BrowserLikeRedirectAgent, readBody
from twisted.web.http_headers import Headers
from twisted.internet import reactor
from twisted.internet.ssl import ClientContextFactory

def cbBody(r):
    print "Response body:"
    print r

def cbRequest(response):
    print "Received response"
    d = readBody(response)
    d.addCallbacks(cbBody, err)
    return d

def err(e):
    try:
        e.raiseException()
    except Exception, err:
        pass
        # display HTTP response
        # print err.response
    e.printTraceback()
    sys.stderr.write(str(e))

def main(reactor):
    startLogging(sys.stdout)
    agent = BrowserLikeRedirectAgent(Agent(reactor))
    d = agent.request("GET", b"http://www.google.com",
            Headers({'User-Agent': ['Twisted Web Client Example']}),
            None)
    d.addCallbacks(cbRequest, err)
    return d

react(main)

2015-04-13 14:13:11-0400 [-] Log opened.
2015-04-13 14:13:11-0400 [-] Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Received response
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Traceback (most recent call last):
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Failure: twisted.web.client.PartialDownloadError: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] [Failure instance: Traceback (failure with no frames): <class 'twisted.web.client.PartialDownloadError'>: 200 OK
2015-04-13 14:13:11-0400 [HTTP11ClientProtocol,client] Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x10303f560>
2015-04-13 14:13:11-0400 [-] Main loop terminated.

Thanks!

Chris


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Guido Winkelmann | 2 Apr 19:49 2015
Picon

treq requests failing with twisted.web._newclient.ResponseNeverReceived

Hi,

Recently, I have been experiencing some problems in our codebase wth treq 
requests failing with error messages like this:

2015-04-02 19:20:22 ERROR    - pf.agent.http.client - POST 
http://127.0.0.1:5000/api/v1/agents/a113b5a8-9822-413b-b1fd-fce3014956cf has 
failed (uid: 78c7ec2a4b4d):
Traceback (most recent call last):
Failure: twisted.web._newclient.ResponseNeverReceived: 
[<twisted.python.failure.Failure <class 
'twisted.internet.error.ConnectionDone'>>]

Unfortunately, I have not been able to successfully produce a reduced test 
case that will reproduce this behaviour.  It only happens under some weird set 
of circumstances, and I am still not sure exactly which.  It appears to be 
highly sensitive to the order of requests.  Funnily enough, this error does 
not show in the codebase as it is on github right now, but if I reorder some 
of the requests, it will show.  Here is one example that has cost me some time 
and head-scratching:

https://github.com/pyfarm/pyfarm-agent/issues/249

I think that these problems have started after upgrading to Twisted 15.0, but 
I'm not completely certain about this.

Here is one of the methods that's affected by this:

https://github.com/pyfarm/pyfarm-agent/blob/master/pyfarm/agent/service.py#L231

(post_direct() is just a thin wrapper over treq.request that adds some headers 
we need in our codebase.  I have tried just using treq.request directly 
instead, the result was the same.)

Can someone tell me under which circumstances treq.request would produce an 
error like this?

Without actually knowing the source in question, I would guess that something 
in there is, for some some reason, erroneously trying to reuse a TCP 
connection that has already been closed.

Regards,

	Guido W.

Gmane