Antonis Kaklis | 1 Jun 15:46 2010
Picon

parsing xml coming from a socket

Hi to all, and since this is my first post, congratulations for the perfect framework.
i have the following code

class SimpleServer(LineReceiver): 

    def connectionMade(self):
        print 'Connection from: ', self.transport.client

    def connectionLost(self, reason):
        print self.transport.client, 'Disconnected'

    def dataReceived(self, line):
        """Here the XML Parser"""

        p = xml.parsers.expat.ParserCreate()

        p.StartElementHandler = start_element
        p.EndElementHandler = end_element
        p.CharacterDataHandler = char_data
        p.Parse(line, 1)

I got the following error
--- <exception caught here> ---
  File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-
x86_64.egg/twisted/internet/selectreactor.py", line 146, in
_doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/lib/python2.6/site-packages/Twisted-10.0.0-py2.6-linux-
x86_64.egg/twisted/internet/tcp.py", line 460, in doRead
    return self.protocol.dataReceived(data)
  File "stdiodemo.py", line 419, in dataReceived
    p.Parse(line, 1)
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

The XML Message is coming in the form of:

POST /test/pcp/Listener HTTP/1.1
user-agent:hjahs
Host:127.0.0.1
Content-Length: 547

<pttv_control_message version="1.0-M4-SNAPSHOT" build="599"
xmlns="http://1270.0.01/pttv">
  <cmdReply>
    <code>200</code>
    <message>OK, found 5 session entries</message>
    <sessionList>
      <session>
        <id>06d4d59bfdfe10139dd874</id>
        <subscriberId>82</subscriberId>
        <deviceClass>and</deviceClass>
      </session>
    </sessionList>
  </cmdReply>
</pttv_control_message>

Please give me some hints.
How to parse the incoming XML and how to ommit the headers from the beginning of the message?

Thanks in advance
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Glyph Lefkowitz | 1 Jun 16:44 2010

Re: parsing xml coming from a socket

Hello, Antonis!

On Jun 1, 2010, at 9:46 AM, Antonis Kaklis wrote:

Hi to all, and since this is my first post, congratulations for the perfect framework.

Thank you :).

i have the following code

This code has two major, obvious problems:

class SimpleServer(LineReceiver):  

    def dataReceived(self, line): 

Here you say "dataReceived" receives a "line" parameter.  But dataReceived does not receive lines, it receives *data*, i.e. arbitrary chunks of bytes.  What you want to do is write a *line*Received method; you are already using LineReceiver, so I assume that's what you were trying to do.

If you fix that, the second problem is that you are creating a new XML parser for every line received.  Is that actually what you want to do?  Your message has some headers, which you will want to skip, and then the body of the message spans multiple lines.

Your "XML message" looks a lot like an HTTP POST (I am assuming the lack of spaces after the headers in your example is a typo).  Perhaps you would be better off using the built-in HTTP server in twisted.web, rather than trying to implement your own wire-level protocol?

Good luck,

-glyph
_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
jannico | 2 Jun 00:16 2010
Picon
Picon

A small getPage problem accessing localhost on Vista

Dear Twistonistas,

I am using 10.0.0 for 2.5 on a Vista Home 'Premium' (yes, yes, I know).

I am accessing some page with getPage(certain_url,method='GET').

Accessing works for non-local urls. Accessing does not work for "http://localhost:8080/..." - windows
error 10061: connection refused (abbreviated from partially localized error message).

This propably sounds like a firewall problem, but all three firewall profiles disabled, only the 'native'
windows vista firewall installed, so it shouldn't be one, I hope. 

When using a synchronous urllib.urlopen('http://localhost:8080/...') / read() pair, it works.

This makes me nervous somehow, I must admit. Perhaps you have any hint / idea what the problem might be? This
would be most welcome!

Btw.: The line "127.0.0.1 localhost" is in the host file in \Windows\System32\drivers\etc.

Thank you very much in advance!

Jannico
--

-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
exarkun | 5 Jun 18:30 2010

Re: A small getPage problem accessing localhost on Vista

On 1 Jun, 10:16 pm, jannico <at> gmx.de wrote:
>Dear Twistonistas,
>
>I am using 10.0.0 for 2.5 on a Vista Home 'Premium' (yes, yes, I know).
>
>I am accessing some page with getPage(certain_url,method='GET').
>
>Accessing works for non-local urls. Accessing does not work for 
>"http://localhost:8080/..." - windows error 10061: connection refused 
>(abbreviated from partially localized error message).
>
>This propably sounds like a firewall problem, but all three firewall 
>profiles disabled, only the 'native' windows vista firewall installed, 
>so it shouldn't be one, I hope.
>
>When using a synchronous urllib.urlopen('http://localhost:8080/...') / 
>read() pair, it works.
>
>This makes me nervous somehow, I must admit. Perhaps you have any hint 
>/ idea what the problem might be? This would be most welcome!
>
>Btw.: The line "127.0.0.1 localhost" is in the host file in 
>\Windows\System32\drivers\etc.
>
>Thank you very much in advance!

Sorry, I don't have anything to suggest.  I hope that when you figure 
this out, though, you'll let us know what the problem was, and if 
necessary file a ticket in the issue tracker.

Jean-Paul
Allen Bierbaum | 14 Jun 21:17 2010
Picon

twisted.web questions

(repost of email earlier today on twisted mailing list.  suggested this may be better place to discuss. sorry if you get this twice.)


We have a large client-server based application based on twisted.  The application has grown to the point where we would like to add a REST-based API to the server side to allow for interfacing with a wider variety of client applications.  Over the weekend I started looking into twisted.web and had a couple of basic questions:

1) Is there any method to stream a large request (ex: PUT or POST with file upload) into the system or does the entire body have to be loaded into memory as part of the request?

2) Has anyone used an existing URL Routing system (ex: Routes [1], Selector [2]) and adapted it for use with Twisted?  (I know I could use the twisted preferred method of parent child hierarchies, but our application has a better fit with the route-based method used by Pylons, Django, Rails, etc)

3) Is there a document anywhere that describes all the environment variables the user can get from the request and how to get to them?  (something like the environ section of the WSGI PEP [3])

4) Are their any wrappers people have developed to make it a bit easier to use deferreds (in particular inlinecallbacks) in the handlers for twisted.web?  (I am considering just writing a wrapper myself that provides an interface similar to google app engine webapp [4] but wanted to check first so I don't reinvent something)

Thanks,
Allen


_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Allen Bierbaum | 15 Jun 00:42 2010
Picon

Re: [Twisted-Python] twisted.web questions

[..] 
>> There's a branch of Twisted that implements this, hopefully it will
>> eventually be merged:
>> http://twistedmatrix.com/trac/ticket/3711
>>
>>
> Is there anything that is currently holding this up?  (ie. is the design
> on
> this branch what will end up in the system or could it change
> dramatically?)

There are a bunch of review comments in the ticket. You could just write
your own until this is ready, I've done it a number of times and it's a
pretty small piece of code.

I started writing one of my own today (basically a wrapper in render_GET that calls an deferrable render_GET).

Something like this:

  def _bg_GET(self, request):
      # Do something slow
      request.write(result)
      return request

   def render_GET(self, request):
      d = deferToThread(self._bg_GET, request)
      d.addCallback(lambda r: r.finish())
      return NOT_DONE_YET 

That seems to work, but I would like to 
   a) not use the deferToThread() and 
   b) do something that doesn't sacrifice too much performance. 

At the end of the day, what I would really like is something in render_GET that checks the return from calling the internal GET method and if it is a deferred it adds a callback to finish the request, if not it simply returns the result.  (this would allow me to use deferToThread, inlineCallbacks, etc as needed in my get methods)

Can you see any issues with doing something like this?  (it seems almost too simple so I expect that I must be missing something)

-Allen

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Christopher Armstrong | 15 Jun 01:28 2010

Re: [Twisted-Python] twisted.web questions

On Mon, Jun 14, 2010 at 5:42 PM, Allen Bierbaum <abierbaum <at> gmail.com> wrote:
> Something like this:
>
>   def _bg_GET(self, request):
>       # Do something slow
>       request.write(result)
>       return request
>    def render_GET(self, request):
>       d = deferToThread(self._bg_GET, request)
>       d.addCallback(lambda r: r.finish())
>       return NOT_DONE_YET
>
> That seems to work, but I would like to
>    a) not use the deferToThread() and
>    b) do something that doesn't sacrifice too much performance.
>
> At the end of the day, what I would really like is something in render_GET
> that checks the return from calling the internal GET method and if it is a
> deferred it adds a callback to finish the request, if not it simply returns
> the result.  (this would allow me to use deferToThread, inlineCallbacks, etc
> as needed in my get methods)
>
> Can you see any issues with doing something like this?  (it seems almost too
> simple so I expect that I must be missing something)

Yeah, don't call request methods (like request.write) from the
function that gets run in a thread. You should be calling
request.write in the callback, or with callFromThread in the threaded
function.

--

-- 
Christopher Armstrong
http://radix.twistedmatrix.com/
http://planet-if.com/

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Allen Bierbaum | 15 Jun 17:12 2010
Picon

Re: [Twisted-Python] twisted.web questions

On Mon, Jun 14, 2010 at 6:28 PM, Christopher Armstrong <radix <at> twistedmatrix.com> wrote:

On Mon, Jun 14, 2010 at 5:42 PM, Allen Bierbaum <abierbaum <at> gmail.com> wrote:
> Something like this:
>
>   def _bg_GET(self, request):
>       # Do something slow
>       request.write(result)
>       return request
>    def render_GET(self, request):
>       d = deferToThread(self._bg_GET, request)
>       d.addCallback(lambda r: r.finish())
>       return NOT_DONE_YET
>
> That seems to work, but I would like to
>    a) not use the deferToThread() and
>    b) do something that doesn't sacrifice too much performance.
>
> At the end of the day, what I would really like is something in render_GET
> that checks the return from calling the internal GET method and if it is a
> deferred it adds a callback to finish the request, if not it simply returns
> the result.  (this would allow me to use deferToThread, inlineCallbacks, etc
> as needed in my get methods)
>
> Can you see any issues with doing something like this?  (it seems almost too
> simple so I expect that I must be missing something)

Yeah, don't call request methods (like request.write) from the
function that gets run in a thread. You should be calling
request.write in the callback, or with callFromThread in the threaded
function.

Good tip.  Looks like the wsgi wrapper has a nice little helper method to support this:


Thanks,
Allen

_______________________________________________
Twisted-web mailing list
Twisted-web <at> twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

Gmane