Graham Dumpleton wrote:
> 2009/5/5 Armin Ronacher <armin.ronacher-GGlT2RywCWtWk0Htik3J/w@public.gmane.org>:
>> Graham Dumpleton wrote:
>>> I can't see but have choice but to pass such settings through as
>>> strings, else more than likely would cause problems for applications.
>>> Problem is it isn't clear what encoding stuff can be in Apache
>>> configuration. At the moment latin-1 is assumed.
>> Because those information does not have a specified encoding I can see
>> nothing wrong with it passing that information as bytestrings.  I would
>> have no problem passing *all* values as bytestrings.
>
> At what point does that become an inconvenience though? I guess that
> is my concern, because if one has to do too many manual conversions in
> an application, people will start to complain it becomes unwieldy to
> use. In other words, you make it easier or more logical for
> frameworks, but do you end up putting more burden on applications for
> stuff outside those core values.
>
> So, for those core CGI values which the framework is going to modify
> even before an application sees them, then fine. Is the framework also
> going to set the rules as to what encoding is used for other values in
> the WSGI environment and convert them per that encoding when an
> application requests them, or is the application always going to have
> to deal with them as bytes?
>
> As I keep saying, you guys who write the frameworks and applications
> are going to know better than I, I am just challenging the notions as
> a way of making people think about it so the end result is what is the
> most logical thing to do.

In short: it's pretty easy for a framework to default to utf-8 for
everything, yet give application developers ways to override that. See,
for example, the cherrypy.tools.encoding Tool in our python3
branch--it's moved from running "sometime" after the page handler, to
wrapping the page handler so all page handlers emit bytes. That makes it
possible for everyone to use unicode strings everywhere, yet still allow
some to specify exact bytes as necessary. In shorter: don't worry about
that part, we've got it covered. ;)


Robert Brewer
fumanchu-Q+9y+cpEbCIdnm+yROfE0A@public.gmane.org

Philip Jenvey brought this to my attention:

  http://www.python.org/dev/peps/pep-0383/

It's a UTF8 encoding and decoding scheme that encodes illegal bytes in such a way that you can decode to get the original bytes object, and thus transcode to another encoding.  It's intended for cases exactly like WSGI.

--
Ian Bicking  |  http://blog.ianbicking.org

On Tue, May 5, 2009 at 10:14 PM, Graham Dumpleton <graham.dumpleton-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Graham Dumpleton wrote:
> Robert, do you have any comments on the restricting of response
> content to bytes and not allow fallback to conversion per latin-1?
>
> I heard that in CherryPy WSGI server you are only allowing bytes. What
> is your rational for that at the moment?


In Python 2.x, one could easily mix unicode strings and byte strings in
the same interface, because they mostly supported the same operations.
Not so in Python 3.x--byte strings are missing everything from
capitalize() to zfill() [1]. I feel that choosing one type or the other
is required in order to avoid mountains of if-statements in middleware
(and lots of 'pass' statements if bytes are found).

I decided that that single type should be byte strings because I want
WSGI middleware and applications to be able to choose what encoding
their output is. Passing unicode to the server would require some
out-of-band method of telling the server which encoding to use per
response, which seemed unacceptable.

The down side, already alluded to, is that middleware cannot then call
e.g. response.capitalize() or any of a number of other methods without
first decoding the response. And it cannot do that reliably unless
(again) the encoding which was used to produce bytes is communicated
down the stack out of band.

The python3 branch of CherryPy is by no means complete. I'd be happy to
explore emitting unicode if we could decide on a method whereby apps
could inform the server which encoding they want. Middleware which
transcoded the response would need a means of overriding that. But of
course, that opens a whole new can of worms if something goes wrong,
because application authors want control over the error response; if the
server is encoding the response, and an error occurs, there would have
to be a way to pass control back up the stack to...what? whichever
component last set the encoding? That road starts to get complicated
very quickly.

If some middleware needs to treat the response as unicode, I'd rather
emit bytes and somehow return the encoding as part of the response.
Perhaps WSGI 2's mythical "return (status, headers, body-iterable,
encoding)". Middleware could then decode/transcode as desired. I can't
think of a downside to that, other than some lost cycles spent
de/encoding, but perhaps there are some I don't yet foresee.


Robert Brewer
fumanchu-Q+9y+cpEbCIdnm+yROfE0A@public.gmane.org

[1] See http://docs.python.org/dev/py3k/library/stdtypes.html#string-methods