Python web programming infrastructure discussion

J.D. Main | 3 Apr 2010 18:32

IIS and Python CGI - how do I see more than just the form data?

Hi Folks,

I hope this question hasn't already been answered...

I'm using IIS 5 and calling a python script directly in the URL of a request.  
Something like:

http://someserver/myscript.py

or even

http://someserver/myscript.py?var1=something&var2=somthingelse

Using the CGI module, I can certainly see and act upon the variables that 
are passed as  GET or POST actions.  What I'm after is something more 
low level.  I want to see the entire HTTP request with everything inside it.

Does IIS actually pass that information to the CGI application or does it just 
pass the variables?

The intent is to write a "RESTFUL" CGI script.  I need to actually "see" the 
URI and the parameters of the incoming request to map the appropriate 
action.  Without short circuiting the IIS webserver, how would my python 
parse the following:

http://someserver/someapp/someuser/someupdate?var1=Charlie

Thanks in advance!

JDM

(Continue reading)

headers

Aaron Watters | 5 Apr 2010 16:14

Re: IIS and Python CGI - how do I see more than just the form data?

I think you should consider using the
WSGI interface.

The WSGI interface puts all the components of a request
into a request environment dictionary which is sent as a
parameter to the function generating the response.  
For example have a look at the test application

http://whiffdoc.appspot.com/tests/misc/testDebugDump?thisVar=thatValue&thisOtherVar=ThatOtherValue

which dumps out the WSGI environment (with WHIFF extensions)
to the response.  All the information you need is somewhere inside
the environment dictionary (but it's not always easy to find).

You could also look at WHIFF which helps combine some of the
features of the CGI module with the WSGI interface.

http://whiffdoc.appspot.com/

Hope that helps,  -- Aaron Watters

===
% man less
less is more.

--- On Sat, 4/3/10, J.D. Main <jdmain@...> wrote:

> From: J.D. Main <jdmain@...>
> Subject: [Web-SIG] IIS and Python CGI - how do I see more than just the form data?
> To: web-sig@...

(Continue reading)

headers

Randy Syring | 6 Apr 2010 22:37

SQLAlchemy Queries & HTML Data Grid

I am planning on building a library that will facilitate creation of custom queries and html display of resulting datasets from SQLAlechemy queries. I have some basic work done here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysapp/branches/0.1/pysapp/modules/datagrid/

But I don't like the API and I don't want the library to be dependent on pysapp. Furthermore, I would like to have a more verbose querying ability akin to Redmine:

http://www.redmine.org/projects/redmine/issues

Including:

Filters
Column Selection
Grouping (multiple levels)
Sorting (multiple columns)
some kind of query saving/loading mechanism with a flexible backend

I have done some basic table generation work here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/

with the tests being the best place to get an idea of how it works:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/pysdatagrid/tests/test_render.py

Looking for comments, pointers to other projects, and/or possibly interest in helping with a project like this. I am currently working in SVN but will most likely move to hg/git if there are others who are interested.
-- -------------------------------------- Randy Syring Intelicom 502-644-4776 "Whether, then, you eat or drink or whatever you do, do all to the glory of God." 1 Cor 10:31

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

J.D. Main | 7 Apr 2010 03:25

Re: IIS and Python CGI - how do I see more than just the form data?

Thanks Aaron,

I think I will explore the WSGI interface.  However, I did learn a trick using 
the OS Module:

import cgi, os

formfields = cgi.FieldStorage()
http_stuff = os.environ

print "Content-type: text/html"
print
print "<html>"
print "<head>"
print "<title>Raw HTTP Test</title>"
print "</head>"
print "<body>"

print formfields
print http_stuff

print "</body>"
print "</html>"

The os.environ variable is a big dictionary containing most (if not all) of the 
values inside the original http GET or POST.

Best Regards,

JDM

I think you should consider using the
WSGI interface.

The WSGI interface puts all the components of a request
into a request environment dictionary which is sent as a
parameter to the function generating the response.  
For example have a look at the test application

http://whiffdoc.appspot.com/tests/misc/testDebugDump?thisVar=thatValue&
thisOtherVar=ThatOtherValue

which dumps out the WSGI environment (with WHIFF extensions)
to the response.  All the information you need is somewhere inside
the environment dictionary (but it's not always easy to find).

You could also look at WHIFF which helps combine some of the
features of the CGI module with the WSGI interface.

http://whiffdoc.appspot.com/

Hope that helps,  -- Aaron Watters

===
% man less
less is more.

--- On Sat, 4/3/10, J.D. Main <jdmain@...> wrote:

> From: J.D. Main <jdmain@...>
> Subject: [Web-SIG] IIS and Python CGI - how do I see more than just the form data?
> To: web-sig@...
> Date: Saturday, April 3, 2010, 12:32 PM
> Hi Folks,
> 
> I hope this question hasn't already been answered...
> 
> I'm using IIS 5 and calling a python script directly in the
> URL of a request.  
> Something like:
> 
> http://someserver/myscript.py
> 
> or even
> 
> http://someserver/myscript.py?var1=something&var2=somthingelse
> 
> Using the CGI module, I can certainly see and act upon the
> variables that 
> are passed as  GET or POST actions.  What I'm
> after is something more 
> low level.  I want to see the entire HTTP request with
> everything inside it.
> 
> Does IIS actually pass that information to the CGI
> application or does it just 
> pass the variables?
> 
> The intent is to write a "RESTFUL" CGI script.  I need
> to actually "see" the 
> URI and the parameters of the incoming request to map the
> appropriate 
> action.  Without short circuiting the IIS webserver,
> how would my python 
> parse the following:
> 
> http://someserver/someapp/someuser/someupdate?var1=Charlie
> 
> Thanks in advance!
> 
> JDM
> 
> 
> _______________________________________________
> Web-SIG mailing list
> Web-SIG@...
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/arw1961%40yahoo.com
> 
_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Aaron Watters | 7 Apr 2010 16:03

Re: SQLAlchemy Queries & HTML Data Grid

Thanks Randy, very interesting.

My initial reaction is that you are building a stack on top of a stack.
It's not clear to me what problem you want to solve
and your requirements are. It's possible that
you could find it easier to abstract directly on top of SQL or alternatively
you could consider using another sort of data model like mongodb.
Building an abstraction on top of SQLAlchemy which hasn't even reached
1.0 strikes me as dubious.

Thanks again, -- Aaron Watters

--- On Tue, 4/6/10, Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org> wrote:

From: Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org>
Subject: [Web-SIG] SQLAlchemy Queries & HTML Data Grid
To: web-sig-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Date: Tuesday, April 6, 2010, 4:37 PM

I am planning on building a library that will facilitate creation of custom queries and html display of resulting datasets from SQLAlechemy queries. I have some basic work done here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysapp/branches/0.1/pysapp/modules/datagrid/

But I don't like the API and I don't want the library to be dependent on pysapp. Furthermore, I would like to have a more verbose querying ability akin to Redmine:

http://www.redmine.org/projects/redmine/issues

Including:

Filters

Column Selection

Grouping (multiple levels)

Sorting (multiple columns)

some kind of query saving/loading mechanism with a flexible backend

I have done some basic table generation work here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/

with the tests being the best place to get an idea of how it works:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/pysdatagrid/tests/test_render.py

Looking for comments, pointers to other projects, and/or possibly interest in helping with a project like this. I am currently working in SVN but will most likely move to hg/git if there are others who are interested.
--
--------------------------------------
Randy Syring
Intelicom
502-644-4776

"Whether, then, you eat or drink or
whatever you do, do all to the glory
of God." 1 Cor 10:31

-----Inline Attachment Follows-----

_______________________________________________
Web-SIG mailing list
Web-SIG-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/arw1961%40yahoo.com

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Aaron Watters | 7 Apr 2010 16:06

Re: IIS and Python CGI - how do I see more than just the form data?

--- On Tue, 4/6/10, J.D. Main <jdmain@...> wrote:

> From: J.D. Main <jdmain@...>
> Subject: Re: [Web-SIG] IIS and Python CGI - how do I see more than just the form data?
> To: web-sig@...
> Date: Tuesday, April 6, 2010, 9:25 PM
> Thanks Aaron,
> 
> I think I will explore the WSGI interface.  However, I
> did learn a trick using 
> the OS Module:
> 
> import cgi, os
> 
> formfields = cgi.FieldStorage()
> http_stuff = os.environ .....

Yes, that will work too.  In fact the CGI interface to WSGI
works like this.  The advantage to using WSGI is that it
makes it possible to move your application to other configurations
more easily (in theory) and it's just a tiny bit more high level.

Best regards,  -- Aaron Watters

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Randy Syring | 7 Apr 2010 19:37

Re: SQLAlchemy Queries & HTML Data Grid

Aaron,

Sorry, I must not really have explained clearly. This isn't an abstraction layer, but more like a UI component or widgit that facilities basic reporting. Look at these pages:

http://www.redmine.org/issues
http://trac.edgewall.org/query

Both pages have a similar structure:

UI Controls

filtering
grouping
column selection
sorting

Paged/Sortable Recordset Display

The library would do the heavy lifting and allow any application using SQLAlchemy to easily create such query/recordset interfaces to underlying data.

You would need to:

Instantiate the DataGrid class
Create a base SQLAlchemy query to be used for the data
Define the filter types associated with the columns (i.e. TextFieldFilter, OptionsFilter('low','medium', 'high'), DateSpanFilter, etc.)
Limit sorting, grouping to appropriate columns
choose which columns of the dataset to show by defaults

The library would then:

parse GET/POST for filter/column/sort/page settings &/or use defaults
compose SQLALchemy query to satisfy the request
execute the query and get the database results
put results into an HTML table
return the HTML form needed for the controls and the recordset table including necessary CSS and JS

Obviously, the library should be easily customizable and the rendering of HTML should be flexible. It should also be flexible enough to work with the different WSGI libraries out there.

I hope that makes better sense. If you or anyone else is interested, I can give you some code I have working from the pysapp project which does some basic filtering, paging, and sorting. The API is awful, but I think it might give you a better idea of what I am talking about.

-------------------------------------- Randy Syring Intelicom 502-644-4776 "Whether, then, you eat or drink or whatever you do, do all to the glory of God." 1 Cor 10:31

Aaron Watters wrote:

Thanks Randy, very interesting.

My initial reaction is that you are building a stack on top of a stack.
It's not clear to me what problem you want to solve
and your requirements are. It's possible that
you could find it easier to abstract directly on top of SQL or alternatively
you could consider using another sort of data model like mongodb.
Building an abstraction on top of SQLAlchemy which hasn't even reached
1.0 strikes me as dubious.

Thanks again, -- Aaron Watters

--- On Tue, 4/6/10, Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org> wrote:

From: Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org>
Subject: [Web-SIG] SQLAlchemy Queries & HTML Data Grid
To: web-sig-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Date: Tuesday, April 6, 2010, 4:37 PM

I am planning on building a library that will facilitate creation of custom queries and html display of resulting datasets from SQLAlechemy queries. I have some basic work done here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysapp/branches/0.1/pysapp/modules/datagrid/

But I don't like the API and I don't want the library to be dependent on pysapp. Furthermore, I would like to have a more verbose querying ability akin to Redmine:

http://www.redmine.org/projects/redmine/issues

Including:

Filters

Column Selection

Grouping (multiple levels)

Sorting (multiple columns)

some kind of query saving/loading mechanism with a flexible backend

I have done some basic table generation work here:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/

with the tests being the best place to get an idea of how it works:

https://svn.rcslocal.com:8443/svn/pysmvt/pysdatagrid/trunk/pysdatagrid/tests/test_render.py

Looking for comments, pointers to other projects, and/or possibly interest in helping with a project like this. I am currently working in SVN but will most likely move to hg/git if there are others who are interested.
-- -------------------------------------- Randy Syring Intelicom 502-644-4776 "Whether, then, you eat or drink or whatever you do, do all to the glory of God." 1 Cor 10:31

-----Inline Attachment Follows-----

_______________________________________________
Web-SIG mailing list
Web-SIG-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/arw1961%40yahoo.com

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Aaron Watters | 7 Apr 2010 20:54

Re: SQLAlchemy Queries & HTML Data Grid

Randy:

It seems you want a sortable HTML table that talks to a back end
query engine. I don't see why this needs to be specific to SQLAlchemy.

Here is a WHIFF middleware which does some of what you are talking
about (the demo formatting is basic/ugly for simplicity purposes).

Demo

http://whiffdoc.appspot.com/tests/misc/testSortTable

Demo source

http://whiffdoc.appspot.com/tests/showText?path=./misc/testSortTable

The documentation

http://whiffdoc.appspot.com/docs/W1200_1400.stdMiddleware#Header83

is not extensive, but here is the source for the
core middleware widget.

http://aaron.oirt.rutgers.edu/cgi-bin/whiffRepo.cgi/file/8c031c68a5a0/whiff/middleware/sortTable.py

As written it requires the whole table as a list of dictionaries and then does
paging from the full list. It certainly needs generalization but maybe it's
a start. Let me know if you have questions or comments.

-- Aaron Watters

--- On Wed, 4/7/10, Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org> wrote:

From: Randy Syring <rsyring-KjVcCr55iNUxpvK4C0GaPQ@public.gmane.org>
Subject: Re: [Web-SIG] SQLAlchemy Queries & HTML Data Grid
To: "Aaron Watters" <arw1961-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
Cc: web-sig-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
Date: Wednesday, April 7, 2010, 1:37 PM

Aaron,

Sorry, I must not really have explained clearly. This isn't an abstraction layer, but more like a UI component or widgit that facilities basic reporting. Look at these pages:

http://www.redmine.org/issues
http://trac.edgewall.org/query]
....

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Randy Syring | 7 Apr 2010 21:07

Re: SQLAlchemy Queries & HTML Data Grid

> It seems you want a sortable HTML table that talks to a back end
> query engine.  I don't see why this needs to be specific to SQLAlchemy.
Well...not just sorting though.  Sorting, filtering, grouping, column 
selection, and paging.  You are right that the backend does not need to 
be SQLAlchemy specific, but since that is what I use, that is what I was 
going to start with.  Ideally, the library would be both sql framework 
and wsgi framework agnostic.

The main point of the library would be to save the time/hassle in 
creating the HTML/CSS/JS for the query controls.

--------------------------------------
Randy Syring
Intelicom
502-644-4776

"Whether, then, you eat or drink or 
whatever you do, do all to the glory
of God." 1 Cor 10:31

_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

headers

Manlio Perillo | 8 Apr 2010 16:08

WSGI and start_response

Hi.

Some time ago I objected the decision to remove start_response function
from next version WSGI, using as rationale the fact that without
start_callable, asynchronous extension are impossible to support.

Now I have found that removing start_response will also make impossible
to support coroutines (or, at least, some coroutines usage).

Here is an example (this is the same example I posted few days ago):
http://paste.pocoo.org/show/199202/

Forgetting about the write callable, the problem is that the application
starts to yield data when tmpl.render_unicode function is called.

Please note that this has *nothing* to do with asynchronus applications.
The code should work with *all* WSGI implementations.

In the pasted example, the Mako render_unicode function is "turned" into
a generator, with a simple function that allows to flush the current buffer.

Can someone else confirm that this code is impossible to support in WSGI
2.0?

If my suspect is true, I once again object against removing start_response.

WSGI 1.0 is really a well designed protocol, since it is able to support
both asynchonous application (with a custom extension) and coroutines,
*even* if this was not considered during protocol design.

Thanks  Manlio
_______________________________________________
Web-SIG mailing list
Web-SIG@...
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org

4	April 2013
4	March 2013
1	January 2013
4	October 2012
2	September 2012
1	August 2012
8	July 2012
16	June 2012
15	May 2012
8	April 2012
16	March 2012
37	February 2012
3	January 2012
2	December 2011
1	October 2011
18	September 2011
7	August 2011
1	July 2011
3	May 2011
55	April 2011
6	March 2011
14	February 2011
218	January 2011
7	December 2010
17	November 2010
34	October 2010
129	September 2010
22	August 2010
76	July 2010
23	June 2010
5	May 2010
78	April 2010
33	March 2010
1	February 2010
13	January 2010
33	December 2009
54	November 2009
16	October 2009
250	September 2009
50	August 2009
18	July 2009
57	June 2009
23	May 2009
88	April 2009
30	March 2009
1	February 2009
17	January 2009
19	December 2008
47	November 2008
2	October 2008
14	September 2008
16	August 2008
92	July 2008
16	June 2008
50	May 2008
27	April 2008
98	March 2008
81	February 2008
92	January 2008
124	December 2007
55	November 2007
154	October 2007
13	September 2007
16	August 2007
23	July 2007
50	June 2007
9	May 2007
16	April 2007
155	March 2007
11	February 2007
24	January 2007
56	December 2006
89	November 2006
45	October 2006
33	September 2006
8	August 2006
2	July 2006
9	June 2006
26	May 2006
50	April 2006
23	March 2006
199	February 2006
68	January 2006
81	December 2005
17	November 2005
17	October 2005
23	September 2005
123	August 2005
93	July 2005
77	June 2005
74	May 2005
167	April 2005
26	March 2005
20	February 2005
19	January 2005
25	December 2004
89	November 2004
85	October 2004
200	September 2004
218	August 2004
1	July 2004
1	June 2004
2	April 2004
19	March 2004
2	February 2004
61	January 2004
51	December 2003
79	November 2003
6	September 2003
3	August 2003
4	July 2003

Mon	Tue	Wed	Thu	Fri	Sat	Sun

			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30