headers
Jeremy Hylton | 1 Dec 2005 04:36
Picon
Favicon

Re: Memory management in the AST parser & compiler

On 11/30/05, Neal Norwitz <nnorwitz <at> gmail.com> wrote:
> On 11/30/05, Thomas Lee <krumms <at> gmail.com> wrote:
> >
> > Quick semi-related question: where are the marshal_* functions called?
> > They're all static in Python-ast.c and don't seem to be actually called
> > anywhere. Can we ditch them?
>
> I *think* they are not necessary.  My guess is that they were there
> for marshaling the AST to disk, though I'm not sure why we would want
> to do that.  It could have been there was the idea of how they would
> be marshalled to PyObjects and exported.
>
> Unless you hear otherwise from Jeremy, I would probably remove them.
>
> I can check your patch into the branch so others can get an idea and
> hopefully provide comments.

The intent was to share the AST objects between C and Python by coping
them.  I still think passing copies is better than sharing live
objects between Python and C, although the specific mechanism may be
different if the C objects are PyObjects.

Jeremy
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
(Continue reading)

Permalink | Reply | 2 comments |
headers
Greg Ewing | 1 Dec 2005 05:05
Picon
Picon
Favicon

Re: Memory management in the AST parser & compiler

Jeremy Hylton wrote:

> I still think passing copies is better than sharing live
> objects between Python and C,

Even if the objects are immutable?

--

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg.ewing <at> canterbury.ac.nz	   +--------------------------------------+
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 1 comment |
headers
Jeremy Hylton | 1 Dec 2005 06:02
Picon
Favicon

Re: Memory management in the AST parser & compiler

Sure.  If they're immutable sharing is fine, but you end up making a
copy anyway if you want to make changes, right?

Jeremy

On 11/30/05, Greg Ewing <greg.ewing <at> canterbury.ac.nz> wrote:
> Jeremy Hylton wrote:
>
> > I still think passing copies is better than sharing live
> > objects between Python and C,
>
> Even if the objects are immutable?
>
> --
> Greg Ewing, Computer Science Dept, +--------------------------------------+
> University of Canterbury,          | A citizen of NewZealandCorp, a       |
> Christchurch, New Zealand          | wholly-owned subsidiary of USA Inc.  |
> greg.ewing <at> canterbury.ac.nz        +--------------------------------------+
> _______________________________________________
> Python-Dev mailing list
> Python-Dev <at> python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
(Continue reading)

Permalink | Reply | 0 comments |
headers
Neal Norwitz | 1 Dec 2005 07:40
Picon

Re: ast-objects branch created

On 11/30/05, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
>
> The bigger chunk of necessary changes is in using these, starting
> with ast.c.

I got a few more files to compile.  The following files (all under
Python/) need some loving care and are looking for a kind soul to
adopt them:

  ast.c, compile.c, future.c, symtable.c

Of these, future.c is by far the easiest to get compiling.

n
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 0 comments |
headers
Brett Cannon | 1 Dec 2005 08:40
Picon

Re: ast-objects branch created

On 11/30/05, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> I created
>
> http://svn.python.org/projects/python/branches/ast-objects/
>
> You can convert your repository to that branch with
>
> svn switch svn+ssh://pythondev <at> svn.python.org/python/branches/ast-objects
>

If you would rather do a separate checkout, do

 svn checkout svn+ssh://pythondev <at> svn.python.org/python/branches/ast-objects

If you want a read-only checkout, see the newly updated entry on
checking out projects in the dev FAQ at
http://www.python.org/dev/devfaq.html#how-do-i-get-a-checkout-of-the-repository-read-only-and-read-write
.

-Brett
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 0 comments |
headers
Martin v. Löwis | 1 Dec 2005 09:22
Picon
Gravatar

PyAST_FromNode returning PyTypeObject* ?

Neal,

Why did you suggest that PyAST_FromNode returns PyTypeObject*?
I can't see why type objects are much used in the AST, unless
I'm missing something essential.

Anyway, I started converting ast.c (two functions only), and
noticed that there is a convention to have nested variables
referring to fresh memory (e.g. inside switch statements). I
started changing these to have all variables at the toplevel.

Then, in either success or failure, you have to release all
of them. Unfortunately, sometimes in failure, an additional
function is called, which isn't called in success. So I
added a success: label.

Also, it is somewhat inconvenient that PyList_SET_ITEM
steals references. Currently, I INCREF the objects added
to the list (as the success: label will DECREF them);
alternatively, clearing the pointer to NULL might also
be appropriate. Perhaps we could have a STEAL_ITEM
macro inside ast.c:

#define STEAL_ITEM(list,index,variable) \
do{PyList_SET_ITEM(list,index,variable);variable=NULL;}while(0)

Regards,
Martin
_______________________________________________
Python-Dev mailing list
(Continue reading)

Permalink | Reply | 1 comment |
headers
Nick Coghlan | 1 Dec 2005 09:48
Picon
Gravatar

Re: Memory management in the AST parser & compiler

Martin v. Löwis wrote:
> Nick Coghlan wrote:
>  > The ast C structs are already auto-generated by a Python script 
> (asdl_c.py, to
>  > be precise). The trick is to make that script generate full PyObjects 
> rather
>  > than the simple C structures that it generates now.
> 
> See the ast-object branch.

Thanks Martin.

>  > The second step is to then modify ast.c to use the new structures. A 
> branch
>  > probably wouldn't help much with initial development (this is a 
> "break the
>  > world, check in when stuff compiles again" kind of change, which is 
> hard to
>  > split amongst multiple people), but I think it would be of benefit when
>  > reviewing the change before moving it back to the trunk.
> 
> Well, there would be a clear two-split right now: one could change
> ast.c, and the other compile.c.

I was focusing too much on the AST production end, and managed to forget that 
compile.c and symtable.c are consumers of the AST, so they'll likely care 
about the change as well ;)

Cheers,
Nick.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jeremy Hylton | 1 Dec 2005 14:11
Picon
Favicon

Re: ast-objects branch created

Martin,

I'm not sure what your intent for this work is, but I'd like to create
a parallel arena branch and compare the results.  I'll start work on
that tomorrow.

Jeremy

On 11/30/05, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> I created
>
> http://svn.python.org/projects/python/branches/ast-objects/
>
> You can convert your repository to that branch with
>
> svn switch svn+ssh://pythondev <at> svn.python.org/python/branches/ast-objects
>
> in the toplevel directory. In particular, this features
>
> http://svn.python.org/projects/python/branches/ast-objects/Parser/asdl_c.py
> http://svn.python.org/projects/python/branches/ast-objects/Include/Python-ast.h
> http://svn.python.org/projects/python/branches/ast-objects/Python/Python-ast.c
>
> The status is currently this:
> - asdl_c generates a type hierarchy: "Sum" productions have one type
>    per constructor, inheriting from a type for the sum; plain products
>    only have a type for the product.
> - attributes are in the base type, accessible through o->_base.attr;
>    projections of the product types are accessible directly through
>    member names.
(Continue reading)

Permalink | Reply | 11 comments |
headers
Neal Norwitz | 1 Dec 2005 19:45
Picon

Re: PyAST_FromNode returning PyTypeObject* ?

On 12/1/05, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> Neal,
>
> Why did you suggest that PyAST_FromNode returns PyTypeObject*?
> I can't see why type objects are much used in the AST, unless
> I'm missing something essential.

It was late and I was trying to make progress.  Assume it was a mistake.
It doesn't seem to make much sense based on the name.  I think I was
replacing all mods with PyTypeObject, but since they are probably
lists, PyObject
would be correct.

n
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 0 comments |
headers
Neal Norwitz | 1 Dec 2005 19:46
Picon

Re: ast-objects branch created

On 12/1/05, Jeremy Hylton <jeremy <at> alum.mit.edu> wrote:
> Martin,
>
> I'm not sure what your intent for this work is, but I'd like to create
> a parallel arena branch and compare the results.  I'll start work on
> that tomorrow.

I think this is a good thing.  It will be much easier to compare
implementations if we have some substantial code reflecting each
technique.

n
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 0 comments |

Gmane