Phillip J. Eby | 3 Sep 2006 07:04
Gravatar

A quasi-interpretation code generation strategy for PEAK-Rules

I was kicking around a bit with the PEAK-Rules code today (after being away 
from it for almost two months), and came up with an idea for how to manage 
code generation more effectively than was possible with my previous ideas.

My original thought was that code generation would be doing the equivalent 
of creating individual "switch" statements representing dispatch tree 
nodes.  This could potentially mean generating quite a large body of code, 
to represent all possible dispatch locations, something like this:

      switch type(expr1):
          case X:
              switch type(expr2):
                  ...
          case Y:
              switch type(expr2):
                  ...

In contrast, RuleDispatch has a short dispatching loop that walks a tree of 
dictionaries, and has some code to get the next expression to be 
computed.  So instead of the dispatch tree being 100% code, it's 100% data.

What I realized today is that it's possible to make a 50/50 split between 
code and data.  I could create a lazily-expanded dispatch tree using 
dictionaries, just like in RuleDispatch.  But, the choice of what 
expression to look up next, could be embedded in code, something like this:

      node, dispatch_expr = start_node, start_expr
      while not node.isleaf:
          switch dispatch_expr:
              case 1:
(Continue reading)

Phillip J. Eby | 6 Sep 2006 06:31
Gravatar

Self-bootstrapping generic functions

Another design thought re: PEAK-Rules...  One of the trickier bits of 
peak.rules.core right now is that there are some special hacks to support 
the use of generic functions to implement generic functions.  It occured to 
me this evening that there is a more "generic" way to deal with the 
problem, such that the generic function framework can easily be made 
incrementally smarter by adding rules to the functions that implement 
generic functions.

Has your head exploded yet?  No?  Read on.  :)

In PEAK-Rules, an Engine object sets the bytecode of a generic function to 
some generated code.  I've been thinking that when the generated code 
becomes invalid (e.g. due to adding a new rule to the function), the engine 
would replace it with some code to trigger a rebuild operation.  The 
problem is that if the rebuild operation needs to *call* the function 
that's being rebuilt, it won't work right because the data structures may 
be in mid-modification.  In other words, a generic function currently isn't 
re-entrant while it's modifying itself.  :)

So, it finally occurred to me that there's a relatively simple way to fix 
this.  When an Engine sets the bytecode of the function to be the "rebuild" 
code instead of the "execute" code, it should *save* the existing "execute" 
code.  The "rebuild" code would acquire a re-entrant lock (for safe 
threading) and check to see if a rebuild is currently in progress for that 
generic function.  If it is, it should release the lock and invoke the 
*old* "execute" code, rather than trying to rebuild again.

The net result is that any generic function that needs to be called during 
its own rebuild or regeneration (by whatever convoluted path) will simply 
use its last valid definition to do so.  And, that "last valid definition" 
(Continue reading)

Kevin Dangoor | 6 Sep 2006 12:47
Picon
Gravatar

Re: Self-bootstrapping generic functions

On Sep 6, 2006, at 12:31 AM, Phillip J. Eby wrote:

> The net result is that any generic function that needs to be called  
> during its own rebuild or regeneration (by whatever convoluted  
> path) will simply use its last valid definition to do so.  And,  
> that "last valid definition" can include a default implementation  
> written as a non-generic function, which is fairly key to  
> bootstrapping the core itself.  That is, I can write a few  
> functions that contain isinstance() tests or simple delegation to  
> object attributes, and that should be sufficient to get the most  
> fundamental operations working in skeletal form (e.g. just enough  
> to do type-only dispatching).

Neat. You're implementing MVCC for Python code :)

Kevin
Alain Poirier | 11 Sep 2006 16:07
Favicon
Gravatar

Error with RuleDispatch on Python 2.5

Hi,

Running this code with Python 2.5rc1 :

"""
import dispatch

 <at> dispatch.generic()
def render(s):
    pass

class C:
    pass

 <at> render.when("isinstance(s, C)")
def render(s):
    pass
"""

I've go the following traceback (same error with test_dispatch.py or 
test_parsing.py). Any idea how to solve this error ?

"""
Traceback (most recent call last):
  File "d.py", line 10, in <module>
     <at> render.when("isinstance(s, C)") 
File
"/opt/python-2.5rc1/lib/python2.5/site-packages/RuleDispatch-0.5a0.dev_r2100-py2.5-linux-i686.egg/dispatch/functions.py", 
line 690, in when
    return self._decorate(cond) 
(Continue reading)

Phillip J. Eby | 11 Sep 2006 17:27
Gravatar

Re: Error with RuleDispatch on Python 2.5

At 04:07 PM 9/11/2006 +0200, Alain Poirier wrote:
>File

>"/opt/python-2.5rc1/lib/python2.5/site-packages/RuleDispatch-0.5a0.dev_r2100-py2.5-linux-i686.egg/dispatch/strategy.py", 
>
>line 541, in __new__
>     self = int.__new__(cls,id(ob))
>OverflowError: long int too large to convert to int
>"""

In 2.5, id() can be a long, so probably RuleDispatch should change to 
subclass long here instead of int.
Marc-Antoine Parent | 17 Sep 2006 06:31
Picon
Favicon
Gravatar

setuptools and subversion 1.4

Good day!
Hope this is the right place to post this.
You may know that subversion now has a different entries file
format... And that the code in get_svn_revision breaks on checkouts
that have been updated to the new format.
I did a quick hack to make it work; it probably needs improvements
(caching the split(), for example) but as an emergency measure it
might do.
I think that, as a more robust, strategy, you might consider calling
'svn info' and parsing the result. It can be done even if the net is
down.
Cheers,
Marc-Antoine

Index: setuptools/command/egg_info.py
===================================================================
--- setuptools/command/egg_info.py      (revision 51897)
+++ setuptools/command/egg_info.py      (working copy)
 <at>  <at>  -193,14 +193,20  <at>  <at> 
             f = open(os.path.join(base,'.svn','entries'))
             data = f.read()
             f.close()
-            dirurl = urlre.search(data).group(1)    # get repository URL
+            if data[0] == '8':
+                dirurl = data.split('\n')[4]
+            else:
+                dirurl = urlre.search(data).group(1)    # get repository URL
             if base==os.curdir:
                 base_url = dirurl+'/'   # save the root url
             elif not dirurl.startswith(base_url):
(Continue reading)

Phillip J. Eby | 17 Sep 2006 06:46
Gravatar

Re: setuptools and subversion 1.4

At 07:31 AM 9/17/2006 +0300, Marc-Antoine Parent wrote:
>Good day!
>Hope this is the right place to post this.
>You may know that subversion now has a different entries file
>format... And that the code in get_svn_revision breaks on checkouts
>that have been updated to the new format.
>I did a quick hack to make it work; it probably needs improvements
>(caching the split(), for example) but as an emergency measure it
>might do.
>I think that, as a more robust, strategy, you might consider calling
>'svn info' and parsing the result. It can be done even if the net is
>down.

Alas, my understanding is that the output from "svn info" is 
locale-specific; in fact we used to use that, then switched to parsing 
"entries".  By the way, folks doing specialized revision control system 
support should probably have a look at:

http://peak.telecommunity.com/DevCenter/setuptools#adding-support-for-other-revision-control-systems

As it's possible to add plugins for just about anything.  I'll add your fix 
or something like it to 0.6c3 (sigh).

Also, given this format change, it sounds like there might be issues with 
detecting what files are under revision control as well.  Have you tried 
doing a fresh checkout of a project with data files (but not an .egg-info 
directory) and then building an "sdist" of it?  Does it detect and include 
all the files?

If not, then the sdist revision control support needs a fix as well.
(Continue reading)

Marc-Antoine Parent | 17 Sep 2006 17:00
Picon
Favicon
Gravatar

Re: setuptools and subversion 1.4

> >I think that, as a more robust, strategy, you might consider calling
> >'svn info' and parsing the result. It can be done even if the net is
> >down.
>
> Alas, my understanding is that the output from "svn info" is
> locale-specific; in fact we used to use that, then switched to parsing
> "entries".

Which ends up being version-specific unfortunately.
I believe the locale-specificity could simply solved by setting
LC_ALL=C in the environment vars of the subprocess, no?

> By the way, folks doing specialized revision control system
> support should probably have a look at:
>
> http://peak.telecommunity.com/DevCenter/setuptools#adding-support-for-other-revision-control-systems
>

Sounds great,  I was thinking about this for bzr!
Cheers,

Marc-Antoine
Phillip J. Eby | 17 Sep 2006 18:23
Gravatar

Re: setuptools and subversion 1.4

At 06:00 PM 9/17/2006 +0300, Marc-Antoine Parent wrote:
>> >I think that, as a more robust, strategy, you might consider calling
>> >'svn info' and parsing the result. It can be done even if the net is
>> >down.
>>
>>Alas, my understanding is that the output from "svn info" is
>>locale-specific; in fact we used to use that, then switched to parsing
>>"entries".
>
>Which ends up being version-specific unfortunately.
>I believe the locale-specificity could simply solved by setting
>LC_ALL=C in the environment vars of the subprocess, no?

Dunno.  But you didn't mention if the new entries format was working 
properly for sdist and related operations (like finding what data files 
exist).  So I downloaded SVN 1.4, and it turns out that sdist needs fixing 
as well.

It also appears as though I could've used 'svnversion' instead of 'svn 
info', and svnversion doesn't have any locale issues.  On the other hand, 
I'd rather not depend on running a command-line tool during egg_info, due 
to the circumstances under which it can run.  For example, if I tarball up 
a project checked out of Subversion and give it to you to do something 
with, and you don't have subversion installed, it would fail.  Whereas the 
parsing approach will work even if you don't have subversion.

Anyway, I've now updated both the 0.7a1 and 0.6c3 trunks to support the svn 
1.4 working copy format, both for finding files and getting the current 
revision.  Warnings will be issued if they come out with a new working copy 
format whose "entries" file doesn't start with either '8' or '<?xml'.
(Continue reading)

paul | 17 Sep 2006 20:03
Favicon

Another RuleDispatch error with python 2.5

Hi,

Running this code with Python 2.5rc1 (RuleDispatch-0.5a0.dev-r2100,
win32, VS2003.NET, XP) fails. The commented convert_date() function
however works. I tested with other types (decimal.Decimal) and it seems
the error occurs as soon as .when() contains multiple tests and 'or'.

thanks
 Paul

"""
import datetime
import dispatch

def convert(obj):
	raise NotImplementedError()
convert = dispatch.generic()(convert)

#def convert_date(obj):
#	return str(obj)
#convert_date = convert.when('isinstance(obj,
datetime.datetime)')(convert_date)

	
def convert_dateobj(obj):
	return str(obj)
convert_dateobj = convert.when(
        'isinstance(obj, datetime.datetime) or '
       'isinstance(obj, datetime.date)')(convert_dateobj)
		
(Continue reading)


Gmane