headers
Christian Tismer | 6 Jun 1999 21:54
Favicon

Stackless Preview (was: Memory leak under Idle?)


Tim Peters wrote:
[see the main list on idle leaks]

> if-pystone-works-ship-it-ly y'rs  - tim

Well, on request of uncle Timmy, I do it.
Although it's very early.

A preview of stackless Python can be found under

ftp://ftp.pns.cc/pub/stackless_990606.zip

Current status:
The main interpreter is completely stackless.
Just for fun, I've set max recursion depth to 30000,
so just try it.

PyStone does of course run. My measures were about
3-5 percent slower than with standard Python. I think
this is quite fair. 

As a side effect, the exec_statement now behaves
better than before, since   
	exec <anything>
without globals and locals should update the current
environment, which worked only for exec "string".

Most of the Run_<thing> functions are stackless as well.
(Continue reading)

Permalink | Reply | 17 comments |
headers
David Ascher | 7 Jun 1999 18:43

ActiveState & fork & Perl

In case you haven't heard about it, ActiveState has recently signed a
contract with Microsoft to do some work on Perl on win32.  

One interesting aspect of this for Python is the specific work being
performed.  From the FAQ on this joint effort, one gets, under "What is
the scope of the work that is being done?":

  fork() 

       This implementation of fork() will clone the running interpreter
  and create a new interpreter with its own thread, but running in the
  same process space. The goal is to achieve functional equivalence to
  fork() on UNIX systems without suffering the performance hit of the
  process creation overhead on Win32 platforms.

     Emulating fork() within a single process needs the ability to run
  multiple interpreters concurrently in separate threads. Perl version
  5.005 has experimental support for this in the form of the PERL_OBJECT
  build option, but it has some shortcomings. PERL_OBJECT needs a C++
  compiler, and currently only works on Windows. ActiveState will be
  working to provide support for revamped support for the PERL_OBJECT
  functionality that will run on every platform that Perl will build on,
  and will no longer require C++ to work. This means that other operating
  systems that lack fork() but have support for threads (such as VMS and
  MacOS) will benefit from this aspect of the work.

Any guesses as to whether we could hijack this work if/when it is released
as Open Source?

--david
(Continue reading)

Permalink | Reply | 46 comments |
headers
Guido van Rossum | 7 Jun 1999 18:49
Picon
Picon

Re: ActiveState & fork & Perl

> In case you haven't heard about it, ActiveState has recently signed a
> contract with Microsoft to do some work on Perl on win32.  

Have I ever heard of it! :-)  David Grove pulled me into one of his
bouts of paranoia.  I think he's calmed down for the moment.

> One interesting aspect of this for Python is the specific work being
> performed.  From the FAQ on this joint effort, one gets, under "What is
> the scope of the work that is being done?":
> 
>   fork() 
> 
>        This implementation of fork() will clone the running interpreter
>   and create a new interpreter with its own thread, but running in the
>   same process space. The goal is to achieve functional equivalence to
>   fork() on UNIX systems without suffering the performance hit of the
>   process creation overhead on Win32 platforms.
> 
>      Emulating fork() within a single process needs the ability to run
>   multiple interpreters concurrently in separate threads. Perl version
>   5.005 has experimental support for this in the form of the PERL_OBJECT
>   build option, but it has some shortcomings. PERL_OBJECT needs a C++
>   compiler, and currently only works on Windows. ActiveState will be
>   working to provide support for revamped support for the PERL_OBJECT
>   functionality that will run on every platform that Perl will build on,
>   and will no longer require C++ to work. This means that other operating
>   systems that lack fork() but have support for threads (such as VMS and
>   MacOS) will benefit from this aspect of the work.
> 
> Any guesses as to whether we could hijack this work if/when it is released
(Continue reading)

Permalink | Reply | 44 comments |
headers
David Ascher | 7 Jun 1999 18:51

Re: ActiveState & fork & Perl

On Mon, 7 Jun 1999, Guido van Rossum wrote:

> When I saw this, my own response was simply "those poor Perl suckers
> are relying too much of fork()."  Am I wrong, and is this also a habit
> of Python programmers?

Well, I find the fork() model to be a very simple one to use, much easier
to manage than threads or full-fledged IPC.  So, while I don't rely on it
in any crucial way, it's quite convenient at times.

--david
Permalink | Reply | 26 comments |
headers
Guido van Rossum | 7 Jun 1999 18:56
Picon
Picon

Re: ActiveState & fork & Perl

> Well, I find the fork() model to be a very simple one to use, much easier
> to manage than threads or full-fledged IPC.  So, while I don't rely on it
> in any crucial way, it's quite convenient at times.

Can you give a typical example where you use it, or is this just a gut
feeling?

It's also dangerous -- e.g. unexpected errors may percolate down the
wrong stack (many mailman bugs had to do with forking), GUI apps
generally won't be cloned, and some extension libraries don't like to
be cloned either (e.g. ILU).

--Guido van Rossum (home page: http://www.python.org/~guido/)
Permalink | Reply | 24 comments |
headers
David Ascher | 7 Jun 1999 19:02

Re: ActiveState & fork & Perl

On Mon, 7 Jun 1999, Guido van Rossum wrote:

> Can you give a typical example where you use it, or is this just a gut
> feeling?

Well, the latest example was that I wanted to spawn a Python process to do
viewing of NumPy arrays with Tk from within the Python interactive shell
(without using a shell wrapper).  It's trivial with a fork(), and
non-trivial with threads.  The solution I had to finalize on was to branch
based on OS and do threads where threads are available and fork()
otherwise.  Likely 2.05 times as many errors as with a single solution =).

> It's also dangerous -- e.g. unexpected errors may percolate down the
> wrong stack (many mailman bugs had to do with forking), GUI apps
> generally won't be cloned, and some extension libraries don't like to
> be cloned either (e.g. ILU).

More dangerous than threads?  Bwaaahaahaa! =).  fork() might be
"deceivingly simple in appearance", I grant you that.  But sometimes
that's good enough.  

It's also possible that fork() without all of its process-handling
relatives isn't useful enough to warrant the effort.

--david
Permalink | Reply | 0 comments |
headers
Barry A. Warsaw | 7 Jun 1999 19:05
X-Face
Picon
Picon

Re: ActiveState & fork & Perl

>>>>> "Guido" == Guido van Rossum <guido <at> cnri.reston.va.us> writes:

    Guido> It's also dangerous -- e.g. unexpected errors may percolate
    Guido> down the wrong stack (many mailman bugs had to do with
    Guido> forking), GUI apps generally won't be cloned, and some
    Guido> extension libraries don't like to be cloned either
    Guido> (e.g. ILU).

Rambling mode on...

Okay, so you can't guarantee that fork will be everywhere you might
want to run an application.  For example, that's one of the main
reasons Mailman hasn't been ported off of Un*x.  But you also can't
guarantee that threads will be everywhere either.  One of the things
I'd (eventually) like to do is to re-architect Mailman so that it uses
a threaded central server instead of the current one-shot process
model.  But there's been debate among the developers because 1)
threads aren't supported everywhere, and 2) thread support isn't
built-in by default anyway.

I wonder if it's feasible or useful to promote threading support in
Python?  Thoughts would include building threads in by default if
possible on the platform, integrating Greg's free threading mods,
etc.  Providing more integrated support for threads might encourage
programmers to reach for that particular tool instead of fork, which
is crude, but pretty damn handy and easy to use.

Rambling mode off...

-Barry
(Continue reading)

Permalink | Reply | 22 comments |
headers
Jim Fulton | 7 Jun 1999 19:07

Re: ActiveState & fork & Perl


David Ascher wrote:
> 
> On Mon, 7 Jun 1999, Guido van Rossum wrote:
> 
> > When I saw this, my own response was simply "those poor Perl suckers
> > are relying too much of fork()."  Am I wrong, and is this also a habit
> > of Python programmers?
> 
> Well, I find the fork() model to be a very simple one to use, much easier
> to manage than threads or full-fledged IPC.  So, while I don't rely on it
> in any crucial way, it's quite convenient at times.

Interesting.  I prefer threads because they eliminate the *need*
for an IPC.  I find locks and the various interesting things you can build
from them to be much easier to deal with and more elegant than IPC.  
I wonder if the perl folks are also going to emulate doing IPC in the 
same process. Hee hee. :)

Jim

--
Jim Fulton           mailto:jim <at> digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
(Continue reading)

Permalink | Reply | 0 comments |
headers
David Ascher | 7 Jun 1999 19:10

Re: ActiveState & fork & Perl

On Mon, 7 Jun 1999, Barry A. Warsaw wrote:

> I wonder if it's feasible or useful to promote threading support in
> Python?  Thoughts would include building threads in by default if
> possible on the platform, 

That seems a good idea to me.  It's a relatively safe thing to enable by
default, no?

> Providing more integrated support for threads might encourage
> programmers to reach for that particular tool instead of fork, which
> is crude, but pretty damn handy and easy to use.

While we're at it, it'd be nice if we could provide a better answer when
someone asks (as "they" often do) "how do I program with threads in
Python" than our usual "the way you'd do it in C".  Threading tutorials
are very hard to come by, I've found (I got the ORA multi-threaded
programming in win32, but it's such a monster I've barely looked at it). I
suggest that we allocate about 10% of TimBot's time to that task.  If
necessary, we can upgrade it to a dual-CPU setup.  With Greg's threading
patches, we could even get it to run on both CPUs efficiently. It could
write about itself.  <unplug>

--david
Permalink | Reply | 21 comments |
headers
Andrew M. Kuchling | 7 Jun 1999 19:20

Re: ActiveState & fork & Perl

David Ascher writes:
>While we're at it, it'd be nice if we could provide a better answer when
>someone asks (as "they" often do) "how do I program with threads in
>Python" than our usual "the way you'd do it in C".  Threading tutorials
>are very hard to come by, I've found (I got the ORA multi-threaded

     Agreed; I'd love to see a HOWTO on thread programming.  I really
liked Andrew Birrell's introduction to threads for Modula-3; see
http://gatekeeper.dec.com/pub/DEC/SRC/research-reports/abstracts/src-rr-035.html
(Postscript and PDF versions available.)  Translating its approach to
Python would be an excellent starting point.

--

-- 
A.M. Kuchling			http://starship.python.net/crew/amk/
    "If you had stayed with us, we could have given you life until death."
    "Don't I get that anyway?"
    -- Stheno and Lyta Hall, in SANDMAN #61: "The Kindly Ones:5"
Permalink | Reply | 2 comments |

Gmane