[PATCH] Cyclic reference preventing garbage collection in 1.9

Hi all

I have started to use SimPy 1.9 to simulate my PhD work, an economic
resource allocation system. So far it has been straight forward, and a
delight to use - my thanks :)

However, I need to simulate some large systems, networks or traders
with 50,000 nodes, and 50,000-100,000 new Processes (messages) being
created every timestep, each Process running for about 6 timesteps on
average. So about 450,000 currently active Processes at any given
time.

This is obviously fairly demanding in terms of memory, but I started
hitting problems on much smaller runs (5000 nodes, 500-1000
Process/s). A bit or manual profiling of my code suggested that none
of my Process objects were ever being deleted and garbage collected
properly

I had a look at Simulation.py, and noticed lines 588 and 594 created a
circular reference to the Process objects, and thus prevent any
Process object from being garbage collected;

588: what._rec=[at,self.sortpr,what,False]

594: what._rec=[at,-self.sortpr,what,False]

A brief look over the code suggests the only reason that a Process
needs a reference to its own queue record is for the _unpost method to
correctly signal a Process to be cancelled. The quickfix patch below
fixed the issue for me in terms of garbage collection, and my

Re: [PATCH] Cyclic reference preventing garbagecollection in 1.9

Hi Simon,
Thank you for this input. You caught a major design flaw/oversight here. 

Let me go to work and see how I can put SimPy right, i.e. cure the cancel
problem.

Klaus Müller

> -----Original Message-----
> From: simpy-users-bounces <at> lists.sourceforge.net 
> [mailto:simpy-users-bounces <at> lists.sourceforge.net] On Behalf 
> Of Wavy Davy
> Sent: Monday, February 18, 2008 9:43 AM
> To: simpy-users <at> lists.sourceforge.net
> Subject: [Simpy-users] [PATCH] Cyclic reference preventing 
> garbagecollection in 1.9
> 
> Hi all
> 
> I have started to use SimPy 1.9 to simulate my PhD work, an 
> economic resource allocation system. So far it has been 
> straight forward, and a delight to use - my thanks :)
> 
> However, I need to simulate some large systems, networks or 
> traders with 50,000 nodes, and 50,000-100,000 new Processes 
> (messages) being created every timestep, each Process running 
> for about 6 timesteps on average. So about 450,000 currently 
> active Processes at any given time.
> 
> This is obviously fairly demanding in terms of memory, but I 

Re: [PATCH] Cyclic reference preventing garbagecollection in 1.9

 Simon,
Yes, my run-time optimization for 1.9 "pessimized" the space/GC situation
:=((.

Your weakref idea sounds good. I will give it a try.

Best,

Klaus Müller

> -----Original Message-----
> From: Wavy Davy [mailto:bloodearnest <at> gmail.com] 
> Sent: Monday, February 18, 2008 10:55 AM
> To: kgmuller
> Subject: Re: [Simpy-users] [PATCH] Cyclic reference 
> preventing garbagecollection in 1.9
> 
> On 18/02/2008, kgmuller <kgmuller <at> xs4all.nl> wrote:
> > Hi Simon,
> >  Thank you for this input. You caught a major design 
> flaw/oversight here.
> 
> Good - I assume this was introduced in the 1.9 optimisations?
> 
> >  Let me go to work and see how I can put SimPy right, i.e. cure the 
> > cancel  problem.
> 
> On the bus on the way to (non-PhD) work this morning, I 
> realised that using weakref is probably a better solution. It 
> should allow the convenience of a Process having a reference 

FW: [PATCH] Cyclic reference preventinggarbagecollection in 1.9

 Simon,
How do you actually determine whether Process objects get garbage-collected?

If the problem is the circular ref between a Process object and _rec, one
possible solution that comes to mind is to get rid of the reference to _rec
when an event notice has been processed, like so (code from SimPy 1.9):

    def _nextev(self):
        """Retrieve next event from event list"""
        global _t, _stop
        noActiveNotice=True
        ## Find next event notice which is not marked cancelled
        while noActiveNotice:
            if self.timestamps:
                 ## ignore priority value         
                (_tnotice, p,nextEvent,cancelled) =
hq.heappop(self.timestamps)
                noActiveNotice=cancelled
            else:
                raise Simerror("No more events at time %s" % _t)
#####  Does the following change cure circular reference problem?
        nextEvent._rec=None
##### end change
        _t=_tnotice
        if _t > _endtime:
            _t = _endtime
            _stop = True
            return (None,)
        try:
            resultTuple = nextEvent._nextpoint.next()

FW: FW: [PATCH] Cyclic referencepreventinggarbagecollection in 1.9

All:
The simple patch proposed below seems to cure the cyclic reference bug in
SimPy 1.9 (according to Simon Davy's tests). I will perform a few tests
myself. If they confirm the cure, I will publish a bug-fix release (1.9.1)
to SimPy 1.9 soonest.

Klaus Müller   

-----Original Message-----
From: simpy-users-bounces <at> lists.sourceforge.net
[mailto:simpy-users-bounces <at> lists.sourceforge.net] On Behalf Of kgmuller
Sent: Monday, February 18, 2008 3:24 PM
To: 'Wavy Davy'; 'Simpy-Developer List'; 'Simpy-Users List'
Subject: [Simpy-users] FW: [PATCH] Cyclic
referencepreventinggarbagecollection in 1.9

 Simon,
How do you actually determine whether Process objects get garbage-collected?

If the problem is the circular ref between a Process object and _rec, one
possible solution that comes to mind is to get rid of the reference to _rec
when an event notice has been processed, like so (code from SimPy 1.9):

    def _nextev(self):
        """Retrieve next event from event list"""
        global _t, _stop
        noActiveNotice=True
        ## Find next event notice which is not marked cancelled
        while noActiveNotice:
            if self.timestamps:

Memory leak in SimPy 1.9

All:
I have to inform you that two users have found a memory leak in SimPy 1.9.
It is the result of my attempt to improve the run time performance of SimPy
--  the leak does not exist in SimPy 1.8. The effect of this leak is that
memory used by a running SimPy program keeps increasing. The reason for this
leak is the apparent inability of the Python garbage collector to collect
Process (generator) instances which have a circular reference. 

I am analyzing this problem situation and will report the availability of a
fix as soon as possible. Soon after, a bug-fix release will be published.

If you use long-running SimPy programs where this memory leak causes
problems, go back to using SimPy 1.8, please.

I am sorry that my optimization attempt in 1.9 made performance worse for
some of you.

Klaus Müller

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Memory leak in SimPy 1.9

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Simpy-users mailing list
Simpy-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simpy-users

Repeating a process

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Simpy-users mailing list
Simpy-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simpy-users

Repeating a process

You could create another process (called disk for example). This process can
then call your read and write processes multiple times. I'm guessing with a
real disk you can't have two reads or two writes or a read and a write at
the same time. To solve this you could wait until your read or write methods
have returned before starting another.

Tim

----------------------------------------------------------------------------
---------------------------------
From: simpy-users-bounces <at> lists.sourceforge.net
[mailto:simpy-users-bounces <at> lists.sourceforge.net] On Behalf Of John
Rominsky
Sent: 25 February 2008 14:23
To: simpy-users <at> lists.sourceforge.net
Subject: [Simpy-users] Repeating a process

I am just starting out in SimPy, but I have found it to be an excellent
tool.  Currently I am trying to set up classes that I can use later on down
the road to plug in for modeling network activity and hard disk usage.  I
tried to set up a process that contained hard drive resources and the
methods to read and write to those resources, but i cant seem to run the
process more than once if i have say two writes.  Here is a slimmed down
version of what I am working on.  Any thoughts on how to or if I can repeat
a process in the same simulation period?

John

#!/usr/bin/env python

from SimPy.Simulation import *

class hard_disk(Process):
        def __init__(self):
                Process.__init__(self)
                self.files = Store(name='disk file table',unitName='files')
                self.storage = Level(name='disk space',unitName='bits')
        def write(self,file,name):
                yield put,self,self.files,[file]
                print self.files.theBuffer
        def read(self,file,name):
                yield get,self,self.files,1
                print self.files.theBuffer

initialize()

disk1 = hard_disk()
activate(disk1,disk1.write('pony','one'))
activate(disk1,disk1.write('dog','two'))

simulate()

No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.21.1/1297 - Release Date: 25/02/2008
09:22

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Repeating a process

I have activated the same process multiple times in a simulation
period so that shouldn't be a problem.
I am also new in SimPy so what I say might not be exactly true. But I 
think your problem is that your yield is not blocking, so after you
put the new file the process is terminated so when the next event comes,
which is scheduled for the same time, there is no process.

Maybe you should consider adding an extra resource (e.g. hard_disk_head) 
in your hard_disk object to simulate the fact that you can't have two 
simulation operations (read/write) and then have a new
process (Reader/Writer) that will have to acquire the hard_disk_head
before continuing and then initialize multiple instances of this process.

--niky

Tim Fish wrote:
> You could create another process (called disk for example). This process can
> then call your read and write processes multiple times. I'm guessing with a
> real disk you can't have two reads or two writes or a read and a write at
> the same time. To solve this you could wait until your read or write methods
> have returned before starting another.
> 
> Tim
> 
> ----------------------------------------------------------------------------
> ---------------------------------
> From: simpy-users-bounces <at> lists.sourceforge.net
> [mailto:simpy-users-bounces <at> lists.sourceforge.net] On Behalf Of John
> Rominsky
> Sent: 25 February 2008 14:23
> To: simpy-users <at> lists.sourceforge.net
> Subject: [Simpy-users] Repeating a process
> 
> I am just starting out in SimPy, but I have found it to be an excellent
> tool.  Currently I am trying to set up classes that I can use later on down
> the road to plug in for modeling network activity and hard disk usage.  I
> tried to set up a process that contained hard drive resources and the
> methods to read and write to those resources, but i cant seem to run the
> process more than once if i have say two writes.  Here is a slimmed down
> version of what I am working on.  Any thoughts on how to or if I can repeat
> a process in the same simulation period?
> 
> John
> 
> #!/usr/bin/env python
> 
> from SimPy.Simulation import *
> 
> class hard_disk(Process):
>         def __init__(self):
>                 Process.__init__(self)
>                 self.files = Store(name='disk file table',unitName='files')
>                 self.storage = Level(name='disk space',unitName='bits')
>         def write(self,file,name):
>                 yield put,self,self.files,[file]
>                 print self.files.theBuffer
>         def read(self,file,name):
>                 yield get,self,self.files,1
>                 print self.files.theBuffer
> 
> initialize()
> 
> disk1 = hard_disk()
> activate(disk1,disk1.write('pony','one'))
> activate(disk1,disk1.write('dog','two'))
> 
> simulate()
> 
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.516 / Virus Database: 269.21.1/1297 - Release Date: 25/02/2008
> 09:22
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Simpy-users mailing list
> Simpy-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simpy-users

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/