Re: How to free unused memory by Python
Anne Archibald <peridot.faceted <at> gmail.com>
2007-09-01 21:25:56 GMT
On 01/09/07, Robert VERGNES <robert.vergnes <at> yahoo.fr> wrote:
> Yes the issue is related with many numpy arrays ( not especially small>2 to
> 7million items in array). And I do have a crash usually while creating a new
> array. (MemoryError). To check this out, I made a small test to understand
> how memory is working in Python and got to see that even with a
> 'mylist=arange()' the memory is not freed back to the OS when 'mylist' is
> deleted...which triggered my original question ' How to free unused memory
> ..'. But as I read from you and other guys, the only way out of this issue -
> ie to avoid crash -probably due to malloc()- then I must free memory before
> and for that I need to process out my recurring calculation process which is
> memory heavy temporarily and must kill my process to release memory after
> work...
> I did notice that if I use huge list -and only a standard python list- ,
> then yes the OS pages normally the memory but when I mix list and numpy
> arrays are involved than I do have a crash when I run near the limit of my
> physical memory - no more paging possible....and a MemoryError crash
> happens. Probably due to the way malloc() request the memory for the numpy
> array...
There are two problems here:
* python or numpy not shinking its virtual memory use
* python crashing during allocation
Why do you think they are related?
The first is a known limitation of most dynamic memory allocation
schemes. Modern malloc()s are generally pretty good about avoiding
memory fragmentation, but the ways in which you can release memory
back to the operating system are often extremely limited. This
problem, that the virtual memory size of a process may remain large
even when most of that memory is unused, arises in raw C programs as
well.
That said, I know that for arrays that large, when they are freed the
glibc malloc() that is used under Linux will definitely release that
memory back to the OS. Are you sure the arrays are actually being
freed? Remember that numpy often creates views of arrays that avoid
copying the data but keep the original array alive:
A = ones(1000000)
B = A[2:4]
del A
Here the memory for A cannot be deallocated because B still points to
it, even though B only needs a few bytes of the many megabytes in A.
To cure this there are various choices, for example:
B = copy(B)
This duplicates the memory and forgets the reference to A (as it is no
longer needed).
As for the crashing, what sort of crash is it? What exception gets
raised (or is it a segfault)?
If it is memory exhaustion, all this business about "not freeing
memory back to the OS" is a red herring. No matter how old your
version of python and how little memory it ever releases back to the
OS, new objects will be allocated from the memory the python process
already has. If your process keeps growing indefinitely, that's not
malloc, that's your code keeping references to more and more data so
that it cannot be free()d. Perhaps look into tools for debugging
memory leaks in python?
Anne
RSS Feed