headers
Ariel Rokem | 1 Jul 2009 01:24
Picon
Favicon
Gravatar

Re: [SciPy-dev] Memory usage of scipy.io.loadmat

Yes - that does seem to free up the memory.

While running this:

In [11]: for i in range(10):
                 a = sio.loadmat('/Users/arokem/Projects/SchizoSpread/Scans/SMR033109_MC/Gray/Original/TSeries/Scan1/tSeries1.mat')
  
causes a memory error, running this:

In [14]: for i in range(10):
                a = sio.loadmat('/Users/arokem/Projects/SchizoSpread/Scans/SMR033109_MC/Gray/Original/TSeries/Scan1/tSeries1.mat')
                gc.collect()

seems like it could go on forever (looking at the memory usage on a memory monitor, it just goes up and down to the same point, without net accumulation).

Thanks a lot!

Ariel









On Tue, Jun 30, 2009 at 3:33 PM, Xavier Saint-Mleux <saintmlx <at> apstat.com> wrote:

>
> So - just invoking an error in the ipython command line has freed up
> 300 MB. Where did they come from? I tried different things - assigning
> other variables doesn't seem to free up this memory. Neither do calls
> to other functions. Except "plot()", which does seem to do the trick
> for some reason. Interestingly, when I run all this in a python
> interactive session (and not ipython), I get a similar memory usage
> initially. Calling a non-existent variable does not free up the
> memory, but other things do. For example, import matplotlib.pylab into
> the namespace did the trick. Does anyone have any idea what is going on?
>

It is probably just the garbage collector being invoked.  If you invoke
it manually, does it always free memory? e.g.:

import gc

gc.collect()



Xavier Saint-Mleux


_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev



--
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
_______________________________________________
SciPy-user mailing list
SciPy-user <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user
Permalink | Reply | 0 comments |
headers
Elias Pampalk | 1 Jul 2009 18:18
Picon

scipy.stats Documentation?

If I don't want to be searching through the source code - where should I be
looking for up-to-date documentation?

Searching via Google for the documentation on scipy.stats Mann Whitney U
test I found:

http://www.scipy.org/doc/api_docs/SciPy.stats.stats.html#mannwhitneyu
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.mannw
hitneyu.html

Both seem outdated compared to what I found here:
http://svn.scipy.org/svn/scipy/trunk/scipy/stats/stats.py
http://projects.scipy.org/scipy/browser/tags/0.7.1/scipy/stats/stats.py

Thank you,
Elias
Permalink | Reply | 11 comments |
headers
David Cournapeau | 1 Jul 2009 18:33
Picon
Gravatar

Re: scipy.stats Documentation?

On Thu, Jul 2, 2009 at 1:18 AM, Elias Pampalk<elias.pampalk <at> gmail.com> wrote:
> If I don't want to be searching through the source code - where should I be
> looking for up-to-date documentation?

The most uptodate documentation is almost always the docstring. If you
use an advanced interpreter such as ipython, you get the docstring
without looking at the sources:

import numpy as np
help np.mean

There is a current effort to bring the official documentation on par
with the docstring, but I don't think it has been done completely for
scipy.stats yet.

David
Permalink | Reply | 9 comments |
headers
Scott David Daniels | 1 Jul 2009 19:21
Picon
Favicon

Re: assign to diagonal values?

Robert Kern wrote:
> On Sat, May 23, 2009 at 16:26, Alan G Isaac <aisaac <at> american.edu> wrote:
>>> On Sat, May 23, 2009 at 16:02, Alan G Isaac wrote:
>>>> On 1/31/2008 1:37 AM Anne Archibald apparently wrote:
>>>>> m[range(n),range(n)]=new_diagonal
>>>> Will that work with range objects (in Python 3)?
>> On 5/23/2009 5:05 PM Robert Kern apparently wrote:
>>> No. The automatic conversion to arrays does not consume iterators (nor
>>> will it when we port to Python 3).
>> Sure, but range objects are not iterators.
>> They are "almost" sequences.
> 
> The answer is still no. Perhaps someone will write special support for
> that type when we do the Python 3 port, but there's nothing in numpy
> that would make it work automatically. For example, xrange() does not
> work as an index with the current numpy.

Well, ranges are more capable than you think in Python 3:
     v = range(25)
     print (v[3], v[0], v[22], v)
prints:
     3 0 22 range(0, 25)

-Scott
Permalink | Reply | 1 comment |
headers
Pauli Virtanen | 1 Jul 2009 19:57
Picon
Picon
Favicon

Re: scipy.stats Documentation?

On 2009-07-01, Elias Pampalk <elias.pampalk <at> gmail.com> wrote:
> If I don't want to be searching through the source code - where should I be
> looking for up-to-date documentation?
>
> Searching via Google for the documentation on scipy.stats Mann Whitney U
> test I found:
>
> http://www.scipy.org/doc/api_docs/SciPy.stats.stats.html#mannwhitneyu
> http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.mannwhitneyu.html

The second one is up-to-date but it's for 
scipy.stats.mstats.mannwhitneyu, not scipy.stats.mannwhitneyu.

Apparently, the main stats.mannwhitneyu function was not included 
in the documentation. Fixed, should appear tomorrow.

--

-- 
Pauli Virtanen
Permalink | Reply | 0 comments |
headers
Robert Kern | 1 Jul 2009 20:38
Picon
Gravatar

Re: assign to diagonal values?

On Wed, Jul 1, 2009 at 12:21, Scott David Daniels<Scott.Daniels <at> acm.org> wrote:
> Robert Kern wrote:
>> On Sat, May 23, 2009 at 16:26, Alan G Isaac <aisaac <at> american.edu> wrote:
>>>> On Sat, May 23, 2009 at 16:02, Alan G Isaac wrote:
>>>>> On 1/31/2008 1:37 AM Anne Archibald apparently wrote:
>>>>>> m[range(n),range(n)]=new_diagonal
>>>>> Will that work with range objects (in Python 3)?
>>> On 5/23/2009 5:05 PM Robert Kern apparently wrote:
>>>> No. The automatic conversion to arrays does not consume iterators (nor
>>>> will it when we port to Python 3).
>>> Sure, but range objects are not iterators.
>>> They are "almost" sequences.
>>
>> The answer is still no. Perhaps someone will write special support for
>> that type when we do the Python 3 port, but there's nothing in numpy
>> that would make it work automatically. For example, xrange() does not
>> work as an index with the current numpy.
>
> Well, ranges are more capable than you think in Python 3:
>     v = range(25)
>     print (v[3], v[0], v[22], v)
> prints:
>     3 0 22 range(0, 25)

No, they are exactly as capable as I think, i.e. as capable as
xrange() is in Python 2:

  In [10]: v = xrange(25)

  In [11]: print v[3], v[0], v[22], v
  3 0 22 xrange(25)

Quite simply, numpy does not support arbitrary sequences and
sequence-like objects as indices. If the eventual numpy port to Python
3 supports range() objects as indices, it will be because someone will
have written special code for it.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
SciPy-user mailing list
SciPy-user <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user
Permalink | Reply | 0 comments |
headers
Fernando Perez | 1 Jul 2009 21:38
Picon
Gravatar

Re: assign to diagonal values?

On Wed, Jan 30, 2008 at 10:28 PM, Anand
Patil<anand.prabhakar.patil <at> gmail.com> wrote:
> Yeah, that's awesome. I have so many for-loops littering my code for
> setting diagonals. Here's an nd-version:
>
> def setdiag(a, d):
>    assert(all([s == len(d) for s in a.shape]))
>    stride = 1+sum(cumprod(a.shape[:-1]))
>    a.flat[::stride] = d
>

If someone feels like reviewing this ticket:

http://projects.scipy.org/numpy/attachment/ticket/1132/numpy-index-funcs.diff

it has the above and a few more utilities, with docs and tests.

Cheers,

f
Permalink | Reply | 2 comments |
headers
Lorenzo Isella | 2 Jul 2009 15:25
Picon

Creating a Matrix from a Sum

Dear All,
I need some help to efficiently write an array manipulation. I suspect 
this could be a one-liner.
Assume that you have a histogram of observations that you store in a 
vector x.
Let us say that its i-th entry, x_i, corresponds to the number of 
observations in the i-th channel, for i=1,2...N. (or 0,1...N-1, please 
let me know if there is any potential 0/1 pitfall in the following).
Now, for any two channels i and j, I want to calculate the probability 
of having an observation  in any other channel k, where  k>=max(i,j).
That is to say

P(i,j)=sum_{k=max(i,j)}^N x_k /C,
where C=sum_{i=1}^N x_i is just a normalization factor.

Does anyone know a good way of writing this in a code to get the P(i,j) 
matrix?
Many thanks

Lorenzo
Permalink | Reply | 6 comments |
headers
Dharhas Pothina | 2 Jul 2009 17:58
Picon
Picon
Favicon

Add a column to masked array.

Hi,

I have a masked array 'data' where data.shape = (1805,4) and another masked array 'a' where a.shape =
(1805,) 

I would like to generate a new masked array with the 'a' as the last column , i.e. the final shape should be
(1805,5). 

I've tried np.hstack which gave an error 'arrays must have same number of dimensions' and ma.hstack just
combines both masked arrays into a list.

how would I go about doing what I need?

thanks

- dharhas
Permalink | Reply | 2 comments |
headers
Pierre GM | 2 Jul 2009 18:57
Picon

Re: Add a column to masked array.


On Jul 2, 2009, at 11:58 AM, Dharhas Pothina wrote:

> Hi,
>
> I have a masked array 'data' where data.shape = (1805,4) and another  
> masked array 'a' where a.shape = (1805,)
>
> I would like to generate a new masked array with the 'a' as the last  
> column , i.e. the final shape should be (1805,5).

Try ma.column_stack

 >>> x=ma.masked_all((10,3))
 >>> y=np.random.rand(10)
 >>> ma.column_stack((x,y)).shape
(10,4)

> I've tried np.hstack which gave an error 'arrays must have same  
> number of dimensions'

The corresponding standard numpy functions don't know how to process  
the mask and tend to fail miserably.

> and ma.hstack just combines both masked arrays into a list.

I've never been able to remember what vstack, hstack and stack do  
respectively, so I use error/trial and the online help...
Permalink | Reply | 1 comment |

Gmane