headers
Darren Dale | 1 Sep 2004 22:02
Picon
Favicon

sum of a masked array

Hi, I'm new to the list. I am having some difficulty finding the sum of 
a masked array. I'm using numarray 1.0 installed on winXP with Python 2.3.4.

 from numarray.ma import *
Rx = ones((2500,2500))
N = make_mask_none((2500,2500))
Rx = array(Rx,mask=N)
print average(Rx) ## works
print sum(Rx) ## gives an error message 16 lines long, the most recent 
being a type error.

Is there something obvious I am doing wrong here?

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
Permalink | Reply | 0 comments |
headers
smpitts | 1 Sep 2004 22:21
Picon
Favicon

Re: sum of a masked array

Darren,
I tried your code on my system: Python 2.2 and numarray 1.0. The type error looks like a bug in the array
display code.

>>> Rx = ones((2500,2500))
>>> N = make_mask_none((2500,2500))
>>> Rx = array(Rx,mask=N)
>>> print average(Rx) ## works
[ 1.  1.  1. ...,  1.  1.  1.]
>>> s = sum(Rx)
>>> print s

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.2/site-packages/numarray/ma/MA.py", line 742, in __str__
    return str(filled(self, f))
  File "/usr/lib/python2.2/site-packages/numarray/generic.py", line 499, in __str__
    return arrayprint.array2string(self, separator=" ", style=str)
  File "/usr/lib/python2.2/site-packages/numarray/arrayprint.py", line 188, in array2string
    separator, prefix)
  File "/usr/lib/python2.2/site-packages/numarray/arrayprint.py", line 137, in _array2string
    data = _leading_trailing(a)
  File "/usr/lib/python2.2/site-packages/numarray/arrayprint.py", line 105, in _leading_trailing
    b = _gen.concatenate((a[:_summaryEdgeItems],
  File "/usr/lib/python2.2/site-packages/numarray/generic.py", line 1028, in concatenate
    return _concat(arrs)
  File "/usr/lib/python2.2/site-packages/numarray/generic.py", line 1012, in _concat
    dest  = arrs[0].__class__(shape=destShape, type=convType)
TypeError: __init__() got an unexpected keyword argument 'type'
(Continue reading)

Permalink | Reply | 1 comment |
headers
Darren Dale | 1 Sep 2004 22:36
Picon
Favicon

Re: sum of a masked array

smpitts <at> ou.edu wrote:

>Darren,
>I tried your code on my system: Python 2.2 and numarray 1.0. The type error looks like a bug in the array
display code.
>
>  
>
I think you are right:

from numarray.ma import *

Rx = ones((2500,2500))
N = make_mask_none((2500,2500))
Rx = array(Rx,mask=N)
s = sum(sum(Rx))
print s ## this works, s is of type int, rather than maskedarray

(Stephen, I'm guessing you know how to pipe the error messages to a 
file. If you know how to do this with DOS/windows, would you write me 
privately and explain? Thanks for writing back so quickly, by the way.)

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
Permalink | Reply | 0 comments |
headers
Darren Dale | 1 Sep 2004 23:51
Picon
Favicon

efficient summation

I am trying to effieciently sum over a subset of the elements of a 
matrix. In Matlab, this could be done like:
a=[1,2,3,4,5,6,7,8,9,10]
b = [1,0,0,0,0,0,0,0,0,1]
res=sum(a(b)) %this sums the elements of a which have corresponding 
elements in b that are true

Is there anything similar in numarray (or numeric)? I thought masked 
arrays looked promising, but I find that masking 90% of the elements 
results in marginal speedups (~5%, instead of 90%) over the unmasked array.

Thanks!
Darren

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
Permalink | Reply | 3 comments |
headers
Stephen Walton | 2 Sep 2004 01:44
Favicon

Re: efficient summation

On Wed, 2004-09-01 at 14:51, Darren Dale wrote:
> I am trying to effieciently sum over a subset of the elements of a 
> matrix. In Matlab, this could be done like:
> a=[1,2,3,4,5,6,7,8,9,10]
> b = [1,0,0,0,0,0,0,0,0,1]
> res=sum(a(b))

This needs to be sum(a(find(b)).

> Is there anything similar in numarray (or numeric)? I thought masked 
> arrays looked promising, but I find that masking 90% of the elements 
> results in marginal speedups (~5%, instead of 90%) over the unmasked
array.

I don't think that's bad, and in fact it is substantially better than
MATLAB.  Consider the following clip from MATLAB Version 7:

>> a=randn(10000000,1);
>> t=cputime;sum(a);e=cputime()-t

e =

    0.1300

>> f=rand(10000000,1)<0.1;
>> t=cputime;sum(a(find(f)));e=cputime()-t

e =

    0.2200
(Continue reading)

Permalink | Reply | 1 comment |
headers
Paulo J. S. Silva | 2 Sep 2004 04:18
Picon
Picon
Favicon

Re: efficient summation

Em Qua, 2004-09-01 às 18:51, Darren Dale escreveu:
> I am trying to effieciently sum over a subset of the elements of a 
> matrix. In Matlab, this could be done like:
> a=[1,2,3,4,5,6,7,8,9,10]
> b = [1,0,0,0,0,0,0,0,0,1]
> res=sum(a(b)) %this sums the elements of a which have corresponding 
> elements in b that are true

If the mask is of boolean type (not integer) you can use it just like in
MATLAB:

>>> from numarray import *
>>> import numarray.random_array as ra
>>> a = ra.random(1000000)
>>> sum(a)
500184.16988508566
>>> b = ra.random(1000000) < 0.1
>>> sum(a[b])
50331.373006955822

This should work for numarray only.

Paulo

--

-- 
Paulo José da Silva e Silva 
Professor Assistente do Dep. de Ciência da Computação
(Assistant Professor of the Computer Science Dept.)
Universidade de São Paulo - Brazil
(Continue reading)

Permalink | Reply | 0 comments |
headers
Darren Dale | 2 Sep 2004 06:34
Picon
Favicon

Re: efficient summation

Stephen Walton wrote:

>In addition, I doubt you can measure CPU time for only a 10 element
>array.  I had to use 1e7 elements in MATLAB on a 2.26MHz P4 just to get
>the CPU time large enough to measure reasonably accurately.  Also recall
>that it is a known characteristic of numarray that it is slow on small
>arrays in general.
>
>  
>
Sorry, I was giving the 10 element example for clarity. I am actually 
using arrays with over 6e6 elements.

 I just discovered compress, it works wonders in my situation.
The following script runs in 1 second on my 2GHz P4, winXP. The same 
calculation using a masked array took 18 seconds:

from numarray import *
from time import clock

clock()
Rx = ones((2500,2500))*12.5
N = zeros((2500,2500),typecode=Bool)
N[:250,:]=1
trans = compress(N,Rx)
temp = exp(2j*pi*(trans+trans))*exp(2j*pi*(trans))
s = sum(temp.real)
print s, clock()

-------------------------------------------------------
(Continue reading)

Permalink | Reply | 0 comments |
headers
Curzio Basso | 2 Sep 2004 11:15
Picon
Picon
Favicon

Re: extracting a random subset of a vector

Chris Barker wrote:

>> and from the hotshot output is looks like it's the indexing, not the 
>> permutation, which takes time.
> 
> not from my tests:

a question about the method: isn't a bit risky to use the clock() for 
timing the performance? The usual argument is that CPU allocates time 
for different processes, and the allocation could vary. That's why I 
used the profiler.

Anyway, I performed another test with the profiler, on the example you 
also used, and I also obtained that most of the time is spent in 
permutation() (2.264 over 2.273 secs).

regards

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
Permalink | Reply | 7 comments |
headers
Karthikesh Raju | 3 Sep 2004 15:33
Picon
Picon

Reshaping along an axis


Hi all,

Suppose i have a tuple or a 1D array as

a = (1,2,3,4,5,6,7,8,9)

presently reshape(a,(3,3)) gives me

1 2 3
4 5 6
7 8 9

i.e reshaping is done column wise. How do one specifiy reshape to work row
wise as: reshape(a,(3,3)) =
1 4 7
2 5 8
3 6 9

With warm regards

karthik

-----------------------------------------------------------------------
Karthikesh Raju,		    email: karthik <at> james.hut.fi
                                           karthikesh.raju <at> gmail.com
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
(Continue reading)

Permalink | Reply | 1 comment |
headers
Tim Hochberg | 3 Sep 2004 15:52
Picon

Re: Reshaping along an axis

Karthikesh Raju wrote:

>Hi all,
>
>Suppose i have a tuple or a 1D array as
>
>a = (1,2,3,4,5,6,7,8,9)
>
>presently reshape(a,(3,3)) gives me
>
>1 2 3
>4 5 6
>7 8 9
>
>i.e reshaping is done column wise. How do one specifiy reshape to work row
>wise as: reshape(a,(3,3)) =
>1 4 7
>2 5 8
>3 6 9
>  
>

Use transpose pluse reshape:

 >>> a = (1,2,3,4,5,6,7,8,9)
 >>> print transpose(reshape(a, (3,3)))
[[1 4 7]
 [2 5 8]
 [3 6 9]]
(Continue reading)

Permalink | Reply | 0 comments |

Gmane