Discussion of Numerical Python

Re: searchsorted bug

> Try out latest SVN.  It should have this problem fixed.
Thanks for this. I've realized that for my case, using object arrays
is probably best. I still think that long term it would be good to
allow comparison functions to take different types, so that one could
compare say integer arrays with floating point arrays without doing an
upcast.

headers

Lars Friedrich | 1 Feb 10:57

histogramdd memory needs

Hello,

I use numpy.histogramdd to compute three dimensional histograms with a 
total number of bins in the order of 1e7. It is clear to me, that such a 
histogram will take a lot of memory. For a dtype=N.float64, it will take 
roughly 80 megabytes. However, I have the feeling that during the 
histogram calculation, much more memory is needed. For example, when I 
have data.shape = (8e6, 3) and do a numpy.histogramdd(d, 280), I expect 
a histogram size of (280**3)*8 = 176 megabytes, but during histogram 
calculation the memory need of pythonw.exe in the Windows Task Manager 
increases up to 687 megabytes over the level before histogram 
calculation. When the calculation is done, the mem usage drops down to 
the expected value. I assume this is due to the internal way, 
numpy.histogramdd works. However, when I need to calculate even bigger 
histograms, I cannot do it this way. So I have the following questions:

1) How can I tell histogramdd to use another dtype than float64? My bins 
will be very little populated so an int16 should be sufficient. Without 
normalization, a Integer dtype makes more sense to me.

2) Is there a way to use another algorithm (at the cost of performance) 
that uses less memory during calculation so that I can generate bigger 
histograms?

My numpy version is '1.0.4.dev3937'

Thanks,
Lars

--

--

(Continue reading)

headers

Andrea Gavana | 1 Feb 12:28

[F2PY]: Allocatable Arrays

Hi All,

    I sent a couple of messages to f2py mailing list, but it seems
like my problem has no simple solution so I thought to ask for some
suggestions here.

Basically, I read some huge unformatted binary files which contain
time-step data from a reservoir simulation. I don't know the
dimensions (i.e., lengths) of the vectors I am going to read, and I
find out this information only when I start reading the file. So, I
thought it would be nice to do something like:

1) Declare outputVector as allocatable;
2) Start reading the file;
3) Find the outputVector dimension and allocate it;
4) Read the data in the outputVector;
5) Return this outputVector.

It works when I compile it and build it in Fortran as an executable
(defining a "main" program in my f90 module), but it bombs when I try
to use it from Python with the error:

C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py
Traceback (most recent call last):
 File "C:\Documents and
Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in <module>
   inteHead, propertyNames, propertyTypes, propertyNumbers =
ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC")
ValueError: failed to create intent(cache|hide)|optional array-- must
have defined dimensions but got (-1,)

(Continue reading)

headers

Lisandro Dalcin | 1 Feb 14:59

Re: [F2PY]: Allocatable Arrays

Sorry if I'm making noise, my knowledge of fortran is really little,
but in your routine AllocateDummy your are fist allocating and next
deallocating the arrays. Are you sure you can then access the contents
of your arrays after deallocating them?

How much complicated is your binary format? For simple formats, you
can just use numpy to read binary data, I use this sometimes, but
again, for simple formats.

On 2/1/08, Andrea Gavana <andrea.gavana <at> gmail.com> wrote:
> Hi All,
>
>     I sent a couple of messages to f2py mailing list, but it seems
> like my problem has no simple solution so I thought to ask for some
> suggestions here.
>
> Basically, I read some huge unformatted binary files which contain
> time-step data from a reservoir simulation. I don't know the
> dimensions (i.e., lengths) of the vectors I am going to read, and I
> find out this information only when I start reading the file. So, I
> thought it would be nice to do something like:
>
> 1) Declare outputVector as allocatable;
> 2) Start reading the file;
> 3) Find the outputVector dimension and allocate it;
> 4) Read the data in the outputVector;
> 5) Return this outputVector.
>
> It works when I compile it and build it in Fortran as an executable
> (defining a "main" program in my f90 module), but it bombs when I try

(Continue reading)

headers

Andrea Gavana | 1 Feb 15:18

Re: [F2PY]: Allocatable Arrays

Hi Lisandro,

On Feb 1, 2008 1:59 PM, Lisandro Dalcin wrote:
> Sorry if I'm making noise, my knowledge of fortran is really little,
> but in your routine AllocateDummy your are fist allocating and next
> deallocating the arrays. Are you sure you can then access the contents
> of your arrays after deallocating them?

Thank you for your answer.

Unfortunately it seems that it doesn't matter whether I deallocate
them or not, I still get the compilation warning and I can't access
those variable in any case. It seems like f2py (or python or whatever)
does not like having more than 1 allocatable array inside a MODULE
declaration.

> How much complicated is your binary format?

*Very* complex. The fact is, I already know how to read those files in
Fortran, is the linking with Python via f2py that is driving me mad. I
can't believe no one has used before allocatable arrays as outputs
(whether from a subroutine or from a module).

> On 2/1/08, Andrea Gavana <andrea.gavana <at> gmail.com> wrote:
> > Hi All,
> >
> >     I sent a couple of messages to f2py mailing list, but it seems
> > like my problem has no simple solution so I thought to ask for some
> > suggestions here.
> >

(Continue reading)

headers

Pearu Peterson | 1 Feb 15:45

Re: [F2PY]: Allocatable Arrays

On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote:
> Hi All,
>
>     I sent a couple of messages to f2py mailing list, but it seems
> like my problem has no simple solution so I thought to ask for some
> suggestions here.

Sorry, I haven't been around there long time.

> Basically, I read some huge unformatted binary files which contain
> time-step data from a reservoir simulation. I don't know the
> dimensions (i.e., lengths) of the vectors I am going to read, and I
> find out this information only when I start reading the file. So, I
> thought it would be nice to do something like:
>
> 1) Declare outputVector as allocatable;
> 2) Start reading the file;
> 3) Find the outputVector dimension and allocate it;
> 4) Read the data in the outputVector;

looks ok.

> 5) Return this outputVector.

What do you mean by "return"? You cannot return allocatable arrays
as far as comes to using f2py for generating wrappers. However,
you can access allocatable array outputVector if it is module data,
as you do below.

> It works when I compile it and build it in Fortran as an executable

(Continue reading)

headers

Pearu Peterson | 1 Feb 15:49

Re: [F2PY]: Allocatable Arrays

On Fri, February 1, 2008 4:18 pm, Andrea Gavana wrote:
> Hi Lisandro,
>
> On Feb 1, 2008 1:59 PM, Lisandro Dalcin wrote:
>> Sorry if I'm making noise, my knowledge of fortran is really little,
>> but in your routine AllocateDummy your are fist allocating and next
>> deallocating the arrays. Are you sure you can then access the contents
>> of your arrays after deallocating them?
>
> Thank you for your answer.
>
> Unfortunately it seems that it doesn't matter whether I deallocate
> them or not, I still get the compilation warning and I can't access
> those variable in any case.

You cannot access those becase they are deallocated. Try to
disable deallocate statements in your fortran code.

> It seems like f2py (or python or whatever)
> does not like having more than 1 allocatable array inside a MODULE
> declaration.

This is not true.

>> How much complicated is your binary format?
>
> *Very* complex. The fact is, I already know how to read those files in
> Fortran, is the linking with Python via f2py that is driving me mad. I
> can't believe no one has used before allocatable arrays as outputs
> (whether from a subroutine or from a module).

(Continue reading)

headers

David Huard | 1 Feb 16:08

Re: histogramdd memory needs

Hi Lars,

[...]

2008/2/1, Lars Friedrich <lfriedri <at> imtek.de>:

1) How can I tell histogramdd to use another dtype than float64? My bins
will be very little populated so an int16 should be sufficient. Without
normalization, a Integer dtype makes more sense to me.

There is no way you'll be able to ask that without tweaking the histogramdd function yourself. The relevant bit of code is the instantiation of hist :

hist = zeros(nbin.prod(), float)

2) Is there a way to use another algorithm (at the cost of performance)
that uses less memory during calculation so that I can generate bigger
histograms?

You could work through your array block by block. Simply fix the range and generate an histogram for each slice of 100k data and sum them up at the end.

The current histogram and histogramdd implementation has the advantage of being general, that is you can work with uniform or non-uniform bins, but it is not particularly efficient, at least for large number of bins (>30).

Cheers,

David

My numpy version is '1.0.4.dev3937'

Thanks,
Lars

--
Dipl.-Ing. Lars Friedrich

Photonic Measurement Technology
Department of Microsystems Engineering -- IMTEK
University of Freiburg
Georges-Köhler-Allee 102
D-79110 Freiburg
Germany

phone: +49-761-203-7531
fax: +49-761-203-7537
room: 01 088
email: lars.friedrich <at> imtek.de
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

headers

Francesc Altet | 1 Feb 19:14

Re: Can not update a submatrix

A Thursday 31 January 2008, Francesc Altet escrigué:
> A Wednesday 30 January 2008, Timothy Hochberg escrigué:
> > [...a fine explanation by Anne and Timothy...]
>
> Ok. As it seems that this subject has interest enough, I went ahead
> and created a small document about views vs copies at:
>
> http://www.scipy.org/Cookbook/ViewsVsCopies

Ooops, I think I've missed the NumPy tutorial:

http://www.scipy.org/Tentative_NumPy_Tutorial

which already talked about copies vs views :-/.  Well, I think my small 
document can complement some parts of the tutorial.  I'll do that as 
soon as I can and remove the recipe from the cookbook.  Sorry for the 
noise.

Cheers,

--

-- 
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   Cárabos Coop. V.   Enjoy Data
 "-"
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

headers

Robert Kern | 1 Feb 19:39

Re: [F2PY]: Allocatable Arrays

Pearu Peterson wrote:
> On Fri, February 1, 2008 1:28 pm, Andrea Gavana wrote:
>> Hi All,
>>
>>     I sent a couple of messages to f2py mailing list, but it seems
>> like my problem has no simple solution so I thought to ask for some
>> suggestions here.
> 
> Sorry, I haven't been around there long time.

Are you going to continue not reading the f2py list? If so, you should point 
everyone there to this list and close the list.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

215	February 2012
366	January 2012
397	December 2011
259	November 2011
475	October 2011
280	September 2011
413	August 2011
536	July 2011
833	June 2011
369	May 2011
340	April 2011
729	March 2011
328	February 2011
359	January 2011
259	December 2010
463	November 2010
623	October 2010
503	September 2010
488	August 2010
787	July 2010
576	June 2010
440	May 2010
502	April 2010
641	March 2010
848	February 2010
441	January 2010
655	December 2009
735	November 2009
497	October 2009
911	September 2009
608	August 2009
594	July 2009
842	June 2009
590	May 2009
665	April 2009
897	March 2009
681	February 2009
587	January 2009
504	December 2008
532	November 2008
557	October 2008
778	September 2008
758	August 2008
1019	July 2008
620	June 2008
1244	May 2008
1110	April 2008
620	March 2008
551	February 2008
672	January 2008
317	December 2007
398	November 2007
391	October 2007
293	September 2007
319	August 2007
351	July 2007
282	June 2007
504	May 2007
617	April 2007
720	March 2007
435	February 2007
594	January 2007
361	December 2006
500	November 2006
873	October 2006
530	September 2006
636	August 2006
803	July 2006
664	June 2006
436	May 2006
543	April 2006
584	March 2006
673	February 2006
439	January 2006
61	December 2005
118	November 2005
145	October 2005
65	September 2005
56	August 2005
120	July 2005
58	June 2005
78	May 2005
205	April 2005
295	March 2005
318	February 2005
185	January 2005
60	December 2004
55	November 2004
131	October 2004
75	September 2004
84	August 2004
150	July 2004
149	June 2004
72	May 2004
49	April 2004
70	March 2004
75	February 2004
110	January 2004
71	December 2003
83	November 2003
61	October 2003
107	September 2003
24	August 2003
48	July 2003
12	June 2003
76	May 2003
6	April 2003
42	March 2003
72	February 2003
110	January 2003
104	December 2002
23	November 2002
36	October 2002
59	September 2002
34	August 2002
36	July 2002
123	June 2002
34	May 2002
62	April 2002
126	March 2002
37	February 2002
72	January 2002
54	December 2001
81	November 2001
33	October 2001
56	September 2001
92	August 2001
52	July 2001
33	June 2001
50	May 2001
67	April 2001
42	March 2001
54	February 2001
56	January 2001
24	December 2000
32	November 2000
60	October 2000
44	September 2000
16	August 2000
16	July 2000
28	June 2000
37	May 2000
28	April 2000
48	March 2000
49	February 2000
8	January 2000

Mon	Tue	Wed	Thu	Fri	Sat	Sun

				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29