Re: reading 10 bit raw data into an array

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: reading 10 bit raw data into an array

On 8/1/07, Danny Chan <chan_dhf <at> yahoo.de> wrote:
> Hi Travis!
> I guess I will still have to pad my data to full bytes before reading it,
> correct?
>
> Travis Oliphant <oliphant.travis <at> ieee.org> schrieb:
> Danny Chan wrote:
> > Hi all!
> > I'm trying to read a data file that contains a raw image file. Every
> > pixel is assigned a value from 0 to 1023, and all pixels are stored from
> > top left to bottom right pixel in binary format in this file. I know the
> > width and the height of the image, so all that would be required is to
> > read 10 bits at a time and store it these as an integer. I played around
> > with the fromstring and fromfile function, and I read the documentation
> > for dtype objects, but I'm still confused. It seems simple enough to
> > read data in a format with a standard bitwidth, but how can I read data
> > in a non-standard format. Can anyone help?
> >
>
> This kind of bit-manipulation must be done using bit operations on
> standard size data types even in C. The file reading and writing
> libraries use bytes as their common denominator.
>
> I would read in the entire image into a numpy array of unsigned bytes
> and then use slicing, masking, and bit-shifting to take 5 bytes at a
> time and convert them to 4 values of a 16-bit unsigned image.
>
> Basically, you would do something like
>
> # read in entire image into 1-d unsigned byte array
> # create 16-bit array of the correct 2-D size
> # use flat indexing to store into the new array
> # new.flat[::4] = old[::5] + bitwise_or(old[1::5], MASK1b) << SHIFT1b
> # new.flat[1::4] = bitwise_or(old[1::5], MASK2a) << SHIFT2a
> + bitwise_or(old[2::5], MASK2b) << SHIFT2b
>
> # etc.
>
>
> The exact MASKS and shifts to use is left as an exercise for the reader 

Quick comment :  are you really sure your  camera produces the 12 bit
data in a "12 bit stream" --- all I have ever seen is that cameras
would just use 16 bit for each pixel.  (All you had to know if it uses
the left or the right part of those.  In other words, you might have
to divide (or use  bit shifting) the data by 16.)
Wasteful yes, but much simpler to handel !?

-Sebastian Haase

fourier with single precision

Hello,

is there a way to tell numpy.fft.fft2 to use complex64 instead of 
complex128 as output dtype to speed the up transformation?

Thanks
Lars

Re: How to implement a 'pivot table?'

I do a lot of this kind of things in SAS. In don't like SAS that much
so it would be great to have functionality like this for numpy
recarray's.

To transplant the approach that SAS takes to a numpy setting you'd
have something like the following 4 steps:
1. Sort the data by date and region
2. Determine the indices for the blocks (e.g., East, 1/1)
3. calculate the summary stats per block

SAS is very efficient at these types of operations i believe. Since it
assumes that the data is sorted, and throws and error if the data is
not sorted appropriately, i assume the indexing can be more efficient.
However, given the earlier comments i am wonder if this approach would
enhance performance.

I would be very interested to see what you come up with so please post
some of the code and/or timing tests to the list if possible.

Best,

Vincent

Re: reading 10 bit raw data into an array

>
> Quick comment :  are you really sure your  camera produces the 12 bit
> data in a "12 bit stream" --- all I have ever seen is that cameras
> would just use 16 bit for each pixel.  (All you had to know if it uses
> the left or the right part of those.  In other words, you might have
> to divide (or use  bit shifting) the data by 16.)
> Wasteful yes, but much simpler to handel !?
>

10 bit, but yes, I am sure. It is an embedded camera system, in fact, I get 
the data stream even before it is handled to any ISP for further processing 
of the picture. In the end, the ISP will convert the data stream to another 
format, but I have to simulate some of the algorithms that will be 
implemented in hardware.

Re: How to implement a 'pivot table?'

Hi,
The hard part is knowing what aggregate function that you want. So a
hard way, even after cheating, to take the data provided is given
below. (The Numpy Example List was very useful especially on the where
function)!

I tried to be a little generic so you can replace the sum by any
suitable function and probably the array type as well. Of course it is
not complete because you still need to know the levels of the 'rows'
and 'columns' and also is not efficient as it has loops.

Bruce

from numpy import *
A=array([[1,1,10],
         [1,1,20],
         [1,2,30],
         [2,1,40],
         [2,2,50],
         [2,2,60] ])
C = zeros((2,2))

for i in range(2):
      crit1 = (A[:,0]==1+i)
      subA=A[crit1,1:]
      for j in range(2):
            crit2 = (subA[:,0]==1+j)
            subB=subA[crit2,1:]
            C[i,j]=subB.sum()

print C

On 7/30/07, Geoffrey Zhu <zyzhu2000 <at> gmail.com> wrote:
> Hi Everyone,
>
> I am wondering what is the best (and fast) way to build a pivot table
> aside from the 'brute force way?'
>
> I want to transform an numpy array into a pivot table. For example, if
> I have a numpy array like below:
>
> Region     Date          # of Units
> ----------    ----------        --------------
> East        1/1             10
> East        1/1             20
> East        1/2             30
> West       1/1             40
> West       1/2             50
> West       1/2             60
>
> I want  to transform this into the following table, where f() is a
> given aggregate function:
>
>            Date
> Region           1/1          1/2
> ----------
> East         f(10,20)         f(30)
> West        f(40)             f(50,60)
>
>
> I can regroup them into 'sets' and do it the brute force way, but that
> is kind of slow to execute. Does anyone know a better way?
>
>
> Thanks,
> Geoffrey
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion <at> scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

f2py self documenting not working

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: How to implement a 'pivot table?'

Greetings,

Speaking of brute force... I've attached a rather ugly module that  
let's you do things with a pretty simple interface (session shown  
below).  I haven't fully tested the performance, but a million  
records with 5 fields takes about 11 seconds on my Mac to do a  
'mean'.  I'm not sure what your performance considerations are, but  
this may be useful.  Record arrays are really nice if they make sense  
for your data.

Travis

(from an ipython command prompt)

In [1]: import testpivot as p

In [2]: a = p.sample_data()

In [3]: a
Out[3]:
recarray([('ACorp', 'Region 1', 'Q1', 20000.0),
        ('ACorp', 'Region 1', 'Q2', 22000.0),
        ('ACorp', 'Region 1', 'Q3', 21000.0),
        ('ACorp', 'Region 1', 'Q4', 26000.0),
        ('ACorp', 'Region 2', 'Q1', 23000.0),
        ('ACorp', 'Region 2', 'Q2', 20000.0),
        ('ACorp', 'Region 2', 'Q3', 22000.0),
        ('ACorp', 'Region 2', 'Q4', 21000.0),
        ('ACorp', 'Region 3', 'Q1', 26000.0),
        ('ACorp', 'Region 3', 'Q2', 23000.0),
        ('ACorp', 'Region 3', 'Q3', 29000.0),
        ('ACorp', 'Region 3', 'Q4', 27000.0),
        ('BCorp', 'Region 1', 'Q1', 20000.0),
        ('BCorp', 'Region 1', 'Q2', 20000.0),
        ('BCorp', 'Region 1', 'Q3', 24000.0),
        ('BCorp', 'Region 1', 'Q4', 24000.0),
        ('BCorp', 'Region 2', 'Q1', 21000.0),
        ('BCorp', 'Region 2', 'Q2', 21000.0),
        ('BCorp', 'Region 2', 'Q3', 22000.0),
        ('BCorp', 'Region 2', 'Q4', 29000.0),
        ('BCorp', 'Region 3', 'Q1', 28000.0),
        ('BCorp', 'Region 3', 'Q2', 25000.0),
        ('BCorp', 'Region 3', 'Q3', 22000.0),
        ('BCorp', 'Region 3', 'Q4', 21000.0)],
       dtype=[('company', '|S5'), ('region', '|S8'), ('quarter', '| 
S2'), ('income', '<f8')])

In [4]: p.pivot(a, 'company', 'region', 'income', p.psum)
######## Summary by company and region ##########
cols:['ACorp' 'BCorp']
rows:['Region 1' 'Region 2' 'Region 3']
[[  89000.   88000.]
[  86000.   93000.]
[ 105000.   96000.]]

In [5]: p.pivot(a, 'company', 'quarter', 'income', p.psum)
######## Summary by company and quarter ##########
cols:['ACorp' 'BCorp']
rows:['Q1' 'Q2' 'Q3' 'Q4']
[[ 69000.  69000.]
[ 65000.  66000.]
[ 72000.  68000.]
[ 74000.  74000.]]

In [6]: p.pivot(a, 'company', 'quarter', 'income', p.pmean)
######## Summary by company and quarter ##########
cols:['ACorp' 'BCorp']
rows:['Q1' 'Q2' 'Q3' 'Q4']
[[ 23000.          23000.        ]
[ 21666.66666667  22000.        ]
[ 24000.          22666.66666667]
[ 24666.66666667  24666.66666667]]

On Aug 1, 2007, at 2:02 PM, Bruce Southey wrote:

> Hi,
> The hard part is knowing what aggregate function that you want. So a
> hard way, even after cheating, to take the data provided is given
> below. (The Numpy Example List was very useful especially on the where
> function)!
>
> I tried to be a little generic so you can replace the sum by any
> suitable function and probably the array type as well. Of course it is
> not complete because you still need to know the levels of the 'rows'
> and 'columns' and also is not efficient as it has loops.
>
> Bruce
>
> from numpy import *
> A=array([[1,1,10],
>          [1,1,20],
>          [1,2,30],
>          [2,1,40],
>          [2,2,50],
>          [2,2,60] ])
> C = zeros((2,2))
>
> for i in range(2):
>       crit1 = (A[:,0]==1+i)
>       subA=A[crit1,1:]
>       for j in range(2):
>             crit2 = (subA[:,0]==1+j)
>             subB=subA[crit2,1:]
>             C[i,j]=subB.sum()
>
>
> print C
>
> On 7/30/07, Geoffrey Zhu <zyzhu2000 <at> gmail.com> wrote:
>> Hi Everyone,
>>
>> I am wondering what is the best (and fast) way to build a pivot table
>> aside from the 'brute force way?'
>>
>> I want to transform an numpy array into a pivot table. For  
>> example, if
>> I have a numpy array like below:
>>
>> Region     Date          # of Units
>> ----------    ----------        --------------
>> East        1/1             10
>> East        1/1             20
>> East        1/2             30
>> West       1/1             40
>> West       1/2             50
>> West       1/2             60
>>
>> I want  to transform this into the following table, where f() is a
>> given aggregate function:
>>
>>            Date
>> Region           1/1          1/2
>> ----------
>> East         f(10,20)         f(30)
>> West        f(40)             f(50,60)
>>
>>
>> I can regroup them into 'sets' and do it the brute force way, but  
>> that
>> is kind of slow to execute. Does anyone know a better way?
>>
>>
>> Thanks,
>> Geoffrey
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion <at> scipy.org
>> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion <at> scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion <at> scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: fourier with single precision

Lars Friedrich wrote:
> Hello,
>
> is there a way to tell numpy.fft.fft2 to use complex64 instead of 
> complex128 as output dtype to speed the up transformation?
>   
As far as I can read from the fft code in numpy, only double is 
supported at the moment, unfortunately. Note that you can get some speed 
by using scipy.fftpack methods instead, if scipy is an option for you.

David

Memory efficient equality test for arrays

Is there a numpy call to test if two large arrays (say 1 Gbyte each) are 
equal (same shape and elements) without creating another large array of 
booleans as happens with "a == b", numpy.equal(a,b), or 
numpy.array_equal(a,b)?

I want a memory efficient and fast comparison.

    Tom