Francesc Alted | 13 Feb 11:51 2015
Picon

Vectorizing computation

Hi,

I would like to vectorize the next computation:

nx, ny, nz = 720, 180, 3
outheight = np.arange(nz) * 3
oro = np.arange(nx * ny).reshape((nx, ny))

def compute1(outheight, oro):
    result = np.zeros((nx, ny, nz))
    for ix in range(nx):
        for iz in range(nz):
            result[ix, :, iz] = outheight[iz] + oro[ix, :]
    return result

I think this should be possible by using an advanced use of broadcasting in numpy.  Anyone willing to post a solution?

Thanks, 
--
Francesc Alted
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Colin J. Williams | 13 Feb 00:44 2015
Picon

Re: Matrix Class

Thanks Ryan.

There are a number of good thoughts in your message.  I'll try to keep track of them.

Another respondent reported different results than mine.  I'm in the process of re-installing to check.

Colin W.

On 11 February 2015 at 16:18, Ryan Nelson <rnelsonchem <at> gmail.com> wrote:
Colin,

I currently use Py3.4 and Numpy 1.9.1. However, I built a quick test conda environment with Python2.7 and Numpy 1.7.0, and I get the same:

############
Python 2.7.9 |Continuum Analytics, Inc.| (default, Dec 18 2014, 16:57:52) [MSC v
.1500 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 2.3.1 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import numpy as np

In [2]: np.__version__
Out[2]: '1.7.0'

In [3]: np.mat([4,'5',6])
Out[3]:
matrix([['4', '5', '6']],
       dtype='|S1')

In [4]: np.mat([4,'5',6], dtype=int)
Out[4]: matrix([[4, 5, 6]])
###############

As to your comment about coordinating with Statsmodels, you should see the links in the thread that Alan posted:
Josef's comments at the time seem to echo the issues the devs (and others) have with the matrix class. Maybe things have changed with Statsmodels.

I know I mentioned Sage and SageMathCloud before. I'll just point out that there are folks that use this for real research problems, not just as a pedagogical tool. They have a Matrix/vector/column_matrix class that do what you were expecting from your problems posted above. Indeed below is a (truncated) cut and past from a Sage Worksheet. (See http://www.sagemath.org/doc/tutorial/tour_linalg.html)
##########
In : Matrix([1,'2',3])
Error in lines 1-1
Traceback (most recent call last):
TypeError: unable to find a common ring for all elements

In : Matrix([[1,2,3],[4,5]])
ValueError: List of rows is not valid (rows are wrong types or lengths)

In : vector([1,2,3])
(1, 2, 3)

In : column_matrix([1,2,3])
[1]
[2]
[3]
##########

Large portions of the custom code and wrappers in Sage are written in Python. I don't think their Matrix object is a subclass of ndarray, so perhaps you could strip out the Matrix stuff from here to make a separate project with just the Matrix stuff, if you don't want to go through the Sage interface. 


On Wed, Feb 11, 2015 at 11:54 AM, cjw <cjw <at> ncf.ca> wrote:

On 11-Feb-15 10:21 AM, Ryan Nelson wrote:
So: In [2]: np.mat([4,'5',6]) Out[2]: matrix([['4', '5', '6']], dtype='<U11') In [3]: np.mat([4,'5',6], dtype=int) Out[3]: matrix([[4, 5, 6]])
Thanks Ryan,

We are not singing from the same hymn book.

Using PyScripter, I get:
*** Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32. ***
>>> import numpy as np
>>> print('Numpy version: ', np.__version__)
('Numpy version: ', '1.9.0')
>>>
Could you say which version you are using please?

Colin W

On Tue, Feb 10, 2015 at 5:07 PM, cjw <cjw <at> ncf.ca> wrote:
It seems to be agreed that there are weaknesses in the existing Numpy Matrix Class. Some problems are illustrated below. I'll try to put some suggestions over the coming weeks and would appreciate comments. Colin W. Test Script: if __name__ == '__main__': a= mat([4, 5, 6]) # Good print('a: ', a) b= mat([4, '5', 6]) # Not the expected result print('b: ', b) c= mat([[4, 5, 6], [7, 8]]) # Wrongly accepted as rectangular print('c: ', c) d= mat([[1, 2, 3]]) try: d[0, 1]= 'b' # Correctly flagged, not numeric except ValueError: print("d[0, 1]= 'b' # Correctly flagged, not numeric", ' ValueError') print('d: ', d) Result: *** Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32. ***
a: [[4 5 6]] b: [['4' '5' '6']] c: [[[4, 5, 6] [7, 8]]] d[0, 1]= 'b' # Correctly flagged, not numeric ValueError d: [[1 2 3]]
-- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Matrix-Class-tp39719.html Sent from the Numpy-discussion mailing list archive at Nabble.com. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion <at> scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion <at> scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Neal Becker | 12 Feb 15:21 2015
Picon

unpacking data values into array of bits

I need to transmit some data values.  These values will be float and long 
values.  I need them encoded into a string of bits.

The only way I found so far to do this seems rather roundabout:

np.unpackbits (np.array (memoryview(struct.pack ('d', pi))))
Out[45]: 
array([0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0,
       0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0,
       0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0], dtype=uint8)

(which I'm not certain is correct)

Also, I don't know how to reverse this process 

--

-- 
-- Those who don't understand recursion are doomed to repeat it
Kartik Kumar Perisetla | 11 Feb 03:38 2015
Picon

Using numpy on hadoop streaming: ImportError: cannot import name multiarray

Hi all,

for one of my projects I am using basically using NLTK for pos tagging, which internally uses a 'english.pickle' file. I managed to package the nltk library with these pickle files to make them available to mapper and reducer for hadoop streaming job using -file option.

However, when nltk library is trying to load that pickle file, it gives error for numpy- since the cluster I am running this job does not have numpy installed. Also, I don't have root access thus, can't install numpy or any other package on cluster. So the only way is to package the python modules to make it available for mapper and reducer. I successfully managed to do that. But now the problem is when numpy is imported, it imports multiarray by default( as seen in init.py) and this is where I am getting the error:

File "/usr/lib64/python2.6/pickle.py", line 1370, in load return Unpickler(file).load() File "/usr/lib64/python2.6/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib64/python2.6/pickle.py", line 1090, in load_global klass = self.find_class(module, name) File "/usr/lib64/python2.6/pickle.py", line 1124, in find_class __import__(module) File "numpy.mod/numpy/__init__.py", line 170, in <module> File "numpy.mod/numpy/add_newdocs.py", line 13, in <module> File "numpy.mod/numpy/lib/__init__.py", line 8, in <module> File "numpy.mod/numpy/lib/type_check.py", line 11, in <module> File "numpy.mod/numpy/core/__init__.py", line 6, in <module> ImportError: cannot import name multiarray

I tried moving numpy directory on my local machine that contains multiarray.pyd, to the cluster to make it available to mapper and reducer but this didn't help.

Any input on how to resolve this(keeping the constraint that I cannot install anything on cluster machines)? 

Thanks!


--
Regards,

Kartik Perisetla
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
cjw | 10 Feb 23:07 2015
Picon

Matrix Class

It seems to be agreed that there are weaknesses in the existing Numpy Matrix
Class.

Some problems are illustrated below.

I'll try to put some suggestions over the coming weeks and would appreciate
comments.

Colin W.

Test Script:

if __name__ == '__main__':
    a= mat([4, 5, 6])                   # Good
    print('a: ', a)
    b= mat([4, '5', 6])                 # Not the expected result
    print('b: ', b)
    c= mat([[4, 5, 6], [7, 8]])         # Wrongly accepted as rectangular
    print('c: ', c)
    d= mat([[1, 2, 3]])
    try:
        d[0, 1]= 'b'                    # Correctly flagged, not numeric
    except ValueError:
        print("d[0, 1]= 'b'             # Correctly flagged, not numeric", '
ValueError')
    print('d: ', d)

Result:

*** Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit
(AMD64)] on win32. ***
>>> 
a:  [[4 5 6]]
b:  [['4' '5' '6']]
c:  [[[4, 5, 6] [7, 8]]]
d[0, 1]= 'b'             # Correctly flagged, not numeric  ValueError
d:  [[1 2 3]]
>>> 

--
View this message in context: http://numpy-discussion.10968.n7.nabble.com/Matrix-Class-tp39719.html
Sent from the Numpy-discussion mailing list archive at Nabble.com.
├ůsmund Hjulstad | 10 Feb 21:02 2015
Picon

How to debugging python crash in ipython notebook


Hello all,

I am struggling with a python crash of an ipython notebook (kernel) that I do not know how to debug.

If I run this:  (valgt is a pandas dataframe with 354 lines and 14 numerical columns)

sns.pairplot(valgt)
plt.savefig('parvis.png', dpi=600)

in the same cell, the kernel consistently crashes. If I run in two separate cells, or exporting to a Python script and running there (changing the inline render to qt), everything is OK.


This is win64 on Win7, both the free build from conda, the accelerate build from conda, and also a version I have built myself. (VS2010 and Intel Fortran XE 2015)

The following is the package list from the 'vanilla' conda:

# packages in environment [...]:
#
dateutil                  2.1                      py34_2
ipython                   2.3.1                    py34_0
jinja2                    2.7.3                    py34_1
markupsafe                0.23                     py34_0
matplotlib                1.4.2                np19py34_0
numexpr                   2.3.1                np19py34_0
numpy                     1.9.1                    py34_0
pandas                    0.15.2               np19py34_0
pip                       6.0.6                    py34_0
pyparsing                 2.0.1                    py34_0
pyqt                      4.10.4                   py34_0
pyreadline                2.0                      py34_0
python                    3.4.2                         1
pytz                      2014.9                   py34_0
pywin32                   219                      py34_0
pyzmq                     14.5.0                   py34_0
rpy2                      2.5.5                     <pip>
scipy                     0.15.1               np19py34_0
seaborn                   0.5.1                np19py34_0
setuptools                12.0.5                   py34_0
six                       1.9.0                    py34_0
tornado                   4.0.2                    py34_0

rpy2 is from the Gohlke python collection. 


The home-built version is a python 3.4.3rc1, with

certifi (14.5.14)
Cython (0.21.2)
ipython (2.4.1)
Jinja2 (2.7.3)
jsonschema (2.4.0)
MarkupSafe (0.23)
matplotlib (1.4.2)
mistune (0.5)
nose (1.3.4)
numexpr (2.4.1.dev0)
numpy (1.10.0.dev0+98a8fe3)
pandas (0.15.2)
pip (6.0.8)
pyparsing (2.0.3)
pyreadline (2.0)
PySide (1.2.2)
python-dateutil (2.4.0)
pytz (2014.10)
pywin32 (219)
pyzmq (14.5.0)
rpy2 (2.5.6)
scipy (0.15.1)
seaborn (0.6.dev0)
setuptools (12.0.5)
six (1.9.0)
tornado (4.1)
wheel (0.24.0)


They crash in the same way, though, the same applies to a test I did with ipython 3


I am wondering how important it is to have everything built with the same compiler and MKL library. Does all packages that use numpy need to be built against the specific numpy+mkl version? For example rpy2 (which is not used near the cells where it crashes), should I always rebuild this as well if I build differently, or...?

And, how can I generate debug information from the kernel that ipython notebook starts?

Any pointers are most appreciated.




--
mvh,
Åsmund Hjulstad
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Antoine Pitrou | 10 Feb 18:10 2015
Picon

Aligned / configurable memory allocation


Hello,

I apologize for pinging the list, but I was wondering if there was
interest in either of https://github.com/numpy/numpy/pull/5457 (make
array data aligned by default) or
https://github.com/numpy/numpy/pull/5470 (make the array data allocator
configurable)?

Regards

Antoine.
Benjamin Root | 9 Feb 17:31 2015
Picon

converting a list of tuples into an array of tuples?

I am trying to write up some code that takes advantage of np.tile() on arbitrary array-like objects. I only want to tile along the first axis. Any other axis, if they exist, should be left alone. I first coerce the object using np.asanyarray(), tile it, and then coerce it back to the original type.

The problem seems to be that some of my array-like objects are being "over-coerced", particularly the list of tuples. I tried doing "np.asanyarray(a, dtype='O')", but that still turns it into a 2-D array.

Am I missing something?

Thanks,
Ben Root
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
R Schumacher | 9 Feb 16:52 2015

Re: Silent Broadcasting considered harmful

The problem would be human resources to implement it on the doc site; I don't know how PHP people control modern spammers.
The current, Pythonic way would be to edit the tutorial when inspired. However
http://wiki.scipy.org/Tentative_NumPy_Tutorial
reads: "Please do not hesitate to click the edit button. You will need to create a User Account first. "
which leads to http://wiki.scipy.org/UserPreferences
"UserPreferences
You are not allowed to view this page."

;)

- Ray Schumacher


At 11:37 PM 2/8/2015, you wrote:
That sounds like a good idea! I didn't see any real good examples of usage
after some googling. Giving more examples of effective usage could also clear
more things up regarding design decisions. Additionally I'm always interested
in learning some new tricks :)
 
Cheers,
Stefan
 
Gesendet: Montag, 09. Februar 2015 um 00:24 Uhr Von: "R Schumacher" <rays <at> blue-cove.com> An: "Discussion of Numerical Python" <numpy-discussion <at> scipy.org> Betreff: Re: [Numpy-discussion] Silent Broadcasting considered harmful
At 02:47 PM 2/8/2015, Simon Wood wrote: >Not quite the same. This is not so much about language semantics as >mathematical definitions. You (the Numpy community) have decided to >overload certain mathematical operators to act in a way that is not >consistent with linear algebra teachings. This can be a bit >confusing for people who develop and implement mathematical >algorithms that have a strong foundation in linear algebra, >irrespective of the language they are migrating from. > >With that said, I do appreciate the comments by Matthew, Eelco and >others. Numpy is *not* a linear algebra package, so it does not >adhere to the same mathematical definitions. This realization has >cleared some things up. Via my (admittedly infrequent use of) numpy.linalg http://docs.scipy.org/doc/numpy/reference/routines.linalg.html#linear-algebra-on-several-matrices-at-once I think it behaves more in line with algebraic thinkers. I do not have any issue with broadcasting, and use it frequently, but I've always wanted to see more examples and discussion directly in the docs, in general. I have over years post/argued for a doc site more like PHP-doc, where users can contribute examples and discuss them. There is a wealth of such examples here in the list and the tutorial, but requires unnecessary time and Google-foo. - Ray Schumacher _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion <at> scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Reiterer | 9 Feb 08:34 2015
Picon
Picon

Re: Silent Broadcasting considered harmful

Ok that are indeed some good reasons to keep the status quo, especially since performance is crucial for numpy.
 
It's a dillemma: Using the matrix class for linear algebra would be the correct way for such thing,
but the matrix API is not that powerful and beautiful as the one of arrays.
On the other hand arrays are beautiful, but not exactly intended to use for linear algebra.
 
So maybe the better way would be not to add warnings to braodcasting operations, but to overhaul the matrix class
to make it more attractive for numerical linear algebra(?)
 
Cheers,
Stefan
Gesendet: Sonntag, 08. Februar 2015 um 23:52 Uhr
Von: "Nathaniel Smith" <njs <at> pobox.com>
An: "Discussion of Numerical Python" <numpy-discussion <at> scipy.org>
Betreff: Re: [Numpy-discussion] Silent Broadcasting considered harmful

On 8 Feb 2015 13:04, "Stefan Reiterer" <domors <at> gmx.net> wrote:
>
> So I suggest that the best would be to throw warnings when arrays get Broadcasted like
> Octave do. Python warnings can be catched and handled, that would be a great benefit.
>  
> Another idea would to provide warning levels for braodcasting, e.g
> 0 = Never, 1=Warn once, 2=Warn always, 3 = Forbid aka throw exception,
> with 0 as default.
> This would avoid breaking other code, and give the user some control over braodcasting.

Unfortunately adding warnings is a non-starter for technical reasons, even before we get into the more subjective debate about ideal API design: issuing a warning is extremely slow (relative to typical array operations), EVEN IF the warning is disabled. (By the time you can figure out it's disabled, it's too late.) So this would cause massive slowdowns in existing code.

Note also that in numpy, even simple expressions like '2 * arr' rely on broadcasting. Do you really want warnings for all these cases?

-n

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion <at> scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
R Schumacher | 9 Feb 00:24 2015

Re: Silent Broadcasting considered harmful

At 02:47 PM 2/8/2015, Simon Wood wrote:
>Not quite the same. This is not so much about language semantics as 
>mathematical definitions. You (the Numpy community) have decided to 
>overload certain mathematical operators to act in a way that is not 
>consistent with linear algebra teachings. This can be a bit 
>confusing for people who develop and implement mathematical 
>algorithms that have a strong foundation in linear algebra, 
>irrespective of the language they are migrating from.
>
>With that said, I do appreciate the comments by Matthew, Eelco and 
>others. Numpy is *not* a linear algebra package, so it does not 
>adhere to the same mathematical definitions. This realization has 
>cleared some things up.

Via my (admittedly infrequent use of) numpy.linalg
http://docs.scipy.org/doc/numpy/reference/routines.linalg.html#linear-algebra-on-several-matrices-at-once
I think it behaves more in line with algebraic thinkers.

I do not have any issue with broadcasting, and use it frequently, but 
I've always wanted to see more examples and discussion directly in 
the docs, in general.
I have over years post/argued for a doc site more like PHP-doc, where 
users can contribute examples and discuss them. There is a wealth of 
such examples here in the list and the tutorial, but requires 
unnecessary time and Google-foo.

- Ray Schumacher 

Gmane