Valentin Haenel | 12 Oct 18:40 2014
Picon

[ANN] bcolz 0.7.2


======================
Announcing bcolz 0.7.2
======================

What's new
==========

This is a maintenance release that fixes various bits and pieces.
Importantly, compatibility with Numpy 1.9 and Cython 0.21 has been fixed
and the test suit no longer segfaults on 32 bit UNIX. Feature-wise a new
``carray.view()`` method has been introduced which allows carrays to
share the same raw data.

``bcolz`` is a renaming of the ``carray`` project.  The new goals for
the project are to create simple, yet flexible compressed containers,
that can live either on-disk or in-memory, and with some
high-performance iterators (like `iter()`, `where()`) for querying them.

Together, bcolz and the Blosc compressor, are finally fulfilling the
promise of accelerating memory I/O, at least for some real scenarios:

http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb#Plots

For more detailed info, see the release notes in:
https://github.com/Blosc/bcolz/wiki/Release-Notes

What it is
==========

(Continue reading)

Warren Weckesser | 12 Oct 00:51 2014
Picon

Request for enhancement to numpy.random.shuffle

I created an issue on github for an enhancement
to numpy.random.shuffle:
    https://github.com/numpy/numpy/issues/5173
I'd like to get some feedback on the idea.

Currently, `shuffle` shuffles the first dimension of an array
in-place.  For example, shuffling a 2D array shuffles the rows:

In [227]: a
Out[227]:
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]])

In [228]: np.random.shuffle(a)

In [229]: a
Out[229]:
array([[ 0,  1,  2],
       [ 9, 10, 11],
       [ 3,  4,  5],
       [ 6,  7,  8]])



To add an axis keyword, we could (in effect) apply `shuffle` to
`a.swapaxes(axis, 0)`.  For a 2-D array, `axis=1` would shuffles
the columns:

In [232]: a = np.arange(15).reshape(3,5)

In [233]: a
Out[233]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

In [234]: axis = 1

In [235]: np.random.shuffle(a.swapaxes(axis, 0))

In [236]: a
Out[236]:
array([[ 3,  2,  4,  0,  1],
       [ 8,  7,  9,  5,  6],
       [13, 12, 14, 10, 11]])


So that's the first part--adding an `axis` keyword.

The other part of the enhancement request is to add a shuffle
behavior that shuffles the 1-d slices *independently*.  That is,
for a 2-d array, shuffling with `axis=0` would apply a different
shuffle to each column.  In the github issue, I defined a
function called `disarrange` that implements this behavior:

In [240]: a
Out[240]:
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11],
       [12, 13, 14]])

In [241]: disarrange(a, axis=0)

In [242]: a
Out[242]:
array([[ 6,  1,  2],
       [ 3, 13, 14],
       [ 9, 10,  5],
       [12,  7,  8],
       [ 0,  4, 11]])


Note that each column has been shuffled independently.

This behavior is analogous to how `sort` handles the `axis`
keyword.  `sort` sorts the 1-d slices along the given axis
independently.

In the github issue, I suggested the following signature
for `shuffle` (but I'm not too fond of the name `independent`):

  def shuffle(a, independent=False, axis=0)

If `independent` is False, the current behavior of `shuffle`
is used.  If `independent` is True, each 1-d slice is shuffled
independently (in the same way that `sort` sorts each 1-d
slice).

Like most functions that take an `axis` argument, `axis=None`
means to shuffle the flattened array.  With `independent=True`,
it would act like `np.random.shuffle(a.flat)`, e.g.

In [247]: a
Out[247]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

In [248]: np.random.shuffle(a.flat)

In [249]: a
Out[249]:
array([[ 0, 14,  9,  1, 13],
       [ 2,  8,  5,  3,  4],
       [ 6, 10,  7, 12, 11]])



A small wart in this API is the meaning of

  shuffle(a, independent=False, axis=None)

It could be argued that the correct behavior is to leave the
array unchanged. (The current behavior can be interpreted as
shuffling a 1-d sequence of monolithic blobs; the axis argument
specifies which axis of the array corresponds to the
sequence index.  Then `axis=None` means the argument is
a single monolithic blob, so there is nothing to shuffle.)
Or an error could be raised.

What do you think?

Warren

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Alan G Isaac | 10 Oct 22:24 2014
Picon

fsin on Intel

This is not NumPy specific but may still interest list members:
http://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion/

Alan Isaac
Benjamin Root | 10 Oct 20:23 2014
Picon

use ufunc for arbitrary positional arguments?

I have a need to "and" together an arbitrary number of boolean arrays. np.logical_and() expects only two positional arguments. There has got to be some sort of easy way to just and these together using the ufunc mechanism, right?

Cheers!
Ben Root
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Julian Taylor | 10 Oct 20:11 2014

parallel compilation with numpy.distutils in numpy 1.10

hi,
To speed up compilation of extensions I have made a PR to compile
extension files in parallel:
https://github.com/numpy/numpy/pull/5161

It adds the --jobs/-j flags to the build command of setup.py which
defines the number of parallel compile processes.
E.g.
python setup.py build --jobs 4 install --prefix /tmp/local

Additionally it adds the environment variable NPY_NUM_BUILD_JOBS which
is used if no commandline is set. This helps e.g. with pip
installations, travis builds (which give you 1.5 cpus) or to put in your
.bashrc.

The parallelization is only with the files of an extension so it is not
super efficient but an uncached numpy build goes down from 1m40s to
1m00s with 3 cores on my machine which is quite decent.
Building scipy from scratch decreased from 10minutes to 6m30s on my machine.

Unfortunately projects using cython will not profit as cython tends to
build an extension from one single file. (You may want to look into gccs
internal parallelization for that, -flto=jobs)-

Does some see issues with the interface I have currently set? Please
speak up soon.

There is still one problem in regards to parallelizing fortran 90. The
ccompiler.py contains following comment:
    # build any sources in same order as they were originally specified
    #   especially important for fortran .f90 files using modules

This indicates the f90 builds cannot be trivially parallelized. I do not
know much fortran, can someone explain to me when ordering of single
file compiles is an issue in f90?

Cheers,
Julian Taylor
Lahiru Samarakoon | 10 Oct 18:51 2014
Picon

Instaling numpy without root access

Dear all,

I am trying to install numpy without root access. So I am building from the source.  I have installed atlas which also has lapack with it.  I changed the site.cfg file as given below

[DEFAULT]
library_dirs = /home/svu/a0095654/ATLAS/build/lib
include_dirs = /home/svu/a0095654/ATLAS/build/include


However, I am getting a segmentation fault when importing numpy.

Please advise. I also put the build log file at the end of the email if necessary.

Thank you,
Best Regards,
Lahiru

Log starts below.

 python2.7 setup.py build  --fcompiler=gnu95
Running from numpy source directory.
/home/svu/a0095654/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite'
  warnings.warn(msg)
non-existing path in 'numpy/f2py': 'docs'
non-existing path in 'numpy/f2py': 'f2py.1'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in ['/home/svu/a0095654/ATLAS/build/lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['/home/svu/a0095654/ATLAS/build/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}

C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/home/svu/a0095654/ATLAS/build/install/lib -lptf77blas -lptcblas -latlas -o _configtest
_configtest.o: In function `main':
/hpctmp/a0095654/Software/numpy-1.9.0/_configtest.c:5: undefined reference to `ATL_buildinfo'
collect2: ld returned 1 exit status
_configtest.o: In function `main':
/hpctmp/a0095654/Software/numpy-1.9.0/_configtest.c:5: undefined reference to `ATL_buildinfo'
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}

C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/home/svu/a0095654/ATLAS/build/install/lib -lptf77blas -lptcblas -latlas -o _configtest
_configtest.o: In function `main':
/hpctmp/a0095654/Software/numpy-1.9.0/_configtest.c:5: undefined reference to `ATL_buildinfo'
collect2: ld returned 1 exit status

Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/home/svu/a0095654/ATLAS/build/install/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/home/svu/a0095654/ATLAS/build/install/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

non-existing path in 'numpy/lib': 'benchmarks'
lapack_opt_info:
openblas_lapack_info:
  libraries openblas not found in ['/home/svu/a0095654/ATLAS/build/lib']
  NOT AVAILABLE

lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in ['/home/svu/a0095654/ATLAS/build/lib']
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /home/svu/a0095654/ATLAS/build/install/lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
/hpctmp/a0095654/Software/numpy-1.9.0/numpy/distutils/system_info.py:1095: UserWarning:
*********************************************************************
    Lapack library (from ATLAS) is probably incomplete:
      size of /home/svu/a0095654/ATLAS/build/install/lib/liblapack.so is 5.0673828125k (expected >4000k)

    Follow the instructions in the KNOWN PROBLEMS section of the file
    numpy/INSTALL.txt.
*********************************************************************

  warnings.warn(message)
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/home/svu/a0095654/ATLAS/build/install/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/home/svu/a0095654/ATLAS/build/install/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

/home/svu/a0095654/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
building library "npymath" sources
customize Gnu95FCompiler
customize Gnu95FCompiler using config
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -c'
gcc: _configtest.c
_configtest.c:1: warning: conflicting types for built-in function ‘exp’
gcc -pthread _configtest.o -o _configtest
_configtest.o: In function `main':
/hpctmp/a0095654/Software/numpy-1.9.0/_configtest.c:6: undefined reference to `exp'
collect2: ld returned 1 exit status
_configtest.o: In function `main':
/hpctmp/a0095654/Software/numpy-1.9.0/_configtest.c:6: undefined reference to `exp'
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -c'
gcc: _configtest.c
_configtest.c:1: warning: conflicting types for built-in function ‘exp’
gcc -pthread _configtest.o -lm -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
building library "npysort" sources
  adding 'build/src.linux-x86_64-2.7/numpy/core/src/private' to include_dirs.
None - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/src/private/npy_partition.h', 'build/src.linux-x86_64-2.7/numpy/core/src/private/npy_binsearch.h']
building extension "numpy.core._dummy" sources
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_numpy_api.py
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h' to sources.
numpy.core - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h']
building extension "numpy.core.multiarray" sources
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_numpy_api.py
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h' to sources.
numpy.core - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h']
building extension "numpy.core.umath" sources
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__ufunc_api.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/src/umath' to include_dirs.
numpy.core - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/src/umath/funcs.inc', 'build/src.linux-x86_64-2.7/numpy/core/src/umath/simd.inc', 'build/src.linux-x86_64-2.7/numpy/core/src/umath/loops.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__ufunc_api.h']
building extension "numpy.core.scalarmath" sources
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h' to sources.
executing numpy/core/code_generators/generate_numpy_api.py
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h' to sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__ufunc_api.h' to sources.
  adding 'build/src.linux-x86_64-2.7/numpy/core/src/private' to include_dirs.
numpy.core - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/src/private/scalarmathmodule.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/_numpyconfig.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__multiarray_api.h', 'build/src.linux-x86_64-2.7/numpy/core/include/numpy/__ufunc_api.h']
building extension "numpy.core._dotblas" sources
  adding 'numpy/core/blasdot/_dotblas.c' to sources.
building extension "numpy.core.umath_tests" sources
building extension "numpy.core.test_rational" sources
building extension "numpy.core.struct_ufunc_test" sources
building extension "numpy.core.multiarray_tests" sources
building extension "numpy.core.operand_flag_tests" sources
building extension "numpy.lib._compiled_base" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
  adding 'numpy/linalg/lapack_litemodule.c' to sources.
  adding 'numpy/linalg/lapack_lite/python_xerbla.c' to sources.
building extension "numpy.linalg._umath_linalg" sources
  adding 'numpy/linalg/umath_linalg.c.src' to sources.
  adding 'numpy/linalg/lapack_lite/python_xerbla.c' to sources.
building extension "numpy.random.mtrand" sources
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
_configtest
failure.
removing: _configtest.c _configtest.o _configtest
building data_files sources
build_src: building npy-pkg config files
running build_py
copying numpy/version.py -> build/lib.linux-x86_64-2.7/numpy
copying build/src.linux-x86_64-2.7/numpy/__config__.py -> build/lib.linux-x86_64-2.7/numpy
copying build/src.linux-x86_64-2.7/numpy/distutils/__config__.py -> build/lib.linux-x86_64-2.7/numpy/distutils
running build_clib
customize UnixCCompiler
customize UnixCCompiler using build_clib
building 'npymath' library
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/npymath
creating build/temp.linux-x86_64-2.7/numpy
creating build/temp.linux-x86_64-2.7/numpy/core
creating build/temp.linux-x86_64-2.7/numpy/core/src
creating build/temp.linux-x86_64-2.7/numpy/core/src/npymath
compile options: '-Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -c'
gcc: numpy/core/src/npymath/halffloat.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npymath/ieee754.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npymath/npy_math.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npymath/npy_math_complex.c
ar: adding 4 object files to build/temp.linux-x86_64-2.7/libnpymath.a
building 'npysort' library
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/numpy/core/src/npysort
compile options: '-Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/home/svu/a0095654/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -c'
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npysort/selection.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npysort/quicksort.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npysort/mergesort.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npysort/binsearch.c
gcc: build/src.linux-x86_64-2.7/numpy/core/src/npysort/heapsort.c
ar: adding 5 object files to build/temp.linux-x86_64-2.7/libnpysort.a
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
running build_scripts
  adding 'build/scripts.linux-x86_64-2.7/f2py2.7' to scripts

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Per.Brodtkorb | 10 Oct 11:34 2014
Picon

Any interest in a generalized piecewise function?

I have worked on a generalized piecewise function (genpiecewise) that are simpler and more general than the current numpy.piecewise implementation. The new generalized piecewise function allows functions of the type f(x0, x1,.. , xn) i.e. to have arbitrary number of input arguments that are evaluated conditionally.

The generalized piecewise function passes all the tests for numpy.piecewise function except the undocumented features of numpy.piecewise which allows condlist to be a single bool list/array or a single int array.

 

A new keyword “fillvalue” opens up the possibility to specify “out of bounds values” to other values than 0 eg. Np.nan.

 

Examples:

Example 1)

>>> x = numpy.linspace(-2,2,5)

>>> numpy.piecewise(x, x<1, [1])  # = numpy.where(x<0, 1, 0)

array([ 1.,  1.,  1.,  0.,  0.])

 

# can be written as

>>> genpiecewise([x<1],[1])

array([1, 1, 1, 0, 0])

# or

>>> genpiecewise([x<1],[1], x)

array([ 1.,  1.,  1.,  0.,  0.])

# or

>>> genpiecewise([x<1],[1.0])

array([ 1.,  1.,  1.,  0.,  0.])

 

Example 2)

>>> numpy.piecewise(x, [x < 0, x >= 0], [lambda x: -x, lambda x: x])

array([ 2.,  1.,  0.,  1.,  2.])

# can be written as

>>> genpiecewise([x < 0, x >= 0], [lambda x: -x, lambda x: x], (x,))

array([ 2.,  1.,  0.,  1.,  2.])

# or

genpiecewise([x < 0,], [lambda x: -x, lambda x: x], x)

array([ 2.,  1.,  0.,  1.,  2.])

 

Example 3)

# New functionality

>>> X,Y = numpy.meshgrid(x,x)

>>> genpiecewise([X*Y<0,], [lambda x,y: -x*y, lambda x,y: x*y], xi=(X,Y))

array([[ 4.,  2., -0.,  2.,  4.],

       [ 2.,  1., -0.,  1.,  2.],

       [-0., -0.,  0.,  0.,  0.],

       [ 2.,  1.,  0.,  1.,  2.],

       [ 4.,  2.,  0.,  2.,  4.]])

>>> genpiecewise([X*Y<-0.5, X*Y>0.5], [lambda x,y: -x*y, lambda x,y: x*y], xi=(X,Y), fillvalue=numpy.nan)

array([[  4.,   2.,  nan,   2.,   4.],

       [  2.,   1.,  nan,   1.,   2.],

       [ nan,  nan,  nan,  nan,  nan],

       [  2.,   1.,  nan,   1.,   2.],

       [  4.,   2.,  nan,   2.,   4.]])

>>> genpiecewise([X*Y<-0.5, X*Y>0.5], [lambda x,y: -x*y, lambda x,y: x*y, numpy.nan], (X,Y))

array([[  4.,   2.,  nan,   2.,   4.],

       [  2.,   1.,  nan,   1.,   2.],

       [ nan,  nan,  nan,  nan,  nan],

       [  2.,   1.,  nan,   1.,   2.],

       [  4.,   2.,  nan,   2.,   4.]])

 

My question is: are there any interest in the community for such a function?

Could some or all of this functionality replace the current numpy.piecewise?

(This function could replace the function _lazywhere (in scipy.stats._distn_infrastructure) which is heavily used in scipy.stats)

 

 

Per A. Brodtkorb

 

The code looks like this:

 

def valarray(shape, value=np.nan, typecode=None):

    """Return an array of all value.

    """

 

    out = ones(shape, dtype=bool) * value

    if typecode is not None:

        out = out.astype(typecode)

    if not isinstance(out, np.ndarray):

        out = asarray(out)

    return out

 

 

def genpiecewise(condlist, funclist, xi=None, fillvalue=0, args=(), **kw):

    """

    Evaluate a piecewise-defined function.

 

    Given a set of conditions and corresponding functions, evaluate each

    function on the input data wherever its condition is true.

 

    Parameters

    ----------

    condlist : list of bool arrays

        Each boolean array corresponds to a function in `funclist`.  Wherever

        `condlist[i]` is True, `funclist[i](x0,x1,...,xn)` is used as the

        output value. Each boolean array in `condlist` selects a piece of `xi`,

        and should therefore be of the same shape as `xi`.

 

        The length of `condlist` must correspond to that of `funclist`.

        If one extra function is given, i.e. if

        ``len(funclist) - len(condlist) == 1``, then that extra function

        is the default value, used wherever all conditions are false.

    funclist : list of callables, f(*(xi + args), **kw), or scalars

        Each function is evaluated over `x` wherever its corresponding

        condition is True.  It should take an array as input and give an array

        or a scalar value as output.  If, instead of a callable,

        a scalar is provided then a constant function (``lambda x: scalar``) is

        assumed.

    xi : tuple

        input arguments to the functions in funclist, i.e., (x0, x1,...., xn)

    fillvalue : scalar

        fillvalue for out of range values. Default 0.

    args : tuple, optional

        Any further arguments given here passed to the functions

        upon execution, i.e., if called ``piecewise(..., ..., args=(1, 'a'))``,

        then each function is called as ``f(x0, x1,..., xn, 1, 'a')``.

    kw : dict, optional

        Keyword arguments used in calling `piecewise` are passed to the

        functions upon execution, i.e., if called

        ``piecewise(..., ..., lambda=1)``, then each function is called as

        ``f(x0, x1,..., xn, lambda=1)``.

 

    Returns

    -------

    out : ndarray

        The output is the same shape and type as x and is found by

        calling the functions in `funclist` on the appropriate portions of `x`,

        as defined by the boolean arrays in `condlist`.  Portions not covered

        by any condition have undefined values.

 

 

    See Also

    --------

    choose, select, where

 

    Notes

    -----

    This is similar to choose or select, except that functions are

    evaluated on elements of `xi` that satisfy the corresponding condition from

    `condlist`.

 

    The result is::

 

            |--

            |funclist[0](x0[condlist[0]],x1[condlist[0]],...,xn[condlist[0]])

      out = |funclist[1](x0[condlist[1]],x1[condlist[1]],...,xn[condlist[1]])

            |...

            |funclist[n2](x0[condlist[n2]],x1[condlist[n2]],...,xn[condlist[n2]])

            |--

 

    Examples

    --------

    Define the sigma function, which is -1 for ``x < 0`` and +1 for ``x >= 0``.

 

    >>> x = np.linspace(-2.5, 2.5, 6)

    >>> genpiecewise([x < 0, x >= 0], [-1, 1])

    array([-1., -1., -1.,  1.,  1.,  1.])

 

    Define the absolute value, which is ``-x`` for ``x <0`` and ``x`` for

    ``x >= 0``.

 

    >>> genpiecewise([x < 0, x >= 0], [lambda x: -x, lambda x: x], (x,))

    array([ 2.5,  1.5,  0.5,  0.5,  1.5,  2.5])

 

    """

    def otherwise_condition(condlist):

        return ~np.logical_or.reduce(condlist, axis=0)

 

    def check_shapes(condlist, funclist):

        nc, nf = len(condlist), len(funclist)

        if nc not in [nf-1, nf]:

            raise ValueError("function list and condition list" +

                             " must be the same length")

 

    check_shapes(condlist, funclist)

    if xi is not None and not isinstance(xi, tuple):

        xi = (xi,)

 

    condlist = np.broadcast_arrays(*condlist)

    if len(condlist) == len(funclist)-1:

        condlist.append(otherwise_condition(condlist))

 

    arrays = dtype = None

    if xi is not None:

        arrays = np.broadcast_arrays(*xi)

        dtype = np.result_type(*arrays)

    else# funclist is a list of scalars only

        dtype = np.result_type(*funclist)

    out = valarray(condlist[0].shape, fillvalue, dtype)

    for cond, func in zip(condlist, funclist):

        if isinstance(func, collections.Callable):

            temp = tuple(np.extract(cond, arr) for arr in arrays) + args

            np.place(out, cond, func(*temp, **kw))

        else# func is a scalar value

            np.place(out, cond, func)

    return out

 

 

 

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Juan | 9 Oct 14:50 2014
Picon

Re: Extracting individual columns in Numpy (suchith)

> Date: Thu, 09 Oct 2014 11:12:31 +0530
> From: suchith <suchithjn22 <at> gmail.com>
> Subject: [Numpy-discussion] Extracting individual columns in Numpy
> To: numpy-discussion <at> scipy.org
> Message-ID: <54362047.10102 <at> gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> How to extract individual columns from a numpy array?
> For example, consider this script
> 
> import numpy as np
> a = np.array([[1,2,3],[4,5,6],[7,8,9]])
> a[0][:]
> a[:][0]
> 
> Now both a[:][0] and a[0][:] are outputting the same result, i.e 
> np.array([1,2,3]). If I want to extract the array [[1],[4],[7]] then 
> what should I do? Is it possible to add this feature? If so, which 
> file(s) should I edit?
> 
> I know C,C++ and Python programming and I am new to open-source software 
> development. Please help me.
> 
Try:

In [2]: import numpy as np

In [3]: a = np.array([[1,2,3],[4,5,6],[7,8,9]])

In [4]: a[0][:]
Out[4]: array([1, 2, 3])

In [5]: a[:][0]
Out[5]: array([1, 2, 3])

In [6]: a[0,:]
Out[6]: array([1, 2, 3])

In [7]: a[:,0]
Out[7]: array([1, 4, 7])

Syntax is different for numpy arrays.
Regards,
	Juan
suchith | 9 Oct 07:42 2014
Picon

Extracting individual columns in Numpy

How to extract individual columns from a numpy array?
For example, consider this script

import numpy as np
a = np.array([[1,2,3],[4,5,6],[7,8,9]])
a[0][:]
a[:][0]

Now both a[:][0] and a[0][:] are outputting the same result, i.e 
np.array([1,2,3]). If I want to extract the array [[1],[4],[7]] then 
what should I do? Is it possible to add this feature? If so, which 
file(s) should I edit?

I know C,C++ and Python programming and I am new to open-source software 
development. Please help me.

I have attached the screenshot with this mail.

Thanks
Suchith.J.N

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Travis Oliphant | 8 Oct 01:32 2014

Re: Copyright status of NumPy binaries on Windows/OS X

Hey Andrew, 

You can use any of the binaries from Anaconda and redistribute them as long as you "cite" Anaconda --- i.e. tell your users that they are using Anaconda-derived binaries.     The Anaconda binaries link against ATLAS.    

The binaries are all at http://repo.continuum.io/pkgs/ 

In case you weren't aware: 

Another way you can build and distribute an "application" is to build a 'conda' meta-package which lists all the dependencies.   If you add to this meta-package 1) an icon and 2) an entry-point, then your application will automatically show up in the "Anaconda Launcher" (see this blog-post:  http://www.continuum.io/blog/new-launcher ) and anyone with the Anaconda Launcher app can install/update your package by clicking on the icon next to it.  

Users can also install your package with conda install or using the conda-gui.  

Best,

-Travis


On Mon, Oct 6, 2014 at 11:54 AM, Andrew Collette <andrew.collette <at> gmail.com> wrote:
Hi all,

I am working with the HDF Group on a new open-source viewer program
for HDF5 files, powered by NumPy, h5py, and wxPython.  On Windows,
since people don't typically have Python installed, we are looking to
distribute the application using PyInstaller, which embeds
dependencies like NumPy.  Likewise for OS X (using Py2App).

We would like to make sure we don't accidentally include
non-open-source components... I recall there was some discussion here
about using the Intel math libraries for binary releases on various
platforms.  Do the releases on SourceForge or PyPI use any proprietary
code?  We'd like to avoid building NumPy ourselves if we can avoid it.

Apologies if this is explained somewhere, but I couldn't find it.

Thanks!
Andrew Collette
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



--

Travis Oliphant
CEO
Continuum Analytics, Inc.
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Julian Taylor | 7 Oct 23:13 2014

Re: Copyright status of NumPy binaries on Windows/OS X

On 06.10.2014 18:54, Andrew Collette wrote:
> Hi all,
> 
> I am working with the HDF Group on a new open-source viewer program
> for HDF5 files, powered by NumPy, h5py, and wxPython.  On Windows,
> since people don't typically have Python installed, we are looking to
> distribute the application using PyInstaller, which embeds
> dependencies like NumPy.  Likewise for OS X (using Py2App).
> 
> We would like to make sure we don't accidentally include
> non-open-source components... I recall there was some discussion here
> about using the Intel math libraries for binary releases on various
> platforms.  Do the releases on SourceForge or PyPI use any proprietary
> code?  We'd like to avoid building NumPy ourselves if we can avoid it.
> 
> Apologies if this is explained somewhere, but I couldn't find it.
> 
> Thanks!
> Andrew Collette

Hi,
the numpy win32 binaries on sourceforge do not contain any proprietary
code. They are build with mingw 3.4.5 and are using a f2c'd version of
netlib blas and lapack which so far I know is public domain.
I think the macos wheels on pypi are built using ATLAS but they do also
contain libquadmath which is LGPL licensed. Its probably pulled in by
fortran (could maybe be removed by a rebuild as neither blas nor numpy
use it)

There are also unofficial win64 binaries floating around, I don't know
what they are using, but its possible they contain MKL, you need to
check with who is building these (Christoph Gohlke I think).

Cheers,
Julian

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Gmane