Valentin Haenel | 6 Jul 20:35 2015
Picon

webinterface down

Hi,

the webinterface at:

http://mail.scipy.org/mailman/listinfo/numpy-discussion

seems down for me. For anyone else too?

V-
Francesc Alted | 6 Jul 17:18 2015
Picon

Question about unaligned access

Hi,

I have stumbled into this:

In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int32)])

In [63]: %timeit sa['f0'].sum()
100 loops, best of 3: 4.52 ms per loop

In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)), dtype=[('f0', np.int64), ('f1', np.int64)])

In [65]: %timeit sa['f0'].sum()
1000 loops, best of 3: 896 µs per loop

The first structured array is made of 12-byte records, while the second is made by 16-byte records, but the latter performs 5x faster.  Also, using an structured array that is made of 8-byte records is the fastest (expected):

In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)), dtype=[('f0', np.int64)])

In [67]: %timeit sa['f0'].sum()
1000 loops, best of 3: 567 µs per loop

Now, my laptop has a Ivy Bridge processor (i5-3380M) that should perform quite well on unaligned data:


So, if 4 years-old Intel architectures do not have a penalty for unaligned access, why I am seeing that in NumPy?  That strikes like a quite strange thing to me.

Thanks,
Francesc

--
Francesc Alted
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Ralf Gommers | 4 Jul 23:40 2015
Picon

ANN: Scipy 0.16.0 release candidate 1

Hi,

I'm pleased to announce the availability of the first release candidate of Scipy 0.16.0. Please try it out and report any issues on the Github issue tracker or on the scipy-dev mailing list.

This first RC is a source-only release. Sources and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.16.0rc1. Note that this is a bit of an experiment - it's the first time we use the Github Releases feature. Feedback welcome.

Thanks to everyone who contributed to this release!

Ralf



========================== SciPy 0.16.0 Release Notes ========================== .. note:: Scipy 0.16.0 is not released yet! .. contents:: SciPy 0.16.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.15.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.6.2 or greater. Highlights of this release include: - A Cython API for BLAS/LAPACK in `scipy.linalg` - A new benchmark suite. It's now straightforward to add new benchmarks, and they're routinely included with performance enhancement PRs. - Support for the second order sections (SOS) format in `scipy.signal`. New features ============ Benchmark suite --------------- The benchmark suite has switched to using `Airspeed Velocity <http://spacetelescope.github.io/asv/>`__ for benchmarking. You can run the suite locally via ``python runtests.py --bench``. For more details, see ``benchmarks/README.rst``. `scipy.linalg` improvements --------------------------- A full set of Cython wrappers for BLAS and LAPACK has been added in the modules `scipy.linalg.cython_blas` and `scipy.linalg.cython_lapack`. In Cython, these wrappers can now be cimported from their corresponding modules and used without linking directly against BLAS or LAPACK. The functions `scipy.linalg.qr_delete`, `scipy.linalg.qr_insert` and `scipy.linalg.qr_update` for updating QR decompositions were added. The function `scipy.linalg.solve_circulant` solves a linear system with a circulant coefficient matrix. The function `scipy.linalg.invpascal` computes the inverse of a Pascal matrix. The function `scipy.linalg.solve_toeplitz`, a Levinson-Durbin Toeplitz solver, was added. Added wrapper for potentially useful LAPACK function ``*lasd4``. It computes the square root of the i-th updated eigenvalue of a positive symmetric rank-one modification to a positive diagonal matrix. See its LAPACK documentation and unit tests for it to get more info. Added two extra wrappers for LAPACK least-square solvers. Namely, they are ``*gelsd`` and ``*gelsy``. Wrappers for the LAPACK ``*lange`` functions, which calculate various matrix norms, were added. Wrappers for ``*gtsv`` and ``*ptsv``, which solve ``A*X = B`` for tri-diagonal matrix ``A``, were added. `scipy.signal` improvements --------------------------- Support for second order sections (SOS) as a format for IIR filters was added. The new functions are: * `scipy.signal.sosfilt` * `scipy.signal.sosfilt_zi`, * `scipy.signal.sos2tf` * `scipy.signal.sos2zpk` * `scipy.signal.tf2sos` * `scipy.signal.zpk2sos`. Additionally, the filter design functions `iirdesign`, `iirfilter`, `butter`, `cheby1`, `cheby2`, `ellip`, and `bessel` can return the filter in the SOS format. The function `scipy.signal.place_poles`, which provides two methods to place poles for linear systems, was added. The option to use Gustafsson's method for choosing the initial conditions of the forward and backward passes was added to `scipy.signal.filtfilt`. New classes ``TransferFunction``, ``StateSpace`` and ``ZerosPolesGain`` were added. These classes are now returned when instantiating `scipy.signal.lti`. Conversion between those classes can be done explicitly now. An exponential (Poisson) window was added as `scipy.signal.exponential`, and a Tukey window was added as `scipy.signal.tukey`. The function for computing digital filter group delay was added as `scipy.signal.group_delay`. The functionality for spectral analysis and spectral density estimation has been significantly improved: `scipy.signal.welch` became ~8x faster and the functions `scipy.signal.spectrogram`, `scipy.signal.coherence` and `scipy.signal.csd` (cross-spectral density) were added. `scipy.signal.lsim` was rewritten - all known issues are fixed, so this function can now be used instead of ``lsim2``; ``lsim`` is orders of magnitude faster than ``lsim2`` in most cases. `scipy.sparse` improvements --------------------------- The function `scipy.sparse.norm`, which computes sparse matrix norms, was added. The function `scipy.sparse.random`, which allows to draw random variates from an arbitrary distribution, was added. `scipy.spatial` improvements ---------------------------- `scipy.spatial.cKDTree` has seen a major rewrite, which improved the performance of the ``query`` method significantly, added support for parallel queries, pickling, and options that affect the tree layout. See pull request 4374 for more details. The function `scipy.spatial.procrustes` for Procrustes analysis (statistical shape analysis) was added. `scipy.stats` improvements -------------------------- The Wishart distribution and its inverse have been added, as `scipy.stats.wishart` and `scipy.stats.invwishart`. The Exponentially Modified Normal distribution has been added as `scipy.stats.exponnorm`. The Generalized Normal distribution has been added as `scipy.stats.gennorm`. All distributions now contain a ``random_state`` property and allow specifying a specific ``numpy.random.RandomState`` random number generator when generating random variates. Many statistical tests and other `scipy.stats` functions that have multiple return values now return ``namedtuples``. See pull request 4709 for details. `scipy.optimize` improvements ----------------------------- A new derivative-free method DF-SANE has been added to the nonlinear equation system solving function `scipy.optimize.root`. Deprecated features =================== ``scipy.stats.pdf_fromgamma`` is deprecated. This function was undocumented, untested and rarely used. Statsmodels provides equivalent functionality with ``statsmodels.distributions.ExpandedNormal``. ``scipy.stats.fastsort`` is deprecated. This function is unnecessary, ``numpy.argsort`` can be used instead. ``scipy.stats.signaltonoise`` and ``scipy.stats.mstats.signaltonoise`` are deprecated. These functions did not belong in ``scipy.stats`` and are rarely used. See issue #609 for details. ``scipy.stats.histogram2`` is deprecated. This function is unnecessary, ``numpy.histogram2d`` can be used instead. Backwards incompatible changes ============================== The deprecated global optimizer ``scipy.optimize.anneal`` was removed. The following deprecated modules have been removed: ``scipy.lib.blas``, ``scipy.lib.lapack``, ``scipy.linalg.cblas``, ``scipy.linalg.fblas``, ``scipy.linalg.clapack``, ``scipy.linalg.flapack``. They had been deprecated since Scipy 0.12.0, the functionality should be accessed as `scipy.linalg.blas` and `scipy.linalg.lapack`. The deprecated function ``scipy.special.all_mat`` has been removed. The deprecated functions ``fprob``, ``ksprob``, ``zprob``, ``randwcdf`` and ``randwppf`` have been removed from `scipy.stats`. Other changes ============= The version numbering for development builds has been updated to comply with PEP 440. Building with ``python setup.py develop`` is now supported.
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith | 4 Jul 01:47 2015
Picon

Readings about governance and decision making for F/OSS projects

Hi all,

As discussed on the call yesterday, here's some links to background
reading on F/OSS project governance, to use as background reading for
our discussion Tuesday. There's a lot out there, but here are the two
I think are the most important/relevant:

The classic book on running a F/OSS project is Karl Fogel's "Producing
OSS", which is available free online. Chapter 4 is the relevant one
for this topic (though the whole book is worth reading):
  http://producingoss.com/en/producingoss.html#social-infrastructure

The IPython governance documents are here:
  https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance

I think it will give us a really good start on Tuesday if everyone is
able to find the time to read Chapter 4 + IPEP 29 ahead of time.
Something to do on the plane, maybe :-).

---

For extra credit, some less crucial links that still might be
interesting (and give something of a sense of the range of options):

The Apache voting system:
  http://www.apache.org/foundation/voting.html

The GNOME foundation's governance documents:
  https://www.gnome.org/foundation/governance/

The Debian constitution, esp. section 6 (the technical committee):
  https://www.debian.org/devel/constitution
  https://www.debian.org/devel/tech-ctte

The GCC steering committee:
  https://www.gnu.org/software/gcc/steering.html

The Node.JS foundation (interesting case -- a foundation in the
process of bootstrapping in order to resolve a fork triggered by
conflict between outside contributors and the company that started the
project):
  https://nodejs.org/foundation/

Jono Bacon's book "The Art of Community", also available free online:
  http://artofcommunityonline.org/Art_of_Community_Second_Edition.pdf
The governance chapter here is much more oriented towards Ubuntu-sized
projects with hundreds-to-thousands of contributors, so not as
relevant for us, but it still contains lots of interesting stuff.

If you have other links on this topic that you are think are
interesting, please add them to the thread!

-n

--

-- 
Nathaniel J. Smith -- http://vorpus.org
James A. Bednar | 3 Jul 22:10 2015
Picon
Picon

ANN: HoloViews 1.3 released

We are pleased to announce the fourth public release of HoloViews,
a Python package for simplifying the exploration of scientific data:

  http://holoviews.org

HoloViews provides composable, sliceable, declarative data
structures for building even complex visualizations easily.
The goal of HoloViews is to let your data just visualize itself,
allowing you to work with large datasets as easily as you work
with simple datatypes at the Python prompt.

You can obtain the new version using conda or pip:

  conda install holoviews

  pip install --upgrade 'holoviews[recommended]'

This release includes a substantial number of new features and
API improvements, most of which have been suggested by our growing
userbase:

- Major optimizations throughout, both for working with HoloViews
  data structures and for visualization.

- Improved widget appearance and greatly reduced flickering
  issues when interactively exploring data in the browser.

- Improved handling of unicode and LaTeX text throughout,
  using Python 3's better unicode support (when available).

- New Polygons, ErrorBars, and Spread Element types.

- Support for multiple matplotlib backends (vanilla matplotlib, mpld3
  and nbagg) with support for other plotting systems (such as Bokeh)
  in development. Easily switching between backends allows you to take
  advantage of the unique features of each one, such as good SVG/PDF
  output, interactive zooming and panning, or 3D viewpoint control.

- Streamlined the API based on user feedback; now even more things
  "just work". This includes new, easy to use constructors for
  common Element types as well as easy conversion between them.

- More customizability of plot and style options, including easier
  control over font sizes, legend positions, background color, and
  multiple color bars. Polar projections now supported throughout.

- More flexible and customizable Layouts, allowing the user to
  define blank spaces (using the Empty object) as well as more
  control over positioning and aspect ratios.

- Support for a holoviews.rc file, integration with IPython Notebook
  interact widgets, improvements to the Pandas interface, easy
  saving and loading of data via pickling, and much more.

And of course we have fixed a number of bugs found by our very
dedicated users; please keep filing Github issues if you find any!

For the full list of changes, see:

  https://github.com/ioam/holoviews/releases

HoloViews remains freely available under a BSD license, is Python 2
and 3 compatible, and has minimal external dependencies, making it
easy to integrate into your workflow. Try out the extensive tutorials
at holoviews.org today, and check out our upcoming SciPy and EuroSciPy
talks in Austin and Cambridge (or read the paper at http://goo.gl/NH9FTB)!

Philipp Rudiger
Jean-Luc R. Stevens
James A. Bednar

The University of Edinburgh
School of Informatics

--

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
Bachir Aoun | 2 Jul 16:37 2015

Biased random number generator

Dear All, 

I have recently developed some features that I think can be useful for others.
I would like to contribute by providing the code of the following definitions

BiasedRandomFloatGenerator
BiasedRandomIntegerGenerator

please find the help of those two classes here
http://bachiraoun.github.io/fullrmc/fullrmc.Core.html#module-fullrmc.Core.Collection

Personally, I am using those generators to model molecular structures by reverse engineering experimental data.  The generators accumulate experience through the whole modelling process and automatically update the generation scheme (numbers probability) according to some success / failure parameter. 

If you think this is something that might be interesting and has the potential to being distributed in the coming Numpy versions please let me know.

regards

--
Bachir AOUN
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
josef.pktd | 1 Jul 16:05 2015
Picon

floats for indexing, reshape - too strict ?

About the deprecation warning for using another type than integers, in ones, reshape, indexing and so on:

Wouldn't it be nicer to accept floats that are equal to an integer?

for example

>>> 5.0 == 5
True

>>> np.ones(10 / 2)
array([ 1.,  1.,  1.,  1.,  1.])
>>> 10 / 2 == 5
True

or the python 2 version

>>> np.ones(10. / 2)
array([ 1.,  1.,  1.,  1.,  1.])
>>> 10. / 2 == 5
True

I'm using now 10 // 2, or int(10./2 + 1)   but this is unconditional and doesn't raise if the numbers are not close or equal to an integer (which would be a bug)


Josef


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
josef.pktd | 1 Jul 05:58 2015
Picon

annoying Deprecation warnings about non-integers

I'm trying to fix some code in statsmodels that creates Deprecation Warnings from numpy

Most of it are quite easy to fix, mainly cases where we use floats to avoid integer division

I have two problems

first, I get Deprecation warnings in the test run that don't specify where they happen.
I try to find them with file searches, but I don't see a `np.ones` that might cause a problem 
(needle in a haystack: Close to 4000 unittests and more than 100,000 lines of numpython)
Also, I'm not sure the warnings are only from statsmodels, they could be in numpy, scipy or pandas, couldn't they?


second, what's wrong with non-integers in `np.r_[[np.nan] * head, x, [np.nan] * tail]` (see below)

I tried to set the warnings filter to `error` but then Python itself errored right away.



Thanks for any clues

Josef


>nosetests  -s --pdb-failures --pdb "M:\j\statsmodels\statsmodels_py34\statsmodels\tsa\tests"

..................C:\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\sit
e-packages\numpy\core\numeric.py:183: DeprecationWarning: using a non-integer nu
mber instead of an integer will result in an error in the future
  a = empty(shape, dtype, order)
..........


.......M:\j\statsmodels\stat
smodels_py34\statsmodels\tsa\filters\filtertools.py:28: DeprecationWarning: usin
g a non-integer number instead of an integer will result in an error in the futu
re
  return np.r_[[np.nan] * head, x, [np.nan] * tail]
..........................


...................C:\WinPython-64bit-3.4.3.1
\python-3.4.3.amd64\lib\site-packages\numpy\lib\twodim_base.py:231: DeprecationW
arning: using a non-integer number instead of an integer will result in an error
 in the future
  m = zeros((N, M), dtype=dtype)
C:\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\numpy\l
ib\twodim_base.py:238: DeprecationWarning: using a non-integer number instead of
 an integer will result in an error in the future
  m[:M-k].flat[i::M+1] = 1
...........


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith | 26 Jun 11:32 2015
Picon

Video meeting this week

Hi all,

In a week and a half, this is happening:

    https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting

It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).

The obligatory doodle:
    http://doodle.com/6b4s6thqt9xt4vnh

Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).

-n

--

-- 
Nathaniel J. Smith -- http://vorpus.org
Charles R Harris | 20 Jun 22:40 2015
Picon

Removal of Deprecated Keywords/functionality

Hi All,

There are three long ago deprecations that I am not sure how to handle.

  • keywords skiprows and missing in genfromtxt, deprecated in 1.5.
  • keyword old_behavior (default False) in correlate. added in 1.5 at least, but default value changed later.

The documentation says they will be removed in numpy 2.0, but we might want to try ealier. The case of the correlation function is trickier, as we probabaly need to provide a function with the old behavior before removing the keyword. I've left these cases as is, but the more old stuff hanging about the greater our technical debt.


Chuck

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Elliot Hallmark | 19 Jun 23:19 2015
Picon

I can't tell if Numpy is configured properly with show_config()

Debian Sid, 64-bit.  I was trying to fix the problem of np.dot running very slow.

I ended up uninstalling numpy, installing libatlas3-base through apt-get and re-installing numpy.  The performance of dot is greatly improved!  But I can't tell from any other method whether numpy is set up correctly.  Consider comparing the faster one to another in a virtual env that is still slow:

###
fast one
###

In [1]: import time, numpy

In [2]: n=1000

In [3]: A = numpy.random.rand(n,n)

In [4]: B = numpy.random.rand(n,n)

In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
0.306427001953

In [6]: numpy.show_config()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

###
slow one
###

In [1]: import time, numpy

In [2]: n=1000

In [3]: A = numpy.random.rand(n,n)

In [4]: B = numpy.random.rand(n,n)

In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
7.88430500031

In [6]: numpy.show_config()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

#####

Further, in the following comparison between Cpython and converting to numpy array for one operation, I get Cpython being faster by the same amount in both environments.  But another user got numpy being faster.

In [1]: import numpy as np In [2]: pts = range(100,1000) In [3]: pts[100] = 0 In [4]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr) 10000 loops, best of 3: 129 µs per loop In [5]: %timeit mini = sorted(enumerate(pts))[0][1] 10000 loops, best of 3: 89.2 µs per loop

The other user got

In [29]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr) 10000 loops, best of 3: 37.7 µs per loop In [30]: %timeit mini = sorted(enumerate(pts))[0][1] 10000 loops, best of 3: 69.2 µs per loop

And I can't help but wonder if there is further configuration I need to make numpy faster, or if this is just a difference between out machines
In the future, should I ignore show_config() and just do this dot product test?

Any guidance would be appreciated.

Thanks,
  Elliot
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Gmane