Discussion of Scientific Python

headers Jacob Biesinger | 1 Aug 2010 02:48 · /arff/

_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

On Sat, Jul 31, 2010 at 8:48 PM, Jacob Biesinger
<jake.biesinger <at> gmail.com> wrote:
> Hi!
> Perhaps I'm using the module incorrectly, but it looks like the x parameter
> in scipy.stats.hypergeom is off by one.  Specifically, I think it's
> one-too-high.
> From the wikipedia article
> http://en.wikipedia.org/wiki/Hypergeometric_distribution#Application_and_example (I
> know they could be wrong-- just hear me out on this),
> scipy.stats.hypergeom?
> Hypergeometric distribution
>
>        Models drawing objects from a bin.
>        M is total number of objects, n is total number of Type I objects.
>        RV counts number of Type I objects in N drawn without replacement
> from
>        population.
> So translating wikipedia's example...
> Pr(x=4; M=50, n=5, N=10)  = (choose(5,4) * choose(50-5, 10-4)) /
> choose(50,10) = .003964583
> Pr(x=5; M=50, n=5, N=10)  = (choose(5,5) * choose(50-5, 10-5)) /
> choose(50,10) = .0001189375
> Which you can check with the python code:
> from scipy import comb as chse   # "combination" => choose
>
> float((chse(5,4, exact=1) * chse(50-5,10-4, exact=1))) / chse(50,10,exact=1)
>  # example one
> 0.0039645830580150654155  # okay!
> float((chse(5,5, exact=1) * chse(50-5,10-5, exact=1))) / chse(50,10,exact=1)
> # example two
> 0.00011893749174045196247  # okay!
> Try example one with scipy.stats.hypergeom:
> # scipy.stats.hypergeom.sf(x, M, n, N)
> scipy.stats.hypergeom.sf(4,50,5,10)
> 0.00011893749169422652     # correct value for x=5, not x=4
> scipy.stats.hypergeom.sf(5,50,5,10)
> -4.6185277824406512e-14    # wrong
> It seems that changing the loc value from =0 (default) to =1 fixes the
> issue...
> scipy.stats.hypergeom.sf(4,50,5,10, loc=1)
> 0.0040835205497095073    # close enough
> scipy.stats.hypergeom.sf(5,50,5,10, loc=1)
> 0.00011893749169422652   # okay!
> Am I using the package wrong?

I don't know why you are using the survival function.

>From some quick examples, hypergeom looks ok:

pmf has identical results to the Wikipedia example you referenced

>>> stats.hypergeom.pmf([4,5],50,5,10)
array([ 0.00396458,  0.00011894])
>>> stats.hypergeom.pmf(4,50,5,10)
0.0039645830580151411
>>> stats.hypergeom.pmf(5,50,5,10)
0.00011893749174045286

consistency of pmf, cdf, sf

>>> stats.hypergeom.pmf(np.arange(5),50,5,10).sum()
0.9998810625082829
>>> stats.hypergeom.cdf(4,50,5,10)
0.9998810625082829
>>> stats.hypergeom.sf(4,50,5,10)
0.00011893749171709711
>>> 1-stats.hypergeom.pmf(np.arange(5),50,5,10).sum()
0.00011893749171709711

I'm always glad when someone verifies the numbers in scipy stats and
appreciate any reports of inconsistencies or bugs. If there are
differences in the parameterization, then we should make sure that
they are sufficiently documented.
Most distributions are internally tested or against numpy.random and
there are very few tests with other packages.

Thanks,

Josef

> --
> Jake Biesinger
> Graduate Student
> Xie Lab, UC Irvine
> (949) 231-7587
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

On Sun, Aug 1, 2010 at 1:16 AM, Jacob Biesinger
<jake.biesinger <at> gmail.com> wrote:
> I have seen the error of my ways and apologize for the noise in the list-- I
> should have been using the probability mass function instead of the survival
> function.  Doing the summation manually shows that all is well:
> def hypergeom(x, fgTotal, bgMatching, bgTotal):
>         return scipy.comb(fgTotal,x) *
> scipy.comb(bgTotal-fgTotal,bgMatching-x) / scipy.comb(bgTotal,bgMatching)
>>>> scipy.stats.hypergeom.sf(4,50,5,10, loc=1)
> 0.0040835205497095073
>>>> sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in
>>>> range(4,min(10+1,50+1)))
> 0.0040835205497557125
>>>> sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(4, min(10+1, 50+1))])
> 0.0040835205497557186
>
>>>> scipy.stats.hypergeom.sf(5,50,5,10, loc=1)
> 0.00011893749169422652
>>>> sum(scipy.stats.hypergeom.pmf(x,50,5,10) for x in
>>>> range(5,min(10+1,50+1)))
> 0.00011893749174045688
>>>> sum([hypergeom(x, 10, 5, 50) for x in scipy.arange(5, min(10+1, 50+1))])
> 0.00011893749174045679

Note that the distribution methods are vectorized. You can use an
array_like for x instead of the list comprehension.

Josef

>
> Thanks anyway!
> --
> Jake Biesinger
> Graduate Student
> Xie Lab, UC Irvine
> (949) 231-7587
>
>
> On Sat, Jul 31, 2010 at 5:48 PM, Jacob Biesinger <jake.biesinger <at> gmail.com>
> wrote:
>>
>> Hi!
>> Perhaps I'm using the module incorrectly, but it looks like the x
>> parameter in scipy.stats.hypergeom is off by one.  Specifically, I think
>> it's one-too-high.
>> From the wikipedia article
>> http://en.wikipedia.org/wiki/Hypergeometric_distribution#Application_and_example (I
>> know they could be wrong-- just hear me out on this),
>> scipy.stats.hypergeom?
>> Hypergeometric distribution
>>
>>        Models drawing objects from a bin.
>>        M is total number of objects, n is total number of Type I objects.
>>        RV counts number of Type I objects in N drawn without replacement
>> from
>>        population.
>> So translating wikipedia's example...
>> Pr(x=4; M=50, n=5, N=10)  = (choose(5,4) * choose(50-5, 10-4)) /
>> choose(50,10) = .003964583
>> Pr(x=5; M=50, n=5, N=10)  = (choose(5,5) * choose(50-5, 10-5)) /
>> choose(50,10) = .0001189375
>> Which you can check with the python code:
>> from scipy import comb as chse   # "combination" => choose
>>
>> float((chse(5,4, exact=1) * chse(50-5,10-4, exact=1))) /
>> chse(50,10,exact=1)  # example one
>> 0.0039645830580150654155  # okay!
>> float((chse(5,5, exact=1) * chse(50-5,10-5, exact=1))) /
>> chse(50,10,exact=1) # example two
>> 0.00011893749174045196247  # okay!
>> Try example one with scipy.stats.hypergeom:
>> # scipy.stats.hypergeom.sf(x, M, n, N)
>> scipy.stats.hypergeom.sf(4,50,5,10)
>> 0.00011893749169422652     # correct value for x=5, not x=4
>> scipy.stats.hypergeom.sf(5,50,5,10)
>> -4.6185277824406512e-14    # wrong
>> It seems that changing the loc value from =0 (default) to =1 fixes the
>> issue...
>> scipy.stats.hypergeom.sf(4,50,5,10, loc=1)
>> 0.0040835205497095073    # close enough
>> scipy.stats.hypergeom.sf(5,50,5,10, loc=1)
>> 0.00011893749169422652   # okay!
>> Am I using the package wrong?
>> --
>> Jake Biesinger
>> Graduate Student
>> Xie Lab, UC Irvine
>> (949) 231-7587
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

I'm happy to announce the first official release of ALGOPY in version 0.2.1.

Rationale:
~~~~~~~~
The purpose of ALGOPY is the evaluation of higher-order derivatives in
the forward and reverse mode of Algorithmic Differentiation (AD) using
univariate Taylor polynomial arithmetic. Particular focus are
functions that contain numerical linear algebra functions (e.g. inv,
dot, eigh, qr, cholesky,...) as they often appear in statistically
motivated functions.

Download:
~~~~~~~~~
http://pypi.python.org/pypi/algopy/0.2.1
or bleeding edge versions from from http://github.com/b45ch1/algopy

Documentation:
~~~~~~~~~~~~
available at http://packages.python.org/algopy/

OS Support:
~~~~~~~~~~
Linux, Windows (tested with pythonxy), should also work on Mac

Software Dependencies:
~~~~~~~~~~~~~~~~~~~~
for the core: numpy, scipy
for testing:  nose

Exampe Session:
~~~~~~~~~~~~~
Consider the contrived example where it is the goal to compute the
directional derivative df/dx_1 :

>>> import numpy; from numpy import log, exp, sin, cos
>>> import algopy; from algopy import UTPM, dot, inv, zeros
>>>
>>> def f(x):
...     A = zeros((2,2),dtype=x)
...     A[0,0] = numpy.log(x[0]*x[1])
...     A[0,1] = numpy.log(x[1]) + exp(x[0])
...     A[1,0] = sin(x[1])**2 + cos(x[0])**3.1
...     A[1,1] = x[0]**cos(x[1])
...     return log( dot(x.T,  dot( inv(A), x)))
...
>>>
>>> x = UTPM(zeros((2,1,2),dtype=float))
>>> x.data[0,0] = [1,2]
>>> x.data[1,0] = [1,0]
>>> y = f(x)
>>>
>>> print 'normal function evaluation f(x) = ',y.data[0,0]
normal function evaluation f(x) =  0.641250189986
>>> print 'directional derivative df/dx1 = ',y.data[1,0]
directional derivative df/dx1 =  1.62982340133

We are pleased to announce the release of PySAL 1.0.0.

PySAL is a library of tools for spatial data analysis written in Python.  This
release comes after an extensive integration and refactoring effort to combine
common analytical functionality in the projects STARS and PySpace into a shared
open source library for geospatial analysis and geocomputation. For more
information, please see the release notes that follow below.

You can download PySAL from here:
http://code.google.com/p/pysal/downloads/list
Python 2.6 binaries for Windows and OS X are available, as well as source
tarballs for other platforms and documentation in html form.

Enjoy,

The PySAL developers

=================================================
Python Spatial Analysis Library
=================================================

.. Contents::

What is PySAL
--------------

PySAL modules
+++++++++++++

 * pysal.core — Core Data Structures and IO
 * pysal.cg — Computational Geometry
 * pysal.esda — Exploratory Spatial Data Analysis
 * pysal.inequality — Spatial Inequality Analysis
 * pysal.spatial_dynamics — Spatial Dynamics
 * pysal.region — Spatially constrained clustering
 * pysal.weights — Spatial Weights
 * pysal.FileIO — PySAL FileIO: Module for reading and writing
                  various file types in a Pythonic way

Documentation
-------------

The documentation site is here
 http://pysal.org/contents.html

Web sites
---------

PySAL's home is here
 http://pysal.org/

The developer's site is here
 http://code.google.com/p/pysal/

Mailing Lists
-------------

Please see the developer's list here
 http://groups.google.com/group/pysal-dev

Help for users is here
 http://groups.google.com/group/openspace-list

Bug reports
-----------

To search for or report bugs, please see
 http://code.google.com/p/pysal/issues/list

License information
-------------------

See the file "LICENSE.txt" for information on the history of this
software, terms & conditions for usage, and a DISCLAIMER OF ALL
WARRANTIES.

--

-- 
Sergio (Serge) Rey
Professor, School of Geographical Sciences and Urban Planning
Arizona State University
http://geoplan.asu.edu/rey

Editor, International Regional Science Review
http://irx.sagepub.com
_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

Here is the last message from trying to install scipy from
sources.(Numpy installed fine)

creating /usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/arff/myfunctools.py ->
/usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/arff/utils.py ->
/usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/arff/__init__.py ->
/usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/arff/setup.py ->
/usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/arff/arffread.py ->
/usr/lib/python2.5/site-packages/scipy/io/arff
copying build/lib.linux-i686-2.5/scipy/io/recaster.py ->
/usr/lib/python2.5/site-packages/scipy/io
error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such
file or directory

I am wondering what causes the last line error?

error: /usr/lib/python2.5/site-packages/scipy/io/recaster.py: No such
file or directory.

Regards,
ernie

_______________________________________________
SciPy-User mailing list
SciPy-User <at> scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

Shapely indeed looks like exactly what I was searching for! Thanks! I
love Python; I'm able to do stuff after a couple of months which I
couldn't figure out how to do in IDL after years of usage. :-D

/Emil

On Mon, 2010-07-26 at 14:28 -0500, Joe Kington wrote:
> On Mon, Jul 26, 2010 at 1:26 PM, Zachary Pincus
> <zachary.pincus <at> yale.edu> wrote:
>         > I am working on a project where I am defining some regions
>         of
>         > interest.
>         > I have a 2200x2200 px 2D Array in which my ROI is defined by
>         a
>         > polygon.
>         > However, my data are smoothed by a gaussian kernel of width
>         300px,
>         > and I
>         > would like to draw some lines indicating this inner 150px
>         distance to
>         > the borders of the ROI. I cannot come up with any way to do
>         this, does
>         > anyone have an idea?
>         
>         
>         Two broad options spring to mind:
>         (1) Geometric -- shrink the polygon along the normals to the
>         vertices.
>         [Oh, I see that eat has given pseudocode for same... good]
>         (2) Gridded -- rasterize the polygon to a binary mask (no
>         tools for
>         this in scipy, I fear... but if you're handy with opengl or
>         something,
>         that's not too hard), and then use scipy.ndimage to erode or
>         dilate
>         the mask as necessary.
>         
>         Zach
>         
>         _______________________________________________
>         SciPy-User mailing list
>         SciPy-User <at> scipy.org
>         http://mail.scipy.org/mailman/listinfo/scipy-user
>         
> 
> If you go with a purely geometric method, look into a module like
> shapely to buffer the polygon. (similar to the code snipped eat
> posted, but more flexible) 
> 
> See this example for a general idea.  
> 
> -Joe
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

50	May 2013
131	April 2013
137	March 2013
185	February 2013
113	January 2013
183	December 2012
170	November 2012
283	October 2012
353	September 2012
213	August 2012
217	July 2012
160	June 2012
145	May 2012
149	April 2012
320	March 2012
286	February 2012
114	January 2012
131	December 2011
214	November 2011
183	October 2011
259	September 2011
275	August 2011
249	July 2011
297	June 2011
332	May 2011
364	April 2011
246	March 2011
283	February 2011
328	January 2011
246	December 2010
483	November 2010
454	October 2010
216	September 2010
292	August 2010
320	July 2010
417	June 2010
310	May 2010
263	April 2010
458	March 2010
415	February 2010
263	January 2010
244	December 2009
415	November 2009
368	October 2009
299	September 2009
447	August 2009
314	July 2009
345	June 2009
442	May 2009
321	April 2009
432	March 2009
450	February 2009
479	January 2009
318	December 2008
292	November 2008
341	October 2008
228	September 2008
264	August 2008
358	July 2008
402	June 2008
324	May 2008
604	April 2008
337	March 2008
338	February 2008
398	January 2008
300	December 2007
379	November 2007
431	October 2007
322	September 2007
378	August 2007
351	July 2007
348	June 2007
475	May 2007
405	April 2007
378	March 2007
313	February 2007
504	January 2007
225	December 2006
361	November 2006
314	October 2006
339	September 2006
302	August 2006
322	July 2006
302	June 2006
329	May 2006
266	April 2006
323	March 2006
399	February 2006
395	January 2006
424	December 2005
291	November 2005
379	October 2005
154	September 2005
235	August 2005
263	July 2005
155	June 2005
133	May 2005
124	April 2005
136	March 2005
77	February 2005
132	January 2005
41	December 2004
297	November 2004
229	October 2004
128	September 2004
167	August 2004
82	July 2004
83	June 2004
117	May 2004
88	April 2004
36	March 2004
124	February 2004
73	January 2004
47	December 2003
73	November 2003
147	October 2003
193	September 2003
116	August 2003
77	July 2003
71	June 2003
154	May 2003
41	April 2003
139	March 2003
69	February 2003
103	January 2003
115	December 2002
129	November 2002
182	October 2002
83	September 2002
86	August 2002
38	July 2002
32	June 2002
97	May 2002
1	January 2002

Off by one bug in Scipy.stats.hypergeom

Re: Off by one bug in Scipy.stats.hypergeom

Re: Off by one bug in Scipy.stats.hypergeom

Re: Off by one bug in Scipy.stats.hypergeom

Re: Off by one bug in Scipy.stats.hypergeom

[ANN] ALGOPY 0.21, algorithmic differentiation in Python

ANN: PySAL 1.0.0

Installing scipy 0.8.0 from sources. (where is recaster.py?)

Re: Installing scipy 0.8.0 from sources. (where is recaster.py?)

Re: Points with given distance to a polygon

Mon	Tue	Wed	Thu	Fri	Sat	Sun

						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31