Sturla Molden | 5 Feb 2009 15:49
Picon
Gravatar

scipy.org is down again


Someone restart the server please?

S.M.
anand.prabhakar.patil | 5 Feb 2009 19:11
Picon
Gravatar

Distutils: incorrect inference of cpu type on Ubuntu Intrepid

Hi all,

I just installed numpy from the debian packages on a virtual machine in a 'cloud' to do some multiprocessing. The numpy version is 1.1.1, but the problem (I think) would happen with 1.3.0.dev6034 as well.

While trying to build a package that incorporates Fortran sources, I ran into the following:


gfortran:f77: mbgw/st_cov_fun/fst_cov_fun.f
f951: error: CPU you selected does not support x86-64 instruction set
f951: error: CPU you selected does not support x86-64 instruction set
f951: error: CPU you selected does not support x86-64 instruction set
f951: error: CPU you selected does not support x86-64 instruction set
error: Command "/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops -march=k6-2 -mmmx -msse2 -msse -msse3 -Ibuild/src.linux-x86_64-2.5 -I/usr/lib/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c -c mbgw/st_cov_fun/fst_cov_fun.f -o build/temp.linux-x86_64-2.5/mbgw/st_cov_fun/fst_cov_fun.o" failed with exit status 1

The problem seems to be the -march=k6-2 flag, which is not right because the chip is a 64-bit Opteron. The flag was set because the method numpy.distutils.cpuinfo.cpu._is_AthlonK6_2 returned True:

def _is_AthlonK6_2(self):
    return self._is_AMD() and self.info[0]['model'] == '2'

def _is_AMD(self):
    return self.info[0]['vendor_id']=='AuthenticAMD'

which is understandable given the machine's /proc/cpuinfo, which follows... but the chip is a 64-bit Opteron, as I said. Should I file a bug report?

Thanks,
Anand


0# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 2
model name      : Quad-Core AMD Opteron(tm) Processor 2352
stepping        : 3
cpu MHz         : 2109.267
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good pni monitor cx16 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips        : 4218.52
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
[...]
_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-dev
Pauli Virtanen | 5 Feb 2009 20:51
Picon
Picon
Favicon

Re: Distutils: incorrect inference of cpu type on Ubuntu Intrepid

Thu, 05 Feb 2009 18:11:00 +0000, anand.prabhakar.patil wrote:

> Hi all,
> 
> I just installed numpy from the debian packages on a virtual machine in
> a 'cloud' to do some multiprocessing. The numpy version is 1.1.1, but
> the problem (I think) would happen with 1.3.0.dev6034 as well.
> 
> While trying to build a package that incorporates Fortran sources, I ran
> into the following:
> 
> gfortran:f77: mbgw/st_cov_fun/fst_cov_fun.f f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set error: Command
> "/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3
> -funroll-loops -march=k6-2 -mmmx -msse2 -msse -msse3
> -Ibuild/src.linux-x86_64-2.5
> -I/usr/lib/python2.5/site-packages/numpy/core/include
> -I/usr/include/python2.5 -c -c mbgw/st_cov_fun/fst_cov_fun.f -o
> build/temp.linux-x86_64-2.5/mbgw/st_cov_fun/fst_cov_fun.o" failed with
> exit status 1

Seems like this is already addressed in SVN trunk:
http://scipy.org/scipy/numpy/changeset/5978

--

-- 
Pauli Virtanen
anand.prabhakar.patil | 5 Feb 2009 21:24
Picon
Gravatar

Re: Distutils: incorrect inference of cpu type on Ubuntu Intrepid

Thanks Pauli, what a relief. I'll give it a whirl.


Anand

On Thu, Feb 5, 2009 at 7:51 PM, Pauli Virtanen <pav <at> iki.fi> wrote:
Thu, 05 Feb 2009 18:11:00 +0000, anand.prabhakar.patil wrote:

> Hi all,
>
> I just installed numpy from the debian packages on a virtual machine in
> a 'cloud' to do some multiprocessing. The numpy version is 1.1.1, but
> the problem (I think) would happen with 1.3.0.dev6034 as well.
>
> While trying to build a package that incorporates Fortran sources, I ran
> into the following:
>
> gfortran:f77: mbgw/st_cov_fun/fst_cov_fun.f f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set f951: error: CPU you
> selected does not support x86-64 instruction set error: Command
> "/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3
> -funroll-loops -march=k6-2 -mmmx -msse2 -msse -msse3
> -Ibuild/src.linux-x86_64-2.5
> -I/usr/lib/python2.5/site-packages/numpy/core/include
> -I/usr/include/python2.5 -c -c mbgw/st_cov_fun/fst_cov_fun.f -o
> build/temp.linux-x86_64-2.5/mbgw/st_cov_fun/fst_cov_fun.o" failed with
> exit status 1

Seems like this is already addressed in SVN trunk:
http://scipy.org/scipy/numpy/changeset/5978

--
Pauli Virtanen

_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-dev

_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-dev
Joe Harrington | 5 Feb 2009 21:44
Picon

updating the numpy/scipy versions in Linux distros

The versions of our stuff about to go out in Ubuntu 9.04 are:

numpy 1.1.1
scipy 0.6.0

Would it be possible to get 1.2.1 and 0.7.0 to go instead?  Who takes
care of pushing these to the various Linux distros?

It would be nice if the Packaging section of Developer Zone on the web
site made the packaging process a little more transparent...  Say what
you do to get a release out and to get it in distribution, who does
it, what the timetables tend to look like, etc.  If you ask for help
there, you're likely to get it.

Thanks,

--jh--
Robert Kern | 5 Feb 2009 21:46
Picon
Gravatar

Re: updating the numpy/scipy versions in Linux distros

On Thu, Feb 5, 2009 at 14:44, Joe Harrington <jh <at> physics.ucf.edu> wrote:
> The versions of our stuff about to go out in Ubuntu 9.04 are:
>
> numpy 1.1.1
> scipy 0.6.0
>
> Would it be possible to get 1.2.1 and 0.7.0 to go instead?  Who takes
> care of pushing these to the various Linux distros?
>
> It would be nice if the Packaging section of Developer Zone on the web
> site made the packaging process a little more transparent...  Say what
> you do to get a release out and to get it in distribution, who does
> it, what the timetables tend to look like, etc.  If you ask for help
> there, you're likely to get it.

My experience is that most distribution packagers aren't on this list.
We don't push; they pull.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
Michael Abshoff | 5 Feb 2009 21:53

Re: updating the numpy/scipy versions in Linux distros

Robert Kern wrote:
> On Thu, Feb 5, 2009 at 14:44, Joe Harrington <jh <at> physics.ucf.edu> wrote:
>> The versions of our stuff about to go out in Ubuntu 9.04 are:
>>
>> numpy 1.1.1
>> scipy 0.6.0
>>
>> Would it be possible to get 1.2.1 and 0.7.0 to go instead?  Who takes
>> care of pushing these to the various Linux distros?
>>
>> It would be nice if the Packaging section of Developer Zone on the web
>> site made the packaging process a little more transparent...  Say what
>> you do to get a release out and to get it in distribution, who does
>> it, what the timetables tend to look like, etc.  If you ask for help
>> there, you're likely to get it.
> 
> My experience is that most distribution packagers aren't on this list.
> We don't push; they pull.
> 

Well, given that Ubuntu gets most of its packages from Debian and 
unstable has some fairly current numpy/scipy I guess you need to wait 
for things to trickle down. But poking the right person in the Ubuntu 
universe might also be a good idea, not that I know who that is.

Cheers,

Michael
Jarrod Millman | 5 Feb 2009 23:10
Picon
Favicon
Gravatar

Re: updating the numpy/scipy versions in Linux distros

On Thu, Feb 5, 2009 at 12:44 PM, Joe Harrington <jh <at> physics.ucf.edu> wrote:
> It would be nice if the Packaging section of Developer Zone on the web
> site made the packaging process a little more transparent...  Say what
> you do to get a release out and to get it in distribution, who does
> it, what the timetables tend to look like, etc.  If you ask for help
> there, you're likely to get it.

I don't use Ubuntu, but a quick google search gave me this:
  http://packages.ubuntu.com/jaunty/python-numpy
  http://packages.ubuntu.com/jaunty/python-scipy

I would rather not have information about Ubuntu's release process on
the SciPy site.  Ubuntu should be the system of record and it is very
easy to find out that information using google.  Anything added to the
SciPy site will either quickly become out of date or will need
maintenance (we have too much out-of-date, conflicting, or duplicate
information on the site).

If you want the Ubuntu developers to use more recent versions of numpy
and scipy, try "asking a question" about whether they will upgrade
through launchpad:
  https://answers.launchpad.net/ubuntu/+source/python-scipy/+addquestion
  https://answers.launchpad.net/ubuntu/+source/python-numpy/+addquestion

Hope that is enough information to get you started.

Good luck,
Jarrod
Werner Hoch | 7 Feb 2009 10:57
Picon
Picon

Re: updating the numpy/scipy versions in Linux distros

Hi all,

here are some notes about openSUSE.

numpy and scipy are not part of the openSUSE core distribution but 
several projects in the openSUSE BuildService are packaging and using 
scipy and numpy. (https://build.opensuse.org/)

I've maintained numpy and scipy for a while in the science project.
Now both packages are maintained in the Education project.

Misc links:
http://download.opensuse.org/repositories/Education/
http://download.opensuse.org/repositories/Education/openSUSE_11.1/repodata/repoview/python-scipy-0-0.6.0-4.5.html
http://download.opensuse.org/repositories/Education/openSUSE_11.1/repodata/repoview/python-numpy-0-1.2.1-2.6.html

The current version are
  numpy 1.2.1
  scipy 0.6.0

I think as soon as scipy 0.7.0 is out the package will be updated in the 
Education project.

Lars has added some patches to scipy 0.6.0 to fix some compilation 
warnings in the Education project. I don't know if he has reported them 
upstream to the scipy project.
openSUSE_11.1 has some strict rules about compilation warnings. The 
compiler warnings are checked after the build some warnings are treated 
as errors.

In my privat project (not published) I've created a test build for scipy 
0.7.0rc2.

The buildservice complains that scipy 0.7.0rc2 has the following errors:
----------
I: Program is using implicit definitions of special functions.
these functions need to use their correct prototypes to allow
the lightweight buffer overflow checking to work.
- Implicit memory/string functions need #include <string.h>.
- Implicit *printf functions need #include <stdio.h>.
- Implicit *printf functions need #include <stdio.h>.
- Implicit *read* functions need #include <unistd.h>.
- Implicit *recv* functions need #include <sys/socket.h>.
E: python-scipy implicit-fortify-decl 
scipy/sparse/linalg/dsolve/SuperLU/SRC/xerbla.c:33

I: Program returns random data in a function
E: python-scipy no-return-in-nonvoid-function 
build/src.linux-x86_64-2.6/scipy/sparse/linalg/isolve/iterative/getbreak.f:74, 
54, 34, 14

I: Program causes undefined operation
(likely same variable used twiceand post/pre incremented in the same 
expression).
e.g. x = x++; Split it in two operations.
E: python-scipy sequence-point 
scipy/sparse/linalg/dsolve/SuperLU/SRC/cutil.c:243
E: python-scipy sequence-point 
scipy/sparse/linalg/dsolve/SuperLU/SRC/zutil.c:243
----------

I'm attaching the patches from Lars (they are against scipy 0.6.0)

Maybe someone can review them and integrate them into scipy.

Regards
Werner
_______________________________________________
Scipy-dev mailing list
Scipy-dev <at> scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-dev
Pauli Virtanen | 8 Feb 2009 18:23
Picon
Picon
Favicon

Bessel functions from Boost


Some of the real-valued Bessel function implementations from the Cephes 
library currently used in scipy.special have problems. (See #503, #851, 
#853, #854.) Fixing some of these (eg. #503) would require implementing
robust computation algorithms from scratch. (The Specfun code is IMHO too
obscure and badly commented to be relied on as an alternative.)

However, the Boost library seems to have good implementations Bessel
(and some other) special functions:

	http://svn.boost.org/svn/boost/trunk/boost/math/special_functions/detail/
	http://www.boost.org/doc/libs/1_37_0/libs/math/doc/sf_and_dist/html/math_toolkit/special.html

Also the license seems Scipy-compatible:

	http://www.boost.org/LICENSE_1_0.txt

So, I'd like to bring these over to Scipy, to replace some of the Cephes
routines.

The only problem is that being in Boost, they are written in C++, and I
guess we can't make Scipy to depend on it.

I see two options:

  A) Bundle the relevant subset of Boost with Scipy. The problem here
     is that the special functions seem to pull in a sizable subset
     of the whole Boost library.

     Also, I don't know how well compilers handle the template-happy
     C++ in boost today on all platforms where Scipy must work on.

  B) Convert the Boost code from C++ to C. This is in fact quite trivial
     search-and-replace operation. One example here:

     http://github.com/pv/scipy/blob/ticket-503-special-iv-fix/scipy/special/cephes/scipy_iv.c

I'd like to see (B) happen in scipy.special. Thoughts?

--

-- 
Pauli Virtanen

Gmane