headers
Stefan Behnel | 3 Aug 08:51
Picon
Favicon
Gravatar

Re: [Pyrex] New Distutils implementation for Pyrex.

Billy G. Allie wrote:
> Stefan Behnel wrote:
>> Billy G. Allie wrote:
>>
>>> I am re-implementing my PostgreSQL interface (pyPgSQL) using Pyrex to
>>> implement an extension for the libpq C-API.  When using the supplied
>>> build_ext, I notices that I had to have the .pxd files in the same
>>> directory as the source .pyx file.  Also, the generated c source file
>>> was placed in the same directory as the Pyrex source.  I also could not
>>> specify Pyrex options to create a listing file, etc.  I wanted to have
>>> the .pxd files in an separate include directory and I also did not want
>>> to clean up the generated files from the Pyrex source directory.  To
>>> correct these (minor) annoyances, I've implemented a new version of
>>> buld_ext.py and created a version of core.py and extension.py.
>>
>> is this an independent implementation or is it based on the old one? If the
>> latter, could you provide a patch against Cython and submit it at the
>> Cython
>> bug tracker? If it's new, could you still provide a single-file patch that
>> includes the new files?
>>
>> https://bugs.launchpad.net/cython
>>
>> http://www.cython.org
>>
>>
> I am attaching a diff generated by the 'diff -cbN <originale Distutils
> directory> <new Distutils directory>' command.
> I also changed all references to Pyrex to Cython and pyrex to cython in the
> files.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Stefan Behnel | 3 Aug 15:41
Picon
Favicon
Gravatar

Re: [Pyrex] New Distutils implementation for Pyrex.

Hi,

Billy G. Allie wrote:
> I've implesmentd a new version of
> buld_ext.py and created a version of core.py and extension.py.

first thing I noticed when running a patched Cython: setuptools break. They
already patch into distutils, so the patching in your code doesn't work. I
would prefer doing without changes to core.py and extensions.py. AFAICT, this
removes the support for configuring import directories at the Extension level.

Alternatively, we could also import things from setuptools first and only fall
back to distutils. However, this would bug people who have setuptools
installed and want to build with distutils (for whatever reason, don't know
how common that is).

I'll take a deeper look into this.

Stefan
Permalink | Reply | 0 comments |
headers
Billy G. Allie | 3 Aug 22:02
Picon

Re: New Distutils implementation for Pyrex.

Stefan Behnel wrote:

Thanks a lot for the patches! I'm resending them to the Cython list for now. You mentioned that you used distutils in Python 2.4 as a base. Python 2.3 compatibility is currently a requirement for Cython. Would you say there are any 2.4 specifics in there? Thanks again, Stefan
Not that I know of.  I just looked at the 2.3 distutils code and didn't see anything that would cause problems with my distutils extension.
_______________________________________________
Pyrex mailing list
Pyrex <at> lists.copyleft.no
http://lists.copyleft.no/mailman/listinfo/pyrex
Permalink | Reply | 0 comments |
headers
Billy G. Allie | 3 Aug 22:31
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefan
Hmmm.  I've never used setuptools, so it wasn't a problem with me :-)
A possible solution would be to try to import setuptools version of core and or extension if they exist, otherwise import the distutils version.   I'll get the source for setuptools and see what they are doing.  (This may take me a couple of days to investigate).
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply | 0 comments |
headers
Billy G. Allie | 4 Aug 05:41
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefan
Stefan,

I've looked at the setuptools source.  Since setuptools is importing Pyrex.Distutils to perform work, setuptools will have to be patched to work with the new code.  From my initial walkthrough of the code (by no meas through), it appeas that the only change would be to change occurances of 'distutils.core' to 'Pyrex.Distutils.core', wrapped in a try ... except statement.  For example:
try:
    from Pyrex.Distutils.core import Extension
except ImportError:
    from distutils.core import Extension
-- or --
try:
    import Pyrex.Distutils.core
except ImportError:
    import distutils.core
The files that would have to change are:
command/upload.py
command/easy_install.py
dist.py
extension.py
__init__.py
package_index.py
tests/__init__.py
Extension is always importted from core, so with the change stated above it will always get the correct Extension class.
build_ext is already imported from Pyrex.Distutils.
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply | 0 comments |
headers
Billy G. Allie | 4 Aug 05:50
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefa
By the way, didn't the package name change from Pyrex to Cython break setuptools?  (It would in the version of setuptools I installed). 
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply | 0 comments |
headers
Phillip J. Eby | 6 Aug 00:37
Gravatar

Re: New Distutils implementation for Pyrex.

At 11:41 PM 8/3/2007 -0400, Billy G. Allie wrote:
>The files that would have to change are:
>command/upload.py
>command/easy_install.py
>dist.py
>extension.py
>__init__.py
>package_index.py
>tests/__init__.py

No need to change all this; setuptools already works correctly with 
Pyrex.Distutils.  If you want it to work with a different extension 
type all you need is a different build_ext command, and for that you 
don't need to patch anything.  Just have your setup script specify 
your build_ext class using setup(... 
cmdclass=dict(build_ext=my_buildext_class), ...).

That's really all you need; no patching is required.

Indeed, the only reason that setuptools does anything with Pyrex at 
all is to support *not* compiling .pyx files when Pyrex is *not* 
installed.  If there are .pyx sources for an Extension and 
Pyrex.Distutils can't be imported, setuptools changes the source file 
extensions to '.c' so that a distributed pre-built .c file can be 
compiled without needing Pyrex.
Permalink | Reply | 0 comments |
headers
Billy G. Allie | 6 Aug 03:44
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Phillip J. Eby wrote:
> At 11:41 PM 8/3/2007 -0400, Billy G. Allie wrote:
>> The files that would have to change are:
>> command/upload.py
>> command/easy_install.py
>> dist.py
>> extension.py
>> __init__.py
>> package_index.py
>> tests/__init__.py
>
> No need to change all this; setuptools already works correctly with 
> Pyrex.Distutils. If you want it to work with a different extension 
> type all you need is a different build_ext command, and for that you 
> don't need to patch anything. Just have your setup script specify your 
> build_ext class using setup(... 
> cmdclass=dict(build_ext=my_buildext_class), ...).
>
> That's really all you need; no patching is required.
>
> Indeed, the only reason that setuptools does anything with Pyrex at 
> all is to support *not* compiling .pyx files when Pyrex is *not* 
> installed. If there are .pyx sources for an Extension and 
> Pyrex.Distutils can't be imported, setuptools changes the source file 
> extensions to '.c' so that a distributed pre-built .c file can be 
> compiled without needing Pyrex.
>
If you want to give new options to extension you need to subclass 
entension.py. If you want to have those new options available as command 
line options (to setup.py), you need to wrap core.py (it can't be 
sub-classed). My new Distutils for Pyrex does both. In order for 
setuptools to compile pyrex extension (.pyx), it needs to use 
Pyrex.Distutils.core instead of distutils.core. If you don't, just 
loading build_ext will fail because of undefined extension variables.
Permalink | Reply | 1 comment |
headers
Phillip J. Eby | 6 Aug 04:11
Gravatar

Re: New Distutils implementation for Pyrex.

At 09:44 PM 8/5/2007 -0400, Billy G. Allie wrote:
>If you want to give new options to extension you need to subclass
>entension.py.

And then just use those classes in your setup.py.  You don't need to 
patch setuptools for that.

>  If you want to have those new options available as command
>line options (to setup.py), you need to wrap core.py (it can't be
>sub-classed).

Why do you need options to setup.py instead of just options to the 
build_ext command?

>  My new Distutils for Pyrex does both. In order for
>setuptools to compile pyrex extension (.pyx), it needs to use
>Pyrex.Distutils.core instead of distutils.core. If you don't, just
>loading build_ext will fail because of undefined extension variables.

Per the above, none of this should be necessary.
Permalink | Reply | 0 comments |
headers
Samuele Kaplun | 9 Aug 14:32
X-Face
Picon
Picon

Segmentation Fault with -O0

Dear Pyrex & Cython lists,
I've found a strange behaviour of my code developed with Pyrex (and Cython) 
which I can't solve.
I wrote and integer bit set extension to Python with Pyrex. It has the 
possibility to be searialized into a string and deserialized from that 
string.
I pass a correctly serialized representation to the constructor of my code (in 
the rhs variable):
[...]
cdef class intbitset:
[...]
    cdef IntBitSet *bitset
[...]
    def __new__(intbitset self, rhs=0, int minsize=-1):
[...]
        cdef size_t size
        cdef void *buf
[...]
                tmp = zlib.decompress(rhs)
                PyObject_AsReadBuffer(tmp, &buf, &size)
                self.bitset = intBitSetCreateFromBuffer(buf, size)
[...]
as you can see, I decompress the rhs string into the tmp Python object, I then 
retrieve a pointer to tmp and then I retrieve a pointer and the size of the 
pointed area, in order to pass it to my C constructor.

When I compile everything without optimization (-O0) it always crashes after 
exiting from __new__.
I tried to selectively commenting-out the code and I found that the 
incriminated row is:
		PyObject_AsReadBuffer(tmp, &buf, &size)
But I really don't know why.
The most interesting thing, is that it never segfaults when compiled with -02 
or -03.
If checked this with latest Cython & Pyrex and I obtain the same behaviour.

Do you maybe see anything obviously wrong in my code?
Best regards,
	Samuele
--

-- 
.O.
..O
OOO
Permalink | Reply | 0 comments |

Gmane