Python 2/3 compatible distutils installation?

Here's an interesting porting issue that came up today.  I'm not crossposting
this to distutils-sig, but I will if no one here has come up with a solution.

I'm looking at testtools[1], which Jerry Seutter has done a good, initial port
of to Python 3.  His merge proposal, along with the diff is here[2].  My
branch which fixes a few of Robert Collins' comments is here[3].

One of the tricky problems is with re-raise syntax, which can't be
compatibly represented syntactically across Python 2 and 3.  The six module
has a hack for this, but testtools uses its own trick, which is to provide a
_compat2x.py and a _compat3x.py file containing the proper syntax.

This works well at execution time, because a simple sys.version_info check can
be used to decide which module gets imported; the other is for all intents and
purposes, ignored.

The problem comes when installing the package with distutils.  The
byte-compilation of _compat2x.py throws a SyntaxError, of course rightly so.

The question I have is how can we avoid trying to byte compile _compat2x.py
when the package is installed via Python 3?  I've tried various distutils
command overrides and hacks (including trying to load a MANIFEST3.in which
exclusions that file), but haven't hit upon the right magic to make it work in
all cases.

Has anybody encountered a similar situation, and if so, how have you worked
around this?  Disgusting kludges welcome. :)

Cheers,
-Barry

Re: Python 2/3 compatible distutils installation?

On Wed, May 16, 2012 at 11:57 AM, Barry Warsaw <barry@...> wrote:
> The question I have is how can we avoid trying to byte compile _compat2x.py
> when the package is installed via Python 3?  I've tried various distutils
> command overrides and hacks (including trying to load a MANIFEST3.in which
> exclusions that file), but haven't hit upon the right magic to make it work in
> all cases.

I also encountered this issue recently.  Can you just avoid importing
anything from your package from within setup.py?  It looks like you're
importing your package mostly just to get the __version__ number in
your package's __init__.py (though I also see you use it in the
cmdclass argument to setup(), which might require a different
work-around).

The consequence for me was that I needed to maintain the version
string for my project in two places: both setup.py and my package's
__init__.py (and I added a unit test to check that they were the
same).  I rationalized this by telling myself that the script
responsible for installing a project shouldn't need to be able to run
the project: it should be independent.

--Chris

Re: Python 2/3 compatible distutils installation?

On May 16, 2012, at 12:24 PM, Chris Jerdonek wrote:

>I also encountered this issue recently.  Can you just avoid importing
>anything from your package from within setup.py?

It's not actually *my* setup.py, but I get what you're saying.  In my own
packages, I do try to avoid those types of imports.

>It looks like you're importing your package mostly just to get the
>__version__ number in your package's __init__.py (though I also see you use
>it in the cmdclass argument to setup(), which might require a different
>work-around).

Yep.  It seems like the testtools.TestCommand is required.

>The consequence for me was that I needed to maintain the version
>string for my project in two places: both setup.py and my package's
>__init__.py (and I added a unit test to check that they were the
>same).  I rationalized this by telling myself that the script
>responsible for installing a project shouldn't need to be able to run
>the project: it should be independent.

In my own packages, I still keep the version string in my __init__.py, but I
don't import it from setup.py to get at it.  Instead, I usually just open the
file and grep its lines for the value.  I also have a helper to make this
easier, e.g.

http://bazaar.launchpad.net/~barry/flufl.enum/trunk/view/head:/setup_helpers.py

and its use in my setup.py

Re: Python 2/3 compatible distutils installation?

On Wed, May 16, 2012 at 2:36 PM, Barry Warsaw <barry@...> wrote:
> On May 16, 2012, at 12:24 PM, Chris Jerdonek wrote:
>
>>I also encountered this issue recently.  Can you just avoid importing
>>anything from your package from within setup.py?
>
> It's not actually *my* setup.py, but I get what you're saying.  In my own
> packages, I do try to avoid those types of imports.

Sorry, I meant to say the package that you're helping with.

I realize now that the issue I had might be completely different,
because I was using 2to3 and testtools protects the import with a
version check, etc.  So perhaps my suggestion won't help.  What causes
the install process to byte-compile the files?

Also, thanks for the idea and pointers re: grabbing the version
number.  That idea had crossed my mind, too, but I didn't implement
it.

--Chris

Re: Python 2/3 compatible distutils installation?

On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:

>What causes the install process to byte-compile the files?

That's the question. :)  By which I mean, the files get byte compiled on
installation, and I've tried various things to have it ignore _compat2x.py
when installing in Python 3, but with no luck.

-Barry

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting

Re: Python 2/3 compatible distutils installation?

Le 16/05/2012 19:34, Barry Warsaw a écrit :
> On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:
>> What causes the install process to byte-compile the files?
> That's the question. :)  By which I mean, the files get byte compiled on
> installation, and I've tried various things to have it ignore _compat2x.py
> when installing in Python 3, but with no luck.

distutils does that.  As far as I know you can either disable
byte-compilation wholesale, or have it for all modules.  I don’t think
even a dirty hack like registering the _compat?.py files as package_data
would work, because the byte-compilation function is applied to the
whole build or install dir (but try it anyway, we never know).

See http://bugs.python.org/issue10530 for a distutils2 feature request
about this same use case.

Cheers
_______________________________________________
Python-porting mailing list
Python-porting <at> python.org
http://mail.python.org/mailman/listinfo/python-porting

Re: Python 2/3 compatible distutils installation?

On Wed, May 16, 2012 at 4:34 PM, Barry Warsaw <barry@...> wrote:
> On May 16, 2012, at 03:16 PM, Chris Jerdonek wrote:
>
> That's the question. :)  By which I mean, the files get byte compiled on
> installation, and I've tried various things to have it ignore _compat2x.py
> when installing in Python 3, but with no luck.

Have you thought about storing the two files with an extension other
than .py?  You could argue that from Python 3's perspective, for
example, the compat2 file isn't really a Python file.  You could write
your setup script so that it copies the correct file into place at
compat.py before calling setup(), and then cleans up afterwards.  This
way, the distutils code could proceed normally.  Also, you wouldn't
need special version-checking code in the actual package because then
both versions would be importing compat when installed.

--Chris

Re: Python 2/3 compatible distutils installation?

On May 16, 2012, at 08:13 PM, Éric Araujo wrote:

>See http://bugs.python.org/issue10530 for a distutils2 feature request
>about this same use case.

Thanks, nosied.

-Barry
_______________________________________________
Python-porting mailing list
Python-porting <at> python.org
http://mail.python.org/mailman/listinfo/python-porting

Re: Python 2/3 compatible distutils installation?

On May 16, 2012, at 06:38 PM, Chris Jerdonek wrote:

>Have you thought about storing the two files with an extension other
>than .py?  You could argue that from Python 3's perspective, for
>example, the compat2 file isn't really a Python file.  You could write
>your setup script so that it copies the correct file into place at
>compat.py before calling setup(), and then cleans up afterwards.  This
>way, the distutils code could proceed normally.  Also, you wouldn't
>need special version-checking code in the actual package because then
>both versions would be importing compat when installed.

It's an interesting idea, thanks.  I'll point upstream at this thread and see
if they want to explore it more or if they want to go in the direction of
using six (which would eliminate the need to use the _compatX.py files in this
particular case).

-Barry

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting

Re: Python 2/3 compatible distutils installation?

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting