Glenn Linderman | 29 Oct 20:34 2014

Impact of Windows PowerShell OneGet ?

New package manager from M$... article here.

It seems doubtful that M$ will eliminate .msi (their obscure, hard to configure and use, installation format), so it seems doubtful that the addition of OneGet will _force_ any changes to Python packaging.

However, it does open the question in my mind about whether there will be any _benefits_ of OneGet that would inspire helpful, useful changes to Python packaging. They speak of "trusted repositories", and the like, and it sounds like a the various *nix package managers (apt-get, et alia), but perhaps allowing multiple repositories rather than just a single source vendor repository (I'm actually not sure if *nix package managers allow multiple repositories or not, but from the way people talk about them, it always sounds like a "distribution" also provides "a repository" of additional packages).

"trusted repositories" sounds more like Perl's CPAN.

One of the links contains this quote: "This first version of OneGet installs and searches software from Chocolatey repositories.  Support of additional repositories will come in subsequent versions."

I have no clue what a Chocolatey repository is (yet, will Google), but unknown others will come, it says... whether it is possible to write a "repository plugin" such that Perl's CPAN or Python's PyPI or other preexisting repositories can be accessed is not clear.

The relationship between PowerShell and OneGet is not clear either... is OneGet written in PowerShell, or is PowerShell just one way to invoke OneGet, or???

Just a heads up.






<div>
    New package manager from M$... article <a href="http://www.neowin.net/news/windows-10-oneget-a-linux-style-package-management-framework">here</a>.<br><br>
    It seems doubtful that M$ will eliminate .msi (their obscure, hard
    to configure and use, installation format), so it seems doubtful
    that the addition of OneGet will _force_ any changes to Python
    packaging.<br><br>
    However, it does open the question in my mind about whether there
    will be any _benefits_ of OneGet that would inspire helpful, useful
    changes to Python packaging. They speak of "trusted repositories",
    and the like, and it sounds like a the various *nix package managers
    (apt-get, et alia), but perhaps allowing multiple repositories
    rather than just a single source vendor repository (I'm actually not
    sure if *nix package managers allow multiple repositories or not,
    but from the way people talk about them, it always sounds like a
    "distribution" also provides "a repository" of additional packages).<br><br>
    "trusted repositories" sounds more like Perl's CPAN.<br><br><a href="http://blogs.technet.com/b/windowsserver/archive/2014/04/03/windows-management-framework-v5-preview.aspx">One
      of the links</a> contains this quote: "This first version of
    OneGet installs and searches software from Chocolatey repositories.
    &nbsp;Support of additional repositories will come in subsequent
    versions."<br><br>
    I have no clue what a Chocolatey repository is (yet, will Google),
    but unknown others will come, it says... whether it is possible to
    write a "repository plugin" such that Perl's CPAN or Python's PyPI
    or other preexisting repositories can be accessed is not clear.<br><br>
    The relationship between PowerShell and OneGet is not clear
    either... is OneGet written in PowerShell, or is PowerShell just one
    way to invoke OneGet, or???<br><br>
    Just a heads up.<br><br><br><br><br><br><br>
</div>
Victor Stinner | 28 Oct 22:07 2014
Picon

PEP 475

Hi,

At the end of August, I sent the PEP 475 which I wrote with
Charles-François Natali:

   https://mail.python.org/pipermail/python-dev/2014-August/136077.html
   https://mail.python.org/pipermail/python-dev/2014-September/136101.html

Antoine Pitrou wrote " I'm +1 on the whole PEP" and R. David Murray
wrote "Personally, I really want Python to handle EINTR for me".

What's the next step? Who wants to handle this PEP? Guido? Antoine?

I will try to answer to questions if previous answers were not enough.

Antoine Pitrou wrote:
>> On Unix, the ``asyncio`` module uses the wakeup file descriptor to
>> wake up its event loop.
> How about Windows?

I modified signal.set_wakeup_fd() in Python 3.5 to support a socket on
Windows. So it becomes possible to support signals with
signal.set_wakeup_fd() on Windows (for SelectorEventLoop and
ProactorEventLoop):

   https://code.google.com/p/tulip/issues/detail?id=191

Antoine Pitrou wrote:
>> Some signals are not interesting and should not interrupt the the
>> application.  There are two options to only interrupt an application
>> on some signals:
>>
>> * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for
>>   ``SIGINT``
>> * Use a I/O multiplexing function like ``select()`` with the Python
>>   signal "wakeup" file descriptor: see the function
>>   ``signal.set_wakeup_fd()``.
>
> This section looks a bit incomplete. Some calls such as os.read() or
> os.write() will (should) return a partial result when interrupted and
> they already handled >0 bytes. Perhaps other functions have a similar
> behaviour?

In Python 3.4, os.read() is dummy: it only calls the C function read() once.

With the PEP 475, read() is only called again on EINTR if the signal
handler did not raise an exception. When read() returns EINTR, there
is "partial read", the read did not start yet.

So I don't understand what should be added to the PEP. There is no
specific change.

Matthew Woodcraft wrote:
> In any case I think PEP 475 should be explaining what is going to happen
> to signal.siginterrupt(). Will setting flag=True be supported? If so,
> will doing so change the behaviour of those parts of the stdlib which
> have already been modified to retry after EINTR?

In Python 3.4, signal.signal() calls siginterrupt(signum, True):
syscalls raise InterruptedError when interrupted by a signal. Calling
explicitly signal.siginterrupt(signum, True) doesn't change anything.

In Python 3.4, signal.siginterrupt(signum, False) reduces the cases
when InterruptedError is raised, but they are still cases when
InterruptedError is raised. The exact behaviour probably depends on
the operating system or even the version of the operating system. It's
better to not rely on siginterrupt(False) to write portable code in
Python 3.4.

With the PEP, signal.siginterrupt(signum, False) is still supported.
The PEP doesn't change the behaviour when the syscall is directly
restarted by the C library. If the function returns EINTR, the
interrupted syscall is retried if the signal handler didn't raise an
exception.

The main problem of siginterrupt(False) is that the Python signal
handler is *not* called if the C library directly retries the
interrupted syscall.

Note: if signals are blocked, the C signal handlers are not called, so
the PEP doesn't change the behaviour neither.

Victor wrote:
> I think that the stdlib should not handle InterruptedError exception
> anymore in the Python code, to simplify the code.

I modified the PEP to mention that:
https://hg.python.org/peps/rev/627fefe0394f

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Stefan Richthofer | 27 Oct 17:23 2014
Picon
Picon

Re: results of id() and weakref.getweakrefs() sometimes break on object resurrection


>You mean Jython deletes instance attributes before calling __del__ ?

No. I think the term of "object resurrection" usually does not mean bringing
back a deleted object in the sense that memory was already freed.
I think it rather means that nothing referred to an object, so it was on the
"kill-list" of gc or zero-ref-count macro.
During its finalizer call, the object managed to get some reference
again. GC or zero-ref-count macro checks refcount again after the
finalizer call (doesn't it?) and then refrains from the originally triggered
deletion-task.

Where things get weired is how to treat objects (e.g. x2) that are
reachable via the original
object (e.g. x) only.

x becomes unreachable => x2 is unreachable too

CPython behavior:
free x's weakrefs, call x.__del__ => x2 is reachable again => free
memory of x; don't touch x2 at all

Java/Jython behavior:
free all weakrefs, call all finalizers of unreachable objects, i.e. call
x.__del__, call x2.__del__ (and maybe more)
=> x2 is reachable again => free memory of x; let x2 survive
(x2 even survives at least for another gc-cycle if the finalizer of x or
x2 only created a weak ref)

At least in Java/Jython case I would call x2 to be "resurrected", i.e.
its finalizer was called and weakrefs were cleared. It was on the
death-list and escaped from it. This finally brings the definition of
the word "resurrection" to its limit in language independent sense as
one can argue there was no resurrection of x2 in CPython although it's
one and the same scenario.

>In which use case exactly?
The one with "indirect resurrection".
Would it have CPython behavior as sketched above or Java/Jython
behavior? (I confirmed the sketched behavior only for ref-drop-to-zero
triggered cleanup)

Best,

Stefan

On 10/27/2014 04:31 PM, Antoine Pitrou wrote:
> On Mon, 27 Oct 2014 16:20:06 +0100
> Stefan Richthofer <stefan.richthofer <at> gmx.de> wrote:
>> I already pointed out
>> "One can surely argue x2 has never been dead, or see it as
>> "it was killed along with x and then resurrected by x"."
>>
>> In Java and thus in Jython, it is treated as the second one.
> You mean Jython deletes instance attributes before calling __del__ ?
> That would make most __del__ implementations quite useless...
> And actually your own example would fail with an AttributeError on
> "X.y = self.z".
>
>> What would still be interesting (at least when Jython 3 is born),
>> is which of the mentioned behaviors occurs if it is
>> performed by CPython's cyclic gc (consistently the first one I would guess).
> In which use case exactly? :-) I've lost track a bit, since you've
> posted several examples...
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev <at> python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de

Stefan Richthofer | 26 Oct 00:20 2014
Picon
Picon

results of id() and weakref.getweakrefs() sometimes break on object resurrection

Hello developers,

I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
regarding object resurrection.
Yes, resurrection is evil, but it is a valid scenario. If an object is
resurrected via its finalizer __del__, sometimes its unique id value as
returned from id() changes. Additionally the list of weak references
pointing to it as returned by weakref.getweakrefs(...) breaks (i.e. is
suddenly empty, assuming it wasn't before).
These issues only arise if the resurrection occurs during
cyclic garbage collection. If the object is finalized because its refcount
drops to zero, everything is fine. See the attached test-file.

Is this behaviour intended or is it a bug? If so, which variant is the bug
and which is right? I can hardly believe that whether id() is preserved
should depend on whether the garbage was cyclic or not.

This might appear of low relevance to you, since no sane program intentionally
performs resurrection. However I originally became aware of the issue
in Jython (where it not only occurs for cyclic garbage but in every
resurrection-case), c.f. http://bugs.jython.org/issue2224.
I am interested in this because I am implementing native gc support
in JyNI and need to understand these details to do it right.

Thanks in advance!

Stefan
Hello developers,

I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
regarding object resurrection.
Yes, resurrection is evil, but it is a valid scenario. If an object is
resurrected via its finalizer __del__, sometimes its unique id value as
returned from id() changes. Additionally the list of weak references
pointing to it as returned by weakref.getweakrefs(...) breaks (i.e. is
suddenly empty, assuming it wasn't before).
These issues only arise if the resurrection occurs during
cyclic garbage collection. If the object is finalized because its refcount
drops to zero, everything is fine. See the attached test-file.

Is this behaviour intended or is it a bug? If so, which variant is the bug
and which is right? I can hardly believe that whether id() is preserved
should depend on whether the garbage was cyclic or not.

This might appear of low relevance to you, since no sane program intentionally
performs resurrection. However I originally became aware of the issue
in Jython (where it not only occurs for cyclic garbage but in every
resurrection-case), c.f. http://bugs.jython.org/issue2224.
I am interested in this because I am implementing native gc support
in JyNI and need to understand these details to do it right.

Thanks in advance!

Stefan
Tony Kelman | 25 Oct 14:45 2014
Picon

Re: Status of C compilers for Python on Windows

I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.

> Cross compilation is a valid issue, but I hope that build services like
> Appveyor also help out here. There is regular talk about the PSF/PyPI
> providing something similar

AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)

> I could step in as maintainer for Cygwin and builds based on GCC using
> mingw* APIs.
>
> Regards,
> Roumen Petrov

A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ignore the problem. Bring the people
working on these patches into the fold as contributors. Review the patches.
It would make Python as a language and a community even more diverse and
welcoming.

> Deprecate/remove support for compiling CPython itself with compilers
> other than MSVC on Windows

I'm not alone in thinking that this would be a bad idea. MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise.

Sincerely,
Tony

David Bolen | 25 Oct 05:47 2014
Picon

XP buildbot problem cloning from hg.python.org

Starting yesterday, my XP buildbot began failing to execute clone
operations against hg.python.org.  There's not a lot of data being
given aside from a transaction abort message (and my buildbot log
showing the hg command exiting), and I'm wondering if something may be
amiss on the server or its configuration?

Note that this is a full clone (which for some reason the Windows
buildbots seem to fall back on with some frequency) and can take quite
a while.  My Windows 7 buildbot is ok so far but it's still doing
incremental pulls over the same time period.

I've got two separate Internet connections here and have tried routing
over both so I don't think it's a network issue.  I've completely
flushed the local build trees and rebooted the buildbot.

Is there anything that might be available on the server to see if
there are errors being logged?  Or anything that could have changed
configuration wise recently (maybe timeout related or something)?  I'm
running a bit low of items to try to change or reset on the buildbot
side.

Thanks.

-- David

Python tracker | 24 Oct 18:08 2014

Summary of Python tracker Issues


ACTIVITY SUMMARY (2014-10-17 - 2014-10-24)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4604 (+17)
  closed 29877 (+44)
  total  34481 (+61)

Open issues with patches: 2144 

Issues opened (40)
==================

#17401: io.FileIO closefd parameter is not documented nor shown in rep
http://bugs.python.org/issue17401  reopened by serhiy.storchaka

#22659: SyntaxError in the configure_ctypes
http://bugs.python.org/issue22659  opened by bill9889

#22662: subprocess.Popen.communicate causing local tty terminal settin
http://bugs.python.org/issue22662  opened by kflavin

#22665: shutil.__all__ incomplete
http://bugs.python.org/issue22665  opened by vadmium

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666  opened by flavio

#22668: memoryview.format is corrupted due to dangling shared pointer
http://bugs.python.org/issue22668  opened by Knio

#22669: Test_venv fails when _ctypes is not available.
http://bugs.python.org/issue22669  opened by terry.reedy

#22671: Typo in class io.BufferedIOBase docs
http://bugs.python.org/issue22671  opened by gigaplastik

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672  opened by jnespolo

#22673: document the special features (eg: fdclose=False) of the stand
http://bugs.python.org/issue22673  opened by snaphat

#22674: strsignal() missing from signal module
http://bugs.python.org/issue22674  opened by Dolda2000

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678  opened by bochecha

#22679: Add encodings of supported in glibc locales
http://bugs.python.org/issue22679  opened by serhiy.storchaka

#22680: unittest discovery is fragile
http://bugs.python.org/issue22680  opened by pitrou

#22681: Add support of KOI8-T encoding
http://bugs.python.org/issue22681  opened by serhiy.storchaka

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682  opened by serhiy.storchaka

#22683: bisect index out of bounds issue
http://bugs.python.org/issue22683  opened by Paul.Ianas

#22684: message.as_bytes() produces recursion depth exceeded
http://bugs.python.org/issue22684  opened by pas

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685  opened by wabu

#22687: horrible performance of textwrap.wrap() with a long word
http://bugs.python.org/issue22687  opened by inkerman

#22689: Posix getenv makes no guarantee of lifetime of returned string
http://bugs.python.org/issue22689  opened by aidanhs

#22695: open() declared deprecated in python 3 docs
http://bugs.python.org/issue22695  opened by Василий.Макаров

#22696: Add a function to know about interpreter shutdown
http://bugs.python.org/issue22696  opened by pitrou

#22697: Deadlock with writing to stderr from forked process
http://bugs.python.org/issue22697  opened by ionel.mc

#22698: Add constants for ioctl request codes
http://bugs.python.org/issue22698  opened by serhiy.storchaka

#22699: cross-compilation of Python3.4
http://bugs.python.org/issue22699  opened by bill9889

#22700: email's header_value_parser missing defect report for 'abc <at> xyz
http://bugs.python.org/issue22700  opened by r.david.murray

#22701: Write unescaped unicode characters (Japanese, Chinese, etc) in
http://bugs.python.org/issue22701  opened by Michael.Kuss

#22702: to improve documentation for join() (str method)
http://bugs.python.org/issue22702  opened by vy0123

#22703: Idle Code Context: separate changing current and future editor
http://bugs.python.org/issue22703  opened by terry.reedy

#22704: Review extension enable options
http://bugs.python.org/issue22704  opened by terry.reedy

#22705: Idle extension configuration: add option-help option
http://bugs.python.org/issue22705  opened by terry.reedy

#22706: Idle extension configuration and key bindings
http://bugs.python.org/issue22706  opened by terry.reedy

#22707: Idle: changed options should take effect immediately
http://bugs.python.org/issue22707  opened by terry.reedy

#22708: httplib/http.client in method _tunnel used HTTP/1.0 CONNECT me
http://bugs.python.org/issue22708  opened by vova

#22709: restore accepting detached stdin in fileinput binary mode
http://bugs.python.org/issue22709  opened by akira

#22711: "legacy" distutils docs better than packaging guide
http://bugs.python.org/issue22711  opened by pitrou

#22714: target of 'import statement' entry in general index for 'i' is
http://bugs.python.org/issue22714  opened by vy0123

#22718: pprint not handline uncomparable dictionary keys, set members 
http://bugs.python.org/issue22718  opened by Andreas.Kostyrka

#22719: os.path.isfile & os.path.exists bug in while loop
http://bugs.python.org/issue22719  opened by hosford42

Most recent 15 issues with no replies (15)
==========================================

#22719: os.path.isfile & os.path.exists bug in while loop
http://bugs.python.org/issue22719

#22714: target of 'import statement' entry in general index for 'i' is
http://bugs.python.org/issue22714

#22706: Idle extension configuration and key bindings
http://bugs.python.org/issue22706

#22704: Review extension enable options
http://bugs.python.org/issue22704

#22703: Idle Code Context: separate changing current and future editor
http://bugs.python.org/issue22703

#22700: email's header_value_parser missing defect report for 'abc <at> xyz
http://bugs.python.org/issue22700

#22699: cross-compilation of Python3.4
http://bugs.python.org/issue22699

#22697: Deadlock with writing to stderr from forked process
http://bugs.python.org/issue22697

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682

#22679: Add encodings of supported in glibc locales
http://bugs.python.org/issue22679

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672

#22671: Typo in class io.BufferedIOBase docs
http://bugs.python.org/issue22671

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666

#22662: subprocess.Popen.communicate causing local tty terminal settin
http://bugs.python.org/issue22662

Most recent 15 issues waiting for review (15)
=============================================

#22709: restore accepting detached stdin in fileinput binary mode
http://bugs.python.org/issue22709

#22708: httplib/http.client in method _tunnel used HTTP/1.0 CONNECT me
http://bugs.python.org/issue22708

#22696: Add a function to know about interpreter shutdown
http://bugs.python.org/issue22696

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682

#22681: Add support of KOI8-T encoding
http://bugs.python.org/issue22681

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672

#22668: memoryview.format is corrupted due to dangling shared pointer
http://bugs.python.org/issue22668

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666

#22665: shutil.__all__ incomplete
http://bugs.python.org/issue22665

#22649: Use _PyUnicodeWriter in case_operation()
http://bugs.python.org/issue22649

#22642: trace module: unclear error message
http://bugs.python.org/issue22642

#22638: ssl module: the SSLv3 protocol is vulnerable ("POODLE" attack)
http://bugs.python.org/issue22638

#22636: avoid using a shell in ctypes.util: replace os.popen with subp
http://bugs.python.org/issue22636

Top 10 most discussed issues (10)
=================================

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685  12 msgs

#17401: io.FileIO closefd parameter is not documented nor shown in rep
http://bugs.python.org/issue17401   6 msgs

#22599: traceback: errors in the linecache module at exit
http://bugs.python.org/issue22599   6 msgs

#22669: Test_venv fails when _ctypes is not available.
http://bugs.python.org/issue22669   6 msgs

#22673: document the special features (eg: fdclose=False) of the stand
http://bugs.python.org/issue22673   6 msgs

#22674: strsignal() missing from signal module
http://bugs.python.org/issue22674   6 msgs

#22711: "legacy" distutils docs better than packaging guide
http://bugs.python.org/issue22711   6 msgs

#10548: Error in setUp not reported as expectedFailure (unittest)
http://bugs.python.org/issue10548   5 msgs

#11820: idle3 shell os.system swallows shell command output
http://bugs.python.org/issue11820   5 msgs

#22680: unittest discovery is fragile
http://bugs.python.org/issue22680   5 msgs

Issues closed (40)
==================

#7186: Document specialness of __doc__, and possibly other "special" 
http://bugs.python.org/issue7186  closed by ethan.furman

#9351: argparse set_defaults on subcommands should override top level
http://bugs.python.org/issue9351  closed by r.david.murray

#16000: test_curses should use unittest
http://bugs.python.org/issue16000  closed by zach.ware

#16863: Python 2 error in Argparse tutorial
http://bugs.python.org/issue16863  closed by terry.reedy

#18853: Got ResourceWarning unclosed file when running Lib/shlex.py de
http://bugs.python.org/issue18853  closed by r.david.murray

#18976: distutils/command/build_ext passes wrong linker flags
http://bugs.python.org/issue18976  closed by Benedikt.Morbach

#19746: No introspective way to detect ModuleImportFailure in unittest
http://bugs.python.org/issue19746  closed by python-dev

#20689: socket.AddressFamily is absent in pydoc output
http://bugs.python.org/issue20689  closed by ethan.furman

#21298: Cheese shop registration stopped working for me recently
http://bugs.python.org/issue21298  closed by georg.brandl

#21991: The new email API should use MappingProxyType instead of retur
http://bugs.python.org/issue21991  closed by r.david.murray

#22160: Windows installers need to be updated following OpenSSL securi
http://bugs.python.org/issue22160  closed by zach.ware

#22186: Typos in .py files
http://bugs.python.org/issue22186  closed by berker.peksag

#22592: Drop support of Borland C compiler
http://bugs.python.org/issue22592  closed by haypo

#22637: avoid using a shell in uuid: replce os.popen with subprocess.P
http://bugs.python.org/issue22637  closed by haypo

#22644: Update Windows installers to OpenSSL 1.0.1j
http://bugs.python.org/issue22644  closed by zach.ware

#22648: Unable to install Python 3.4.2 amd64 on Windows 8.1
http://bugs.python.org/issue22648  closed by brp-log

#22653: Crash in insertdict
http://bugs.python.org/issue22653  closed by pitrou

#22655: Build error on CentOS 5.4
http://bugs.python.org/issue22655  closed by skrah

#22658: IMAP4 Example lacking host information
http://bugs.python.org/issue22658  closed by r.david.murray

#22660: Review ssl docs for security recommendations
http://bugs.python.org/issue22660  closed by pitrou

#22661: WinXP concerns
http://bugs.python.org/issue22661  closed by r.david.murray

#22663: patchcheck alters content of .png files
http://bugs.python.org/issue22663  closed by python-dev

#22664: IDLE: Standard output and error from multiprocessing vanishes
http://bugs.python.org/issue22664  closed by serhiy.storchaka

#22667: Incorrect evaluation of variables with names containing supple
http://bugs.python.org/issue22667  closed by benjamin.peterson

#22670: wrong site-package installation even if correct libdir passed
http://bugs.python.org/issue22670  closed by georg.brandl

#22675: typo in argparse.py
http://bugs.python.org/issue22675  closed by python-dev

#22676: _pickle's whichmodule() is slow
http://bugs.python.org/issue22676  closed by brett.cannon

#22677: IDLE: icon not loaded
http://bugs.python.org/issue22677  closed by Ahmad.El-Komey

#22686: random.randint does not include endpoint
http://bugs.python.org/issue22686  closed by mark.dickinson

#22688: Use the subprocess module in the uuid module
http://bugs.python.org/issue22688  closed by serhiy.storchaka

#22690: importing Gtk breaks strptime
http://bugs.python.org/issue22690  closed by sigzegv

#22691: A Better Help File
http://bugs.python.org/issue22691  closed by r.david.murray

#22692: Problems with Python's help()
http://bugs.python.org/issue22692  closed by r.david.murray

#22694: The help file issue I'm having.
http://bugs.python.org/issue22694  closed by r.david.murray

#22710: doctest exceptions include nondeterministic namespace
http://bugs.python.org/issue22710  closed by r.david.murray

#22712: Add 'input' argument to subprocess.check_call and subprocess.c
http://bugs.python.org/issue22712  closed by serhiy.storchaka

#22713: print()
http://bugs.python.org/issue22713  closed by georg.brandl

#22715: PEP 257: drop the recommendation for a blank line between the 
http://bugs.python.org/issue22715  closed by r.david.murray

#22716: Add reference to the object missing an attribute to AttributeE
http://bugs.python.org/issue22716  closed by serhiy.storchaka

#22717: PySSL segmentation fault
http://bugs.python.org/issue22717  closed by r.david.murray

ACTIVITY SUMMARY (2014-10-17 - 2014-10-24)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4604 (+17)
  closed 29877 (+44)
  total  34481 (+61)

Open issues with patches: 2144 

Issues opened (40)
==================

#17401: io.FileIO closefd parameter is not documented nor shown in rep
http://bugs.python.org/issue17401  reopened by serhiy.storchaka

#22659: SyntaxError in the configure_ctypes
http://bugs.python.org/issue22659  opened by bill9889

#22662: subprocess.Popen.communicate causing local tty terminal settin
http://bugs.python.org/issue22662  opened by kflavin

#22665: shutil.__all__ incomplete
http://bugs.python.org/issue22665  opened by vadmium

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666  opened by flavio

#22668: memoryview.format is corrupted due to dangling shared pointer
http://bugs.python.org/issue22668  opened by Knio

#22669: Test_venv fails when _ctypes is not available.
http://bugs.python.org/issue22669  opened by terry.reedy

#22671: Typo in class io.BufferedIOBase docs
http://bugs.python.org/issue22671  opened by gigaplastik

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672  opened by jnespolo

#22673: document the special features (eg: fdclose=False) of the stand
http://bugs.python.org/issue22673  opened by snaphat

#22674: strsignal() missing from signal module
http://bugs.python.org/issue22674  opened by Dolda2000

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678  opened by bochecha

#22679: Add encodings of supported in glibc locales
http://bugs.python.org/issue22679  opened by serhiy.storchaka

#22680: unittest discovery is fragile
http://bugs.python.org/issue22680  opened by pitrou

#22681: Add support of KOI8-T encoding
http://bugs.python.org/issue22681  opened by serhiy.storchaka

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682  opened by serhiy.storchaka

#22683: bisect index out of bounds issue
http://bugs.python.org/issue22683  opened by Paul.Ianas

#22684: message.as_bytes() produces recursion depth exceeded
http://bugs.python.org/issue22684  opened by pas

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685  opened by wabu

#22687: horrible performance of textwrap.wrap() with a long word
http://bugs.python.org/issue22687  opened by inkerman

#22689: Posix getenv makes no guarantee of lifetime of returned string
http://bugs.python.org/issue22689  opened by aidanhs

#22695: open() declared deprecated in python 3 docs
http://bugs.python.org/issue22695  opened by Василий.Макаров

#22696: Add a function to know about interpreter shutdown
http://bugs.python.org/issue22696  opened by pitrou

#22697: Deadlock with writing to stderr from forked process
http://bugs.python.org/issue22697  opened by ionel.mc

#22698: Add constants for ioctl request codes
http://bugs.python.org/issue22698  opened by serhiy.storchaka

#22699: cross-compilation of Python3.4
http://bugs.python.org/issue22699  opened by bill9889

#22700: email's header_value_parser missing defect report for 'abc <at> xyz
http://bugs.python.org/issue22700  opened by r.david.murray

#22701: Write unescaped unicode characters (Japanese, Chinese, etc) in
http://bugs.python.org/issue22701  opened by Michael.Kuss

#22702: to improve documentation for join() (str method)
http://bugs.python.org/issue22702  opened by vy0123

#22703: Idle Code Context: separate changing current and future editor
http://bugs.python.org/issue22703  opened by terry.reedy

#22704: Review extension enable options
http://bugs.python.org/issue22704  opened by terry.reedy

#22705: Idle extension configuration: add option-help option
http://bugs.python.org/issue22705  opened by terry.reedy

#22706: Idle extension configuration and key bindings
http://bugs.python.org/issue22706  opened by terry.reedy

#22707: Idle: changed options should take effect immediately
http://bugs.python.org/issue22707  opened by terry.reedy

#22708: httplib/http.client in method _tunnel used HTTP/1.0 CONNECT me
http://bugs.python.org/issue22708  opened by vova

#22709: restore accepting detached stdin in fileinput binary mode
http://bugs.python.org/issue22709  opened by akira

#22711: "legacy" distutils docs better than packaging guide
http://bugs.python.org/issue22711  opened by pitrou

#22714: target of 'import statement' entry in general index for 'i' is
http://bugs.python.org/issue22714  opened by vy0123

#22718: pprint not handline uncomparable dictionary keys, set members 
http://bugs.python.org/issue22718  opened by Andreas.Kostyrka

#22719: os.path.isfile & os.path.exists bug in while loop
http://bugs.python.org/issue22719  opened by hosford42

Most recent 15 issues with no replies (15)
==========================================

#22719: os.path.isfile & os.path.exists bug in while loop
http://bugs.python.org/issue22719

#22714: target of 'import statement' entry in general index for 'i' is
http://bugs.python.org/issue22714

#22706: Idle extension configuration and key bindings
http://bugs.python.org/issue22706

#22704: Review extension enable options
http://bugs.python.org/issue22704

#22703: Idle Code Context: separate changing current and future editor
http://bugs.python.org/issue22703

#22700: email's header_value_parser missing defect report for 'abc <at> xyz
http://bugs.python.org/issue22700

#22699: cross-compilation of Python3.4
http://bugs.python.org/issue22699

#22697: Deadlock with writing to stderr from forked process
http://bugs.python.org/issue22697

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682

#22679: Add encodings of supported in glibc locales
http://bugs.python.org/issue22679

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672

#22671: Typo in class io.BufferedIOBase docs
http://bugs.python.org/issue22671

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666

#22662: subprocess.Popen.communicate causing local tty terminal settin
http://bugs.python.org/issue22662

Most recent 15 issues waiting for review (15)
=============================================

#22709: restore accepting detached stdin in fileinput binary mode
http://bugs.python.org/issue22709

#22708: httplib/http.client in method _tunnel used HTTP/1.0 CONNECT me
http://bugs.python.org/issue22708

#22696: Add a function to know about interpreter shutdown
http://bugs.python.org/issue22696

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685

#22682: Add support of KZ1048 (RK1048) encoding
http://bugs.python.org/issue22682

#22681: Add support of KOI8-T encoding
http://bugs.python.org/issue22681

#22678: An OSError subclass for "no space left on device" would be nic
http://bugs.python.org/issue22678

#22672: float arguments in scientific notation not supported by argpar
http://bugs.python.org/issue22672

#22668: memoryview.format is corrupted due to dangling shared pointer
http://bugs.python.org/issue22668

#22666: email.Header no encoding of unicode strings containing newline
http://bugs.python.org/issue22666

#22665: shutil.__all__ incomplete
http://bugs.python.org/issue22665

#22649: Use _PyUnicodeWriter in case_operation()
http://bugs.python.org/issue22649

#22642: trace module: unclear error message
http://bugs.python.org/issue22642

#22638: ssl module: the SSLv3 protocol is vulnerable ("POODLE" attack)
http://bugs.python.org/issue22638

#22636: avoid using a shell in ctypes.util: replace os.popen with subp
http://bugs.python.org/issue22636

Top 10 most discussed issues (10)
=================================

#22685: memory leak: no transport for pipes by create_subprocess_exec/
http://bugs.python.org/issue22685  12 msgs

#17401: io.FileIO closefd parameter is not documented nor shown in rep
http://bugs.python.org/issue17401   6 msgs

#22599: traceback: errors in the linecache module at exit
http://bugs.python.org/issue22599   6 msgs

#22669: Test_venv fails when _ctypes is not available.
http://bugs.python.org/issue22669   6 msgs

#22673: document the special features (eg: fdclose=False) of the stand
http://bugs.python.org/issue22673   6 msgs

#22674: strsignal() missing from signal module
http://bugs.python.org/issue22674   6 msgs

#22711: "legacy" distutils docs better than packaging guide
http://bugs.python.org/issue22711   6 msgs

#10548: Error in setUp not reported as expectedFailure (unittest)
http://bugs.python.org/issue10548   5 msgs

#11820: idle3 shell os.system swallows shell command output
http://bugs.python.org/issue11820   5 msgs

#22680: unittest discovery is fragile
http://bugs.python.org/issue22680   5 msgs

Issues closed (40)
==================

#7186: Document specialness of __doc__, and possibly other "special" 
http://bugs.python.org/issue7186  closed by ethan.furman

#9351: argparse set_defaults on subcommands should override top level
http://bugs.python.org/issue9351  closed by r.david.murray

#16000: test_curses should use unittest
http://bugs.python.org/issue16000  closed by zach.ware

#16863: Python 2 error in Argparse tutorial
http://bugs.python.org/issue16863  closed by terry.reedy

#18853: Got ResourceWarning unclosed file when running Lib/shlex.py de
http://bugs.python.org/issue18853  closed by r.david.murray

#18976: distutils/command/build_ext passes wrong linker flags
http://bugs.python.org/issue18976  closed by Benedikt.Morbach

#19746: No introspective way to detect ModuleImportFailure in unittest
http://bugs.python.org/issue19746  closed by python-dev

#20689: socket.AddressFamily is absent in pydoc output
http://bugs.python.org/issue20689  closed by ethan.furman

#21298: Cheese shop registration stopped working for me recently
http://bugs.python.org/issue21298  closed by georg.brandl

#21991: The new email API should use MappingProxyType instead of retur
http://bugs.python.org/issue21991  closed by r.david.murray

#22160: Windows installers need to be updated following OpenSSL securi
http://bugs.python.org/issue22160  closed by zach.ware

#22186: Typos in .py files
http://bugs.python.org/issue22186  closed by berker.peksag

#22592: Drop support of Borland C compiler
http://bugs.python.org/issue22592  closed by haypo

#22637: avoid using a shell in uuid: replce os.popen with subprocess.P
http://bugs.python.org/issue22637  closed by haypo

#22644: Update Windows installers to OpenSSL 1.0.1j
http://bugs.python.org/issue22644  closed by zach.ware

#22648: Unable to install Python 3.4.2 amd64 on Windows 8.1
http://bugs.python.org/issue22648  closed by brp-log

#22653: Crash in insertdict
http://bugs.python.org/issue22653  closed by pitrou

#22655: Build error on CentOS 5.4
http://bugs.python.org/issue22655  closed by skrah

#22658: IMAP4 Example lacking host information
http://bugs.python.org/issue22658  closed by r.david.murray

#22660: Review ssl docs for security recommendations
http://bugs.python.org/issue22660  closed by pitrou

#22661: WinXP concerns
http://bugs.python.org/issue22661  closed by r.david.murray

#22663: patchcheck alters content of .png files
http://bugs.python.org/issue22663  closed by python-dev

#22664: IDLE: Standard output and error from multiprocessing vanishes
http://bugs.python.org/issue22664  closed by serhiy.storchaka

#22667: Incorrect evaluation of variables with names containing supple
http://bugs.python.org/issue22667  closed by benjamin.peterson

#22670: wrong site-package installation even if correct libdir passed
http://bugs.python.org/issue22670  closed by georg.brandl

#22675: typo in argparse.py
http://bugs.python.org/issue22675  closed by python-dev

#22676: _pickle's whichmodule() is slow
http://bugs.python.org/issue22676  closed by brett.cannon

#22677: IDLE: icon not loaded
http://bugs.python.org/issue22677  closed by Ahmad.El-Komey

#22686: random.randint does not include endpoint
http://bugs.python.org/issue22686  closed by mark.dickinson

#22688: Use the subprocess module in the uuid module
http://bugs.python.org/issue22688  closed by serhiy.storchaka

#22690: importing Gtk breaks strptime
http://bugs.python.org/issue22690  closed by sigzegv

#22691: A Better Help File
http://bugs.python.org/issue22691  closed by r.david.murray

#22692: Problems with Python's help()
http://bugs.python.org/issue22692  closed by r.david.murray

#22694: The help file issue I'm having.
http://bugs.python.org/issue22694  closed by r.david.murray

#22710: doctest exceptions include nondeterministic namespace
http://bugs.python.org/issue22710  closed by r.david.murray

#22712: Add 'input' argument to subprocess.check_call and subprocess.c
http://bugs.python.org/issue22712  closed by serhiy.storchaka

#22713: print()
http://bugs.python.org/issue22713  closed by georg.brandl

#22715: PEP 257: drop the recommendation for a blank line between the 
http://bugs.python.org/issue22715  closed by r.david.murray

#22716: Add reference to the object missing an attribute to AttributeE
http://bugs.python.org/issue22716  closed by serhiy.storchaka

#22717: PySSL segmentation fault
http://bugs.python.org/issue22717  closed by r.david.murray
Frank, Matthew I | 23 Oct 21:22 2014
Picon

Cross compiling Python (for Android)

This email is about my experience getting CPython (3.4.1) to

cross-compile and run on x86 Android (4.4.2 with sdk 19 and ndk-r9).

I know that Android is not a supported architecture (and I won't

regale you with stories about the complete locale and mbstowcs support

I had to borrow from FreeBSD to get it working).  The purpose of this

email is that several things I found are arguably "bugs" in the Python

build system or code when it comes to cross-compiling that are exposed

by Android's poor Posix support.  I'd like some advice about what kind

of patch (if any) would be most suitable for fixing the problems on

the Python side.

 

Just to be complete:  I'm configuring with

 

     CPPFLAGS=-I../my-locale ../Python-3.4.1/configure --enable-shared

     --prefix=/path/to/install/dir --build=x86_64-linux-gnu

     --host=i686-linux-android --disable-ipv6 ac_cv_file__dev_ptmx=no

     ac_cv_file__dev_ptc=no ac_cv_little_endian_double=yes

 

(The CPPFLAGS addition is to get the headers for my locale fixes

instead of the default Android ones.  ac_cv_file__dev_ptmx=no and

ac_cv_file__dev_ptc=no are because I don't have /dev/whatever on my

build machine.  ac_cv_little_endian_double is because configure for

cross builds can't figure out the endianness of doubles on the host

(because it is running on the build machine not the host.)  (For ARM

it would be ac_cv_mixed_endian_double=yes.)

 

I've gotten to the point where `make; make install` succeeds up to the

point of building something that runs on my Android system (from the

command line) and `python -m test` runs 388 tests, with 321 ok, 24

test failures and 43 tests skipped (the skips mostly due, I think, to

me not yet having installed the right cross-building support for

things like bz2 and dbm.)

 

1. `make` succeeds but `make install` always fails at the end with

   something having to do with being unable to run "ensurepip"

   (presumably because ensurepip requires modules that only run on the

   host, not the build module.)  So it seems this should be wrapped in

   a test for cross compilation, but I haven't looked at exactly what

   yet.  The error is:

 

   /linux-python/bin/python3.4: Error while finding spec for

   'ensurepip.__main__' (<class 'ImportError'>:

   /build-directory/build/lib.linux-i686-3.4/time.cpython-34m.so:

   wrong ELF class: ELFCLASS32); 'ensurepip' is a package and cannot

   be directly executed make: *** [install] Error 1

 

2. setup.py is missing -lm flag for several modules.  On Linux this

   problem is hidden because libm is already loaded by the executable

   calling dlopen(), but Android's loader deals with unknown symbols

   differently (searches only the libs explicitly linked against the

   module being loaded.)  http://bugs.python.org/issue21668 reports

   the problem for selectmodule (can't find ceil()) and timemodule

   (fmod() and floor()).  But there are at least two more: audioop

   fails to load because it needs floor() and ctypes_test fails to

   load because it needs sqrt().  I'll happily update the patch in

   21668.

 

   Is there any fundamental objection to adding the -lm flag to the

   link step where it is necessary?

 

3. What is ossaudiodev?  It tries to include "sys/soundcard.h", which

   I don't have on my system.   (The rule in setup.py is

   wrapped in a test for host of Linux/FreeBSD/Darwin, but Android x86

   gets configured with --host=i686-linux-android so to turn it off

   requires an extra test for "and not cross_compiling".)

 

   Can I just turn off ossaudiodev for cross compiling or might

   someone want it in a different type of cross build?  (In which case

   I think I'll have to write some kind autoconf rule for it, which I

   don't quite know how to do yet.)

 

4. Module _decimal is failing to compile.  The problem is that it has

   a header called memory.h.  Android's libc has the problem that

   /usr/include/stdlib.h includes <memory.h>.  But the build system

   puts -I. on the include path before the system dirs (as it should)

   so when compiling _decimal, Modules/_decimal/libmpdec/memory.h gets

   found instead of /usr/include/memory.h.  Shiz has a patch here:

   https://github.com/rave-engine/python3-android/blob/master/mk/python/3.3.5/p\

ython-3.3.5-android-libmpdec.patch

   (which renames memory.h -> mpmemory.h) but I don't know

 

   a.  Is there a tracker for this yet?  and

   b.  Is Shiz's fix the desired one or should I be looking for

       another approach?  (Maybe modifying the -I flags for the build

       of just the build of _decimal or something?)

 

5. I'm not sure what test configure is actually doing for gethostby*()

   in a cross-compile environment.  In any case Android has a bug

   where gethostbyaddr_r() is declared in the headers, but not

   actually implemented in libc.  So I have to modify my pyconfig.h by

   hand to define HAVE_GETHOSTBYNAME and undef HAVE_GETHOSTBYNAME_R

   and HAVE_GETHOSTBYNAME_R_6_ARG.

 

   Is there a variable (like ac_cv_little_endian_double) that I can

   give to `configure` to make it set HAVE_GETHOSTBYNAME* the way I

   need?  If so I've been unable to figure it out.

 

6. Android's <pwd.h> header mysteriously leaves the pw_gecos field out

   of struct passwd.  Is a fix like defining a new variable

   HAVE_BROKEN_GECOS_FIELD the appropriate way to go with this?  (If

   this is an okay solution then the patch to Modules/pwdmodule.c is

   shown below, but I still have to figure out how to patch

   configure.ac to test for the condition and set the variable

   appropriately, so a pointer to a similar block of code in

   configure.ac would be appreciated.)

 

Sorry for the TL;DR.  I appreciate your having taken the time to read

this far.

 

Thanks,

-Matt

 

Proposed patch for pwdmodule.c:

 

--- a/Modules/pwdmodule.c 2014-05-19 00:19:39.000000000 -0500

+++ b/Modules/pwdmodule.c       2014-10-21 18:00:35.676331205 -0500

<at> <at> -57,6 +57,10 <at> <at>

   }

}

 

+#if defined(HAVE_BROKEN_GECOS_FIELD)

+static char fakePwGecos[256] = "";

+#endif

+

static PyObject *

mkpwent(struct passwd *p)

{

<at> <at> -72,7 +76,11 <at> <at>

     SETS(setIndex++, p->pw_passwd);

     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromUid(p->pw_uid));

     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromGid(p->pw_gid));

+#if !defined(HAVE_BROKEN_GECOS_FIELD)

     SETS(setIndex++, p->pw_gecos);

+#else

+    SETS(setIndex++, fakePwGecos);

+#endif

     SETS(setIndex++, p->pw_dir);

     SETS(setIndex++, p->pw_shell);

 

<div>
<div class="WordSection1">
<p class="MsoNormal"><span>This email is about my experience getting CPython (3.4.1) to<p></p></span></p>
<p class="MsoNormal"><span>cross-compile and run on x86 Android (4.4.2 with sdk 19 and ndk-r9).<p></p></span></p>
<p class="MsoNormal"><span>I know that Android is not a supported architecture (and I won't<p></p></span></p>
<p class="MsoNormal"><span>regale you with stories about the complete locale and mbstowcs support<p></p></span></p>
<p class="MsoNormal"><span>I had to borrow from FreeBSD to get it working).&nbsp; The purpose of this<p></p></span></p>
<p class="MsoNormal"><span>email is that several things I found are arguably "bugs" in the Python<p></p></span></p>
<p class="MsoNormal"><span>build system or code when it comes to cross-compiling that are exposed<p></p></span></p>
<p class="MsoNormal"><span>by Android's poor Posix support.&nbsp; I'd like some advice about what kind<p></p></span></p>
<p class="MsoNormal"><span>of patch (if any) would be most suitable for fixing the problems on<p></p></span></p>
<p class="MsoNormal"><span>the Python side.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Just to be complete:&nbsp; I'm configuring with<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; CPPFLAGS=-I../my-locale ../Python-3.4.1/configure --enable-shared<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; --prefix=/path/to/install/dir --build=x86_64-linux-gnu<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; --host=i686-linux-android --disable-ipv6 ac_cv_file__dev_ptmx=no<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; ac_cv_file__dev_ptc=no ac_cv_little_endian_double=yes<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>(The CPPFLAGS addition is to get the headers for my locale fixes<p></p></span></p>
<p class="MsoNormal"><span>instead of the default Android ones.&nbsp; ac_cv_file__dev_ptmx=no and<p></p></span></p>
<p class="MsoNormal"><span>ac_cv_file__dev_ptc=no are because I don't have /dev/whatever on my<p></p></span></p>
<p class="MsoNormal"><span>build machine.&nbsp; ac_cv_little_endian_double is because configure for<p></p></span></p>
<p class="MsoNormal"><span>cross builds can't figure out the endianness of doubles on the host<p></p></span></p>
<p class="MsoNormal"><span>(because it is running on the build machine not the host.)&nbsp; (For ARM<p></p></span></p>
<p class="MsoNormal"><span>it would be ac_cv_mixed_endian_double=yes.)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>I've gotten to the point where `make; make install` succeeds up to the<p></p></span></p>
<p class="MsoNormal"><span>point of building something that runs on my Android system (from the<p></p></span></p>
<p class="MsoNormal"><span>command line) and `python -m test` runs 388 tests, with 321 ok, 24<p></p></span></p>
<p class="MsoNormal"><span>test failures and 43 tests skipped (the skips mostly due, I think, to<p></p></span></p>
<p class="MsoNormal"><span>me not yet having installed the right cross-building support for<p></p></span></p>
<p class="MsoNormal"><span>things like bz2 and dbm.)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>1. `make` succeeds but `make install` always fails at the end with<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; something having to do with being unable to run "ensurepip"<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; (presumably because ensurepip requires modules that only run on the<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; host, not the build module.)&nbsp; So it seems this should be wrapped in<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; a test for cross compilation, but I haven't looked at exactly what<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; yet.&nbsp; The error is:<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; /linux-python/bin/python3.4: Error while finding spec for<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; 'ensurepip.__main__' (&lt;class 'ImportError'&gt;:<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; /build-directory/build/lib.linux-i686-3.4/time.cpython-34m.so:<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; wrong ELF class: ELFCLASS32); 'ensurepip' is a package and cannot<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; be directly executed make: *** [install] Error 1<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>2. setup.py is missing -lm flag for several modules.&nbsp; On Linux this<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; problem is hidden because libm is already loaded by the executable<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; calling dlopen(), but Android's loader deals with unknown symbols<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; differently (searches only the libs explicitly linked against the<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; module being loaded.)&nbsp; http://bugs.python.org/issue21668 reports<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; the problem for selectmodule (can't find ceil()) and timemodule<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; (fmod() and floor()).&nbsp; But there are at least two more: audioop<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; fails to load because it needs floor() and ctypes_test fails to<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; load because it needs sqrt().&nbsp; I'll happily update the patch in<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; 21668.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; Is there any fundamental objection to adding the -lm flag to the<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; link step where it is necessary?<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>3. What is ossaudiodev?&nbsp; It tries to include "sys/soundcard.h", which<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; I don't have on my system.&nbsp;&nbsp; (The rule in setup.py is<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; wrapped in a test for host of Linux/FreeBSD/Darwin, but Android x86<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; gets configured with --host=i686-linux-android so to turn it off<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; requires an extra test for "and not cross_compiling".)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; Can I just turn off ossaudiodev for cross compiling or might<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; someone want it in a different type of cross build?&nbsp; (In which case<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; I think I'll have to write some kind autoconf rule for it, which I<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; don't quite know how to do yet.)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>4. Module _decimal is failing to compile.&nbsp; The problem is that it has<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; a header called memory.h.&nbsp; Android's libc has the problem that<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; /usr/include/stdlib.h includes &lt;memory.h&gt;.&nbsp; But the build system<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; puts -I. on the include path before the system dirs (as it should)<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; so when compiling _decimal, Modules/_decimal/libmpdec/memory.h gets<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; found instead of /usr/include/memory.h.&nbsp; Shiz has a patch here:<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; https://github.com/rave-engine/python3-android/blob/master/mk/python/3.3.5/p\<p></p></span></p>
<p class="MsoNormal"><span>ython-3.3.5-android-libmpdec.patch<p></p></span></p>
<p class="MsoNormal"><span>&nbsp; &nbsp;(which renames memory.h -&gt; mpmemory.h) but I don't know<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; a.&nbsp; Is there a tracker for this yet?&nbsp; and<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; b.&nbsp; Is Shiz's fix the desired one or should I be looking for<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; another approach?&nbsp; (Maybe modifying the -I flags for the build<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; of just the build of _decimal or something?)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>5. I'm not sure what test configure is actually doing for gethostby*()<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; in a cross-compile environment.&nbsp; In any case Android has a bug<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; where gethostbyaddr_r() is declared in the headers, but not<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; actually implemented in libc.&nbsp; So I have to modify my pyconfig.h by<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; hand to define HAVE_GETHOSTBYNAME and undef HAVE_GETHOSTBYNAME_R<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; and HAVE_GETHOSTBYNAME_R_6_ARG.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; Is there a variable (like ac_cv_little_endian_double) that I can<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; give to `configure` to make it set HAVE_GETHOSTBYNAME* the way I<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; need?&nbsp; If so I've been unable to figure it out.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>6. Android's &lt;pwd.h&gt; header mysteriously leaves the pw_gecos field out<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; of struct passwd.&nbsp; Is a fix like defining a new variable<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; HAVE_BROKEN_GECOS_FIELD the appropriate way to go with this?&nbsp; (If<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; this is an okay solution then the patch to Modules/pwdmodule.c is<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; shown below, but I still have to figure out how to patch<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; configure.ac to test for the condition and set the variable<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; appropriately, so a pointer to a similar block of code in<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; configure.ac would be appreciated.)<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Sorry for the TL;DR.&nbsp; I appreciate your having taken the time to read<p></p></span></p>
<p class="MsoNormal"><span>this far.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Thanks,<p></p></span></p>
<p class="MsoNormal"><span>-Matt<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Proposed patch for pwdmodule.c:<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>--- a/Modules/pwdmodule.c 2014-05-19 00:19:39.000000000 -0500<p></p></span></p>
<p class="MsoNormal"><span>+++ b/Modules/pwdmodule.c&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2014-10-21 18:00:35.676331205 -0500<p></p></span></p>
<p class="MsoNormal"><span> <at>  <at>  -57,6 +57,10  <at>  <at> <p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp; }<p></p></span></p>
<p class="MsoNormal"><span>}<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>+#if defined(HAVE_BROKEN_GECOS_FIELD)<p></p></span></p>
<p class="MsoNormal"><span>+static char fakePwGecos[256] = "";<p></p></span></p>
<p class="MsoNormal"><span>+#endif<p></p></span></p>
<p class="MsoNormal"><span>+<p></p></span></p>
<p class="MsoNormal"><span>static PyObject *<p></p></span></p>
<p class="MsoNormal"><span>mkpwent(struct passwd *p)<p></p></span></p>
<p class="MsoNormal"><span>{<p></p></span></p>
<p class="MsoNormal"><span> <at>  <at>  -72,7 +76,11  <at>  <at> <p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; SETS(setIndex++, p-&gt;pw_passwd);<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromUid(p-&gt;pw_uid));<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromGid(p-&gt;pw_gid));<p></p></span></p>
<p class="MsoNormal"><span>+#if !defined(HAVE_BROKEN_GECOS_FIELD)<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; SETS(setIndex++, p-&gt;pw_gecos);<p></p></span></p>
<p class="MsoNormal"><span>+#else<p></p></span></p>
<p class="MsoNormal"><span>+&nbsp;&nbsp;&nbsp; SETS(setIndex++, fakePwGecos);<p></p></span></p>
<p class="MsoNormal"><span>+#endif<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; SETS(setIndex++, p-&gt;pw_dir);<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;&nbsp;&nbsp;&nbsp; SETS(setIndex++, p-&gt;pw_shell);<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
</div>
</div>
Pekka Klärck | 22 Oct 12:15 2014
Picon
Picon

Re: Backporting ensurepip to 2.7, Which commands to install?

[Replying to a mail that was sent before I joined this list. Quoting,
headers, etc. aren't exactly right.]

Nick Coghlan wrote:
>On 4 October 2014 10:51, Donald Stufft <donald at stufft.io> wrote:
>> Whoops, I misred.
>>
>> So to be clear, you think:
>>
>> install -> pip, pip2, pip2.7
>> altinstall -> pip2.7
>
> To spell out the assumption I didn't make clear when helping with the
> backport PEP, the difference comes from PEP 394, which specifies the
> following behaviour when installing Python itself:
>
> Python 2.7: python, python2, python2.7
> Python 3.4: python3, python3.4
>
> That maps to ensurepip as:
>
> Python 2.7: pip, pip2, pip2.7
> Python 3.4: pip3, pip3.4

I just installed Python 3.4.2 on Windows and noticed that its Scripts
directory has these files out-of-the-box:

easy_install.exe
easy_install-3.4.exe.
pip.exe
pip3.exe
pip3.4.exe

Based on Nick's explanation above, having pip.exe there looks like bug
in the installer and could easily cause a conflict with other pip
installations. I don't understand why easy_install is included there
in the first place, but easy_install.exe can obviously cause a similar
conflict.

Cheers,
    .peke
--

-- 
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org
Philip Jenvey | 21 Oct 19:03 2014

PyPy3 2.4.0 released

=================================================
PyPy3 2.4 - Snow White
=================================================

We're pleased to announce PyPy3 2.4, which contains significant performance
enhancements and bug fixes.

You can download the PyPy3 2.4.0 release here:

    http://pypy.org/download.html

PyPy3 Highlights
================

Issues reported with our previous release were fixed after reports from users on
our new issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at
#pypy. Here is a summary of the user-facing PyPy3 specific changes:

* Better Windows compatibility, e.g. the nt module functions _getfinalpathname
  & _getfileinformation are now supported (the former is required for the
  popular pathlib library for example)

* Various fsencode PEP 383 related fixes to the posix module (readlink, uname,
  ttyname and ctermid) and improved locale handling

* Switched default binary name os POSIX distributions to 'pypy3' (which
  symlinks to to 'pypy3.2')

* Fixed a couple different crashes related to parsing Python 3 source code

Further Highlights (shared w/ PyPy2)
====================================

Benchmarks improved after internal enhancements in string and
bytearray handling, and a major rewrite of the GIL handling. This means
that external calls are now a lot faster, especially the CFFI ones. It also
means better performance in a lot of corner cases with handling strings or
bytearrays. The main bugfix is handling of many socket objects in your
program which in the long run used to "leak" memory.

We fixed a memory leak in IO in the sandbox_ code

We welcomed more than 12 new contributors, and conducted two Google
Summer of Code projects, as well as other student projects not
directly related to Summer of Code.

* Reduced internal copying of bytearray operations

* Tweak the internal structure of StringBuilder to speed up large string
  handling, which becomes advantageous on large programs at the cost of slightly
  slower small *benchmark* type programs.

* Boost performance of thread-local variables in both unjitted and jitted code,
  this mostly affects errno handling on linux, which makes external calls
  faster.

* Move to a mixed polling and mutex GIL model that make mutlithreaded jitted
  code run *much* faster

* Optimize errno handling in linux (x86 and x86-64 only)

* Remove ctypes pythonapi and ctypes.PyDLL, which never worked on PyPy

* Classes in the ast module are now distinct from structures used by
  the compiler, which simplifies and speeds up translation of our
  source code to the PyPy binary interpreter

* Win32 now links statically to zlib, expat, bzip, and openssl-1.0.1i.
  No more missing DLLs

* Many issues were resolved_ since the 2.3.1 release in June

.. _`whats-new`: http://doc.pypy.org/en/latest/whatsnew-2.4.0.html
.. _resolved: https://bitbucket.org/pypy/pypy/issues?status=resolved
.. _sandbox: http://doc.pypy.org/en/latest/sandbox.html

We have further improvements on the way: rpython file handling,
numpy linalg compatibility, as well
as improved GC and many smaller improvements.

Please try it out and let us know what you think. We especially welcome
success stories, we know you are using PyPy, please tell us about it!

Cheers

The PyPy Team
Andreas Maier | 21 Oct 18:43 2014
Picon

isinstance() on old-style classes in Py 2.7


Hi. Today, I ran across this, in Python 2.7.6:

>>> class C:
...   pass
...
>>> issubclass(C,object)
False
>>> isinstance(C(),object)
True   <-- ???

The description of isinstance() in Python 2.7 does not reveal this result
(to my reading).

From a duck-typing perspective, one would also not guess that an instance
of C would be considered an instance of object:

>>> dir(C())
['__doc__', '__module__']
>>> dir(object())
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__',
'__hash__', '__init__', '__new__', '__reduce__
', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__']

-> What is the motivation for isinstance(C,object) to return True in Python
2.7?

Andy

Andreas Maier
IBM Senior Technical Staff Member, Systems Management Architecture & Design
IBM Research & Development Laboratory Boeblingen, Germany
maiera <at> de.ibm.com, +49-7031-16-3654
________________________________________________________________________
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


Gmane