Paul Sokolovsky | 29 Mar 20:16 2015
Picon

Builtin functions are magically static methods?

Hello,

I looked into porting Python3 codecs module to MicroPython and saw
rather strange behavior, which is best illustrated with following
testcase:

==========
def foo(a):
    print("func:", a)

import _codecs
fun = _codecs.utf_8_encode
#fun = hash
#fun = str.upper
#fun = foo

class Bar:
    meth = fun

print(fun)
print(fun("foo"))
b = Bar()
print(b.meth("bar"))
==========

Uncommenting either _codecs.utf_8_encode or hash (both builtin
functions) produces 2 similar output lines, which in particular means
that its possible to call a native function as (normal) object method,
which then behaves as if it was a staticmethod - self is not passed to
a native function.
(Continue reading)

Victor Stinner | 28 Mar 10:57 2015
Picon

Buildbot PPC64 AIX 3.x failures

Hi,

There are many failures on the AIX buildbot. Can someone try to fix
them? Or would it be possible to turn off the buildbot to quickly see
regressions?

http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/3426/steps/test/logs/stdio

The buildbot has not enough memory, and some tests are failing since
more than 3 months.

Victor
Victor Stinner | 28 Mar 10:53 2015
Picon

OpenBSD buildbot has many failures

Hi,

The OpenBSD buildbot always fail with the same failures since many
months (ex: test_socket). Is someone interested to fix them? If no,
would it be possible to turn it off to get a better view of
regressions? Currently, they are too many red buildbots to quickly see
regressions introduced by recent commits.

http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x

Victor
Victor Stinner | 28 Mar 10:51 2015
Picon

Buildbot x86 XP-4 3.x doesn't compile anymore: drop it?

Hi,

The buildbot x86 XP-4 3.x doesn't compile anymore since 3 months or
more (maybe when Steve upgraded the Visual Studio project to VS 2015?
I don't know).

Would it be possible to fix this buildbot, or to turn it off?

By the way, do we seriously want to support Windows XP? I mean, *who*
will maintain it (no me sorry!). I saw recent changes to explicitly
*drop* support for Windows older than Visa (stop using GetTickCount,
always call GetTickCount64, for time.monotonic).

Victor
Victor Stinner | 28 Mar 10:48 2015
Picon

MemoryError and other bugs on AMD64 OpenIndiana 3.x

Hi,

The buildbot AMD64 OpenIndiana 3.x is always red since at least 6
months. Almost always, tests fail because the buildbot has not enough
memory. Well, it's fun to see how Python handles MemoryError (it's
pretty good, there is no hard crash! my work on failmalloc was maybe
useful ;-)), but it doesn't help to detect regressions.

Would it be possible to enhance this buildbot?

I copy this email to the owner of buildbot. But I also sent it to
python-dev because MemoryError is not the only error, please take care
of our buildbots!

Thanks,
Victor
Victor Stinner | 28 Mar 10:39 2015
Picon

Sporadic failures of test_subprocess and test_multiprocessing_spawn

Hi,

Can you please take a look at the following issue and try to reproduce it?
http://bugs.python.org/issue23771

The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and
"AMD64 Debian root 3.x" buildbots:

- test_notify_all() of test_multiprocessing_spawn
- test_double_close_on_error() of test_subprocess
- other sporadic failures of test_subprocess

I'm quite sure that they are regressions, maybe related to the
implementation of the PEP 475. In the middle of all PEP 475 changes, I
changed some functions to release the GIL on I/O, it wasn't the case
before. I may be related.

Are you able to reproduce these issues? I'm unable to reproduce them
on Fedora 21. Maybe they are more likely on Debian-like operating
systems?

Victor
Python tracker | 27 Mar 18:08 2015

Summary of Python tracker Issues


ACTIVITY SUMMARY (2015-03-20 - 2015-03-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4820 (+22)
  closed 30732 (+47)
  total  35552 (+69)

Open issues with patches: 2248 

Issues opened (55)
==================

#23571: Raise SystemError if a function returns a result with an excep
http://bugs.python.org/issue23571  reopened by berker.peksag

#23573: Avoid redundant allocations in str.find and like
http://bugs.python.org/issue23573  reopened by serhiy.storchaka

#23720: __del__() order is broken since 3.4.0
http://bugs.python.org/issue23720  reopened by Alexey Kazantsev

#23723: Provide a way to disable bytecode staleness checks
http://bugs.python.org/issue23723  opened by brett.cannon

#23725: update tempfile docs to say that TemporaryFile is secure
(Continue reading)

Victor Stinner | 26 Mar 13:30 2015
Picon

RFC: PEP 490 - Chain exceptions at C level

Hi,

Here is the first draft of my PEP to chain automatically exceptions at
C level in Python 3.5. Plain text version of the PEP below. HTML
version of the PEP:

   https://www.python.org/dev/peps/pep-0490/

It has already an implementation, the pyerr_chain.patch file attached to
http://bugs.python.org/issue23763

The patch is very short.

We may modify more functions to hide the previous exception when the
new exception contains as much or more information. Keeping the
previous exception waste memory, make reference cycles more likely and
is useless when displaying the full exception chain.

I wrote a huge patch modifying a lot of functions to explicitly *not*
chain exceptions: pyerr_match_clear-2.patch of the issue #23763. I
wrote the patch to show where exceptions will be chained with
pyerr_chain.patch applied. But we might apply it if we want to keep
exactly the same behaviour between Python 3.4 and 3.5 for C modules
the stdlib.

pyerr_match_clear-2.patch adds also checks on the type of the previous
exception to not replace an "unexpected" exception. For example, we
can replace an expected TypeError with a ValueError (without chaining
them), but we should not raise a ValueError if we got a
KeyboardInterrupt, a MemoryError or other unexpected exceptions. But
(Continue reading)

π | 24 Mar 16:53 2015
Picon

PiCxx

Hello PythonDevvers,

I apologise in advance. This is slightly off topic. This will be my only post on the subject.

PyCXX (http://sourceforge.net/projects/cxx/) accomplishes roughly the same as Boost::Python (C++ wrapper for CPython), only without requiring Boost.

The original code has become tangled and bloated.  
Over the winter I've rewritten it using C++11.  New C++11 features have allowed for more direct/compact/concise/readable code.

I've called my rewrite PiCxx and put it up here: https://github.com/p-i-/PiCxx

PiCxx only supports Python3 currently, as that is all I need for my own use.  It wouldn't be too much work to add 2.x support.  Also I've only tested it on my OSX system (although it is designed to be cross-platform).  It offers an alternative to Boost::Python that doesn't require Boost.

Improvements, suggestions, bug fixes, etc are all welcome.

Well, that's all. I originally subscribed to this list because I thought I might need some help navigating some of the dark corners of the CPython API, but thanks to a handful of Samurai on SO and IRC I seem to have scraped through.

I will unsubscribe in due course; it's been charming to spend awhile in the belly of the snake, and humbling to witness how hard you guys work.

Happy spring everyone,

π
<div>Hello PythonDevvers,<div class=""><br class=""></div>
<div class="">I apologise in advance. This is slightly off topic. This will be my only post on the subject.</div>
<div class=""><br class=""></div>
<div class="">PyCXX (<a href="http://sourceforge.net/projects/cxx/" class="">http://sourceforge.net/projects/cxx/</a>) accomplishes roughly the same as Boost::Python (C++ wrapper for CPython), only without requiring Boost.</div>
<div class=""><br class=""></div>
<div class="">The original code has become tangled and bloated. &nbsp;</div>
<div class="">Over the winter I've rewritten it using C++11. &nbsp;New C++11 features have allowed for more direct/compact/concise/readable code.</div>
<div class=""><br class=""></div>
<div class="">I've called my rewrite PiCxx and put it up here:&nbsp;<a href="https://github.com/p-i-/PiCxx" class="">https://github.com/p-i-/PiCxx</a>
</div>
<div class=""><br class=""></div>
<div class="">PiCxx only supports Python3 currently, as that is all I need for my own use. &nbsp;It wouldn't be too much work to add 2.x support. &nbsp;Also I've only tested it on my OSX system (although it is designed to be cross-platform). &nbsp;It offers an alternative to Boost::Python that doesn't require Boost.</div>
<div class=""><br class=""></div>
<div class="">Improvements, suggestions, bug fixes, etc are all welcome.</div>
<div class=""><br class=""></div>
<div class="">Well, that's all. I originally subscribed to this list because I thought I might need some help navigating some of the dark corners of the CPython API, but thanks to a handful of Samurai on SO and IRC I seem to have scraped through.</div>
<div class=""><br class=""></div>
<div class="">I will unsubscribe in due course; it's been charming to spend awhile in the belly of the snake, and humbling to witness how hard you guys work.</div>
<div class=""><br class=""></div>
<div class="">Happy spring everyone,</div>
<div class=""><br class=""></div>
<div class="">&pi;</div>
</div>
Karl Pickett | 24 Mar 15:28 2015

How is obmalloc safe with "Invalid read of size 4" ?

We are having random, rare, nonreproducible segfaults/hangs with python2 on ubuntu 14.04 in EC2.  I've managed to attach GDB to some hung ones and there looks like clear memory corruption in the 'interned' hash table, causing lookdict_string() to spin forever because all remaining slots have a garbage 'key' pointer.  This happens just loading the 'site' module dependencies, like 're' or 'codecs', before any of our code even gets run. 

So we then tried running it under valgrind, and we got a lot of nasty errors.  Even after reading the Misc/README.valgrind, which talks about *uninitialized* reads being ok, I still don't see how reading from *freed* memory would ever be safe, and why the suppression file thinks thats ok:

$ valgrind   ./pymd79/bin/python -c ""
==14651== Memcheck, a memory error detector
==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==14651== Command: ./pymd79/bin/python -c 
==14651== 
==14651== Invalid read of size 4
==14651==    at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911)
==14651==    by 0x461EA3: PyObject_Free (obmalloc.c:994)
==14651==    by 0x4789AB: tupledealloc (tupleobject.c:235)
==14651==    by 0x5225BA: code_dealloc (codeobject.c:309)
==14651==    by 0x4CFFC3: load_source_module (import.c:1100)
==14651==    by 0x4D0E16: import_submodule (import.c:2700)
==14651==    by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515)
==14651==    by 0x4AE49A: builtin___import__ (bltinmodule.c:49)
==14651==    by 0x422C89: PyObject_Call (abstract.c:2529)
==14651==    by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902)
==14651==    by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265)
==14651==    by 0x4B6B71: PyEval_EvalCode (ceval.c:667)
==14651==  Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801 free'd
==14651==    at 0x4C28577: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14651==    by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145)
==14651==    by 0x4CFE71: load_source_module (import.c:801)

- Karl
<div><div dir="ltr">We are having random, rare, nonreproducible segfaults/hangs with python2 on ubuntu 14.04 in EC2.&nbsp; I've managed to attach GDB to some hung ones and there looks like clear memory corruption in the 'interned' hash table, causing lookdict_string() to spin forever because all remaining slots have a garbage 'key' pointer.&nbsp; This happens just loading the 'site' module dependencies, like 're' or 'codecs', before any of our code even gets run.&nbsp;<div>
<div><br></div>
<div>So we then tried running it under valgrind, and we got a lot of nasty errors.&nbsp; Even after reading the Misc/README.valgrind, which talks about *uninitialized* reads being ok, I still don't see how reading from *freed* memory would ever be safe, and why the suppression file thinks thats ok:<br>
</div>
<div><br></div>
<div>
<div>$ valgrind &nbsp; ./pymd79/bin/python -c ""</div>
<div>==14651== Memcheck, a memory error detector</div>
<div>==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.</div>
<div>==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info</div>
<div>==14651== Command: ./pymd79/bin/python -c&nbsp;</div>
<div>==14651==&nbsp;</div>
<div>==14651== Invalid read of size 4</div>
<div>==14651== &nbsp; &nbsp;at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911)</div>
<div>==14651== &nbsp; &nbsp;by 0x461EA3: PyObject_Free (obmalloc.c:994)</div>
<div>==14651== &nbsp; &nbsp;by 0x4789AB: tupledealloc (tupleobject.c:235)</div>
<div>==14651== &nbsp; &nbsp;by 0x5225BA: code_dealloc (codeobject.c:309)</div>
<div>==14651== &nbsp; &nbsp;by 0x4CFFC3: load_source_module (import.c:1100)</div>
<div>==14651== &nbsp; &nbsp;by 0x4D0E16: import_submodule (import.c:2700)</div>
<div>==14651== &nbsp; &nbsp;by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515)</div>
<div>==14651== &nbsp; &nbsp;by 0x4AE49A: builtin___import__ (bltinmodule.c:49)</div>
<div>==14651== &nbsp; &nbsp;by 0x422C89: PyObject_Call (abstract.c:2529)</div>
<div>==14651== &nbsp; &nbsp;by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902)</div>
<div>==14651== &nbsp; &nbsp;by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265)</div>
<div>==14651== &nbsp; &nbsp;by 0x4B6B71: PyEval_EvalCode (ceval.c:667)</div>
<div>==14651== &nbsp;Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801 free'd</div>
<div>==14651== &nbsp; &nbsp;at 0x4C28577: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)</div>
<div>==14651== &nbsp; &nbsp;by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145)</div>
<div>==14651== &nbsp; &nbsp;by 0x4CFE71: load_source_module (import.c:801)</div>
</div>
<div><br></div>
<div>- Karl</div>
</div>
</div></div>
Andrea Griffini | 23 Mar 18:59 2015
Picon

--with-valgrind and --enable-shared

Hello,

does it have any sense for a linux distribution (arch to be specific) to provide default Python package compiled with valgrind support? I thought this flag was just about silencing false positives generated by valgrind (in other words a workaround for "bugs" of another software) and useful only when developing Python itself or C extensions.

The same distribution also compiles by default to a shared library and this has a quite noticeable impact on performance on x64 (surprisingly for me) for CPU bound processing; in a few test cases I measured valgrind+shared Python running at 66% the speed of plain ./configure && make Python on my system. Is this setting reasonable for general users?

If they are good defaults, why aren't them the default?

Andrea Griffini
<div><div dir="ltr">Hello,<div><br></div>
<div>does it have any sense for a linux distribution (arch to be specific) to provide default Python package compiled with valgrind support? I thought this flag was just about silencing false positives generated by valgrind (in other words a workaround for "bugs" of another software) and useful only when developing Python itself or C extensions.</div>
<div><br></div>
<div>The same distribution also compiles by default to a shared library and this has a quite noticeable impact on performance on x64 (surprisingly for me) for CPU bound processing; in a few test cases I measured valgrind+shared Python running at 66% the speed of plain ./configure &amp;&amp; make Python on my system. Is this setting reasonable for general users?</div>
<div><br></div>
<div>If they are good defaults, why aren't them the default?</div>
<div><br></div>
<div>Andrea Griffini</div>
</div></div>

Gmane