Victor Stinner | 9 Feb 14:41
Gravatar

patch


Attachment (patch): application/octet-stream, 23 KiB
David Goulet | 9 Feb 00:03
Gravatar

ctypes/utils.py problem


Hi everyone,

I'm working with the LTTng (Linux Tracing) team and we came across a problem
with our user-space tracer and Python default behavior. We provide a libc
wrapper that instrument free() and malloc() with a simple ld_preload of that lib.

This lib *was* named "liblttng-ust-libc.so" and we came across python software
registering to our trace registry daemon (meaning that somehow the python binary
is using our in-process library). We dig a bit and found this:

Lib/ctypes/utils.py:

def _findLib_ldconfig(name):
	# XXX assuming GLIBC's ldconfig (with option -p)
        expr = r'/[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name)
        res = re.search(expr,
                        os.popen('/sbin/ldconfig -p 2>/dev/null').read())

and, at least, also found in _findLib_gcc(name) and _findSoname_ldconfig(name).

This cause Python to use any library ending with "libc.so" to be loaded....

I don't know the reasons behind this but we are concerned about "future issues"
that can occur with this kind of behavior.

For now, we renamed our lib so everything is fine.

Thanks a lot guys.
David
(Continue reading)

Mark Shannon | 8 Feb 20:18
Gravatar

PEP for new dictionary implementation

Proposed PEP for new dictionary implementation, PEP 410?
is attached.

Cheers,
Mark.
PEP: XXX
Title: Key-Sharing Dictionary
Version: $Revision$
Last-Modified: $Date$
Author: Mark Shannon <mark <at> hotpy.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Feb-2012
Python-Version: 3.3 or 3.4
Post-History: 08-Feb-2012

Abstract
========

This PEP proposes a change in the implementation of the builtin dictionary
type ``dict``. The new implementation allows dictionaries which are used as
attribute dictionaries (the ``__dict__`` attribute of an object) to share
keys with other attribute dictionaries of instances of the same class.

Motivation
==========

(Continue reading)

Mark Shannon | 8 Feb 18:13
Gravatar

Code review tool uses my old email address

Hi,

I changed my email address (about a year ago) and updated my bug tracker 
settings to my new address (late last year).
However, the code review tool still shows my old email address.
How do I change it?

Cheers,
Mark.
Eli Bendersky | 8 Feb 04:59
Picon
Gravatar

folding cElementTree behind ElementTree in 3.3

Hello,

Here's a note from "What's new in Python 3.0":

"""A common pattern in Python 2.x is to have one version of a module
implemented in pure Python, with an optional accelerated version
implemented as a C extension; for example, pickle and cPickle. This
places the burden of importing the accelerated version and falling
back on the pure Python version on each user of these modules. In
Python 3.0, the accelerated versions are considered implementation
details of the pure Python versions. Users should always import the
standard version, which attempts to import the accelerated version and
falls back to the pure Python version. The pickle / cPickle pair
received this treatment. The profile module is on the list for 3.1.
The StringIO module has been turned into a class in the io module."""

Is there a good reason why xml.etree.ElementTree /
xml.etree.cElementTree did not "receive this treatment"?

In the case of this module, it's quite unfortunate because:

1. The accelerated module is much faster and memory efficient (see
recent benchmarks here: http://bugs.python.org/issue11379), and XML
processing is an area where processing matters
2. The accelerated module implements the same API
3. It's very hard to even find out about the existence of the
accelerated module. Its sole mention in the docs is this un-emphasized
line in http://docs.python.org/dev/py3k/library/xml.etree.elementtree.html:

"A C implementation of this API is available as xml.etree.cElementTree."
(Continue reading)

Victor Stinner | 8 Feb 01:02
Gravatar

Add a new "locale" codec?

Hi,

I added PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() and
PyUnicode_EncodeLocale() to Python 3.3 to fix bugs. I hesitate to
expose this codec in Python: it can be useful is some cases,
especially if you need to interact with C functions.

The glib library has functions using the *current* locale encoding,
g_locale_from_utf8() for example.

Related issue with more information:
http://bugs.python.org/issue13619

Victor
Gregory P. Smith | 7 Feb 23:24
Favicon

Re: which C language standard CPython must conform to

On Tue, Feb 7, 2012 at 1:41 PM, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> Am 07.02.2012 20:10, schrieb Gregory P. Smith:
>> Why do we still care about C89?  It is 2012 and we're talking about
>> Python 3.  What compiler on what platform that anyone actually cares
>> about does not support C99?
>
> As Amaury says: Visual Studio still doesn't support C99. The story is
> both funny and sad: In Visual Studio 2002, the release notes included
> a comment that they couldn't consider C99 (in 2002), because of lack of
> time, and the standard came so quickly. In 2003, they kept this notice.
> In VS 2005 (IIRC), they said that there is too little customer demand
> for C99 so that they didn't implement it; they recommended to use C++
> or C#, anyway. Now C2011 has been published.

Thanks!  I've probably asked this question before.  Maybe I'll learn
this time. ;)

Some quick searching shows that there is at least hope Microsoft is on
board with C++11x (not so surprising, their crown jewels are written
in C++).  We should at some point demand a C++ compiler for CPython
and pick of subset of C++ features to allow use of but that is likely
reserved for the Python 4 timeframe (a topic for another thread and
time entirely, it isn't feasible for today's codebase).

In that timeframe another alternative Question may make sense to ask:
Do we need a single unified all-platform-from-one-codebase python
interpreter?

If we can get other VM implementations up to date language feature
wise and manage to sufficiently decouple standard library development
(Continue reading)

Brett Cannon | 7 Feb 21:07
Favicon

requirements for moving __import__ over to importlib?

I'm going to start this off with the caveat that hg.python.org/sandbox/bcannon#bootstrap_importlib is not completely at feature parity, but getting there shouldn't be hard. There is a FAILING file that has a list of the tests that are not passing because importlib bootstrapping and a comment as to why (I think) they are failing. But no switch would ever happen until the test suite passes.


Anyway, to start this conversation I'm going to open with why I think removing most of the C code in Python/import.c and replacing it with importlib/_bootstrap.py is a positive thing.

One is maintainability. Antoine mentioned how if change occurs everyone is going to have to be able to fix code  in importlib, and that's the point! I don't know about the rest of you but I find Python code easier to work with than C code (and if you don't you might be subscribed to the wrong mailing list =). I would assume the ability to make changes or to fix bugs will be a lot easier with importlib than import.c. So maintainability should be easier when it comes to imports.

Two is APIs. PEP 302 introduced this idea of an API for objects that can perform imports so that people can control it, enhance it, introspect it, etc. But as it stands right now, import.c implements none of PEP 302 for any built-in import mechanism. This mostly stems from positive thing #1 I just mentioned. but since I was able to do this code from scratch I was able to design for (and extend) PEP 302 compliance in order to make sure the entire import system was exposed cleanly. This means it is much easier now to write a custom importer for quirky syntax, a different storage mechanism, etc.

Third is multi-VM support. IronPython, Jython, and PyPy have all said they would love importlib to become the default import implementation so that all VMs have the same implementation. Some people have even said they will use importlib regardless of what CPython does simply to ease their coding burden, but obviously that still leads to the possibility of subtle semantic differences that would go away if all VMs used the same implementation. So switching would lead to one less possible semantic difference between the various VMs.

So, that is the positives. What are the negatives? Performance, of course.

Now I'm going to be upfront and say I really did not want to have this performance conversation now as I have done *NO* profiling or analysis of the algorithms used in importlib in order to tune performance (e.g. the function that handles case-sensitivity, which is on the critical path for importing source code, has a platform check which could go away if I instead had platform-specific versions of the function that were assigned to a global variable at startup). I also know that people have a bad habit of latching on to micro-benchmark numbers, especially for something like import which involves startup or can easily be measured. I mean I wrote importlib.test.benchmark to help measure performance changes in any algorithmic changes I might make, but it isn't a real-world benchmark like what Unladen Swallow gave us (e.g. the two start-up benchmarks that use real-world apps -- hg and bzr -- aren't available on Python 3 so only normal_startup and nosite_startup can be used ATM).

IOW I really do not look forward to someone saying "importlib is so much slower at importing a module containing ``pass``" when (a) that never happens, and (b) most programs do not spend their time importing but instead doing interesting work.

For instance, right now importlib does ``python -c "import decimal"`` (which, BTW, is the largest module in the stdlib) 25% slower on my machine with a pydebug build (a non-debug build would probably be in my favor as I have more Python objects being used in importlib and thus more sanity checks). But if you do something (very) slightly more interesting like ``python -m calendar`` where is a slight amount of work then importlib is currently only 16% slower. So it all depends on how we measure (as usual).

So, if there is going to be some baseline performance target I need to hit to make people happy I would prefer to know what that (real-world) benchmark is and what the performance target is going to be on a non-debug build. And if people are not worried about the performance then I'm happy with that as well. =)
Mark Janssen | 7 Feb 17:55
Picon

[Python-ideas] matrix operations on dict :)

On Mon, Feb 6, 2012 at 6:12 PM, Steven D'Aprano <steve <at> pearwood.info> wrote:
On Mon, Feb 06, 2012 at 09:01:29PM +0100, julien tayon wrote:
> Hello,
>
> Proposing vector operations on dict, and acknowledging there was an
> homeomorphism from rooted n-ary trees to dict, was inducing the
> possibility of making matrix of dict / trees.

This seems interesting to me, but I don't see that they are important
enough to be built-in to dicts. [...]


Otherwise, this looks rather like a library of functions looking for a
use. It might help if you demonstrate what concrete problems this helps
you solve.


I have the problem looking for this solution!

The application for this functionality is in coding a fractal graph (or "multigraph" in the literature).  This is the most powerful structure that Computer Science has ever conceived.  If you look at the evolution of data structures in compsci, the fractal graph is the ultimate.  From lists to trees to graphs to multigraphs.  The latter elements can always encompass the former with only O(1) extra cost.

It has the potential to encode *any* relationship from the very small to the very large (as well as across or *laterally*) in one unified structure.  Optimize this one data structure and the whole standard library could be refactored and simplified by an order of magnitude.  Not only that, it will pave the way for the "re-factored" internet that's being worked on which creates a content-centric Internet beyond the graph-level, hypertext internet.

Believe, it will be awesome.

Slowing down....

mark

Brett Cannon | 6 Feb 15:57
Favicon

need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping

So my grand quest for bootstrapping importlib into CPython is damn close to coming to fruition; I have one nasty bug blocking my way and I can't figure out what could be causing it. I'm hoping someone here will either know the solution off the top of their head or will have the time to have a quick look to see if they can figure it out as my brain is mush at this point.


First, the bug tracking all of this is http://bugs.python.org/issue2377 and the repo where I have been doing my work is ssh://hg <at> hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed if you want an HTTPS checkout). Everything works fine as long as you don't use sub-interpreters via test_capi (sans some test failures based on some assumptions which can easily be fixed; the bug I'm talking about is the only real showstopper at this point).

Here is the issue: if you run test_capi the code triggers an assertion of ``test_subinterps (__main__.TestPendingCalls) ... Assertion failed: (gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line 327.``. If you run the test under gdb you will discover that the assertion is related to ref counts when collecting for a generation (basically the ref updating is hitting 0 when it shouldn't).

Now the odd thing is that this is happening while importing frozen module code (something I didn't touch) which is calling marshal (something else I didn't touch) and while it is in the middle of unmarshaling the frozen module code it is triggering the assertion.

Does anyone have any idea what is going on? Am I possibly doing something stupid with refcounts which is only manifesting when using sub-interpreters? All relevant code for bootstrapping is contained in Python/pythonrun.c:import_init() (with a little tweaking in the _io module to delay importing the os module and making import.c always use __import__ instead of using the C code). I'm storing the __import__ function in the PyInterpreterState to keep separate state from the other interpreters (i.e. separate sys modules so as to use the proper sys.modules, etc.). But as I said, this all works in a single interpreter view of the world (the entire test suite doesn't trigger a nasty error like this).

Thanks for any help people can provide me on this now 5 year quest to get this work finished.

-Brett
Blockheads Oi Oi | 5 Feb 18:44
Picon
Favicon
Gravatar

Volunteer

You may remember me from a couple of years ago when I was trying to help 
out with Python.  Unfortunately I trod on a few toes.  I now know why. 
I have been diagnosed with Asperger Syndrome at 55 years old.
I would like to give it another go.
--

-- 
Cheers.

Mark Lawrence.


Gmane