Benjamin Peterson | 24 Nov 04:18 2014

ubuntu buildbot

Hi David,
I noticed you run the "Builder x86 Ubuntu Shared" buildbot. It seems
it's running a very old version of Ubuntu. Is there any chance of
getting that updated?

Regards,
Benjamin
Glenn Linderman | 24 Nov 00:00 2014

Re: Move selected documentation repos to PSF BitBucket account?

On 11/23/2014 3:18 AM, Nick Coghlan wrote:
Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again.
Exactly. None of the patches I care about have gone anywhere for years. Two of them were done by others, apparently using the right processes, and still went nowhere, so it doesn't encourage me to attempt to even learn the process to make a more appropriate contribution of mine (which, sadly, I fixed too many different bugs in one file or two files, and have not split them into individual patches). Another has recently been making some progress... I guess that means I don't care enough about them, to fight to force them through the system, it is easier just to apply the patches to my installation... but that is a sad state of affairs.
<div>
    <div class="moz-cite-prefix">On 11/23/2014 3:18 AM, Nick Coghlan
      wrote:<br>
</div>
    <blockquote cite="mid:CADiSq7c_yVuK__woYrdKWr5QBwf_tJmP_G6pB6aahbxGuimBcQ <at> mail.gmail.com" type="cite">Patches getting held up in the review queue for weeks
      or months is a *huge* barrier to contribution, as it prevents the
      formation of the positive feedback cycle where having a
      contribution accepted feels good, so folks are more likely to want
      to contribute again.</blockquote>
    Exactly. None of the patches I care about have gone anywhere for
    years. Two of them were done by others, apparently using the right
    processes, and still went nowhere, so it doesn't encourage me to
    attempt to even learn the process to make a more appropriate
    contribution of mine (which, sadly, I fixed too many different bugs
    in one file or two files, and have not split them into individual
    patches). Another has recently been making some progress... I guess
    that means I don't care enough about them, to fight to force them
    through the system, it is easier just to apply the patches to my
    installation... but that is a sad state of affairs.<br>
</div>
Mark Shannon | 23 Nov 21:18 2014

Please reconsider PEP 479.

Hi,

I have serious concerns about this PEP, and would ask you to reconsider it.

[ Very short summary:
     Generators are not the problem. It is the naive use of next() in an 
iterator that is the problem. (Note that all the examples involve calls 
to next()).
     Change next() rather than fiddling with generators.
]

I have five main concerns with PEP 479.
1. Is the problem, as stated by the PEP, really the problem at all?
2. The proposed solution does not address the underlying problem.
3. It breaks a fundamental aspect of generators, that they are iterators.
4. This will be a hindrance to porting code from Python 2 to Python 3.
5. The behaviour of next() is not considered, even though it is the real 
cause of the problem (if there is a problem).

1. The PEP states that "The interaction of generators and StopIteration 
is currently somewhat surprising, and can conceal obscure bugs."
I don't believe that to be the case; if someone knows what StopIteration 
is and how it is used, then the interaction is entirely as expected.

I believe the naive use of next() in an iterator to be the underlying 
problem.
The interaction of generators and next() is just a special case of this.

StopIteration is not a normal exception, indicating a problem, rather it 
exists to signal exhaustion of an iterator.
However, next() raises StopIteration for an exhausted iterator, which 
really is an error.
Any iterator code (generator or __next__ method) that calls next() 
treats the StopIteration as a normal exception and propogates it.
The controlling loop then interprets StopIteration as a signal to stop 
and thus stops.
*The problem is the implicit shift from signal to error and back to signal.*

2. The proposed solution does not address this issue at all, but rather 
legislates against generators raising StopIteration.

3. Generators and the iterator protocol were introduced in Python 2.2, 
13 years ago.
For all of that time the iterator protocol has been defined by the 
__iter__(), next()/__next__() methods and the use of StopIteration to 
terminate iteration.

Generators are a way to write iterators without the clunkiness of 
explicit __iter__() and next()/__next__() methods, but have always 
obeyed the same protocol as all other iterators. This has allowed code 
to rewritten from one form to the other whenever desired.

Do not forget that despite the addition of the send() and throw() 
methods and their secondary role as coroutines, generators have 
primarily always been a clean and elegant way of writing iterators.

4. Porting from Python 2 to Python 3 seems to be hard enough already.

5. I think I've already covered this in the other points, but to 
reiterate (excuse the pun):
Calling next() on an exhausted iterator is, I would suggest, a logical 
error.
However, next() raises StopIteration which is really a signal to the 
controlling loop.
The fault is with next() raising StopIteration.
Generators raising StopIteration is not the problem.

It also worth noting that calling next() is the only place a 
StopIteration exception is likely to occur outside of the iterator protocol.

An example
----------

Consider a function to return the value from a set with a single member.
def value_from_singleton(s):
     if len(s) < 2:  #Intentional error here (should be len(s) == 1)
        return next(iter(s))
     raise ValueError("Not a singleton")

Now suppose we pass an empty set to value_from_singleton(s), then we get 
a StopIteration exception, which is a bit weird, but not too bad.

However it is when we use it in a generator (or in the __next__ method 
of an iterator) that we get a serious problem.
Currently the iterator appears to be exhausted early, which is wrong.
However, with the proposed change we get RuntimeError("generator raised 
StopIteration") raised, which is also wrong, just in a different way.

Solutions
---------
My preferred "solution" is to do nothing except improving the 
documentation of next(). Explain that it can raise StopIteration which, 
if allowed to propogate can cause premature exhaustion of an iterator.

If something must be done then I would suggest changing the behaviour of 
next() for an exhausted iterator.
Rather than raise StopIteration it should raise ValueError (or IndexError?).

Also, it might be worth considering making StopIteration inherit from 
BaseException, rather than Exception.

Cheers,
Mark.

P.S. 5 days seems a rather short time to respond to a PEP.
Could we make it at least a couple of weeks in the future,
or better still specify a closing date for comments.

Brett Cannon | 23 Nov 17:55 2014

Re: Move selected documentation repos to PSF BitBucket account?



On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan <ncoghlan <at> gmail.com> wrote:


On 23 Nov 2014 18:11, "Donald Stufft" <donald <at> stufft.io> wrote:
> > On Nov 23, 2014, at 2:35 AM, Nick Coghlan <ncoghlan <at> gmail.com> wrote:
> >

> > In the absence of a proposal to change version control systems
> > (again), the lack of Mercurial hosting on GitHub makes it rather a
> > moot point. Given that we can barely muster up any enthusiasm for
> > rehosting *without* changing version control systems (and the number
> > of CI systems that integrate with hg.python.org repos other than the
> > main CPython one is exactly zero), any proposal that involves doing
> > even *more* work seems doubly doomed.
> >
>
> I’d volunteer to do the work to get the PEPs, and possibly other repositories
> onto Github if we so decided to do so. Don’t let the lack of volunteer stop
> that because I will find the time to do it if need be.

It's the other way around: someone would have to volunteer to make the case that switching version control systems will actually help in any way whatsoever with the core reviewer bandwidth problem.

We do *not* have a shortage of people wanting to contribute. While ongoing outreach efforts are essential to promote increased diversity in the contributor base and to counter natural attrition, there is currently no major problem that needs solving on that front. CPython is high profile enough that folks are willing to do battle with the current complicated contribution process, so we're already one of the most active open source projects in the world, in *spite* of the problems with the existing workflow.


The immediate problem is making it easier to accept contributions from people. The long-term, never-ending problem is making the whole process of submitting a patch and getting it accepted as easy as possible for everyone involved, contributor and committer alike. If the goal is to make it so we can accept PRs for easier patch acceptances then that can be accomplished on either Bitbucket or GitHub. But if we want to make it easier for people to make contributions then GitHub is arguably better than Bitbucket, whether it's through familiarity of GitHub for most people thanks to other FLOSS projects or from the superior tooling around GitHub (both the platform itself and the ecosystem that has sprung up around it).
 

This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work).

Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch & upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward).

Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again.

In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online.

Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way.


I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee.

Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks.
 

From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure.


That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit.
 

It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL.


What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be.
 

Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day job, and you almost always want to go for the lowest impact, lowest risk approach that will alleviate the bottleneck you're worried about while changing the smallest possible number of elements in the overall workflow management system.


Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy.
 

That underlying calculation doesn't really change much even when the units shift from budget dollars to volunteers' time and energy.

So here is what I want to know to focus this discussion:

First, what new workflow are you proposing regardless of repo hosting provider? Are you proposing we maintain just mirrors and update the devguide to tell people to fork on the hosting provider, make their changes, generate a patch (which can be as simple as telling people how find the raw diff on the hosting provider), and then upload the patch the issue tracker just like today? Are you going farther and saying we have people send PRs on the hosting site, have them point to their PR in the issue tracker, and then we accept PRs (I'm going on the assumption we are not dropping our issue tracker)?

Second, to properly gauge the impact of switching from git to hg from an infrastructure perspective, what automation scripts do we have and how difficult would it be to update them to use git instead of hg? This is necessary simply to know where we would need to update URLs, let alone change in DVCS.

Third, do our release managers care about hg vs. git strongly? They probably use the DVCS the most directly and at a lower level by necessity compared to anyone else.

Fourth, do any core developers feel strongly about not using GitHub? Now please notice I said "GitHub" and not "git"; I think the proper way to frame this whole discussion is we are deciding if we want to switch to Bitbucket or GitHub who provide a low-level API for their version control storage service through hg or git, respectively. I personally dislike git, but I really like GitHub and I don't even notice git there since I use GitHub's OS X app; as I said, I view this as choosing a platform and not the underlying DVCS as I have happily chosen to access the GitHub hosting service through an app that is not git (it's like accessing a web app through it's web page or its REST API).

At least for me, until we get a clear understanding of what workflow changes we are asking for both contributors and core developers and exactly what work would be necessary to update our infrastructure for either Bitbucket or GitHub we can't really have a reasonable discussion that isn't going to be full of guessing.

And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases).

-Brett
 

Regards,
Nick.

>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
<div>
<br><br><div class="gmail_quote">On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan &lt;<a href="mailto:ncoghlan <at> gmail.com">ncoghlan <at> gmail.com</a>&gt; wrote:<br><blockquote class="gmail_quote">
<p dir="ltr"><br>
On 23 Nov 2014 18:11, "Donald Stufft" &lt;<a href="mailto:donald <at> stufft.io" target="_blank">donald <at> stufft.io</a>&gt; wrote:<br>
&gt; &gt; On Nov 23, 2014, at 2:35 AM, Nick Coghlan &lt;<a href="mailto:ncoghlan <at> gmail.com" target="_blank">ncoghlan <at> gmail.com</a>&gt; wrote:<br>
&gt; &gt;<br></p>
<p dir="ltr">
&gt; &gt; In the absence of a proposal to change version control systems<br>
&gt; &gt; (again), the lack of Mercurial hosting on GitHub makes it rather a<br>
&gt; &gt; moot point. Given that we can barely muster up any enthusiasm for<br>
&gt; &gt; rehosting *without* changing version control systems (and the number<br>
&gt; &gt; of CI systems that integrate with <a href="http://hg.python.org" target="_blank">hg.python.org</a> repos other than the<br>
&gt; &gt; main CPython one is exactly zero), any proposal that involves doing<br>
&gt; &gt; even *more* work seems doubly doomed.<br>
&gt; &gt;<br>
&gt;<br>
&gt; I&rsquo;d volunteer to do the work to get the PEPs, and possibly other repositories<br>
&gt; onto Github if we so decided to do so. Don&rsquo;t let the lack of volunteer stop<br>
&gt; that because I will find the time to do it if need be.</p>
<p dir="ltr"></p>
<p dir="ltr">It's the other way around: someone would have to volunteer to make the case that switching version control systems will actually help in any way whatsoever with the core reviewer bandwidth problem.</p>
<p dir="ltr">We do *not* have a shortage of people wanting to contribute. While ongoing outreach efforts are essential to promote increased diversity in the contributor base and to counter natural attrition, there is currently no major problem that needs solving on that front. CPython is high profile enough that folks are willing to do battle with the current complicated contribution process, so we're already one of the most active open source projects in the world, in *spite* of the problems with the existing workflow.</p>
</blockquote>
<div><br></div>
<div>The immediate problem is making it easier to accept contributions from people. The long-term, never-ending problem is making the whole process of submitting a patch and getting it accepted as easy as possible for everyone involved, contributor and committer alike. If the goal is to make it so we can accept PRs for easier patch acceptances then that can be accomplished on either Bitbucket or GitHub. But if we want to make it easier for people to make contributions then GitHub is arguably better than Bitbucket, whether it's through familiarity of GitHub for most people thanks to other FLOSS projects or from the superior tooling around GitHub (both the platform itself and the ecosystem that has sprung up around it).</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr"> </p>
<p dir="ltr">This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work).</p>
<p dir="ltr">Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch &amp; upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward).</p>
<p dir="ltr">Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again.</p>
<p dir="ltr">In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online.</p>
<p dir="ltr">Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way.</p>
</blockquote>
<div><br></div>
<div>I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee.</div>
<div><br></div>
<div>Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks.</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr">From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure.</p>
</blockquote>
<div><br></div>
<div>That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit.</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr">It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL.</p>
</blockquote>
<div><br></div>
<div>What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be.</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr">Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day job, and you almost always want to go for the lowest impact, lowest risk approach that will alleviate the bottleneck you're worried about while changing the smallest possible number of elements in the overall workflow management system.</p>
</blockquote>
<div><br></div>
<div>Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy.</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr">That underlying calculation doesn't really change much even when the units shift from budget dollars to volunteers' time and energy.</p>
</blockquote>
<div>So here is what I want to know to focus this discussion:</div>
<div><br></div>
<div>First, what new workflow are you proposing regardless of repo hosting provider? Are you proposing we maintain just mirrors and update the devguide to tell people to fork on the hosting provider, make their changes, generate a patch (which can be as simple as telling people how find the raw diff on the hosting provider), and then upload the patch the issue tracker just like today? Are you going farther and saying we have people send PRs on the hosting site, have them point to their PR in the issue tracker, and then we accept PRs (I'm going on the assumption we are not dropping our issue tracker)?</div>
<div><br></div>
<div>Second, to properly gauge the impact of switching from git to hg from an infrastructure perspective, what automation scripts do we have and how difficult would it be to update them to use git instead of hg? This is necessary simply to know where we would need to update URLs, let alone change in DVCS.</div>
<div><br></div>
<div>Third, do our release managers care about hg vs. git strongly? They probably use the DVCS the most directly and at a lower level by necessity compared to anyone else.</div>
<div><br></div>
<div>Fourth, do any core developers feel strongly about not using GitHub? Now please notice I said "GitHub" and not "git"; I think the proper way to frame this whole discussion is we are deciding if we want to switch to Bitbucket or GitHub who provide a low-level API for their version control storage service through hg or git, respectively. I personally dislike git, but I really like GitHub and I don't even notice git there since I use GitHub's OS X app; as I said, I view this as choosing a platform and not the underlying DVCS as I have happily chosen to access the GitHub hosting service through an app that is not git (it's like accessing a web app through it's web page or its REST API).</div>
<div><br></div>
<div>At least for me, until we get a clear understanding of what workflow changes we are asking for both contributors and core developers and exactly what work would be necessary to update our infrastructure for either Bitbucket or GitHub we can't really have a reasonable discussion that isn't going to be full of guessing.</div>
<div><br></div>
<div>And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases).</div>
<div><br></div>
<div>-Brett</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<p dir="ltr">Regards,<br>
Nick.</p>
<p dir="ltr">&gt;<br>
&gt; ---<br>
&gt; Donald Stufft<br>
&gt; PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br>
&gt;<br></p>
_______________________________________________<br>
Python-Dev mailing list<br><a href="mailto:Python-Dev <at> python.org" target="_blank">Python-Dev <at> python.org</a><br><a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/brett%40python.org" target="_blank">https://mail.python.org/mailman/options/python-dev/brett%40python.org</a><br>
</blockquote>
</div>
</div>
Benjamin Peterson | 23 Nov 05:52 2014

2.7.9 delayed

As you may have noticed, 2.7.9rc1 wasn't tagged today. I've been much
busier than I expected to be, so I haven't had the time to fix up some
2.7.9 lose ends. Hopefully, the delay won't be more than a few days.
Steve Dower | 22 Nov 23:10 2014
Picon

PCBuild project updates


​Hi all

Just attracting some attention to http://bugs.python.org/issue22919 for those of us who build
Python out of the PCBuild folder. More details/patches there, but the tl;dr is

* Still works with VS 2010 (and now VS 2013 and VS 2015 too)
* Build time should be reduced
* Tools\buildbot\*.bat are also updated, so buildbots shouldn't notice any change

If you're concerned/interested, I'll be watching the tracker for comments. Ideally I'd like at least a
couple of people to test build with whatever VS version they have, but if nobody can do that then I'll assume
that nobody will notice anything break sooner than me :)

Cheers,
Steve
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Tshepang Lekhonkhobe | 22 Nov 06:26 2014
Picon

Re: Move selected documentation repos to PSF BitBucket account?

On Fri, Nov 21, 2014 at 8:46 PM, Guido van Rossum <guido <at> python.org> wrote:
> Like it or not, github is easily winning this race.

Are you considering moving  CPython development to Github?
Python tracker | 21 Nov 18:08 2014

Summary of Python tracker Issues


ACTIVITY SUMMARY (2014-11-14 - 2014-11-21)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4658 (+11)
  closed 30014 (+28)
  total  34672 (+39)

Open issues with patches: 2176 

Issues opened (30)
==================

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869  reopened by pitrou

#22872: multiprocessing.Queue raises AssertionError
http://bugs.python.org/issue22872  opened by Joseph.Siddall

#22873: Re: SSLsocket.getpeercert - return ALL the fields of the certi
http://bugs.python.org/issue22873  opened by nagle

#22875: asyncio: call_soon() documentation unclear on timing
http://bugs.python.org/issue22875  opened by Corbin.Simpson

#22876: ip_interface can't be broadcast or number net
http://bugs.python.org/issue22876  opened by Вячеслав

#22881: show median in benchmark results
http://bugs.python.org/issue22881  opened by scoder

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882  opened by Ludovic.Gasc

#22883: Get rid of references to PyInt in Py3 sources
http://bugs.python.org/issue22883  opened by serhiy.storchaka

#22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std
http://bugs.python.org/issue22884  opened by keviv

#22885: Arbitrary code execution vulnerability due to unchecked eval()
http://bugs.python.org/issue22885  opened by stephen.farris

#22886: TestProgram leaves defaultTestLoader.errors dirty
http://bugs.python.org/issue22886  opened by rbcollins

#22888: ensurepip and distutils' build_scripts fails on Windows when p
http://bugs.python.org/issue22888  opened by gmljosea

#22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x
http://bugs.python.org/issue22890  opened by serhiy.storchaka

#22891: code removal from urllib.parse.urlsplit()
http://bugs.python.org/issue22891  opened by Alexander.Todorov

#22893: Idle: __future__ does not work in startup code.
http://bugs.python.org/issue22893  opened by terry.reedy

#22894: unittest.TestCase.subTest causes all subsequent tests to be sk
http://bugs.python.org/issue22894  opened by Trey.Cucco

#22895: test failure introduced by the fix for issue #22462
http://bugs.python.org/issue22895  opened by doko

#22896: Don't use PyObject_As*Buffer() functions
http://bugs.python.org/issue22896  opened by serhiy.storchaka

#22897: IDLE hangs on OS X with Cocoa Tk if encoding dialog is require
http://bugs.python.org/issue22897  opened by steipe

#22898: segfault during shutdown attempting to log ResourceWarning
http://bugs.python.org/issue22898  opened by emptysquare

#22899: http.server.BaseHTTPRequestHandler.parse_request (TypeError: d
http://bugs.python.org/issue22899  opened by recharti

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901  opened by bru

#22902: Use 'ip' for uuid.getnode()
http://bugs.python.org/issue22902  opened by bru

#22903: unittest creates non-picklable errors
http://bugs.python.org/issue22903  opened by pitrou

#22904: make profile-opt includes -fprofile* flags in _sysconfigdata C
http://bugs.python.org/issue22904  opened by gregory.p.smith

#22906: PEP 479: Change StopIteration handling inside generators
http://bugs.python.org/issue22906  opened by Rosuav

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907  opened by peko

#22908: ZipExtFile in zipfile can be seekable
http://bugs.python.org/issue22908  opened by Iridium.Yang

#22909: [argparse] Using parse_known_args, unknown arg with space in v
http://bugs.python.org/issue22909  opened by TabAtkins

#22910: test_pydoc test_synopsis_sourceless is a flaky test
http://bugs.python.org/issue22910  opened by gregory.p.smith

Most recent 15 issues with no replies (15)
==========================================

#22909: [argparse] Using parse_known_args, unknown arg with space in v
http://bugs.python.org/issue22909

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907

#22896: Don't use PyObject_As*Buffer() functions
http://bugs.python.org/issue22896

#22893: Idle: __future__ does not work in startup code.
http://bugs.python.org/issue22893

#22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x
http://bugs.python.org/issue22890

#22886: TestProgram leaves defaultTestLoader.errors dirty
http://bugs.python.org/issue22886

#22885: Arbitrary code execution vulnerability due to unchecked eval()
http://bugs.python.org/issue22885

#22883: Get rid of references to PyInt in Py3 sources
http://bugs.python.org/issue22883

#22872: multiprocessing.Queue raises AssertionError
http://bugs.python.org/issue22872

#22865: Allow pty.spawn to ignore data to copy
http://bugs.python.org/issue22865

#22859: unittest.TestProgram.usageExit no longer invoked
http://bugs.python.org/issue22859

#22853: Multiprocessing.Queue._feed deadlocks on import
http://bugs.python.org/issue22853

#22844: test_gdb failure on Debian Wheezy for Z
http://bugs.python.org/issue22844

#22832: Tweak parameter names for fcntl module
http://bugs.python.org/issue22832

#22831: Use "with" to avoid possible fd leaks
http://bugs.python.org/issue22831

Most recent 15 issues waiting for review (15)
=============================================

#22908: ZipExtFile in zipfile can be seekable
http://bugs.python.org/issue22908

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907

#22906: PEP 479: Change StopIteration handling inside generators
http://bugs.python.org/issue22906

#22904: make profile-opt includes -fprofile* flags in _sysconfigdata C
http://bugs.python.org/issue22904

#22902: Use 'ip' for uuid.getnode()
http://bugs.python.org/issue22902

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901

#22898: segfault during shutdown attempting to log ResourceWarning
http://bugs.python.org/issue22898

#22894: unittest.TestCase.subTest causes all subsequent tests to be sk
http://bugs.python.org/issue22894

#22891: code removal from urllib.parse.urlsplit()
http://bugs.python.org/issue22891

#22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std
http://bugs.python.org/issue22884

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882

#22881: show median in benchmark results
http://bugs.python.org/issue22881

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869

#22865: Allow pty.spawn to ignore data to copy
http://bugs.python.org/issue22865

#22848: Subparser help does not respect SUPPRESS argument
http://bugs.python.org/issue22848

Top 10 most discussed issues (10)
=================================

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869  12 msgs

#11145: '%o' % user-defined instance
http://bugs.python.org/issue11145  11 msgs

#18813: Speed up slice object processing
http://bugs.python.org/issue18813   8 msgs

#22536: subprocess should include filename in FileNotFoundError except
http://bugs.python.org/issue22536   6 msgs

#22823: Use set literals instead of creating a set from a list
http://bugs.python.org/issue22823   5 msgs

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882   5 msgs

#22895: test failure introduced by the fix for issue #22462
http://bugs.python.org/issue22895   5 msgs

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901   5 msgs

#14099: ZipFile.open() should not reopen the underlying file
http://bugs.python.org/issue14099   4 msgs

#17293: uuid.getnode() MAC address on AIX
http://bugs.python.org/issue17293   4 msgs

Issues closed (26)
==================

#18637: mingw: export _PyNode_SizeOf as PyAPI for parser module
http://bugs.python.org/issue18637  closed by serhiy.storchaka

#20434: Fix error handler of _PyString_Resize() on allocation failure
http://bugs.python.org/issue20434  closed by kristjan.jonsson

#20450: hg touch fails on System Z Linux buildbot
http://bugs.python.org/issue20450  closed by serhiy.storchaka

#20604: Missing invalid mode in error message of socket.makefile.
http://bugs.python.org/issue20604  closed by serhiy.storchaka

#20662: Pydoc doesn't escape parameter defaults in html
http://bugs.python.org/issue20662  closed by serhiy.storchaka

#20736: test_socket: testSendmsgDontWait needlessly skipped on Linux
http://bugs.python.org/issue20736  closed by serhiy.storchaka

#21266: test_zipfile fails to run in the installed location
http://bugs.python.org/issue21266  closed by serhiy.storchaka

#21963: 2.7.8 backport of Issue1856 (avoid daemon thread problems at s
http://bugs.python.org/issue21963  closed by pitrou

#22370: pathlib OS detection
http://bugs.python.org/issue22370  closed by python-dev

#22453: PyObject_REPR macro causes refcount leak
http://bugs.python.org/issue22453  closed by serhiy.storchaka

#22796: Support for httponly/secure cookies reintroduced lax parsing b
http://bugs.python.org/issue22796  closed by pitrou

#22825: Modernize TextFile
http://bugs.python.org/issue22825  closed by serhiy.storchaka

#22827: Backport ensurepip to 2.7 (PEP 477)
http://bugs.python.org/issue22827  closed by dstufft

#22847: Improve method cache efficiency
http://bugs.python.org/issue22847  closed by pitrou

#22850: Backport ensurepip Windows installer changes to 2.7
http://bugs.python.org/issue22850  closed by steve.dower

#22871: datetime documentation incomplete
http://bugs.python.org/issue22871  closed by belopolsky

#22874: gzip bug in python 2.7.3?
http://bugs.python.org/issue22874  closed by jbonerz

#22877: Backport ensurepip OS X installer changes to 2.7
http://bugs.python.org/issue22877  closed by ned.deily

#22878: PEP 477 (ensurepip backport to 2.7.9): "make install" and "mak
http://bugs.python.org/issue22878  closed by ned.deily

#22879: Make set's add and discard methods return useful information
http://bugs.python.org/issue22879  closed by rhettinger

#22880: hmac.new docs show optional args incorrectly
http://bugs.python.org/issue22880  closed by berker.peksag

#22887: ensurepip and distutils' build_scripts fails on Windows when p
http://bugs.python.org/issue22887  closed by ezio.melotti

#22889: set a timeout for DNS lookups
http://bugs.python.org/issue22889  closed by r.david.murray

#22892: Typo in Library's 'threading' module section
http://bugs.python.org/issue22892  closed by r.david.murray

#22900: decimal.Context Emin, Emax limits restrict functionality witho
http://bugs.python.org/issue22900  closed by ncoghlan

#22905: spam
http://bugs.python.org/issue22905  closed by ned.deily

ACTIVITY SUMMARY (2014-11-14 - 2014-11-21)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4658 (+11)
  closed 30014 (+28)
  total  34672 (+39)

Open issues with patches: 2176 

Issues opened (30)
==================

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869  reopened by pitrou

#22872: multiprocessing.Queue raises AssertionError
http://bugs.python.org/issue22872  opened by Joseph.Siddall

#22873: Re: SSLsocket.getpeercert - return ALL the fields of the certi
http://bugs.python.org/issue22873  opened by nagle

#22875: asyncio: call_soon() documentation unclear on timing
http://bugs.python.org/issue22875  opened by Corbin.Simpson

#22876: ip_interface can't be broadcast or number net
http://bugs.python.org/issue22876  opened by Вячеслав

#22881: show median in benchmark results
http://bugs.python.org/issue22881  opened by scoder

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882  opened by Ludovic.Gasc

#22883: Get rid of references to PyInt in Py3 sources
http://bugs.python.org/issue22883  opened by serhiy.storchaka

#22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std
http://bugs.python.org/issue22884  opened by keviv

#22885: Arbitrary code execution vulnerability due to unchecked eval()
http://bugs.python.org/issue22885  opened by stephen.farris

#22886: TestProgram leaves defaultTestLoader.errors dirty
http://bugs.python.org/issue22886  opened by rbcollins

#22888: ensurepip and distutils' build_scripts fails on Windows when p
http://bugs.python.org/issue22888  opened by gmljosea

#22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x
http://bugs.python.org/issue22890  opened by serhiy.storchaka

#22891: code removal from urllib.parse.urlsplit()
http://bugs.python.org/issue22891  opened by Alexander.Todorov

#22893: Idle: __future__ does not work in startup code.
http://bugs.python.org/issue22893  opened by terry.reedy

#22894: unittest.TestCase.subTest causes all subsequent tests to be sk
http://bugs.python.org/issue22894  opened by Trey.Cucco

#22895: test failure introduced by the fix for issue #22462
http://bugs.python.org/issue22895  opened by doko

#22896: Don't use PyObject_As*Buffer() functions
http://bugs.python.org/issue22896  opened by serhiy.storchaka

#22897: IDLE hangs on OS X with Cocoa Tk if encoding dialog is require
http://bugs.python.org/issue22897  opened by steipe

#22898: segfault during shutdown attempting to log ResourceWarning
http://bugs.python.org/issue22898  opened by emptysquare

#22899: http.server.BaseHTTPRequestHandler.parse_request (TypeError: d
http://bugs.python.org/issue22899  opened by recharti

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901  opened by bru

#22902: Use 'ip' for uuid.getnode()
http://bugs.python.org/issue22902  opened by bru

#22903: unittest creates non-picklable errors
http://bugs.python.org/issue22903  opened by pitrou

#22904: make profile-opt includes -fprofile* flags in _sysconfigdata C
http://bugs.python.org/issue22904  opened by gregory.p.smith

#22906: PEP 479: Change StopIteration handling inside generators
http://bugs.python.org/issue22906  opened by Rosuav

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907  opened by peko

#22908: ZipExtFile in zipfile can be seekable
http://bugs.python.org/issue22908  opened by Iridium.Yang

#22909: [argparse] Using parse_known_args, unknown arg with space in v
http://bugs.python.org/issue22909  opened by TabAtkins

#22910: test_pydoc test_synopsis_sourceless is a flaky test
http://bugs.python.org/issue22910  opened by gregory.p.smith

Most recent 15 issues with no replies (15)
==========================================

#22909: [argparse] Using parse_known_args, unknown arg with space in v
http://bugs.python.org/issue22909

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907

#22896: Don't use PyObject_As*Buffer() functions
http://bugs.python.org/issue22896

#22893: Idle: __future__ does not work in startup code.
http://bugs.python.org/issue22893

#22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x
http://bugs.python.org/issue22890

#22886: TestProgram leaves defaultTestLoader.errors dirty
http://bugs.python.org/issue22886

#22885: Arbitrary code execution vulnerability due to unchecked eval()
http://bugs.python.org/issue22885

#22883: Get rid of references to PyInt in Py3 sources
http://bugs.python.org/issue22883

#22872: multiprocessing.Queue raises AssertionError
http://bugs.python.org/issue22872

#22865: Allow pty.spawn to ignore data to copy
http://bugs.python.org/issue22865

#22859: unittest.TestProgram.usageExit no longer invoked
http://bugs.python.org/issue22859

#22853: Multiprocessing.Queue._feed deadlocks on import
http://bugs.python.org/issue22853

#22844: test_gdb failure on Debian Wheezy for Z
http://bugs.python.org/issue22844

#22832: Tweak parameter names for fcntl module
http://bugs.python.org/issue22832

#22831: Use "with" to avoid possible fd leaks
http://bugs.python.org/issue22831

Most recent 15 issues waiting for review (15)
=============================================

#22908: ZipExtFile in zipfile can be seekable
http://bugs.python.org/issue22908

#22907: Misc/python-config.sh.in: ensure sed invocations only match be
http://bugs.python.org/issue22907

#22906: PEP 479: Change StopIteration handling inside generators
http://bugs.python.org/issue22906

#22904: make profile-opt includes -fprofile* flags in _sysconfigdata C
http://bugs.python.org/issue22904

#22902: Use 'ip' for uuid.getnode()
http://bugs.python.org/issue22902

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901

#22898: segfault during shutdown attempting to log ResourceWarning
http://bugs.python.org/issue22898

#22894: unittest.TestCase.subTest causes all subsequent tests to be sk
http://bugs.python.org/issue22894

#22891: code removal from urllib.parse.urlsplit()
http://bugs.python.org/issue22891

#22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std
http://bugs.python.org/issue22884

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882

#22881: show median in benchmark results
http://bugs.python.org/issue22881

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869

#22865: Allow pty.spawn to ignore data to copy
http://bugs.python.org/issue22865

#22848: Subparser help does not respect SUPPRESS argument
http://bugs.python.org/issue22848

Top 10 most discussed issues (10)
=================================

#22869: Split pylifecycle.c out from pythonrun.c
http://bugs.python.org/issue22869  12 msgs

#11145: '%o' % user-defined instance
http://bugs.python.org/issue11145  11 msgs

#18813: Speed up slice object processing
http://bugs.python.org/issue18813   8 msgs

#22536: subprocess should include filename in FileNotFoundError except
http://bugs.python.org/issue22536   6 msgs

#22823: Use set literals instead of creating a set from a list
http://bugs.python.org/issue22823   5 msgs

#22882: Document Linux packages you need to compile Python with all de
http://bugs.python.org/issue22882   5 msgs

#22895: test failure introduced by the fix for issue #22462
http://bugs.python.org/issue22895   5 msgs

#22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre
http://bugs.python.org/issue22901   5 msgs

#14099: ZipFile.open() should not reopen the underlying file
http://bugs.python.org/issue14099   4 msgs

#17293: uuid.getnode() MAC address on AIX
http://bugs.python.org/issue17293   4 msgs

Issues closed (26)
==================

#18637: mingw: export _PyNode_SizeOf as PyAPI for parser module
http://bugs.python.org/issue18637  closed by serhiy.storchaka

#20434: Fix error handler of _PyString_Resize() on allocation failure
http://bugs.python.org/issue20434  closed by kristjan.jonsson

#20450: hg touch fails on System Z Linux buildbot
http://bugs.python.org/issue20450  closed by serhiy.storchaka

#20604: Missing invalid mode in error message of socket.makefile.
http://bugs.python.org/issue20604  closed by serhiy.storchaka

#20662: Pydoc doesn't escape parameter defaults in html
http://bugs.python.org/issue20662  closed by serhiy.storchaka

#20736: test_socket: testSendmsgDontWait needlessly skipped on Linux
http://bugs.python.org/issue20736  closed by serhiy.storchaka

#21266: test_zipfile fails to run in the installed location
http://bugs.python.org/issue21266  closed by serhiy.storchaka

#21963: 2.7.8 backport of Issue1856 (avoid daemon thread problems at s
http://bugs.python.org/issue21963  closed by pitrou

#22370: pathlib OS detection
http://bugs.python.org/issue22370  closed by python-dev

#22453: PyObject_REPR macro causes refcount leak
http://bugs.python.org/issue22453  closed by serhiy.storchaka

#22796: Support for httponly/secure cookies reintroduced lax parsing b
http://bugs.python.org/issue22796  closed by pitrou

#22825: Modernize TextFile
http://bugs.python.org/issue22825  closed by serhiy.storchaka

#22827: Backport ensurepip to 2.7 (PEP 477)
http://bugs.python.org/issue22827  closed by dstufft

#22847: Improve method cache efficiency
http://bugs.python.org/issue22847  closed by pitrou

#22850: Backport ensurepip Windows installer changes to 2.7
http://bugs.python.org/issue22850  closed by steve.dower

#22871: datetime documentation incomplete
http://bugs.python.org/issue22871  closed by belopolsky

#22874: gzip bug in python 2.7.3?
http://bugs.python.org/issue22874  closed by jbonerz

#22877: Backport ensurepip OS X installer changes to 2.7
http://bugs.python.org/issue22877  closed by ned.deily

#22878: PEP 477 (ensurepip backport to 2.7.9): "make install" and "mak
http://bugs.python.org/issue22878  closed by ned.deily

#22879: Make set's add and discard methods return useful information
http://bugs.python.org/issue22879  closed by rhettinger

#22880: hmac.new docs show optional args incorrectly
http://bugs.python.org/issue22880  closed by berker.peksag

#22887: ensurepip and distutils' build_scripts fails on Windows when p
http://bugs.python.org/issue22887  closed by ezio.melotti

#22889: set a timeout for DNS lookups
http://bugs.python.org/issue22889  closed by r.david.murray

#22892: Typo in Library's 'threading' module section
http://bugs.python.org/issue22892  closed by r.david.murray

#22900: decimal.Context Emin, Emax limits restrict functionality witho
http://bugs.python.org/issue22900  closed by ncoghlan

#22905: spam
http://bugs.python.org/issue22905  closed by ned.deily
Nick Coghlan | 21 Nov 13:36 2014
Picon

Move selected documentation repos to PSF BitBucket account?

For those that aren't aware, PEP 474 is a PEP I wrote a while back
suggesting we set up a "forge.python.org" service that provides easier
management of Mercurial repos that don't have the complex branching
requirements of the main CPython repo. Think repos like the PEPs repo,
or the developer guide repo.

The primary objective of the PEP is to enable an online editing + pull
request style workflow for these pure documentation projects that only
have a single active branch.

I'd been taking "must be hosted in PSF infrastructure" as a hard
requirement, but MAL pointed out earlier this evening that in the age
of DVCS's, that requirement may not make sense: if you avoid tightly
coupling your automation to a particular DVCS host's infrastructure,
then reverting back to self-hosting (if that becomes necessary for
some reason) is mostly just a matter of "hg push".

If that "must be self-hosted" constraint is removed, then the obvious
candidate for Mercurial hosting that supports online editing + pull
requests is the PSF's BitBucket account.

There'd still be some work in such a change to make sure we didn't
break automated regeneration of associated site elements, but that's
still a lot simpler than adding an entirely new piece of
infrastructure.

If folks are amenable to that variant of the idea, I'll undefer PEP
474 and revise it accordingly, with the developer guide and the PEP's
repo as the initially proposed candidates for transfer.

Regards,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
Guido van Rossum | 19 Nov 21:10 2014

PEP 479: Change StopIteration handling inside generators

There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate.

The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools.

In Python 3.5 the proposed change is conditional on:

    from __future__ import replace_stopiteration_in_generators

This would affect all generators (including generator expressions) compiled under its influence. The feature would become standard in Python 3.6 or 3.7.

To avoid a lot of requests for clarification you may also want to read up on the python-ideas discussion, e.g. here:

    https://groups.google.com/forum/#!topic/python-ideas/yJi1gRot9yY

I am leaning towards approving this PEP, but not until we've had a review here at python-dev. I would like to thank Chris Angelico for writing the original PEP draft.

--
--Guido van Rossum (python.org/~guido)
<div><div dir="ltr">
<div>
<div>
<div>There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate.<br><br>The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools.<br><br>
</div>In Python 3.5 the proposed change is conditional on:<br><br>
</div>&nbsp;&nbsp;&nbsp; from __future__ import replace_stopiteration_in_generators<br><br>
</div>This would affect all generators (including generator expressions) compiled under its influence. The feature would become standard in Python 3.6 or 3.7.<br><div><div>
<br><div>
<div>The PEP is here:<br><br>&nbsp;&nbsp;&nbsp; <a href="https://www.python.org/dev/peps/pep-0479/">https://www.python.org/dev/peps/pep-0479/</a><br><br>
</div>
<div>To avoid a lot of requests for clarification you may also want to read up on the python-ideas discussion, e.g. here:<br><br>&nbsp;&nbsp;&nbsp; <a href="https://groups.google.com/forum/#!topic/python-ideas/yJi1gRot9yY">https://groups.google.com/forum/#!topic/python-ideas/yJi1gRot9yY</a><br clear="all">
</div>
<div>
<div><br></div>
<div>I am leaning towards approving this PEP, but not until we've had a review here at python-dev. I would like to thank Chris Angelico for writing the original PEP draft.<br>
</div>
<div>
<br>-- <br><div class="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido">python.org/~guido</a>)</div>
</div>
</div>
</div>
</div></div>
</div></div>
Francis Giraldeau | 17 Nov 23:09 2014
Picon

Support for Linux perf

Hi, 

The PEP-418 is about performance counters, but there is no mention of performance management unit (PMU) counters, such as cache misses and instruction counts.

The Linux perf tool aims at recording these samples at the system level. I ran linux perf on CPython for profiling. The resulting callstack is inside libpython.so, mostly recursive calls to PyEval_EvalFrameEx(), because the tool works at the ELF level. Here is an example with a dummy program (linux-tools on Ubuntu 14.04): 

$ perf record python crunch.py
$ perf report --stdio
# Overhead  Command       Shared Object                            Symbol
# ........  .......  ..................  ................................
#
    32.37%   python  python2.7           [.] PyEval_EvalFrameEx          
    13.70%   python  libm-2.19.so        [.] __sin_avx                   
     5.25%   python  python2.7           [.] binary_op1.5010             
     4.82%   python  python2.7           [.] PyObject_GetAttr            

While this may be insightful for the interpreter developers, it it not so for the average Python developer. The report should display Python code instead. It seems obvious, still I haven't found the feature for that.

When a performance counter reaches a given value, a sample is recorded. The most basic sample only records a timestamps, thread ID and the program counter (%rip). In addition, all executable memory maps of libraries are recorded. For the callstack, frame pointers are traversed, but most of the time, they are optimized on x86, so there is a fall back to unwind, which requires saving register values and a chunk of the stack. The memory space of the process is reconstructed offline.

CPython seems to allocates code and frames on mmap() pages. If the data is outside about 1k from the top of stack, it is not available offline in the trace. We need some way to reconstitute this memory space of the interpreter to resolve the symbols, probably by  dumping the data on disk.

In Java, there is a small HotSpot agent that spits out the symbols of JIT code:


The problem is that CPython does not JIT code, and executed code is the ELF library itself. The executed frames are parameters of functions of the interpreter. I don't think the same approach can be used (maybe this can be applied to PyPy?). 

I looked at how Python frames are handled in GDB (file cpython/Tools/gdb/libpython.py). A python frame is detected in Frame(gdbframe).is_evalframeex() by a C call to PyEval_EvalFrameEx(). However, the traceback accesses PyFrameObject on the heap (at least for f->f_back = 0xa57460), which is possible in GDB when the program is paused and the whole memory space is available, but is not recorded for offline use in perf. Here is an example of callstack from GDB:

#0  PyEval_EvalFrameEx (f=Frame 0x7ffff7f1b060, for file crunch.py, line 7, in bar (num=466829), 
    throwflag=0) at ../Python/ceval.c:1039
#1  0x0000000000527877 in fast_function (func=<function at remote 0x7ffff6ec45a0>, 
    pp_stack=0x7fffffffd280, n=1, na=1, nk=0) at ../Python/ceval.c:4106
#2  0x0000000000527582 in call_function (pp_stack=0x7fffffffd280, oparg=1) at ../Python/ceval.c:4041


We could add a kernel module that "knows" how to make samples of CPython, but it means python structures becomes sort of ABI, and kernel devs won't allow a python interpreter in kernel mode ;-).

What we really want is f_code data and related objects:

(gdb) print (void *)(f->f_code)
$8 = (void *) 0x7ffff7e370f0

Maybe we could save these pages every time some code is loaded from the interpreter? (the memory range is about 1.7MB, but )

Anyway, I think we must change CPython to support tools such as perf. Any thoughts? 

Cheers,

Francis

<div><div dir="ltr">Hi,&nbsp;<div><br></div>
<div><span>The PEP-418 is about performance counters, but there is no mention of performance management unit (PMU) counters, such as cache misses and instruction counts.</span></div>
<div><span><br></span></div>
<div>
<span>The Linux perf tool aims at recording these samples at the system level.&nbsp;</span><span>I ran linux perf on CPython for profiling. The resulting callstack is inside libpython.so, mostly recursive calls to&nbsp;</span><span>PyEval_EvalFrameEx(), because the tool works at the ELF level. Here is an example with a dummy program (linux-tools on Ubuntu 14.04):&nbsp;</span>
</div>
<div><span><br></span></div>
<div><span>$ perf record python crunch.py</span></div>
<div><span>$ perf report --stdio</span></div>
<div>
<div><span># Overhead &nbsp;Command &nbsp; &nbsp; &nbsp; Shared Object &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Symbol</span></div>
<div><span># ........ &nbsp;....... &nbsp;.................. &nbsp;................................</span></div>
<div><span>#</span></div>
<div><span>&nbsp; &nbsp; 32.37% &nbsp; python &nbsp;python2.7 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [.] PyEval_EvalFrameEx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</span></div>
<div><span>&nbsp; &nbsp; 13.70% &nbsp; python &nbsp;<a href="http://libm-2.19.so" target="_blank">libm-2.19.so</a> &nbsp; &nbsp; &nbsp; &nbsp;[.] __sin_avx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span></div>
<div><span>&nbsp; &nbsp; &nbsp;5.25% &nbsp; python &nbsp;python2.7 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [.] binary_op1.5010 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</span></div>
<div><span>&nbsp; &nbsp; &nbsp;4.82% &nbsp; python &nbsp;python2.7 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [.] PyObject_GetAttr &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</span></div>
<div><br></div>
<div>While this may be insightful for the interpreter developers, it it not so for the average Python developer. The report should display Python code instead.&nbsp;<span>It seems obvious, still I haven't found the feature for that.</span>
</div>
<div><br></div>
<div>When a performance counter reaches a given value, a sample is recorded. The most basic sample only records a timestamps, thread ID and the program counter (%rip). In addition, all executable memory maps of libraries are recorded. For the callstack, frame pointers are traversed, but most of the time, they are optimized on x86, so there is a fall back to unwind, which requires saving register values and a chunk of the stack. The memory space of the process is reconstructed offline.</div>
<div><br></div>
<div>CPython seems to allocates code and frames on mmap() pages. If the data is outside about 1k from the top of stack, it is not available offline in the trace. We need some way to reconstitute this memory space of the interpreter to resolve the symbols, probably by &nbsp;dumping the data on disk.</div>
<div><br></div>
<div>In Java, there is a small HotSpot agent that spits out the symbols of JIT code:</div>
<div><br></div>
<div>
<span><a href="https://github.com/jrudolph/perf-map-agent" target="_blank">https://github.com/jrudolph/perf-map-agent</a></span><br>
</div>
<div><span><br></span></div>
<div><span>The problem is that CPython does not JIT code, and executed code is the ELF library itself. The executed frames are parameters of functions of the interpreter. I don't think the same approach can be used (maybe this can be applied to PyPy?).&nbsp;</span></div>
<div><span><br></span></div>
<div>I looked at how Python frames are handled in GDB (file&nbsp;cpython/Tools/gdb/libpython.py). A python frame is detected in Frame(gdbframe).is_evalframeex() by a C call to&nbsp;PyEval_EvalFrameEx(). However, the traceback accesses PyFrameObject on the heap (at least for f-&gt;f_back = 0xa57460), which is possible in GDB when the program is paused and the whole memory space is available, but is not recorded for offline use in perf. Here is an example of callstack from GDB:</div>
<div><br></div>
<div>
<div>#0 &nbsp;PyEval_EvalFrameEx (f=Frame 0x7ffff7f1b060, for file crunch.py, line 7, in bar (num=466829),&nbsp;</div>
<div>&nbsp; &nbsp; throwflag=0) at ../Python/ceval.c:1039</div>
<div>#1 &nbsp;0x0000000000527877 in fast_function (func=&lt;function at remote 0x7ffff6ec45a0&gt;,&nbsp;</div>
<div>&nbsp; &nbsp; pp_stack=0x7fffffffd280, n=1, na=1, nk=0) at ../Python/ceval.c:4106</div>
<div>#2 &nbsp;0x0000000000527582 in call_function (pp_stack=0x7fffffffd280, oparg=1) at ../Python/ceval.c:4041</div>
</div>
<div><br></div>
<div><br></div>
<div>We could add a kernel module that "knows" how to make samples of CPython, but it means python structures becomes sort of ABI, and kernel devs won't allow a python interpreter in kernel mode ;-).<br>
</div>
<div><br></div>
<div>What we really want is f_code data and related objects:</div>
<div>
<div><br></div>
<div>(gdb) print (void *)(f-&gt;f_code)</div>
<div>$8 = (void *) 0x7ffff7e370f0</div>
</div>
<div><br></div>
<div>Maybe we could save these pages every time some code is loaded from the interpreter? (the memory range is about 1.7MB, but )</div>
<div><br></div>
<div>Anyway, I think we must change CPython to support tools such as perf. Any thoughts?&nbsp;</div>
<div><br></div>
<div>Cheers,</div>
<div><br></div>
<div>Francis</div>
<div><br></div>
</div>
</div></div>

Gmane