Brett Cannon | 15 Apr 23:26 2014
Picon

Timing breakdown of Py_InitializeEx_Private()

To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E:

INIT: 
setlocale: 11
envvar: 2
random init: 2
interp creation: 15
thread creation: 6
GIL: 10
_Py_ReadyTypes(): 930
more types: 44
builtins: 141
exceptions: 287
sys: 258
_PyImport_Init: 15
import hooks: 4
_PyWarnings_Init(): 15
ENTERING import_init():
  PyImport_ImportFrozenModule(_frozen_importlib): 1186
  interp->importlib: 6
  PyInit_imp(): 15
  _imp: 3
  importlib._install(): 876
  _PyImportZip_Init(): 130
_PyFaulthandler_Init(): 13
time: 3
ENTERING initfsencoding():
  codec lookup: 2179
signals: 120
tracemalloc: 7
__main__: 13
initstdio(): 1568
warnings module: 4
initsite(): 9552
<div><div dir="ltr">To finish my timing work I decided to see where&nbsp;Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E:<div><br></div>
<div>
<div>INIT:&nbsp;</div>
<div>setlocale: 11</div>

<div>envvar: 2</div>
<div>random init: 2</div>
<div>interp creation: 15</div>
<div>thread creation: 6</div>
<div>GIL: 10</div>
<div>_Py_ReadyTypes(): 930</div>
<div>more types: 44</div>
<div>builtins: 141</div>
<div>exceptions: 287</div>

<div>sys: 258</div>
<div>_PyImport_Init: 15</div>
<div>import hooks: 4</div>
<div>_PyWarnings_Init(): 15</div>
<div>ENTERING import_init():</div>
<div>&nbsp; PyImport_ImportFrozenModule(_frozen_importlib): 1186</div>
<div>&nbsp; interp-&gt;importlib: 6</div>

<div>&nbsp; PyInit_imp(): 15</div>
<div>&nbsp; _imp: 3</div>
<div>&nbsp; importlib._install(): 876</div>
<div>&nbsp; _PyImportZip_Init(): 130</div>
<div>_PyFaulthandler_Init(): 13</div>
<div>time: 3</div>
<div>ENTERING initfsencoding():</div>
<div>

&nbsp; codec lookup: 2179</div>
<div>signals: 120</div>
<div>tracemalloc: 7</div>
<div>__main__: 13</div>
<div>initstdio(): 1568</div>
<div>warnings module: 4</div>
<div>initsite(): 9552</div>
</div>
</div></div>
Skip Montanaro | 15 Apr 17:34 2014
Picon

Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

Eric wrote:

> Perhaps not so much "a very real advantage" as "less of a
> distraction".  It's still significantly slower than 2.7.  :)

I'm still confused. I peeked in /usr/bin/hg. The only "system" modules
it imports directly are os and system (maybe I'm using an ancient
version?). After that, it imports its own lazy import module. This
suggests to me that Mercurial's import slowness is mostly in its own
code (I counted 104 Python modules and six shared objects in its
mercurial package, which isn't going to be affected (much?) by
freezing the Python standard modules.

I'm not trying to be difficult here. I thought that way back in the
day a huge amount of work was done to remove needless filesystem
activity, and zip packaging has been around for quite awhile.

As an experiment, I ran hg pull as

    /usr/bin/python -v /usr/bin/hg pull

in my cpython repo then looked at the -v output. Summarizing the
output I saw the following:

    30 imports (0 dlopens)

    Python banner printed

    86 imports (18 dlopens)

    adding changesets message

    5 imports (2 dlopens)

    adding manifests message

    1 import (0 dlopens)

    adding file changes message

    7 imports (3 dlopens)

    added ... changesets message

    4 imports (0 dlopens)

    run 'hg update' message

(I missed a "searching" message in there somewhere.)

That's a total of 133 imports, 23 of which were satisfied by loading
an extension module. The imports break down as follows:

51 imports (4 dlopens) from the mercurial package
5 imports from the hgext package
7 imports from the encodings package

Everything else is imported from the top level, and at a glance
appears to be all Python stdlib stuff.  The key period of execution
looks to be between the printing of the Python banner and the printing
of the adding changesets message. I see 46 imports (2 dlopens) from
the mercurial or hgext packages. That leaves 40 stdlib imports, of
which 16 were satisfied by dlopen.

As a final check, I saved all the stdlib import statements from the -v
output (77 statements) to a file and timed its execution:

    % time /usr/bin/python ~/tmp/pyimp.py

    real    0m0.162s
    user    0m0.034s
    sys     0m0.010s

It doesn't take much time to import every stdlib module that Mercurial
needs.  I don't know how much slower all this import machinery is in
3.x (and can't test at work, as we don't have a copy laying about). It
would probably have to be 3x or more slower for it to have much
visible impact on even simple hg commands.  I find it hard to believe
that freezing the stdlib is going to lower the barrier enough for the
Mercurial folks, if, in fact, import slowness is their main reason for
not moving to 3.x.

Skip
Brett Cannon | 14 Apr 23:51 2014
Picon

this is what happens if you freeze all the modules required for startup

It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get.

Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email.

Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent).

Thoughts?


--------------------------------------

default vs the freezing:

### normal_startup ###

Min: 0.524812 -> 0.473339: 1.11x faster

Avg: 0.534403 -> 0.481245: 1.11x faster

Significant (t=61.80)

Stddev: 0.00466 -> 0.00391: 1.1909x smaller


### startup_nosite ###

Min: 0.307359 -> 0.291939: 1.05x faster

Avg: 0.317667 -> 0.300156: 1.06x faster

Significant (t=26.29)

Stddev: 0.00543 -> 0.00385: 1.4099x smaller

---------

2.7 vs the freezing:

### normal_startup ###

Min: 0.367571 -> 0.465264: 1.27x slower

Avg: 0.374404 -> 0.476662: 1.27x slower

Significant (t=-90.26)

Stddev: 0.00313 -> 0.00738: 2.3603x larger


### startup_nosite ###

Min: 0.164510 -> 0.290544: 1.77x slower

Avg: 0.169833 -> 0.301109: 1.77x slower

Significant (t=-286.30)

Stddev: 0.00211 -> 0.00407: 1.9310x larger

---------

As a baseline, 2.7 vs default:

### normal_startup ###

Min: 0.368916 -> 0.521758: 1.41x slower

Avg: 0.376784 -> 0.531883: 1.41x slower

Significant (t=-172.82)

Stddev: 0.00423 -> 0.00474: 1.1207x larger


### startup_nosite ###

Min: 0.165156 -> 0.309090: 1.87x slower

Avg: 0.171516 -> 0.319004: 1.86x slower

Significant (t=-283.45)

Stddev: 0.00334 -> 0.00399: 1.1948x larger

<div><div dir="ltr">
<div>It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get.</div>

<div><br></div>
<div>Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email.</div>

<div><br></div>
<div>Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent).</div>

<div><br></div>
<div>Thoughts?</div>
<div><br></div>
<div><br></div>
<div>--------------------------------------</div>
<div><br></div>default vs the freezing:<div>

<p class="">### normal_startup ###</p>
<p class="">Min: 0.524812 -&gt; 0.473339: 1.11x faster</p>
<p class="">Avg: 0.534403 -&gt; 0.481245: 1.11x faster</p>
<p class="">Significant (t=61.80)</p>
<p class="">Stddev: 0.00466 -&gt; 0.00391: 1.1909x smaller</p>
<p class=""><br></p>
<p class="">### startup_nosite ###</p>
<p class="">Min: 0.307359 -&gt; 0.291939: 1.05x faster</p>
<p class="">Avg: 0.317667 -&gt; 0.300156: 1.06x faster</p>
<p class="">Significant (t=26.29)</p>
<p class="">Stddev: 0.00543 -&gt; 0.00385: 1.4099x smaller</p>
<p class="">---------</p>
<p class="">2.7 vs the freezing:</p>
<p class="">### normal_startup ###</p>
<p class="">Min: 0.367571 -&gt; 0.465264: 1.27x slower</p>
<p class="">

Avg: 0.374404 -&gt; 0.476662: 1.27x slower</p>
<p class="">Significant (t=-90.26)</p>
<p class="">Stddev: 0.00313 -&gt; 0.00738: 2.3603x larger</p>
<p class=""><br></p>
<p class="">### startup_nosite ###</p>
<p class="">Min: 0.164510 -&gt; 0.290544: 1.77x slower</p>

<p class="">Avg: 0.169833 -&gt; 0.301109: 1.77x slower</p>
<p class="">Significant (t=-286.30)</p>
<p class="">

</p>
<p class="">Stddev: 0.00211 -&gt; 0.00407: 1.9310x larger</p>
<p class="">---------</p>
<p class="">As a baseline, 2.7 vs default:</p>
<p class="">### normal_startup ###</p>
<p class="">Min: 0.368916 -&gt; 0.521758: 1.41x slower</p>

<p class="">Avg: 0.376784 -&gt; 0.531883: 1.41x slower</p>
<p class="">Significant (t=-172.82)</p>
<p class="">Stddev: 0.00423 -&gt; 0.00474: 1.1207x larger</p>
<p class=""><br></p>
<p class="">### startup_nosite ###</p>
<p class="">

Min: 0.165156 -&gt; 0.309090: 1.87x slower</p>
<p class="">Avg: 0.171516 -&gt; 0.319004: 1.86x slower</p>
<p class="">Significant (t=-283.45)</p>
<p class="">

</p>
<p class="">Stddev: 0.00334 -&gt; 0.00399: 1.1948x larger</p>
</div>
</div></div>
Steve Dower | 14 Apr 17:32 2014
Picon

Python "2migr8"


Just in case there's anyone out there who isn't yet sick of discussing how to proceed with Python 2.7, I have
some more inputs to contribute.

To put it up front, I'm totally against "CPython 2.8" ever becoming a real thing. Anything that comes out
should be seen as a migration path, not an upgrade path. I'll also admit I'm not heavily invested in working
on it myself, but I had a number of conversations during PyCon (as well as being at the language summit) that
puts me in a position to share the ideas and concerns that have been raised.

The main trigger was a conversation I had with two employees of a very large bank that has about 3000 Python
users (not developers - mostly financial analysts) and 16 million lines of code running on 2.7. They are
keen to migrate to 3.x but cannot afford to stop work entirely while their code is updated. (There was much
more to the conversation than I'm relating here - I'm keeping to the directly relevant bits.)

In describing the approach they'd like to take, they made me realise that there is definitely a place for a
Python that is different but mostly compatible with 2.7, in a way that 2.7.x could not be. For the sake of
having a name, I'll refer to this as "Python 2migr8" (pronounced "to migrate" :) ).

The two important components of Python 2migr8 would be the ability to disable 2.7-only features, and to do
so on a module-by-module basis.

My best idea so far would be to have a magic comment (to ensure 2.7 compatibility better than a "from
__future__ ...") near the top of the file that marks that file as "must straddle 2.7 and 3.3". Adding this
comment causes (for example) the parser to treat "except x, y" as a syntax error in this file, forces "from
__future__ import ...", hides "dict.iterkeys", undefines "basestring", etc., but only for this file.
(I haven't thought through all the possibilities or implications - Eric Snow said he was going to sprint on
this today/tomorrow, so he'll soon have a better idea just what can be done.)

In effect, 2migr8 would be the version that *only* supports "single-source" files. This allows large code
bases to progressively migrate modules from 2.x to single-source while continuing to run against Python
2.7. As files are updated, they are marked as being single-source. When all files have this marker, it
should be possible to flip the switch and run with Python 3.3 or later.

You could also think of this as enabling "-3 --warnings-as-errors" for individual modules, though since
the user has already opted in to 2migr8, it isn't unreasonable to make more significant changes, like
having dict.keys returning a list that warns if it is mutated. This sort of warning can only really be done
by changing the interpreter - static analysis just can't catch everything - and only when users accept a
potential performance hit and low probability of breakage when they move to 2migr8 (followed by a
not-quite-as-low probability of breaking when they eventually move from 2migr8 to 3.x, but it's still
better than guaranteed breakage).

As a fork, it would also be possible to bundle the modules that have been backported, and possibly also to
disallow importing deprecated stdlib modules when 2.7 functionality is disabled. As I said, I haven't
thought through all the possibilities, but the general idea is to take 2.7 and *remove* features so it
becomes easier to migrate.

Where does python-dev come in? Obviously this is where a fork like this would have to start - there has been
such strong and public opposition to any significant changes like this that you'd be hard pressed to find
someone willing to start and promote it from outside. There is also a good opportunity to make a start and
directly invite those using it to contribute the rules or warnings that they need - the 3000 Python "users"
I mentioned earlier are backed by a team of true developers who are more than capable of contributing, and
this would be a great opportunity to directly invite them. However unfair and incorrect it may be, there is
a perception in some businesses that open-source projects do not want contributions from them. I invited
more than one business to have someone join python
 -dev and get involved during PyCon, and I heard that others did the same - it may not be at the level of
employing a core developer full time, but it's the starting point that some companies will ne
 ed to be able to become comfortable with employing a core dev.

I'm not pretending to have a full plan on how this will work. I was privileged to have some private
conversations during PyCon that are directly relevant, so I'm bringing it here to promote the
discussion. Thanks to everyone I had a chance to chat to, and to everyone generally for a great PyCon.

Cheers,
Steve

 
Nathaniel Smith | 14 Apr 07:39 2014
Picon

[numpy wishlist] PyMem_*Calloc

Hi all,

The new tracemalloc infrastructure in python 3.4 is super-interesting
to numerical folks, because we really like memory profiling. Numerical
programs allocate a lot of memory, and sometimes it's not clear which
operations allocate memory (some numpy operations return views of the
original array without allocating anything; others return copies). So
people actually use memory tracking tools[1], even though
traditionally these have been pretty hacky (i.e., just checking RSS
before and after each line is executed), and numpy has even grown its
own little tracemalloc-like infrastructure [2], but it only works for
numpy data.

BUT, we also really like calloc(). One of the basic array creation
routines in numpy is numpy.zeros(), which returns an array full of --
you guessed it -- zeros. For pretty much all the data types numpy
supports, the value zero is represented by the bytestring consisting
of all zeros. So numpy.zeros() usually uses calloc() to allocate its
memory.

calloc() is more awesome than malloc()+memset() for two reasons.
First, calloc() for larger allocations is usually implemented using
clever VM tricks, so that it doesn't actually allocate any memory up
front, it just creates a COW mapping of the system zero page and then
does the actual allocation one page at a time as different entries are
written to. This means that in the somewhat common case where you
allocate a large array full of zeros, and then only set a few
scattered entries to non-zero values, you can end up using much much
less memory than otherwise. It's entirely possible for this to make
the difference between being able to run an analysis versus not.
memset() forces the whole amount of RAM to be committed immediately.

Secondly, even if you *are* going to touch all the memory, then
calloc() is still faster than malloc()+memset(). The reason is that
for large allocations, malloc() usually does a calloc() no matter what
-- when you get a new page from the kernel, the kernel has to make
sure you can't see random bits of other processes's memory, so it
unconditionally zeros out the page before you get to see it. calloc()
knows this, so it doesn't bother zeroing it again. malloc()+memset(),
by contrast, zeros the page twice, producing twice as much memory
traffic, which is huge.

SO, we'd like to route our allocations through PyMem_* in order to let
tracemalloc "see" them, but because there is no PyMem_*Calloc, doing
this would force us to give up on the calloc() optimizations.

The obvious solution is to add a PyMem_*Calloc to the API. Would this
be possible? Unfortunately it would require adding a new field to the
PyMemAllocator struct, which would be an ABI/API break; PyMemAllocator
is exposed directly in the C API and passed by value:
  https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator
(Too bad we didn't notice this a few months ago before 3.4 was
released :-(.) I guess we could just rename the struct in 3.5, to
force people to update their code. (I guess there aren't too many
people who would have to update their code.)

Thoughts?

-n

[1] http://scikit-learn.org/stable/developers/performance.html#memory-usage-profiling
[2] https://github.com/numpy/numpy/pull/309

--

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
Benjamin Peterson | 14 Apr 04:46 2014

your next regular installment of python 2.7

Yep, it's almost that time of the year again. My proposed schedule for
2.7.7 is

May 12th, 2.7.7rc1
May 26th, 2.7.7 final

That means if you want your 2.7 improvement to see the light of day
before November, the next few weeks are the time to do it. (PEP 446, I'm
looking at you.)

Also, I doubt anyone cares, but I'm going to do the last security
release of python 3.1 concurrently. That will, of course, be a
source-only release.

Regards,
Benjamin
Greg Mildenstein | 13 Apr 17:31 2014
Picon

python-3.4.0

Hi,
 
I'm running windows 8.1 64 bit. I was using 'python-3.3.1.amd64' but have uninstalled it and installed the above version. However, when I try running 'PyScripter', I get an error stating, 'ERROR - unable to open python, it will now exit'. Is there something I'm doing wrong?
 
Thanks
 
Greg Mildenstein
<div><div dir="ltr">Hi,<br>&nbsp;<br>I'm running windows 8.1 64 bit. I was using 'python-3.3.1.amd64' but have uninstalled it and installed the above version. However, when I try running 'PyScripter', I get an error stating, 'ERROR - unable to open python, it will now exit'. Is there something I'm doing wrong?<br>&nbsp;<br>Thanks<br>&nbsp;<br>Greg Mildenstein<br>
</div></div>
Nikolaus Rath | 12 Apr 20:58 2014

Appeal for reviews

Hello,

I've accumulated a number of patches in the issue tracker that are
waiting for someone to review/commit/reject them. I'm eager to make
corrections as necessary, I just need someone to look the work that I've
done so far:

* http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for
  non-blocking socket)

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)

* http://bugs.python.org/issue7776 (http.client.HTTPConnection
  tunneling is broken)

* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

* http://bugs.python.org/issue19414 (iter(ordered_dict) yields keys
  not in dict in some circumstances)

* http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
  missing)

* http://bugs.python.org/issue21057 (TextIOWrapper does not support
  reading bytearrays or memoryviews)

  
I realize that core developer time is scarce, so I have started to only
work on patches after I've confirmed that someone is available and
interested to review them. However, it would be great if some people
could find time to look at the issues above as well. Having your
contributions just languish in the bugtracker is really dispiriting... I
*want* to contribute, and I can't believe that Python is the one
open-source project that is not in need of more manpower. But for some
reason it seems really hard to actually find something to do..

Best,
Nikolaus

--

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Michael Foord | 12 Apr 17:56 2014
Picon

Fwd: Jython Report

Below is the Jython "status update" report on Jython I received from Jim Baker and summarised in the Language Summit. It comes with one addendum from Frank:

Jim's list is fantastic - the one bit I'd like to add to the list:

Jython now supports a buffer protocol that parallels CPython's C API buffer protocol. This provided the basis for support of buffer() and memoryview(). The work was done with Jython3 in mind and will be a huge boost to that eventual effort.

Begin forwarded message:

From: Jim Baker <jbaker <at> zyasoft.com>
Subject: Re: Jython Report
Date: 7 April 2014 06:42:51 BST
To: Michael Foord <michael <at> voidspace.org.uk>
Cc: Frank Wierzbicki <fwierzbicki <at> gmail.com>

Recent changes to trunk (last 6 months)

* Recently tagged a soft beta 2!
* Java 7 JVM is now the minimum version, which gives a larger base of functionality to work with (such as using Java 7's AutoCloseable to imply corresponding context manager support in using Python code)
* Enable mixing Python and Java types in the bases of a class when using a metaclass
* Added support for buffer, memoryview, although not complete yet with respect to Java integration
* Console and encoding support, such as unicodedata/idna updates
* Many, many small fixes

About to be in trunk, to support beta 3

* socket-reboot reimplements socket/select/ssl on top of Netty 4, a popular event loop networking framework for the JVM (used by a large number of performant projects in Java space and originally part of JBoss). There was no ssl support before, but now socket and especially select semantics are much closer to CPython as well (basically close to the Windows socket model). 
* socket-reboot in turn enables requests and thereby pip. A branch of pip currently works, actually modifying an upstream vendor lib (html5lib) so that it doesn't use isolated UTF-16 surrogates in literals, since this is not actually legal unicode, nor does it work in Jython's UTF-16 based representation. Ironically this usage is to detect such illegal use in input streams.
* Relative star imports, which seems to impact a number of interesting projects.
* Performance tuning of sre. Jython has a port of CPython's sre, however our use of UTF-16 requires expansion into an array of codepoints. Currently this is done on demand, which can potentially add another O(n) factor in evaluating regexes. A pull request we will apply memoizes. In the future, we will rewrite the logic in sre so that it does next/prev, much like JRuby currently does for similar encoding issues.

Related work

* Other PyPA tooling including virtualenv and wheel needs more diagnosis to see why they currently fail on Jython, but our hope is that this is minor. 
* New project jythontools by a number of Jython developers (including Frank and Jim). This includes a number of projects that will help evolve Jython, but outside the usual release schedule and the usual problem of being in core (such as eventual deprecation):
      - Clamp - precise integration with Java, enabling such capabilities as Java directly importing Python modules without explicitly initializing the Jython runtime or using object factories. Future work will enable Java annotation integration, as decorators. Integrates with setuptools; future integration as well with Maven via Aether.
      - Jiffy - provide a CFFI backend for Jython. Right now it is pure vaporware, but cursory examination of cffi.backend_ctypes suggests that it should be straightforward and of modest effort to provide a similar backend by using JFFI, which Jython and JRuby both use to access native runtime services (such as Posix API) as part of the Java native runtime project.
* The Patois project has been started to collect examples for cross-implementation support, as seen in surrogate support, but it will be a good question to get that really going, vs just talking about it.
* JyNI - simply adding this jar to the classpath enables C extension API support. Note that this project has been licensed by its developer (not a Jython committer) under an LGPL license.

Release schedule

* Complete beta 2
* Beta 3 is forthcoming, likely in 2 weeks
* For beta 4, need to perform a comprehensive bug triage - what will be in, not in for 2.7.0
* EuroPython sprint to finalize a release candidate for 2.7.0?

Future

* Mostly around performance, Java integration, and of course the usual bug fixes
* Python bytecode compiler remains important, including for support targeting Android and removing restriction on getting too large a method for the JVM
* More hooks for Java integration, including managing generated bytecode
* Integrating Zippy could provide for PyPy-like performance, but requires Graal JVM
* Supporting invokedynamic is a more realistic solution, but without the use of annotations (eg turn off Python frames) is going to be limited (maybe 2x?) based on earlier analysis

Jython 3.x?

* This comes up periodically, and it would be super nice for us to complete this support. At the very least it would make unicode strings and bytestrings correspond directly to how they are represented in Java, so that will be a nice cleanup.
* Release schedule: we will get there at some point!


On Sun, Apr 6, 2014 at 5:20 PM, Jim Baker <jbaker <at> zyasoft.com> wrote:
Michael,

I was thinking about this very topic this morning! Will send you the latest status in the next 24h, specifically our work to support pypa (setuptools, pip, virtualenv, wheel) and related tooling.

- Jim


On Sun, Apr 6, 2014 at 11:30 AM, Michael Foord <michael <at> voidspace.org.uk> wrote:
Hey guys,

Would you be able to write up a brief report on the current and future status of Jython, for me to read out at the Python language summit on Wednesday? (Unless someone who works on Jython will be there - but as far as I know that isn't the case.)

All the best,

Michael Foord

--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing
http://www.sqlite.org/different.html










--http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html





<div>Below is the Jython "status update" report on Jython I received from Jim Baker and summarised in the Language Summit. It comes with one addendum from Frank:<div><br></div>
<div>Jim's list is fantastic - the one bit I'd like to add to the list:<br><br>Jython now supports a buffer protocol that parallels CPython's C API buffer protocol. This provided the basis for support of buffer() and memoryview(). The work was done with Jython3 in mind and will be a huge boost to that eventual effort.<br><div>
<br><div>Begin forwarded message:</div>
<br class="Apple-interchange-newline"><blockquote type="cite">
<div>
<span>From: </span><span>Jim Baker &lt;<a href="mailto:jbaker <at> zyasoft.com">jbaker <at> zyasoft.com</a>&gt;<br></span>
</div>
<div>
<span>Subject: </span><span>Re: Jython Report<br></span>
</div>
<div>
<span>Date: </span><span>7 April 2014 06:42:51 BST<br></span>
</div>
<div>
<span>To: </span><span>Michael Foord &lt;<a href="mailto:michael <at> voidspace.org.uk">michael <at> voidspace.org.uk</a>&gt;<br></span>
</div>
<div>
<span>Cc: </span><span>Frank Wierzbicki &lt;<a href="mailto:fwierzbicki <at> gmail.com">fwierzbicki <at> gmail.com</a>&gt;<br></span>
</div>
<br><div>
<div dir="ltr">
<div>Recent changes to trunk (last 6 months)</div>
<div><br></div>
<div>* Recently tagged a soft beta 2!</div>
<div>* Java 7 JVM is now the minimum version, which gives a larger base of functionality to work with (such as using Java 7's AutoCloseable to imply corresponding context manager support in using Python code)</div>

<div>* Enable mixing Python and Java types in the bases of a class when using a metaclass</div>
<div>* Added support for buffer, memoryview, although not complete yet with respect to Java integration</div>
<div>* Console and encoding support, such as unicodedata/idna updates</div>

<div>* Many, many small fixes</div>
<div><br></div>
<div>About to be in trunk, to support beta 3</div>
<div><br></div>
<div>* socket-reboot reimplements socket/select/ssl on top of Netty 4, a popular event loop networking framework for the JVM (used by a large number of performant projects in Java space and originally part of JBoss). There was no ssl support before, but now socket and especially select semantics are much closer to CPython as well (basically close to the Windows socket model).&nbsp;</div>

<div>* socket-reboot in turn enables requests and thereby pip. A branch of pip currently works, actually modifying an upstream vendor lib (html5lib) so that it doesn't use isolated UTF-16 surrogates in literals, since this is not actually legal unicode, nor does it work in Jython's UTF-16 based representation. Ironically this usage is to detect such illegal use in input streams.</div>

<div>* Relative star imports, which seems to impact a number of interesting projects.</div>
<div>* Performance tuning of sre. Jython has a port of CPython's sre, however our use of UTF-16 requires expansion into an array of codepoints. Currently this is done on demand, which can potentially add another O(n) factor in evaluating regexes. A pull request we will apply memoizes. In the future, we will rewrite the logic in sre so that it does next/prev, much like JRuby currently does for similar encoding issues.</div>

<div><br></div>
<div>Related work</div>
<div><br></div>
<div>* Other PyPA tooling including virtualenv and wheel needs more diagnosis to see why they currently fail on Jython, but our hope is that this is minor.&nbsp;</div>
<div>
* New project jythontools by a number of Jython developers (including Frank and Jim). This includes a number of projects that will help evolve Jython, but outside the usual release schedule and the usual problem of being in core (such as eventual deprecation):</div>

<div>&nbsp; &nbsp; &nbsp; - Clamp - precise integration with Java, enabling such capabilities as Java directly importing Python modules without explicitly initializing the Jython runtime or using object factories. Future work will enable Java annotation integration, as decorators. Integrates with setuptools; future integration as well with Maven via Aether.</div>

<div>&nbsp; &nbsp; &nbsp; - Jiffy - provide a CFFI backend for Jython. Right now it is pure vaporware, but cursory examination of cffi.backend_ctypes suggests that it should be straightforward and of modest effort to provide a similar backend by using JFFI, which Jython and JRuby both use to access native runtime services (such as Posix API) as part of the Java native runtime project.</div>

<div>* The Patois project has been started to collect examples for cross-implementation support, as seen in surrogate support, but it will be a good question to get that really going, vs just talking about it.</div>
<div>
* JyNI - simply adding this jar to the classpath enables C extension API support. Note that this project has been licensed by its developer (not a Jython committer) under an LGPL license.</div>
<div><br></div>
<div>Release schedule</div>
<div><br></div>
<div>* Complete beta 2</div>
<div>* Beta 3 is forthcoming, likely in 2 weeks</div>
<div>* For beta 4, need to perform a comprehensive bug triage - what will be in, not in for 2.7.0</div>

<div>* EuroPython sprint to finalize a release candidate for 2.7.0?</div>
<div><br></div>
<div>Future</div>
<div><br></div>
<div>* Mostly around performance, Java integration, and of course the usual bug fixes</div>
<div>* Python bytecode compiler remains important, including for support targeting Android and removing restriction on getting too large a method for the JVM<br>
</div>
<div>* More hooks for Java integration, including managing generated bytecode</div>
<div>* Integrating Zippy could provide for PyPy-like performance, but requires Graal JVM</div>
<div>* Supporting invokedynamic is a more realistic solution, but without the use of annotations (eg turn off Python frames) is going to be limited (maybe 2x?) based on earlier analysis</div>

<div><br></div>
<div>Jython 3.x?</div>
<div><br></div>
<div>* This comes up periodically, and it would be super nice for us to complete this support. At the very least it would make unicode strings and bytestrings correspond directly to how they are represented in Java, so that will be a nice cleanup.</div>

<div>* Release schedule: we will get there at some point!</div>
</div>
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Sun, Apr 6, 2014 at 5:20 PM, Jim Baker <span dir="ltr">&lt;<a href="mailto:jbaker <at> zyasoft.com" target="_blank">jbaker <at> zyasoft.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">
<div dir="ltr">Michael,<div><br></div>
<div>I was thinking about this very topic this morning! Will send you the latest status in the next 24h, specifically our work to support pypa (setuptools, pip, virtualenv, wheel) and related tooling.</div>

<span class="HOEnZb">
<div><br></div>
<div>- Jim</div></span>
</div>
<div class="HOEnZb"><div class="h5">
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Sun, Apr 6, 2014 at 11:30 AM, Michael Foord <span dir="ltr">&lt;<a href="mailto:michael <at> voidspace.org.uk" target="_blank">michael <at> voidspace.org.uk</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">Hey guys,<br><br>
Would you be able to write up a brief report on the current and future status of Jython, for me to read out at the Python language summit on Wednesday? (Unless someone who works on Jython will be there - but as far as I know that isn't the case.)<br><br>
All the best,<br><br>
Michael Foord<br><br>
--<br><a href="http://www.voidspace.org.uk/" target="_blank">http://www.voidspace.org.uk/</a><br><br><br>
May you do good and not evil<br>
May you find forgiveness for yourself and forgive others<br>
May you share freely, never taking more than you give.<br>
-- the sqlite blessing<br><a href="http://www.sqlite.org/different.html" target="_blank">http://www.sqlite.org/different.html</a><br><br><br><br><br><br><br>
</blockquote>
</div>
<br>
</div>
</div></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br><div apple-content-edited="true">
<span class="Apple-style-span"><span class="Apple-style-span"><div>
<span class="Apple-style-span"><div>
<br class="Apple-interchange-newline">--<a class="moz-txt-link-freetext" href="http://www.voidspace.org.uk/">http://www.voidspace.org.uk/</a>

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing <a class="moz-txt-link-freetext" href="http://www.sqlite.org/different.html">http://www.sqlite.org/different.html</a><div><br></div>
</div></span><br class="Apple-interchange-newline">
</div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
Brett Cannon | 11 Apr 21:27 2014

Re: Receiving email updates for the bug tracker




On Fri, Apr 11, 2014 at 3:25 PM, Antony Lee <antony.lee <at> berkeley.edu> wrote:
Nope, the email is correct...

Then you can try reporting a bug at http://psf.upfronthosting.co.za/roundup/meta/
 

2014-04-11 12:12 GMT-07:00 Brett Cannon <brett <at> python.org>:




On Fri, Apr 11, 2014 at 1:47 PM, Antony Lee <antony.lee <at> berkeley.edu> wrote:
Hi,
Sorry for the slightly off-topic(?) question but I would like to know how to receive email notifications for activity on bugs I've opened on the bugs.python.org -- so far I don't receive anything even though I'm on the nosy list.

Should be automatic. I would check your email address in your account settings.

-Brett
 
Thanks,
Antony

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org




<div><div dir="ltr">
<br><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Apr 11, 2014 at 3:25 PM, Antony Lee <span dir="ltr">&lt;<a href="mailto:antony.lee <at> berkeley.edu" target="_blank">antony.lee <at> berkeley.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote"><div dir="ltr"><div class="gmail_extra">Nope, the email is correct...<br>
</div></div></blockquote>
<div><br></div>
<div>Then you can try reporting a bug at&nbsp;<a href="http://psf.upfronthosting.co.za/roundup/meta/">http://psf.upfronthosting.co.za/roundup/meta/</a>
</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">

<div dir="ltr"><div class="gmail_extra">
<br><div class="gmail_quote">2014-04-11 12:12 GMT-07:00 Brett Cannon <span dir="ltr">&lt;<a href="mailto:brett <at> python.org" target="_blank">brett <at> python.org</a>&gt;</span>:<div><div class="h5">

<br><blockquote class="gmail_quote">
<div dir="ltr">
<br><div class="gmail_extra">
<br><br><div class="gmail_quote">

<div>On Fri, Apr 11, 2014 at 1:47 PM, Antony Lee <span dir="ltr">&lt;<a href="mailto:antony.lee <at> berkeley.edu" target="_blank">antony.lee <at> berkeley.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote"><div dir="ltr">Hi,<div>Sorry for the slightly off-topic(?) question but I would like to know how to receive email notifications for activity on bugs I've opened on the <a href="http://bugs.python.org" target="_blank">bugs.python.org</a> -- so far I don't receive anything even though I'm on the nosy list.</div>

</div></blockquote>
<div><br></div>
</div>
<div>Should be automatic. I would check your email address in your account settings.</div>
<div><br></div>
<div>-Brett</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">

<div dir="ltr">
<div>Thanks,</div>
<div>Antony</div>
</div>
<br>_______________________________________________<br>
Python-Dev mailing list<br><a href="mailto:Python-Dev <at> python.org" target="_blank">Python-Dev <at> python.org</a><br><a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/brett%40python.org" target="_blank">https://mail.python.org/mailman/options/python-dev/brett%40python.org</a><br><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div></div>
</div>
<br>
</div></div>
</blockquote>
</div>
<br>
</div>
</div></div>
Antony Lee | 11 Apr 19:47 2014
Picon

Receiving email updates for the bug tracker

Hi,
Sorry for the slightly off-topic(?) question but I would like to know how to receive email notifications for activity on bugs I've opened on the bugs.python.org -- so far I don't receive anything even though I'm on the nosy list.
Thanks,
Antony
<div><div dir="ltr">Hi,<div>Sorry for the slightly off-topic(?) question but I would like to know how to receive email notifications for activity on bugs I've opened on the <a href="http://bugs.python.org">bugs.python.org</a> -- so far I don't receive anything even though I'm on the nosy list.</div>
<div>Thanks,</div>
<div>Antony</div>
</div></div>

Gmane