Magnus Morton | 9 Feb 00:01 2016

Question about "__extend__" hacks (in pypyjit module)


In the for the ‘pypyjit’ module there’s a comment in the
‘setup_after_space_initialization’ method "force the __extend__ hacks to occur early” (line
34) followed by two imports. What are these __extend__ hacks?

Many thanks,

pypy-dev mailing list
pypy-dev <at>
Robert Grosse | 7 Feb 05:55 2016

What is the last good build of Pypy?

I am trying to update Pypy. I want to build Pypy from source using the instructions at in order to get the latest bugfixes and improvements.

However, all of the last 5 nightly builds shown at failed their tests. So I can't tell what a good revision to update to is.

What is the last good revision of Pypy?
pypy-dev mailing list
pypy-dev <at>
Richard Plangger | 2 Feb 18:11 2016

s390x backend gil and threading issue


I'm currently searching for a problem, I have debugged for quite a long
time. I think this is the root problem why the pypy translation with
pypy is still slower than cpython. Here are some of my findings

The list of last tests that fail all have one thing in common: They have
an issue with the gil/threading. (See [1])
Most interesting ones are the last five.

* test_gc_locking (2x) fail on the build bot (only using cpython), but
not on my machine. This is strange because bbot and my vm use the same
distro, same compiler version, .... the only difference is that bbot has
better hardware and the tests are run with testrunner.
Is there another way I can reproduce this?

* test_ping_pong. (-A test) ping pong from one thread to another
stressing locking and the GIL switch. On s390x and the translated VM
this takes really long (10 seconds and on bbot it seems to exceed 30
seconds when run in parallel). However if I run the same test with
PYPYLOG=jit:- it completes in ~0.96 seconds (in gdb it is the same). If
you subtract the time needed for printing you might end up with the same
speed as x86 has for this test.
What does the printing/gdbing trigger to let the GIL switch happen that

* I have placed memory fences at the same positions as on ppc (2x isync
and lwsync). Are there any other places that need to complete all
pending the memory operations?

* There is one path in call_release_gil (just after the call) where
rpy_fastgil was acquired (because it as 0) and the shadowstack is not
the one of the current thread. Then *rpy_fastgil = 0 is set for the
slowpath function.
Wouldn't it be possible to steal the gil at this point? Would that lead
to a problem?



pypy-dev mailing list
pypy-dev <at>
Carl Friedrich Bolz | 2 Feb 13:11 2016

Call for Participation: Virtual Machine Summer School 2016

Call for Participation

Virtual Machines Summer School 2016 (#vmss16)
May 31 - June 3 2016
Cumberland Lodge, UK

Early Bird Registration Period: January 11 - February 15, 2016
Regular Registration Period: February 16 - April 30, 2016

The Virtual Machines Summer School (VMSS) will take place in
Cumberland Lodge, UK in summer 2016 just outside London. The aim of
the summer school is to give early career researchers (PhD students
and postdocs particularly) an overview of the field, and to meet some
of its top thinkers.

Venue: Cumberland Lodge is a former royal hunting lodge in Windsor
Great Park, a few miles south of Windsor castle. Since all
participants will be staying at the Lodge, a concentrated and
inspiring atmosphere is guaranteed.

Registration is subsidised by the EPRSC, so costs are low,
particularly for those who register early. There are travel grants
available for students that might otherwise be unable to attend the
summer school.

For further information and registration please visit:

Confirmed Speakers:

Antony Hosking (Purdue University)
Ben Titzer (Google)
Carl Friedrich Bolz (King's College London)
Chris Seaton (Oracle Labs)
Cliff Click (H2O)
Edd Barrett (King's College London)
Gilles Duboscq (Oracle Labs)
Jennifer Sartor (Vrije Universiteit Brussel)
Mario Wolczko (Oracle Labs)
Paul Tarau (University of North Texas)
Richard Jones (University of Kent)
Steve Blackburn (Australian National University)
Vyacheslav Egorov (Google)


Laurence Tratt & Carl Friedrich Bolz (King's College London)
Matti Picus | 29 Jan 15:49 2016

windows and python27.lib

When linking to a dll with MSVC, one needs to provide not the dll itself 
rather an import library. Also, the linker must be able to resolve all 
function definitions in order to run to completion, so the linker must 
be able to find the import library.
On cpython, the import library used to link to python27.dll is called 
Python27.lib, it is located in a 'libs' directory at the top level in a 
cpython distribution together with things like pyexpat.lib, _socket.lib 
etc. distutils (and setuptools) do not need to know about the import 
library, since a #pragma is used to issue a link directive for the 
import library whenever pyconfig.h is included in a c file. A debug 
build will create and use python27_d.lib that corresponds to python27_d.dll

What do we do in PyPy?
At translation we build a libpypy-c.dll, and an export library 
libpypy-c.lib. These are copied together with pypy-c.exe to the 
directory where translation occurred (on the build bots this is in 
pypy/goal) Then  the package script copies the libpypy-c.lib to pypy's 
include directory as python27.lib. A debug build will use these same 
names. The same pragma is used as in cpython to force linking with the 
import library whenever pyconfig.h is included

So what you ask?
I think the exe should be created as pypy.exe, the dll should be called 
pypy27.dll, and the import library should be consistently named 
pypy27.lib. There should be no renaming in This has 
implications in the following places:
- the exe_name in targetpypystandalone should drop the -%(backend) modifier
- pyconfig.h and the package script should be modified to use pypy27 
- probably some tests will fail, they should be fixed
- cffi/ needs tweaking in _apply_embedding_fix
- should not rename (what do we do on linux about pypy-c -> 
what did I forget?

We should also handle a debug build of pypy, we should be creating a 
pypy27_d.lib, pypy27_d.dll, today the usual names are reused.

Are there compelling reasons _not_ to make the naming consistent with 
cpython? As a fallback, we could just rename the import library to 

My current motivation to do this is that _apply_embedding_fix does not 
work for win32 pypy

Richard Plangger | 28 Jan 14:37 2016

Stack limit in the jit backends


the file rpython/translator/c/src/stack.h defines MAX_STACK_SIZE. PPC
has a bigger limit than e.g. x86. I experienced that on s390x there is
as well a higher consumption of memory for stack frames (they are
variable sized with a pretty high minimum limit (160 bytes) by the ABI).

I have two questions:

1) The OS (i.e. linux) defines a stack limit (ulimit -s), does pypy
overwrite this value with MAX_STACK_SIZE?

2) How would I determine which size is best for s390x? Or how did we
come up with 768kb for x86, and 2.8mb for ppc?


pypy-dev mailing list
pypy-dev <at>
Sergey Matyunin | 23 Jan 17:24 2016

partition in numpypy


Need a little help with numpypy. I want to implement partition method
for numpy array. Let's say I can compile npy_partition.h.src, import
it through CFFI. Therefore I can write a python function
my_partition(numpy_array, other_arguments...) that performs
partitioning for a give numpy array.

Now I want to create partition method for ndarray.
As far as I understand methods of ndarray are defined in
pypy/module/micronumpy/ in a special way and
ndarray.partition = my_partition doesn't work.

Is it possible to add method to ndarray inside numpypy, not inside
pypy's micronumpy module?


Nagy, Attila | 21 Jan 10:47 2016

TypeError: expected string, got NoneType object with setuptools and co with pypy 4


After installing pypy 4.0.1 on FreeBSD (from ports), I get the above 
exception when trying to install setuptools, or when installing anything 
with pip (after pypy -m ensurepip).

This all worked with the previous version (2.6).

I could find this problem elsewhere:
so it doesn't seem to be related to FreeBSD.

An example full trace:
# pypy -m pip install --upgrade pip
You are using pip version 6.1.1, however version 8.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip
   Traceback (most recent call last):
     File "/usr/local/pypy-4.0/site-packages/pip/", line 
246, in main
       status =, args)
     File "/usr/local/pypy-4.0/site-packages/pip/commands/", 
line 342, in run
     File "/usr/local/pypy-4.0/site-packages/pip/req/", line 
345, in prepare_files
       functools.partial(self._prepare_file, finder))
     File "/usr/local/pypy-4.0/site-packages/pip/req/", line 
290, in _walk_req_to_install
       more_reqs = handler(req_to_install)
     File "/usr/local/pypy-4.0/lib_pypy/", line 42, in __call__
       return self._func(*(self._args + fargs), **fkeywords)
     File "/usr/local/pypy-4.0/site-packages/pip/req/", line 
487, in _prepare_file
       download_dir, do_download, session=self.session,
     File "/usr/local/pypy-4.0/site-packages/pip/", line 827, 
in unpack_url
     File "/usr/local/pypy-4.0/site-packages/pip/", line 673, 
in unpack_http_url
       from_path, content_type = _download_http_url(link, session, temp_dir)
     File "/usr/local/pypy-4.0/site-packages/pip/", line 887, 
in _download_http_url
       with open(file_path, 'wb') as content_file:
   TypeError: expected string, got NoneType object

And the one for setuptools:
# pypy install
running install
Checking .pth file support in /usr/local/pypy-4.0/site-packages/
/usr/local/bin/pypy -E -c pass
TEST PASSED: /usr/local/pypy-4.0/site-packages/ appears to support .pth 
running bdist_egg
running egg_info
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
writing requirements to setuptools.egg-info/requires.txt
writing top-level names to setuptools.egg-info/top_level.txt
Traceback (most recent call last):
   File "", line 169, in <module>
     dist = setuptools.setup(**setup_params)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 
151, in setup
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 
953, in run_commands
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 
972, in run_command
   File "/tmp/setuptools-19.4/setuptools/command/", line 67, 
in run
   File "/tmp/setuptools-19.4/setuptools/command/", line 109, 
in do_egg_install
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 326, 
in run_command
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 
972, in run_command
   File "/tmp/setuptools-19.4/setuptools/command/", line 
152, in run
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 326, 
in run_command
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/", line 
972, in run_command
   File "/tmp/setuptools-19.4/setuptools/command/", line 186, 
in run
   File "/tmp/setuptools-19.4/setuptools/command/", line 209, 
in find_sources
   File "/tmp/setuptools-19.4/setuptools/command/", line 293, 
in run
   File "/tmp/setuptools-19.4/setuptools/command/", line 322, 
in add_defaults
   File "/tmp/setuptools-19.4/setuptools/command/", line 100, in 
   File "/tmp/setuptools-19.4/setuptools/command/", line 236, 
in append
     if self._safe_path(path):
   File "/tmp/setuptools-19.4/setuptools/command/", line 256, 
in _safe_path
     u_path = unicode_utils.filesys_decode(path)
   File "/tmp/setuptools-19.4/setuptools/", line 31, in 
     return path.decode(enc)
TypeError: expected string, got NoneType object

In this case the context is:
def filesys_decode(path):
     Ensure that the given path is decoded,
     NONE when no expected encoding works

     fs_enc = sys.getfilesystemencoding()
     if isinstance(path, six.text_type):
         return path

     for enc in (fs_enc, "utf-8"):
             return path.decode(enc)
         except UnicodeDecodeError:

On python 2.7, fs_enc here is:
Python 2.7.11 (default, Dec 20 2015, 01:15:21)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 
208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> repr(sys.getfilesystemencoding())

While on pypy:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Jan 16 2016, 
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> repr(sys.getfilesystemencoding())

Returning None here is fine according to the docs, but the above code 
snippet doesn't handle the TypeError, which it gets when doing 
How this could work?

On pypy 2.6 on the same machine:
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jun 12 2015, 
[PyPy 2.6.0] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
No exception!

On pypy 4.0.1:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Dec 10 2015, 
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: expected string, got NoneType object

Unhandled TypeError exception!

python 2.7 works the same way:
>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: decode() argument 1 must be string, not None

Which installs setuptools just fine (although it doesn't get to the 
above snippet, because it doesn't return None to 
If I set fs_enc to None in the above snippet, even python 2.7 fails.

So the key here (if I'm not completely lost) seems to be not returning 
None to the getfilesystemencoding to fix broken(?) software.

What do you think about this?

Richard Plangger | 19 Jan 12:12 2016

s390x the last failing tests


I wanted to give a quick update on the state of the implementation. Good
news! I think there is not that much left to be done!

I'm currently waiting for a bigger VM (already wrote an email to
linux1 <at>, 2 days ago? They are maybe on holiday?) to translate
the full project.

There are approx. 20 Failing tests that are left (own-linux-s390x). All
other pass on my virtual machine. They are mostly related to big endian
issues. Here are some questions:

1) Generally I got the impression that there are some tests that do not
consider endianess (e.g. micronumpy). I guess it is time to change them
to handle this? What about PPC? Did those not come up there?

2) It seems that the gcc on the build bot is quite old? It can for
instance not assemble the instruction LAY (load address), but the VM I
got (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4)) is able to. As soon
as I can get my hands on a Debian machine that is configured similarly I
can say more (end of Jan?).


pypy-dev mailing list
pypy-dev <at>
Siddhartha Gairola | 17 Jan 09:44 2016

Need Assistance

Dear Developers,

I am new to this community and would like to get started.
I have forked the pypy repository on bitbucket and have cloned it on my local machine.

Would appreciate some guidance.

Thank You.

pypy-dev mailing list
pypy-dev <at>
Phyo Arkar | 6 Jan 13:43 2016

More Push towards Python 3.x?

Pypy, Python 3.x is lagging behind a lot. 
Python 3.5 is a version we should be Evolving to so Is there any planned timeline for Pypy Python 3.x ?

pypy-dev mailing list
pypy-dev <at>