Phyo Arkar | 29 Aug 11:56 2014

pypy 2.3.1 json encoding performnce is Extremely slow (30x slower ).

$ pypy

Array with 256 doubles:
json encode : 5044.67291 calls/sec
json decode : 19591.44018 calls/sec
Array with 256 utf-8 strings:
decode UTF : 71.03748 calls/sec
json decode UTF : 482.03748 calls/sec

$ /usr/bin/python

Array with 256 doubles:

json encode : 4292.39818 calls/sec
json decode : 15089.87792 calls/sec

Array with 256 utf-8 strings:

json encode UTF : 2062.16175 calls/sec
json decode UTF : 479.04892 calls/sec

Test using ultra json:
$ /usr/bin/python ujson/test/

Array with 256 doubles:
ujson encode      : 4386.51907 calls/sec
simplejson encode : 4269.30241 calls/sec
yajl  encode      : 4268.15286 calls/sec
                                                      ujson decode
(Continue reading)

Wilfred Hughes | 25 Aug 15:32 2014

Pidigits performance without FFI

Hi folks

I've been looking at the pidigits benchmark from the Computer Language Benchmarks, but measuring performance without FFI (so native Python numbers instead of GMP).

You can see my results here:

I'm seeing better performance for CPython than pypy3 on simple arithmetic code. Is this to be expected? Could I make pypy3 perform better?

pypy-dev mailing list
pypy-dev <at>
Mike Kaplinskiy | 25 Aug 09:20 2014

Adding a feature to re

Hey folks,

One of the projects I'm working on in CPython is becoming a little CPU bound and I was hoping to use pypy. One problem though - one of the pieces uses the regex library (which claims to be CPython's re-next). Running regex through cpyext works, but is deadly slow.

From reading the docs it seems like I have a few options:
 - rewrite all of regex in Python - seems like a bad idea
 - rewrite regex to be non-python specific & use cppyy or cffi to interface with it. I actually looked into this & unfortunately the CPython API seems quite deep in there.
 - get rid of the dependency somehow. What I'm missing are named lists (basically "L<a>", a=["1","2"] will match 1 or 2). Unfortunately creating one really long re string is out of the question - I have not seen compile() finish with that approach. Writing a custom DFA could be on the table, but I was hoping to avoid that error prone step.
 - somehow factor out the part using regex and keep using CPython for it.
 - add the missing functionality to pypy's re. This seems like the path of least resistance.

I've started looking into the sre module and it looks like quite a few bits (parsing & compiling to byte code mostly) are reused from CPython. I would have to change some of those bits. My question is then - is there any hope of getting these changes upstream then? Do stdlib pieces have a "no touch" policy?

pypy-dev mailing list
pypy-dev <at>
Martin Matusiak | 23 Aug 22:03 2014

Fwd: For py3k...


So just to get a quick sanity check before I start.

We are missing lzma in py3.3 (or to be more precise we have the
beginnings of the module, but it's not passing any tests yet). Philip
hinted to me recently that it would make sense to integrate lzmaffi. I
would like to work on that, but I will probably get stuck a few times,
so I'm planning to do some writeups on my blog as I go along (both for
note taking and debugging). Since it's an external project I was
planning to follow the same procedure that we use for stdlib, which is
described in stdlib-upgrade.txt. So start out with vendor/lzmaffi,
then branch off to py3.3-lzmaffi for integration work, finally merge
into py3.3.

Does that seem like a good approach?


2014-07-23 19:20 GMT+02:00 Peter Cock <p.j.a.cock <at>>:
> Thanks - that will be a useful companion to my backport
> for using lzma on C Python 2.6, 2.7 and early Python 3 :)
> Peter
> On Wed, Jul 23, 2014 at 5:14 PM, Armin Rigo <arigo <at>> wrote:
>> Hi all,
>> A module mentioned today in a EuroPython lightning talk: "lzma"
>> reimplemented in cffi (compatible with the one from Python 3.3's
>> stdlib).
>> A bientôt,
>> Armin.
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev <at>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev <at>
pypy-dev mailing list
pypy-dev <at>
Armin Rigo | 14 Aug 09:26 2014

Mini-sprint August 25-27 in Switzerland

Hi all,

We're running a 3-days mini-sprint the 25-27 of August at the
University of Neuchatel, Switzerland, research group of Pascal Felber.
This is not a general PyPy sprint, but one on the topic of STM
specifically.  Nevertheless, in case there is someone nearby who
listens to this mailing list and who would like to show up (from one
day to the full 3 days), he is welcome to.

More details: the mini-sprint is about anything related to PyPy-STM,
but more precisely, now would be a good time to try to push
the idea forward, now that we have an almost reasonable core working
(however rough).  This would involve in general the question: what
does it mean for the user?  Should we scrap atomic blocks and instead
focus on lock elision (as a more backward-compatible idea with nicer
fallbacks in case we try to do I/O in the atomic block)?  We
definitely should make dictionaries more aware of multithreading, but
is that enough to avoid most "unexpected" conflicts?  Can we try to
adapt a larger single-threaded framework (my favourite is Twisted) to
run on multiple threads transparently for the user's applications?

A bientôt,

pypy-dev mailing list
pypy-dev <at>
Armin Rigo | 11 Aug 09:17 2014

Re: trouble compiling p4python api with pypy3

Hi Russ,

On 10 August 2014 19:39, Russ Tremain <russt <at>> wrote:
> Would it be better to customize the api to comply with the "native" pypy interface?

The "native" pypy interface is cffi (which also works on CPython,
btw).  It would definitely give better stability and performance on
PyPy.  It's more than just adapting some C code here and there,

Russ Tremain | 10 Aug 19:56 2014

failure installing pygit2 on pypy3 raspbian

Hi, just wondering if pip install of pygit2 is expected to work, or 
if this is a problem related to ARM.
Also, is there a difference using "pip" vs. pip3 or pip3.2?


Linux raspberrypi 3.12.22+ #691 PREEMPT Wed Jun 18 18:29:58 BST 2014 armv6l
pi <at> raspberrypi:~$ pip3 install pygit2
Downloading/unpacking pygit2
   Downloading pygit2-0.21.2.tar.gz (416kB): 416kB downloaded
   Running (path:/tmp/pip_build_pi/pygit2/ egg_info 
for package pygit2
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c:26:34: error: 
unknown type name 'git_buf'
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c: In function 
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c:35:10: error: 
unknown type name 'git_buf'
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c:37:12: error: 
'git_buf' undeclared (first use in this function)
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c:1649:21: error: 
(near initialization for 'nums[13]')

[snip - thousands more errors]

     Traceback (most recent call last):
line 131, in _compile
line 909, in spawn
         spawn(cmd, dry_run=self.dry_run)
line 32, in spawn
         _spawn_posix(cmd, search_path, dry_run=dry_run)
line 163, in _spawn_posix
         % (cmd[0], exit_status))
     distutils.errors.DistutilsExecError: command 'cc' failed with exit status 1

     During handling of the above exception, another exception occurred:

     Traceback (most recent call last):
line 47, in _build
line 936, in run_command

line 46, in run
line 354, in run
line 463, in build_extensions

line 182, in build_extension
line 518, in build_extension
line 574, in compile
         self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
line 133, in _compile
         raise CompileError(msg)
     distutils.errors.CompileError: command 'cc' failed with exit status 1

     During handling of the above exception, another exception occurred:

     Traceback (most recent call last):
       File "<string>", line 17, in <module>
       File "/tmp/pip_build_pi/pygit2/", line 178, in <module>
         from ffi import ffi
       File "pygit2/", line 59, in <module>
         include_dirs=include_dirs, library_dirs=library_dirs)
line 341, in verify
         lib = self.verifier.load_library()
line 74, in load_library
line 139, in _compile_module
         outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
line 25, in compile
         outputfilename = _build(tmpdir, ext)
line 50, in _build
         raise VerificationError('%s: %s' % (e.__class__.__name__, e))
     cffi.ffiplatform.VerificationError: CompileError: command 'cc' 
failed with exit status 1
     Complete output from command python egg_info:
     pygit2/__pycache__/_cffi__gab826d3dx2bbed4fa.c:26:34: error: 
unknown type name 'git_buf'


Cleaning up...
Command python egg_info failed with error code 1 in 
Storing debug log for failure in /home/pi/.pip/pip.log
pi <at> raspberrypi:~$
Russ Tremain | 9 Aug 22:27 2014

trouble compiling p4python api with pypy3


I'm trying to get p4python (perforce python API) compiling with pypy 
and I notice that the include headers in pypy are very different than 
the standard 3.2.5 python headers.  Is the C api different for pypy3 
from python 3.2.5?

For example, note the following diffs:

diff -c `find . -name moduleobject.h`
*** ./pypy3-2.3.1-linux-armhf-raspbian/include/moduleobject.h 
	2014-06-19 09:20:58.000000000 -0700
--- ./Python-3.2.5/Include/moduleobject.h	2013-05-15 
09:33:41.000000000 -0700
*** 1,3 ****
--- 1,4 ----
   /* Module object interface */

   #ifndef Py_MODULEOBJECT_H
*** 6,11 ****
--- 7,30 ----
   extern "C" {

+ PyAPI_DATA(PyTypeObject) PyModule_Type;
+ #define PyModule_Check(op) PyObject_TypeCheck(op, &PyModule_Type)
+ #define PyModule_CheckExact(op) (Py_TYPE(op) == &PyModule_Type)
+ PyAPI_FUNC(PyObject *) PyModule_New(
+     const char *name            /* UTF-8 encoded string */
+     );
+ PyAPI_FUNC(PyObject *) PyModule_GetDict(PyObject *);
+ PyAPI_FUNC(const char *) PyModule_GetName(PyObject *);
+ PyAPI_FUNC(const char *) PyModule_GetFilename(PyObject *);
+ PyAPI_FUNC(PyObject *) PyModule_GetFilenameObject(PyObject *);
+ #ifndef Py_LIMITED_API
+ PyAPI_FUNC(void) _PyModule_Clear(PyObject *);
+ #endif
+ PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*);
+ PyAPI_FUNC(void*) PyModule_GetState(PyObject*);
   typedef struct PyModuleDef_Base {
     PyObject* (*m_init)(void);
*** 32,37 ****
--- 51,57 ----
     freefunc m_free;

   #ifdef __cplusplus

When I compile against the distributed pypy3 headers, I get errors:

08/09/14.12:11:29: Building p4python ...

08/09/14.12:11:29: pypy build --apidir 
../p4api-2014.1.886167.main/ --ssl
API Release 2014.1
running build
running build_py
creating build
creating build/lib.linux-armv6l-3.2
copying -> build/lib.linux-armv6l-3.2
running build_ext
building 'P4API' extension
creating build/temp.linux-armv6l-3.2
cc -O2 -fPIC -Wimplicit -DID_OS="LINUX31ARM" -DID_REL="2012.2" 
-DID_PATCH="549493" -DID_API="2014.1.main/886167" -DID_Y="2012" 
-DID_M="11" -DID_D="05" -I../p4api-2014.1.886167.main/ 
-I/usr/local/pypy3-2.3.1-linux-armhf-raspbian/include -c P4API.cpp -o 
build/temp.linux-armv6l-3.2/P4API.o -DOS_LINUX -DOS_LINUX31 
cc1plus: warning: command line option '-Wimplicit' is valid for 
C/ObjC but not for C++ [enabled by default]
P4API.cpp: In function 'int P4API_traverse(PyObject*, visitproc, void*)':
P4API.cpp:984:5: error: 'PyModule_GetState' was not declared in this scope
P4API.cpp: In function 'int P4API_clear(PyObject*)':
P4API.cpp:989:5: error: 'PyModule_GetState' was not declared in this scope
P4API.cpp: In function 'PyObject* PyInit_P4API()':
P4API.cpp:1045:30: error: 'PyModule_GetState' was not declared in this scope
error: command 'cc' failed with exit status 1

raspberrypi{gfinstall.5} pypy --version
Python 3.2.5 (986752d005bb, Jun 19 2014, 16:20:03)
[PyPy 2.3.1 with GCC 4.7.2 20120731 (prerelease)]

If I hack around the macro problems, I can get a compile, but then I 
get a runtime error:

raspberrypi{gfinstall.42} cat
import P4

raspberrypi{gfinstall.43} pypy
Traceback (most recent call last):
   File "", line 1, in <module>
     import P4
   File "/home/pi/git-fusion/bin/p4python-2012.2.549493/", line 
359, in <module>
     import P4API
ImportError: unable to load extension module 
undefined symbol: __cxa_pure_virtual

Any help appreciated..

Toni Mattis | 4 Aug 18:06 2014

Calling lambdas inside loop causes significant slowdown


I'm trying to figure out the fastest way in PyPy to introduce
abstractions into loops, e.g. refactoring the following code:

def sum_direct(data):
    s = 0
    for i in data:
        if i < 5:
            s += i + 1
    return s

to something like:

def sum_lambda(data):
    filter_func = lambda x: x < 5
    map_func = lambda x: x + 1

    s = 0
    for i in data:
        if filter_func(i):
            s += map_func(i)
    return s

and then turning both lambdas into arguments, class members and so on.
However, the refactoring mentioned above already introduces about 50% of
runtime overhead and is not getting better with further refactorings.
Shoudn't the tracing/inlining eliminate most of this overhead or is
there a mistake on my part?

I timed both methods on a large array:

from array import array
import time

data = array('i')
for i in xrange(100000000):
    data.append(i % 10)

t = time.time()
result = sum_lambda(data)  # or sum_direct
print result, time.time() - t

Calling sum_direct() takes about 0.43 seconds, sum_lambda() is at 0.64s
on average.
(I'm at changeset 72674:78d5d873a260 from Aug 3 2014, translated and run
on Ubuntu 14.04)

The JIT trace of the lambda code basically adds two force_token()
operations and potentially more expensive guards. Is there any chance to
avoid these without excessive metaprogramming? If no, which speedup
tricks (speaking of jit hooks, code generation, etc.) can you recommend
for implementing such APIs?

Thanks in advance,
Dima Tisnek | 4 Aug 17:04 2014

use as benchmark pypy vs python if you please

Attached is n-queens solver (pardon my naive algorithm), it runs:
python 2.7.6: 17s
pypy 2.4.0 alpha: 23s
same nojit: 32s

I've tried similar-looking algorithm for another problem before, and has similar results -- somehow pypy was slower.

feel free to investigate / tweak or even use on
Attachment ( text/x-python, 4530 bytes
pypy-dev mailing list
pypy-dev <at>
Anton Gulenko | 3 Aug 12:57 2014

AssertionError in rpython_jit_metainterp_resume.c


I am working on the RSqueak VM (
Usually I debug assertion errors by executing the VM in interpreted mode. Recently I started getting an error where that doesn't work. See below for the only output on the console.
I am working on Windows, so the VM is compiled with Visual Studio.
The VM is translated using the pypy commit tagged "release-2.3.1".

RPython traceback:
  File "rpython_jit_metainterp_compile.c", line 379, in ResumeGuardForcedDescr_handle_async_forcing
  File "rpython_jit_metainterp_resume.c", line 256, in force_from_resumedata
  File "rpython_jit_metainterp_virtualizable.c", line 426, in write_from_resume_data_partial
  File "rpython_jit_metainterp_resume.c", line 768, in ResumeDataDirectReader_decode_ref
Fatal RPython error: AssertionError

Since this seems to be related to virtualizable objects: during translation we get 2 warnings that the "fresh_virtualizable" hints are ignored (there are 2 places where virtualizable frame objects are created.
Also, we have one virtualizable list[*] field in our Context class, which should never be resized. I applied the make_sure_not_resized hint on it, but maybe I'm using it wrong... I'm just calling make_sure_not_resized once, after setting the list field.

Any ideas?
Thanks in advance!

pypy-dev mailing list
pypy-dev <at>