Magnus Morton | 9 Feb 00:01 2016

Question about "__extend__" hacks (in pypyjit module)

Hi,

In the __init__.py for the ‘pypyjit’ module there’s a comment in the
‘setup_after_space_initialization’ method "force the __extend__ hacks to occur early” (line
34) followed by two imports. What are these __extend__ hacks?

Many thanks,
Magnus 

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Robert Grosse | 7 Feb 05:55 2016
Picon

What is the last good build of Pypy?

I am trying to update Pypy. I want to build Pypy from source using the instructions at http://doc.pypy.org/en/latest/build.html in order to get the latest bugfixes and improvements.

However, all of the last 5 nightly builds shown at http://buildbot.pypy.org/summary?branch=%3Ctrunk%3E failed their tests. So I can't tell what a good revision to update to is.

What is the last good revision of Pypy?
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Richard Plangger | 2 Feb 18:11 2016
Picon
Gravatar

s390x backend gil and threading issue

hi,

I'm currently searching for a problem, I have debugged for quite a long
time. I think this is the root problem why the pypy translation with
pypy is still slower than cpython. Here are some of my findings
(+questions):

The list of last tests that fail all have one thing in common: They have
an issue with the gil/threading. (See [1])
Most interesting ones are the last five.

* test_gc_locking (2x) fail on the build bot (only using cpython), but
not on my machine. This is strange because bbot and my vm use the same
distro, same compiler version, .... the only difference is that bbot has
better hardware and the tests are run with testrunner.
Is there another way I can reproduce this?

* test_ping_pong. (-A test) ping pong from one thread to another
stressing locking and the GIL switch. On s390x and the translated VM
this takes really long (10 seconds and on bbot it seems to exceed 30
seconds when run in parallel). However if I run the same test with
PYPYLOG=jit:- it completes in ~0.96 seconds (in gdb it is the same). If
you subtract the time needed for printing you might end up with the same
speed as x86 has for this test.
What does the printing/gdbing trigger to let the GIL switch happen that
smoothly?

* I have placed memory fences at the same positions as on ppc (2x isync
and lwsync). Are there any other places that need to complete all
pending the memory operations?

* There is one path in call_release_gil (just after the call) where
rpy_fastgil was acquired (because it as 0) and the shadowstack is not
the one of the current thread. Then *rpy_fastgil = 0 is set for the
slowpath function.
Wouldn't it be possible to steal the gil at this point? Would that lead
to a problem?

Cheers,
Richard

[1] http://buildbot.pypy.org/summary?builder=own-linux-s390x

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Carl Friedrich Bolz | 2 Feb 13:11 2016
Picon
Picon
Gravatar

Call for Participation: Virtual Machine Summer School 2016

Call for Participation

Virtual Machines Summer School 2016 (#vmss16)
May 31 - June 3 2016
Cumberland Lodge, UK
http://soft-dev.org/events/vmss16/

Early Bird Registration Period: January 11 - February 15, 2016
Regular Registration Period: February 16 - April 30, 2016

The Virtual Machines Summer School (VMSS) will take place in
Cumberland Lodge, UK in summer 2016 just outside London. The aim of
the summer school is to give early career researchers (PhD students
and postdocs particularly) an overview of the field, and to meet some
of its top thinkers.

Venue: Cumberland Lodge is a former royal hunting lodge in Windsor
Great Park, a few miles south of Windsor castle. Since all
participants will be staying at the Lodge, a concentrated and
inspiring atmosphere is guaranteed.

Registration is subsidised by the EPRSC, so costs are low,
particularly for those who register early. There are travel grants
available for students that might otherwise be unable to attend the
summer school.

For further information and registration please visit:
http://soft-dev.org/events/vmss16/

Confirmed Speakers:

Antony Hosking (Purdue University)
Ben Titzer (Google)
Carl Friedrich Bolz (King's College London)
Chris Seaton (Oracle Labs)
Cliff Click (H2O)
Edd Barrett (King's College London)
Gilles Duboscq (Oracle Labs)
Jennifer Sartor (Vrije Universiteit Brussel)
Mario Wolczko (Oracle Labs)
Paul Tarau (University of North Texas)
Richard Jones (University of Kent)
Steve Blackburn (Australian National University)
Vyacheslav Egorov (Google)

Organizers:

Laurence Tratt & Carl Friedrich Bolz (King's College London)
Matti Picus | 29 Jan 15:49 2016
Picon

windows and python27.lib

When linking to a dll with MSVC, one needs to provide not the dll itself 
rather an import library. Also, the linker must be able to resolve all 
function definitions in order to run to completion, so the linker must 
be able to find the import library.
On cpython, the import library used to link to python27.dll is called 
Python27.lib, it is located in a 'libs' directory at the top level in a 
cpython distribution together with things like pyexpat.lib, _socket.lib 
etc. distutils (and setuptools) do not need to know about the import 
library, since a #pragma is used to issue a link directive for the 
import library whenever pyconfig.h is included in a c file. A debug 
build will create and use python27_d.lib that corresponds to python27_d.dll

What do we do in PyPy?
At translation we build a libpypy-c.dll, and an export library 
libpypy-c.lib. These are copied together with pypy-c.exe to the 
directory where translation occurred (on the build bots this is in 
pypy/goal) Then  the package script copies the libpypy-c.lib to pypy's 
include directory as python27.lib. A debug build will use these same 
names. The same pragma is used as in cpython to force linking with the 
import library whenever pyconfig.h is included

So what you ask?
I think the exe should be created as pypy.exe, the dll should be called 
pypy27.dll, and the import library should be consistently named 
pypy27.lib. There should be no renaming in package.py. This has 
implications in the following places:
- the exe_name in targetpypystandalone should drop the -%(backend) modifier
- pyconfig.h and the package script should be modified to use pypy27 
consistently
- probably some tests will fail, they should be fixed
- cffi/api.py needs tweaking in _apply_embedding_fix
- package.py should not rename (what do we do on linux about pypy-c -> 
pypy?)
what did I forget?

We should also handle a debug build of pypy, we should be creating a 
pypy27_d.lib, pypy27_d.dll, today the usual names are reused.

Are there compelling reasons _not_ to make the naming consistent with 
cpython? As a fallback, we could just rename the import library to 
pypy27.lib

My current motivation to do this is that _apply_embedding_fix does not 
work for win32 pypy

Matti
Richard Plangger | 28 Jan 14:37 2016
Picon
Gravatar

Stack limit in the jit backends

Hi,

the file rpython/translator/c/src/stack.h defines MAX_STACK_SIZE. PPC
has a bigger limit than e.g. x86. I experienced that on s390x there is
as well a higher consumption of memory for stack frames (they are
variable sized with a pretty high minimum limit (160 bytes) by the ABI).

I have two questions:

1) The OS (i.e. linux) defines a stack limit (ulimit -s), does pypy
overwrite this value with MAX_STACK_SIZE?

2) How would I determine which size is best for s390x? Or how did we
come up with 768kb for x86, and 2.8mb for ppc?

Cheers,
Richard

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Sergey Matyunin | 23 Jan 17:24 2016
Picon

partition in numpypy

Hello.

Need a little help with numpypy. I want to implement partition method
for numpy array. Let's say I can compile npy_partition.h.src, import
it through CFFI. Therefore I can write a python function
my_partition(numpy_array, other_arguments...) that performs
partitioning for a give numpy array.

Now I want to create partition method for ndarray.
As far as I understand methods of ndarray are defined in
pypy/module/micronumpy/ndarray.py in a special way and
ndarray.partition = my_partition doesn't work.

Is it possible to add method to ndarray inside numpypy, not inside
pypy's micronumpy module?

--

-- 
Sergey
Nagy, Attila | 21 Jan 10:47 2016
Picon

TypeError: expected string, got NoneType object with setuptools and co with pypy 4

Hi,

After installing pypy 4.0.1 on FreeBSD (from ports), I get the above 
exception when trying to install setuptools, or when installing anything 
with pip (after pypy -m ensurepip).

This all worked with the previous version (2.6).

I could find this problem elsewhere:
http://stackoverflow.com/questions/34566676/failed-to-install-pip-for-pypy-on-ubuntu
so it doesn't seem to be related to FreeBSD.

An example full trace:
# pypy -m pip install --upgrade pip
You are using pip version 6.1.1, however version 8.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip
   Exception:
   Traceback (most recent call last):
     File "/usr/local/pypy-4.0/site-packages/pip/basecommand.py", line 
246, in main
       status = self.run(options, args)
     File "/usr/local/pypy-4.0/site-packages/pip/commands/install.py", 
line 342, in run
       requirement_set.prepare_files(finder)
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
345, in prepare_files
       functools.partial(self._prepare_file, finder))
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
290, in _walk_req_to_install
       more_reqs = handler(req_to_install)
     File "/usr/local/pypy-4.0/lib_pypy/_functools.py", line 42, in __call__
       return self._func(*(self._args + fargs), **fkeywords)
     File "/usr/local/pypy-4.0/site-packages/pip/req/req_set.py", line 
487, in _prepare_file
       download_dir, do_download, session=self.session,
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 827, 
in unpack_url
       session,
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 673, 
in unpack_http_url
       from_path, content_type = _download_http_url(link, session, temp_dir)
     File "/usr/local/pypy-4.0/site-packages/pip/download.py", line 887, 
in _download_http_url
       with open(file_path, 'wb') as content_file:
   TypeError: expected string, got NoneType object

And the one for setuptools:
# pypy setup.py install
running install
Checking .pth file support in /usr/local/pypy-4.0/site-packages/
/usr/local/bin/pypy -E -c pass
TEST PASSED: /usr/local/pypy-4.0/site-packages/ appears to support .pth 
files
running bdist_egg
running egg_info
writing setuptools.egg-info/PKG-INFO
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
writing requirements to setuptools.egg-info/requires.txt
writing top-level names to setuptools.egg-info/top_level.txt
Traceback (most recent call last):
   File "setup.py", line 169, in <module>
     dist = setuptools.setup(**setup_params)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/core.py", line 
151, in setup
     dist.run_commands()
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
953, in run_commands
     self.run_command(cmd)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/install.py", line 67, 
in run
     self.do_egg_install()
   File "/tmp/setuptools-19.4/setuptools/command/install.py", line 109, 
in do_egg_install
     self.run_command('bdist_egg')
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, 
in run_command
     self.distribution.run_command(command)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/bdist_egg.py", line 
152, in run
     self.run_command("egg_info")
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/cmd.py", line 326, 
in run_command
     self.distribution.run_command(command)
   File "/usr/local/pypy-4.0/lib-python/2.7/distutils/dist.py", line 
972, in run_command
     cmd_obj.run()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 186, 
in run
     self.find_sources()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 209, 
in find_sources
     mm.run()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 293, 
in run
     self.add_defaults()
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 322, 
in add_defaults
     sdist.add_defaults(self)
   File "/tmp/setuptools-19.4/setuptools/command/sdist.py", line 100, in 
add_defaults
     self.filelist.append(fn)
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 236, 
in append
     if self._safe_path(path):
   File "/tmp/setuptools-19.4/setuptools/command/egg_info.py", line 256, 
in _safe_path
     u_path = unicode_utils.filesys_decode(path)
   File "/tmp/setuptools-19.4/setuptools/unicode_utils.py", line 31, in 
filesys_decode
     return path.decode(enc)
TypeError: expected string, got NoneType object

In this case the context is:
def filesys_decode(path):
     """
     Ensure that the given path is decoded,
     NONE when no expected encoding works
     """

     fs_enc = sys.getfilesystemencoding()
     if isinstance(path, six.text_type):
         return path

     for enc in (fs_enc, "utf-8"):
         try:
             return path.decode(enc)
         except UnicodeDecodeError:
             continue

On python 2.7, fs_enc here is:
Python 2.7.11 (default, Dec 20 2015, 01:15:21)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 
208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> repr(sys.getfilesystemencoding())
"'US-ASCII'"

While on pypy:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Jan 16 2016, 
01:16:36)
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> repr(sys.getfilesystemencoding())
'None'

Returning None here is fine according to the docs, but the above code 
snippet doesn't handle the TypeError, which it gets when doing 
path.decode(None).
How this could work?

On pypy 2.6 on the same machine:
Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Jun 12 2015, 
19:25:58)
[PyPy 2.6.0] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
u'test'
No exception!

On pypy 4.0.1:
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Dec 10 2015, 
01:17:03)
[PyPy 4.0.1 with GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: expected string, got NoneType object

Unhandled TypeError exception!

python 2.7 works the same way:
>>> 'test'.decode(None)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: decode() argument 1 must be string, not None

Which installs setuptools just fine (although it doesn't get to the 
above snippet, because it doesn't return None to 
sys.getfilesystemencoding()).
If I set fs_enc to None in the above snippet, even python 2.7 fails.

So the key here (if I'm not completely lost) seems to be not returning 
None to the getfilesystemencoding to fix broken(?) software.

What do you think about this?

Thanks,
Richard Plangger | 19 Jan 12:12 2016
Picon
Gravatar

s390x the last failing tests

hi,

I wanted to give a quick update on the state of the implementation. Good
news! I think there is not that much left to be done!

I'm currently waiting for a bigger VM (already wrote an email to
linux1 <at> us.ibm.com, 2 days ago? They are maybe on holiday?) to translate
the full project.

There are approx. 20 Failing tests that are left (own-linux-s390x). All
other pass on my virtual machine. They are mostly related to big endian
issues. Here are some questions:

1) Generally I got the impression that there are some tests that do not
consider endianess (e.g. micronumpy). I guess it is time to change them
to handle this? What about PPC? Did those not come up there?

2) It seems that the gcc on the build bot is quite old? It can for
instance not assemble the instruction LAY (load address), but the VM I
got (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4)) is able to. As soon
as I can get my hands on a Debian machine that is configured similarly I
can say more (end of Jan?).

Cheers,
Richard

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Siddhartha Gairola | 17 Jan 09:44 2016
Picon

Need Assistance

Dear Developers,

I am new to this community and would like to get started.
I have forked the pypy repository on bitbucket and have cloned it on my local machine.

Would appreciate some guidance.

Thank You.

Regards,
Sid
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Phyo Arkar | 6 Jan 13:43 2016
Picon

More Push towards Python 3.x?

Pypy, Python 3.x is lagging behind a lot. 
Python 3.5 is a version we should be Evolving to so Is there any planned timeline for Pypy Python 3.x ?


_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Gmane