Dino Viehland | 1 Sep 05:28 2011
Picon

Here's a fun one...

This came up on an internal discussion, I thought it was fun, especially given that we all behave differently:

 

Paste this into the REPL:

 

class PS1(object):

  def __init__(self): self.count = 0

  def __str__(self):

    self.count += 1

    return "%d >>>" % self.count

 

import sys

sys.ps1 = PS1()

 

 

CPython  - calls __str__

Jython – calls __repr__

IronPython – ignores ps1

PyPy - unsupported operand type for unary buffer: 'PS1'

 

(note I don’t necessarily have the latest versions for everyone)

 

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Fenn Bailey | 1 Sep 07:29 2011
Picon

djangobench performance

Hi all,


As an experiment, I thought I'd test JKM's djangobench (https://github.com/jacobian/djangobench) under pypy as a way of determining a (hopefully) more useful benchmark than the template-only "django" benchmark that's standard on speed.pypy.org and also to get an idea as to whether switching to pypy for production django apps could (currently) be a good idea.

djangobench is designed to fairly comprehensively compare the performance of different aspects of differing versions of django in an effort to detect performance degradation/regression/etc.

It's based on perf.py from the unladen swallow project, so it was fairly easy to crudely hack up to instead compare a single django version running under cpython 2.6 vs pypy 1.6.

---
$ python -V
Python 2.6.5
$ pypy -V
Python 2.7.1 (d8ac7d23d3ec, Aug 17 2011, 11:51:19)
[PyPy 1.6.0 with GCC 4.4.3]
---

The results were a little surprising (and not in a good way): http://pastie.org/2463906

Based on the highly degraded performance (>2 orders of magnitude in some cases) I'm guessing there's some sort of issue in the way I'm benchmarking things.

Code can be found here: https://github.com/fennb/djangobench

Environment is ubuntu 10.04 64bit running in a VM on a macbook pro. cpython was the current ubuntu binary package, pypy was 1.6 precompiled binary from pypy.org. It's quite possible memory size issues may have impacted some of the benchmarks (but not all).

Any ideas as to why the performance drop-off would be so significant?

Cheers,

  Fenn.
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Tennessee Leeuwenburg | 1 Sep 08:45 2011
Picon

Re: [Speed] Co-ordinating benchmarking

Okay, so all of a sudden there seem to be a *lot* of people looking at
this. This was a long thread quickly, and I only just got up to speed
with it. There are a lot of new names, and I don't know what existing
skills, interest and territories exist. Apologies for any faux pas.

I would like to organise the list of tasks a bit more clearly if that
is okay. I may be less familiar with the parts of this process than
others, so I just want to get it down clearly.

What I've done:
  -- Clone the Python repo into /home/speedracer/cpython/cpython
(updated to 2.7)
  -- Installed os packages to support a reasonable build.
  -- Built and installed python2.7 into /home/speedraver/cpython/27_bin

Presumably, people would like to be monitoring both PyPy and Cpython
as they progress over time. This means some kind of auto runner which
updates from the repo, re-runs the timings, and submits them to
codespeed. I am unclear on whether there is a "clear winner" for this
piece of technology.

List of Required Infrastructure on speed.python.org:
  -- A home for the cpython repo (check)
  -- An installed and running codespeed server (pending)
  -- A buildbot / automation for keeping up to date with the PyPy and
cpython repos (???)
  -- Automation to execute a process which must (???)
     1) Run the benchmarks
     2) Submit results to codespeed

I would suggest that the codespeed server be installed into
speedracer's home directory.

This must all be installable and configurable from Chef (which appears
to me like Fabric, i.e. an automation tool for deployment and
management of systems). This is not yet accomplished.

We also clearly need some kind of wiki documentation on what is going
on, so that contributors (or just newbies like me) can figure out
where things are at and what is going on. The bitbucket project is
great, but the task titles are currently rather brief if someone isn't
already totally up-to-speed on what is going on.

There appear to me to be two unresolved questions:
  1) What piece of technology should we use for a buildbot / build automation
  2) What piece of technology should we use for the benchmark runner?

I have no suggestions on (1), it's not a strong point for me.

As regards (2), I am ignorant to what others might already be using,
except to say the landscape seems unclear and fractured to me. My
work, benchmarker.py, is likely to be adaptable to our needs and I am
more than happy to support the package so it can be applied here. As I
understand it, the GSOC project was about the actual benchmarking
functions, not so much about automation and support for managing the
results of benchmarking. If an already-working alternative to
benchmarker.py exists and makes more sense to use, then that is fine
by me. I would still like to help out to learn more about
benchmarking.

The main issue with me as a contributor will be time. I have a full
plate as a result of Real Life, so I will sometimes go dark for a week
or so. However, I'm motivated and interested, and can put in a few
hours a week most weeks.

Do I have this right? Is that a reasonable description of the work
breakdown? Do we have clear names against tasks so that co-ordination
can be done through those people (rather than via the whole list)?

Regards,
-Tennessee

On Thu, Sep 1, 2011 at 6:28 AM, Noah Kantrowitz <noah <at> coderanger.net> wrote:
> Its all branches all the way down, so we can start work anywhere and push it to an "official" PSF bin later I
think. I'm sure we will want to host a mirror of it on the python.org hg server too, just for discoverability.
>
> --Noah
>
> On Aug 31, 2011, at 1:12 PM, Miquel Torres wrote:
>
>> Oh, cool, so there will be an Opscode hosted account for the PSF,
>> right? Then the Chef repo should be for the PSF. Maybe in a current
>> account somewhere? What do you propose?
>>
>> Miquel
>>
>>
>> 2011/8/31 Noah Kantrowitz <noah <at> coderanger.net>:
>>> Opscode has already agreed to donate a Hosted account as long we keep it under ~20 clients :-) I can hand
out the info for it to anyone that wants. As for setting up the Chef repo, just remember we are trying to not
manage this system in isolation and that it will be part of a bigger PSF infrastructure management effort.
>>>
>>> --Noah
>>>
>>> On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:
>>>
>>>> Hi all,
>>>>
>>>> though I took up on the task of installing a Codespeed instance
>>>> myself, I didn't have time until now. This weekend I will definitely
>>>> have  a *lot* of time to work on this, so count on that task being
>>>> done by then.
>>>>
>>>> The bitbucket issue tracker is a start (though a organization account
>>>> would be better) and the splash page is great. So let's get started
>>>> organizing things.
>>>>
>>>> Regarding the deployment strategy, it turns out I use Chef at work, so
>>>> I am in full agreement with Noah here (yey!). Actually, I am the
>>>> author of LittleChef (which we can use as a tool to execute Chef on
>>>> the node).
>>>>
>>>> So, Configuration Management. I would propose that Noah starts the
>>>> repo with the Chef cookbooks (preferably a complete LittleChef
>>>> kitchen, but that is not a must :), and gets the main recipes (apache,
>>>> django) going, while I create a cookbook for Codespeed. What do you
>>>> think?
>>>>
>>>> The benchmark runner question is still open. We need to clarify that.
>>>> Use the pypy runner? Tennessee's work?
>>>>
>>>> Regarding repositories and issues, we could maybe have a "speed"
>>>> organization account (not sure on Bitbucket, you can do that in
>>>> Github), where we have a wiki, issues, and runner + config management
>>>> repo + other stuff.
>>>>
>>>> Cheers,
>>>> Miquel
>>>>
>>>> 2011/8/31 Jesse Noller <jnoller <at> gmail.com>:
>>>>> I've put up a splash page for the project this AM:
>>>>>
>>>>> http://speed.python.org/
>>>>>
>>>>> jesse
>>>>> _______________________________________________
>>>>> pypy-dev mailing list
>>>>> pypy-dev <at> python.org
>>>>> http://mail.python.org/mailman/listinfo/pypy-dev
>>>>>
>>>> _______________________________________________
>>>> Speed mailing list
>>>> Speed <at> python.org
>>>> http://mail.python.org/mailman/listinfo/speed
>>>
>>>
>
>
> _______________________________________________
> Speed mailing list
> Speed <at> python.org
> http://mail.python.org/mailman/listinfo/speed
>
>

--

-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"
William ML Leslie | 1 Sep 09:23 2011
Picon

Re: djangobench performance

On 1 September 2011 15:29, Fenn Bailey <fenn.bailey <at> gmail.com> wrote:
> The results were a little surprising (and not in a good way):
> http://pastie.org/2463906
...
> Any ideas as to why the performance drop-off would be so significant?

N = 200 means most of the benchmarks probably won't even JIT, so that
might be a start.  The threshold in the released pypy is N = 1000.

But even without JIT, 20+ fold slowdowns are very interesting:
10n_render, query_all and query_raw.

I wonder if anyone has benchmarked sqlite under pypy - that would have
the most dramatic effect here.

--

-- 
William Leslie
Fenn Bailey | 1 Sep 09:33 2011
Picon

Re: djangobench performance

Hi William,



N = 200 means most of the benchmarks probably won't even JIT, so that
might be a start.  The threshold in the released pypy is N = 1000.


Yeah, I suspected that might be the case, and did a few test individual benchmarks with a much higher N (ie: >20,000). It definitely improved things comparatively quite a lot, but ultimately still resulted in a 3-4x slowdown over CPython.

  Fenn.
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Antonio Cuni | 1 Sep 09:50 2011
Picon

Re: [Speed] Moving the project forward

On 31/08/11 22:11, Brett Cannon wrote:
> The PyPy folk could answer this as they have their repo on bitbucket
> already. Else I guess we can just create a standalone account that
> represents the official speed.python.org account.

for pypy we do exactly that. There is a bitbucket user named "pypy" whose 
credentials are shared among all the core devs.

ciao,
Anto
Antonio Cuni | 1 Sep 09:58 2011
Picon

Re: Here's a fun one...

On 01/09/11 05:28, Dino Viehland wrote:
> This came up on an internal discussion, I thought it was fun, especially given
> that we all behave differently:
>
> Paste this into the REPL:
[cut]

it seems to work fine with pypy 1.6.  Note that str() is called twice for each 
line, so we get 1, 3, 5, 7..., but this happens only on cpython.

 >>>> class PS1(object):
....   def __init__(self):
....       self.count = 0
....   def __str__(self):
....       self.count += 1
....       return "%d >>>" % self.count
....
 >>>> import sys
 >>>> sys.ps1 = PS1()
1 >>>
3 >>>
5 >>>
7 >>>
9 >>>

ciao,
Anto
Nick Coghlan | 1 Sep 10:10 2011
Picon

Re: [Speed] Moving the project forward

On Thu, Sep 1, 2011 at 5:50 PM, Antonio Cuni <anto.cuni <at> gmail.com> wrote:
> On 31/08/11 22:11, Brett Cannon wrote:
>>
>> The PyPy folk could answer this as they have their repo on bitbucket
>> already. Else I guess we can just create a standalone account that
>> represents the official speed.python.org account.
>
> for pypy we do exactly that. There is a bitbucket user named "pypy" whose
> credentials are shared among all the core devs.

The security auditing part of my brain has its fingers in its ears and
is singing "La La La" rather loudly :)

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
Antonio Cuni | 1 Sep 10:37 2011
Picon

Re: djangobench performance

On 01/09/11 09:23, William ML Leslie wrote:
> I wonder if anyone has benchmarked sqlite under pypy - that would have
> the most dramatic effect here.

I'm doing it right now. It seems that for some reasons the JIT does not remove 
the ctypes overhead of sqlite calls, thus they are much slower than they 
should be.

ciao,
Anto
Armin Rigo | 1 Sep 10:57 2011

Re: Here's a fun one...

Hi,

On Thu, Sep 1, 2011 at 9:58 AM, Antonio Cuni <anto.cuni <at> gmail.com> wrote:
> it seems to work fine with pypy 1.6.  Note that str() is called twice for
> each line, so we get 1, 3, 5, 7..., but this happens only on cpython.

...but this happens only on PyPy, you mean.  It works as expected on
CPython 2.7.  Is it a bug? :-)

A bientôt,

Armin.

Gmane