Re: Interested in becoming a PHP/QA Team Member

Michelangelo van Dam wrote:
> Hi,
>
> Since I'm heavily involved in organizing test fest for Belgium and try 
> to work on a solid solution for running tests, I would like to express 
> my interest in becoming member of the PHP QA team.
Welcome - and thank you for all the hard work  

Many people have helped with TestFest this year and it has made me 
wonder if we should have a named team. We haven't right now, and the 
names listed in PHP Credits under QA are http://www.php.net/credits.php 
don't reflect the people that are currently involved in test writing 
although some  have certainly contributed a great deal to PHP.

If we do have a set of names I think the best place for it is probably 
the QA site, but maybe it would be best to start with a discussion about 
what  being a member of the PHP QA team means?

I think, for example, that it would mean that your primary role in the 
PHP project is to test PHP. That might encompass helping with TestFest, 
contributing to the QA pages, writing tests, writing test tools, running 
tests regularly and making the results available. Maybe there are other 
things?

One advantage of a named team is to provide a set of go-to people for 
questions on testing PHP. But that would mean it would have to stay 
current, so maybe there would need to be a way to take names off such a 
list too?

These are just ideas, I'd welcome discussion of them, particularly from 

Re: Interested in becoming a PHP/QA Team Member

Hi,

On Mon, Jun 1, 2009 at 9:10 PM, zoe <zoe.slattery <at> googlemail.com> wrote:
> Michelangelo van Dam wrote:
>>
>> Hi,
>>
>> Since I'm heavily involved in organizing test fest for Belgium and try to
>> work on a solid solution for running tests, I would like to express my
>> interest in becoming member of the PHP QA team.
>
> Welcome - and thank you for all the hard work  
>
> Many people have helped with TestFest this year and it has made me wonder if
> we should have a named team. We haven't right now, and the names listed in
> PHP Credits under QA are http://www.php.net/credits.php don't reflect the
> people that are currently involved in test writing although some  have
> certainly contributed a great deal to PHP.
>
> If we do have a set of names I think the best place for it is probably the
> QA site, but maybe it would be best to start with a discussion about what
>  being a member of the PHP QA team means?
>
> I think, for example, that it would mean that your primary role in the PHP
> project is to test PHP. That might encompass helping with TestFest,
> contributing to the QA pages, writing tests, writing test tools, running
> tests regularly and making the results available. Maybe there are other
> things?
>
> One advantage of a named team is to provide a set of go-to people for

Re: Interested in becoming a PHP/QA Team Member

Hi,

This subject started as "being a QA team member", but I think this can be
also treated as "What is the QA team".

I'll speak about my personal case, because I think many people can share
this point of view.

I started reviewing php code and php tests when PHP 5.3 was "usable",
because I needed some 5.3 features for a big project. I was (and still is)
the project leader on this project, and my first task was to choose whether
to use 5.3 or not, considering primarily the stability of the sapi.
Then I discovered the mailing list and all the hard work done so far. I knew
in a minute that I could help this community because I'm able to take a few
hours a month dedicated to QA in PHP because this is directly connected to
my project. A stable PHP is vital to me (and to everybody!). As my knowledge
in C/C++ is non-existent, the QA came to be a good way to help PHP, as it
can be done only with PHP skills.

- But - I felt lost on all those projects connected to PHPQA, and this is
the majors questions I had :
  ** How to help ?
      Writing tests ? Writing/conceiving tools ? Send daily output of
run-tests in latest cvs ?
  ** Where to help ?
      Use only the mailing list ? Completing the wiki ?
  ** Where to start ?
      What are the priorities ? Writing tests on a specific module ?
Reviewing tests ? Completing wiki ? ...

cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.php

hjalle		Tue Jun  2 13:19:40 2009 UTC

  Added files:                 
    /php-benchmarks/benchcli/tests	test_whirl.php 
    /php-benchmarks/benchcli/tests/phpWhirl	99bottles.wr AUTHORS 
                                           	COPYING keymaker.wr 
                                           	whirl.php 
    /php-benchmarks/benchcli/tests/phpWhirl/classes	WhirlMathRing.php 
                                                   	WhirlMemory.php 
                                                   	WhirlOpsRing.php 
                                                   	WhirlParser.php 
                                                   	WhirlRing.php 
  Log:
  Whirl parser test

  

http://cvs.php.net/viewvc.cgi/php-benchmarks/benchcli/tests/test_whirl.php?view=markup&rev=1.1
Index: php-benchmarks/benchcli/tests/test_whirl.php
+++ php-benchmarks/benchcli/tests/test_whirl.php
<?php

/**
 * Parses and runs the keymaker.wr program using
 * the php Whirl parser, made by Jakob Westhoff
 * <jakob <at> westhoffswelt.de>
 */

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

This benchmark, when running parsing keymaker.wr, takes pretty long time for
me (about 1 minute on a slow machine). If I run 99bottles instead, which is
a smaller program, it takes about 10 seconds instead. Is 1 minute acceptable
or should we go for the other, 99bottles?

On Tue, Jun 2, 2009 at 3:19 PM, Alexander Hjalmarsson <hjalle <at> php.net>wrote:

> hjalle          Tue Jun  2 13:19:40 2009 UTC
>
>  Added files:
>    /php-benchmarks/benchcli/tests      test_whirl.php
>    /php-benchmarks/benchcli/tests/phpWhirl     99bottles.wr AUTHORS
>                                                COPYING keymaker.wr
>                                                whirl.php
>    /php-benchmarks/benchcli/tests/phpWhirl/classes     WhirlMathRing.php
>                                                        WhirlMemory.php
>                                                        WhirlOpsRing.php
>                                                        WhirlParser.php
>                                                        WhirlRing.php
>  Log:
>  Whirl parser test
>
>
> --
> PHP Quality Assurance Mailing List <http://www.php.net/>
> To unsubscribe, visit: http://www.php.net/unsub.php
>

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

On Tue, Jun 2, 2009 at 2:32 PM, Alexander Hjalmarsson <hjalle <at> sgh.se> wrote:
> This benchmark, when running parsing keymaker.wr, takes pretty long time for
> me (about 1 minute on a slow machine). If I run 99bottles instead, which is
> a smaller program, it takes about 10 seconds instead. Is 1 minute acceptable
> or should we go for the other, 99bottles?

I've found in my benchmarking that its useful to have a set of
benchmarks which run very quickly, and another set where I can be
surer of the results. How long do the other benchmarks run for?

Paul

>
> On Tue, Jun 2, 2009 at 3:19 PM, Alexander Hjalmarsson <hjalle <at> php.net>wrote:
>
>> hjalle          Tue Jun  2 13:19:40 2009 UTC
>>
>>  Added files:
>>    /php-benchmarks/benchcli/tests      test_whirl.php
>>    /php-benchmarks/benchcli/tests/phpWhirl     99bottles.wr AUTHORS
>>                                                COPYING keymaker.wr
>>                                                whirl.php
>>    /php-benchmarks/benchcli/tests/phpWhirl/classes     WhirlMathRing.php
>>                                                        WhirlMemory.php
>>                                                        WhirlOpsRing.php
>>                                                        WhirlParser.php
>>                                                        WhirlRing.php
>>  Log:
>>  Whirl parser test
>>

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

On Tue, Jun 2, 2009 at 3:43 PM, Paul Biggar <paul.biggar <at> gmail.com> wrote:

> On Tue, Jun 2, 2009 at 2:32 PM, Alexander Hjalmarsson <hjalle <at> sgh.se>
> wrote:
> > This benchmark, when running parsing keymaker.wr, takes pretty long time
> for
> > me (about 1 minute on a slow machine). If I run 99bottles instead, which
> is
> > a smaller program, it takes about 10 seconds instead. Is 1 minute
> acceptable
> > or should we go for the other, 99bottles?
>
> I've found in my benchmarking that its useful to have a set of
> benchmarks which run very quickly, and another set where I can be
> surer of the results. How long do the other benchmarks run for?
>

The tests are running at about 3-10 seconds. Maybe it's a good idea to have
an argument for a longer run (like running the keymaker instead of
99bottles), but in that case, the micro benchmarks would be pretty useless
as they would only increase the iteration-times. The other test I just
uploaded also have a similar problem, though its the opposite. I just have
working input thats like 1000 characters long, which makes the benchmark to
only take like 3/4 of a second. It would be good to have better input data
for that test, but it didn't work with more advanced javascripts such as the
V8-benchmarks and similar. Do anyone know any "simple" input that might
work, but that yet is pretty comprehensive?

>
> Paul

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

On Tue, Jun 2, 2009 at 4:26 PM, Paul Biggar <paul.biggar <at> gmail.com> wrote:

> On Tue, Jun 2, 2009 at 3:09 PM, Alexander Hjalmarsson <hjalle <at> sgh.se>
> wrote:
> > On Tue, Jun 2, 2009 at 3:43 PM, Paul Biggar <paul.biggar <at> gmail.com>
> wrote:
> >>
> >> On Tue, Jun 2, 2009 at 2:32 PM, Alexander Hjalmarsson <hjalle <at> sgh.se>
> >> wrote:
> >> > This benchmark, when running parsing keymaker.wr, takes pretty long
> time
> >> > for
> >> > me (about 1 minute on a slow machine). If I run 99bottles instead,
> which
> >> > is
> >> > a smaller program, it takes about 10 seconds instead. Is 1 minute
> >> > acceptable
> >> > or should we go for the other, 99bottles?
> >>
> >> I've found in my benchmarking that its useful to have a set of
> >> benchmarks which run very quickly, and another set where I can be
> >> surer of the results. How long do the other benchmarks run for?
> >
> >
> > The tests are running at about 3-10 seconds. Maybe it's a good idea to
> have
> > an argument for a longer run (like running the keymaker instead of
> > 99bottles), but in that case, the micro benchmarks would be pretty
> useless
> > as they would only increase the iteration-times.

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

On Tue, Jun 2, 2009 at 3:09 PM, Alexander Hjalmarsson <hjalle <at> sgh.se> wrote:
> On Tue, Jun 2, 2009 at 3:43 PM, Paul Biggar <paul.biggar <at> gmail.com> wrote:
>>
>> On Tue, Jun 2, 2009 at 2:32 PM, Alexander Hjalmarsson <hjalle <at> sgh.se>
>> wrote:
>> > This benchmark, when running parsing keymaker.wr, takes pretty long time
>> > for
>> > me (about 1 minute on a slow machine). If I run 99bottles instead, which
>> > is
>> > a smaller program, it takes about 10 seconds instead. Is 1 minute
>> > acceptable
>> > or should we go for the other, 99bottles?
>>
>> I've found in my benchmarking that its useful to have a set of
>> benchmarks which run very quickly, and another set where I can be
>> surer of the results. How long do the other benchmarks run for?
>
>
> The tests are running at about 3-10 seconds. Maybe it's a good idea to have
> an argument for a longer run (like running the keymaker instead of
> 99bottles), but in that case, the micro benchmarks would be pretty useless
> as they would only increase the iteration-times.

I wouldn't worry too much about the micro-benchmarks. They wouldnt be
much worse than they are now.

> The other test I just
> uploaded also have a similar problem, though its the opposite. I just have
> working input thats like 1000 characters long, which makes the benchmark to
> only take like 3/4 of a second. It would be good to have better input data

Re: cvs: php-benchmarks /benchcli/tests test_whirl.php /benchcli/tests/phpWhirl 99bottles.wr AUTHORS COPYING keymaker.wr whirl.php /benchcli/tests/phpWhirl/classes WhirlMathRing.php WhirlMemory.php WhirlOpsRing.php WhirlParser.php WhirlRing.

On Tue, Jun 2, 2009 at 3:37 PM, Alexander Hjalmarsson <hjalle <at> sgh.se> wrote:
>> I wouldn't worry too much about the micro-benchmarks. They wouldnt be
>> much worse than they are now.
>
>
> That might be correct. :)
> When adding up the cachegrind time (or valgrind), the time for the benchmark
> would be as you know, about 20x longe. How long is reasonable for a
> benchmark to run? I'm mostly thinking of when enabling valgrind/cachegrind
> or any other tool.

Ah, of course. That is a real problem. That would probably make it
impractical to run 1-minute long benchmarks in valgrind.

A few options:
- Allow it, but run only short things by default in valgrind
- Seek out faster ways to do the things that valgrind gets you. Options include:
For memory allocation:
  - There might be settings to ask valgrind to just record the memory allocated?
  - Does glibc allow you to set an environmental variable giving the
allocated memory?
  - Maybe its simple to link in another memory allocator like jemalloc
or Google's tcmalloc which might support that?
  - Google's unladen-swallow project measures memory usage by sampling
(see http://code.google.com/p/unladen-swallow/source/browse/tests/perf.py).
That may be a better approach.
For cache/branch behaviour:
  - One option is to use PAPI, but that requires a patched kernel.
Perhaps allow both options, and the cachegrind one would just be slow,
and people wanting fast results should use PAPI? Not the best solution