Re: parallel autotest [2/3]: Implement 'testsuite --jobs'.

Hi Eric,

* Eric Blake wrote on Tue, Sep 30, 2008 at 04:16:33AM CEST:
> According to Ralf Wildenhues on 9/29/2008 3:35 PM:
> >>> + <at> item --jobs <at> r{[}= <at> var{n} <at> r{]}
> >>> + <at> itemx -j <at> ovar{n}
> >>> + <at> itemx -j
> >> This second line is redundant, since you used  <at> ovar on the line above.
> > 
> > The first line addresses --jobs=N, the second line addresses -jN.
> > What is redundant there?  (see below at --help output, for more info)
> 
> I guess I didn't phrase that well.  I meant that the second -j line:
> 
> - -j[N]
> - -j
> 
> looks redundant;

D'oh.  I simply misunderstood you.

> I think it could be better rendered as:
> 
> - --jobs[=N]
> - -jN
> - -j
> 
> in the manual ( <at> var, not  <at> ovar), and

Or just

Re: [PATCH 1/5] add a separate diversion for shell functions

Paolo Bonzini <bonzini <at> gnu.org> writes:

> 
> > Thanks for your effort on this series!  I agree that we are finally ready
> > to require shell functions in ./configure scripts.  It will take me some
> > time to review them all, but let's start at the beginning.
> 
> Thanks.

Just so you don't worry, I haven't forgotten about this series; now that Ralf's 
first cut of parallel autotest is in, I'll start looking at it in more detail.

--

-- 
Eric Blake

Solaris tr bug

I'm committing this.  The first hunk is a bit obvious, by the fact that Posix 
only requires sed to handle text files (the Posix definition of a text file is 
no NUL bytes, no lines longer than LINE_MAX, and either empty or ending in a 
trailing newline), but the reminder can't hurt.  The second is an actual Posix 
compliance bug encountered at least by people who insist on using ancient 
tools, although I don't yet know if other modern platforms besides Solaris 
share this tr bug.

From: Eric Blake <ebb9 <at> byu.net>
Date: Thu, 2 Oct 2008 09:02:37 -0600
Subject: [PATCH] Document more binary file portability traps.

* doc/autoconf.texi (Limitations of Usual Tools) <sed>: Remind
reader that NUL and sed don't always mix.
<tr>: Mention Solaris /usr/ucb/tr bug with \0.

Signed-off-by: Eric Blake <ebb9 <at> byu.net>
---
 ChangeLog         |    7 +++++++
 doc/autoconf.texi |   17 ++++++++++++++++-
 2 files changed, 23 insertions(+), 1 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 6b3f95b..dbba2f8 100644
--- a/ChangeLog
+++ b/ChangeLog
 <at>  <at>  -1,3 +1,10  <at>  <at> 
+2008-10-02  Eric Blake  <ebb9 <at> byu.net>
+
+	Document more binary file portability traps.

Re: Solaris tr bug

Eric Blake <ebb9 <at> byu.net> writes:

> +Posix requires  <at> command{tr} to operate on binary files.  But at least
> +Solaris  <at> command{/usr/ucb/tr} still fails to handle  <at> samp{\0} as the
> +octal escape for  <at> code{NUL}.

Those historic implementations just drop NUL characters from the input,
see
<http://www.opengroup.org/onlinepubs/009695399/utilities/tr.html#tag_04_145_18>.

Andreas.

--

-- 
Andreas Schwab, SuSE Labs, schwab <at> suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: parallel autotest [2/3]: Implement 'testsuite --jobs'.

* Ralf Wildenhues wrote on Thu, Oct 02, 2008 at 08:21:16AM CEST:
> 
> However, there are still races in the code.  I found two separate
> issues:
> - sometimes, a parallel run would fail to output all test results on
>   stdout; they would appear in the log file, however.  This happens
>   maybe 20% of the time on one host, but never occurred on another.
> - I found one hang (among many many test runs) at the end where the
>   master process hangs waiting for a token.

The latter seems to be fixed with this, pushed (or rather: one source of
hangs is fixed).

I don't understand how I could overlook that, it consistently hangs the
'Keywords and ranges' test in the case where the sub testsuite './k' is
run with no tests to execute (-k is not matched).  The build daemons
show the failure, too.

Apologies also for forgetting the ChangeLog entry in my last commit.
:-/

Cheers,
Ralf

   Fix hang with `testsuite -k notmatched'.

   * lib/autotest/general.m4: Do not reset $at_jobs if it is equal
   to one.  Fixes hang with `-k notmatched'.

diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4

Re: parallel autotest [2/3]: Implement 'testsuite --jobs'.

* Ralf Wildenhues wrote on Thu, Oct 02, 2008 at 08:46:24PM CEST:
> 
>    Fix hang with `testsuite -k notmatched'.
> 
>    * lib/autotest/general.m4: Do not reset $at_jobs if it is equal
>    to one.  Fixes hang with `-k notmatched'.

And of course more corner cases of the same kind.  Pushed.

Cheers,
Ralf

    Fix more testsuite hang corner cases.

    * lib/autotest/general.m4: Use the serial code path if no test
    is to be run.
    * tests/autotest.at (parallel test execution): Test -j and -jN
    with `-k notmatched'.

diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index be135c1..f1966ce 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
 <at>  <at>  -1018,6 +1018,9  <at>  <at>  rm -f "$at_suite_dir/at-source-lines"

 # Set number of jobs for `-j'; avoid more jobs than test groups.
 set X $at_groups; shift; at_max_jobs=$[ <at> %: <at> ]
+if test $at_max_jobs -eq 0; then
+  at_jobs=1
+fi

Re: [PATCH 2/5] avoid duplicate tests for shell functions

Paolo Bonzini wrote:
> This one-line patch ensures that if a feature is deemed required,
> its test is not emitted in the suggested tests.  It is not 
> perfect, but it works for the case at hand which is right now
> the most complex use of the _AS_DETECT_BETTER_SHELL machinery.

Ping (also 2..5/5).

Paolo

Intermittent parallel test failures

Hello Bob, all,

as I noted earlier in
<http://thread.gmane.org/gmane.comp.sysutils.autoconf.patches/5563/focus=5786>,
there are intermittent test failures for the parallel tests.  These
failures also show up in the build bot output:
<http://buildbot.proulx.com:9003/>
<http://buildbot.proulx.com:9003/i686%20gnu-linux/builds/909/step-test/0>
<http://buildbot.proulx.com:9003/amd64%20gnu-linux/builds/878/step-test/0>

I am trying to find out
1) whether the exact failure on your systems is the same that I saw,
2) what differences there are to the system I have where I do not see
   any failures.

For (1), would it be possible to include tests/testsuite.log in the
logged state for a failed test suite?  Thanks.

For (2), it would be nice to know a bit more about the systems.  Which
shell (and version) do the systems use?  Which kernel version?  Are they
dual or uniprocessor/-core systems?  Also, it looks to me like the amd64
one prints system info that looks like it's an x86:
<http://buildbot.proulx.com:9003/amd64%20gnu-linux>

Thanks for this very helpful service!

Cheers,
Ralf

Re: parallel autotest [2/3]: Implement 'testsuite --jobs'.

Eric Blake <ebb9 <at> byu.net> writes:

> 
> > +if test $at_jobs -ne 1 &&
> > +     rm -f "$at_job_fifo" &&
> > +     ( mkfifo "$at_job_fifo" ) 2>/dev/null &&
> > +     exec AT_JOB_FIFO_FD<> "$at_job_fifo"
> 
> Is exec n<> portable?  Yes, POSIX specifies it, but I haven't seen it used
> anywhere else in autoconf.  I guess we commit it now, and if it is not
> portable, we wrap it inside an eval alongside the mkfifo check as a reason
> to skip the parallel testing.

Unfortunately, it is NOT portable.  According to POSIX open [1], under 
O_RDWR, "The result is undefined if this flag is applied to a FIFO."  On 
cygwin, FIFOs can only be read-only or write-only, there is no way to get a 
read-write FIFO, and using <> in the shell defaults to a read-only FIFO.

[1] http://www.opengroup.org/onlinepubs/009695399/functions/open.html

As a result, the parallel tests invariably fail:

./autotest.at:837: $CONFIG_SHELL ./micro-suite -j4
--- /dev/null   2006-11-30 17:00:00.000000000 -0700
+++ /home/eblake/autoconf/tests/testsuite.dir/at-groups/133/stderr      2008-10-
02 09:42:30.020100900 -0600
 <at>  <at>  -0,0 +1,8  <at>  <at> 
+./micro-suite: line 1703: read: read error: 0: Communication error on send
+./micro-suite: line 1716: read: read error: 0: Communication error on send
+./micro-suite: line 1716: read: read error: 0: Communication error on send

Re: [Mingw-users] cfg.mk:20: *** Recursive variable `PATH' references itself (eventually). Stop. while compiling autoconf

According to Keith Marshall on 9/30/2008 3:24 PM:
>> Sounds like the case-insensitive make behavior is still biting us,
>> then (which we previously determined only afflicts in-place
>> builds); I guess Stepan's patch didn't work as expected [1],[2]. 
> 
>> Also, does this patch help?
>> ...
> 
> Yes; it resolves the issue I have observed, allowing the in-place 
> build to complete successfully, (both `make all' and `make install').
> 
> Thanks.

Good to hear.  I'm installing this:

--
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9 <at> byu.net

From c1a0d34e262ce8179fe3386041e59c263ea3f52b Mon Sep 17 00:00:00 2001
From: Eric Blake <ebb9 <at> byu.net>
Date: Mon, 6 Oct 2008 20:06:00 -0600
Subject: [PATCH] Fix build with case-insensitive make, again.

* Makefile.am (pkgdata_DATA): Protect by MAKE_CASE_SENSITIVE.
Reported via Keith Marshall, originally by newthinker in
<http://thread.gmane.org/gmane.comp.gnu.mingw.user/27725>.