GNU logs - #12620, boring messages

Message sent to bug-automake@HIDDEN:

Subject: bug#12620: Parallel tests vs fast tests (and beyond)
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-automake@HIDDEN
Resent-Date: Wed, 10 Oct 2012 19:55:01 +0000
Resent-Message-ID: <handler.12620.B.134989888412895 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 12620 <at> debbugs.gnu.org
MIME-Version: 1.0
Date: Wed, 10 Oct 2012 20:53:46 +0100
Message-ID: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

With the recent work on parallel tests in automake I thought it was
time to give them a spin, so I did, for the "zee" branch of GNU Zile.
This has about 100 tests, the total wall clock time being around 8s on
my 2.5GHz 4-core Sandy Bridge machine, with the following target:

 check-local: $(builddir)/src/zee
	echo $(TESTS) | $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) xargs $(RUNLUATESTS)

So, I rewrote it as:

LOG_COMPILER = $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) $(RUNLUATESTS)

and lo! it ran in 2/3 of the time. Not bad for a few minutes' work.
But then I looked closer, and noticed that although the wall-clock
time had gone down, the user+system time had almost doubled. Oops: I
was only winning because I had four cores. Developers using
single-core or dual-core machines might well lose out with parallel
tests.

The test harness is a Lua script, so the parallel tests were starting
an extra Lua interpreter per test, so  this is not a huge surprise,
but it is a pity.

In the end, I found I got a much bigger speed up (down to under 3s), with:

check-local: $(builddir)/src/zee
	NPROC=`nproc`; \
	echo $(LUA_TESTS) | $(LUA_ENV) $(LUA_TESTS_ENVIRONMENT) xargs
--max-procs=$$NPROC --max-args=$$(( `echo $(LUA_TESTS) | wc -w` /
$$NPROC + 1 )) $(RUNLUATESTS) > /dev/null

That would be relatively straightforward to make portable (or rather,
bail out if non-GNU xargs is used, but it's still much clumsier than
parallel-tests, and of course uses the old serial-tests. (I'm working
with automake 1.11.6, in case it matters.) The implication of the 1.12
release notes is that serial-tests will be dropped at some point
post-1.13.

This whole problem is an instance of a more general problem, of which
another example is parallel make: for best performance, it should
probably batch up calls to gcc, for example, so that multiple source
files are compiled by each invocation.

I can imagine a view that we're rapidly increasing the number of
cores, so in fact there's little point working on these intermediate
solutions, but it seems to me that it's not just less-well-off
developers who could benefit, but also compile farms.

Comments? Solutions I've missed?

-- 
http://rrt.sc3d.org

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.428 (Entity 5.428)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Reuben Thomas <rrt@HIDDEN>
Subject: bug#12620: Acknowledgement (Parallel tests vs fast tests (and
 beyond))
Message-ID: <handler.12620.B.134989888412895.ack <at> debbugs.gnu.org>
References: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
X-Gnu-PR-Message: ack 12620
X-Gnu-PR-Package: automake
Reply-To: 12620 <at> debbugs.gnu.org
Date: Wed, 10 Oct 2012 19:55:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-automake@HIDDEN

If you wish to submit further information on this problem, please
send it to 12620 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
12620: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D12620
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-automake@HIDDEN:

Subject: bug#12620: Parallel tests vs fast tests (and beyond)
Resent-From: Jack Kelly <jack@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 11 Oct 2012 21:14:02 +0000
Resent-Message-ID: <handler.12620.B12620.134999003619193 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Reuben Thomas <rrt@HIDDEN>
Cc: 12620 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
References: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
Date: Fri, 12 Oct 2012 08:12:59 +1100
Message-ID: <CAEnY7O-rikuU3_W_6HTGoy6Q_zedHsFhNXoX=_wFm45eu24AWA@HIDDEN>
From: Jack Kelly <jack@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

On Thu, Oct 11, 2012 at 6:53 AM, Reuben Thomas <rrt@HIDDEN> wrote:
> This whole problem is an instance of a more general problem, of which
> another example is parallel make: for best performance, it should
> probably batch up calls to gcc, for example, so that multiple source
> files are compiled by each invocation.

We had a discussion along these lines when refactoring elisp
compilation: in the past it was all done in one big batch. Now it's
done with an emacs invocation per .el file. The result of that
discussion was that while you slow down single core machines, the new
way of doing things fits make's model much better: instead of fiddling
around with stamp files, the Makefile just has a suffix rule.

IMHO, your point is valid, but to do it safely would require a tool
smarter than make.

-- Jack

Message sent to bug-automake@HIDDEN:

Subject: bug#12620: Parallel tests vs fast tests (and beyond)
Resent-From: Reuben Thomas <rrt@HIDDEN>
Original-Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 11 Oct 2012 21:42:01 +0000
Resent-Message-ID: <handler.12620.B12620.134999167121520 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Jack Kelly <jack@HIDDEN>
Cc: 12620 <at> debbugs.gnu.org
MIME-Version: 1.0
In-Reply-To: <CAEnY7O-rikuU3_W_6HTGoy6Q_zedHsFhNXoX=_wFm45eu24AWA@HIDDEN>
References: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
	<CAEnY7O-rikuU3_W_6HTGoy6Q_zedHsFhNXoX=_wFm45eu24AWA@HIDDEN>
Date: Thu, 11 Oct 2012 22:40:16 +0100
Message-ID: <CAOnWdoiX4mGp1Z+vppDJ=DswceGZWutH_HgjK5wM4g86Tt_Ncg@HIDDEN>
From: Reuben Thomas <rrt@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Precedence: list
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org

On 11 October 2012 22:12, Jack Kelly <jack@HIDDEN> wrote:
>
> We had a discussion along these lines when refactoring elisp
> compilation: in the past it was all done in one big batch. Now it's
> done with an emacs invocation per .el file. The result of that
> discussion was that while you slow down single core machines, the new
> way of doing things fits make's model much better: instead of fiddling
> around with stamp files, the Makefile just has a suffix rule.

That's very interesting, of course that's the way make works naturally
for C programs, and I never thought of changing it (although I read
about how much faster for compiling several files it is, but it only
ever occurred to me that it would be nice to batch up compilations for
building from source, not for development where typically you may only
need to recompile one file).

-- 
http://rrt.sc3d.org

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#12620: Parallel tests vs fast tests (and beyond)
References: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
In-Reply-To: <CAOnWdoix5PAoUrXPwyq8zeTjpgkpGEnz4VwWVxFUebhKk+pg9g@HIDDEN>
Resent-From: Stefano Lattarini <stefano.lattarini@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 26 Dec 2013 16:47:02 +0000
Resent-Message-ID: <handler.12620.B12620.13880764198295 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 12620
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: 12620 <at> debbugs.gnu.org
Cc: Reuben Thomas <rrt@HIDDEN>
Received: via spool by 12620-submit <at> debbugs.gnu.org id=B12620.13880764198295
          (code B ref 12620); Thu, 26 Dec 2013 16:47:02 +0000
Received: (at 12620) by debbugs.gnu.org; 26 Dec 2013 16:46:59 +0000
Received: from localhost ([127.0.0.1]:44130 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1VwE5L-00029h-Cs
	for submit <at> debbugs.gnu.org; Thu, 26 Dec 2013 11:46:59 -0500
Received: from mail-ee0-f42.google.com ([74.125.83.42]:65465)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stefano.lattarini@HIDDEN>)
 id 1VwE5I-00029T-Py; Thu, 26 Dec 2013 11:46:57 -0500
Received: by mail-ee0-f42.google.com with SMTP id e53so3776623eek.29
 for <multiple recipients>; Thu, 26 Dec 2013 08:46:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=message-id:date:from:mime-version:to:cc:subject:content-type
 :content-transfer-encoding;
 bh=Do07e+vDgl+T5P9XNLvqI/8UIsuL9HjDGM1/Zz72PJ0=;
 b=os48ecGC4L31+DeoJlVC5HDVEVQlOflN+/5NHVpoHyNzZA+5YlJSvHxknDRFMta9xx
 SaeYFhqHCvBpRY2YFDYeC8gwB1dYcedjoi/EgKMvJ549J7FF4r74x7HSzLG0PlyEKLT1
 mWzOUgO6JLm++TOsA2qQ049lkrGUo6hNmKudipKVmnnnf+WkqqEUTWQhpMbqhkNI3ZnB
 tlizhG3iQ1xlLcvZ5k1UNx1rZnqlk5ByzlbTbDP2JbLZVR7JKe0hHpDJrTyY91vu9N0x
 a+aKbxT34Z+zclr+Doy7/akG/WbOhshQ991A6qKS9IAlNOebFgIk0QSWf1LxBlGnz9uY
 KIcg==
X-Received: by 10.15.45.135 with SMTP id b7mr8408407eew.88.1388076415906;
 Thu, 26 Dec 2013 08:46:55 -0800 (PST)
Received: from [192.168.0.101]
 (host143-4-dynamic.5-87-r.retail.telecomitalia.it. [87.5.4.143])
 by mx.google.com with ESMTPSA id b41sm74201656eef.16.2013.12.26.08.46.51
 for <multiple recipients>
 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
 Thu, 26 Dec 2013 08:46:55 -0800 (PST)
Message-ID: <52BC5D78.6000006@HIDDEN>
Date: Thu, 26 Dec 2013 17:46:48 +0100
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

severity 12620 wishlist
stop

Reference:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12620>

Marking this as a "wishlist" rather than a bug, since the current
implementation of parallel tests is working correctly and is mostly
"good enough", and your proposal is thus geared towards improving an
already decent situation rather than fixing errors or misbehaviors
(this doesn't mean I'm denying or belittling the merits of your
proposal, of course).

Just one nit:

Reuben Thomas wrote:
>
> The implication of the 1.12 release notes is that serial-tests
> will be dropped at some point post-1.13.
>
I've been convinced that any idea I had in such a direction was
misguided; the old serial harness is now marked merely as
"discouraged", but there no plans to deprecate pr remove it in
the foreseeable future.

Thanks,
  Stefano

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 26 Dec 2013 16:47:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 26 11:47:00 2013
Received: from localhost ([127.0.0.1]:44132 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1VwE5L-00029k-PN
	for submit <at> debbugs.gnu.org; Thu, 26 Dec 2013 11:47:00 -0500
Received: from mail-ee0-f42.google.com ([74.125.83.42]:65465)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stefano.lattarini@HIDDEN>)
 id 1VwE5I-00029T-Py; Thu, 26 Dec 2013 11:46:57 -0500
Received: by mail-ee0-f42.google.com with SMTP id e53so3776623eek.29
 for <multiple recipients>; Thu, 26 Dec 2013 08:46:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=message-id:date:from:mime-version:to:cc:subject:content-type
 :content-transfer-encoding;
 bh=Do07e+vDgl+T5P9XNLvqI/8UIsuL9HjDGM1/Zz72PJ0=;
 b=os48ecGC4L31+DeoJlVC5HDVEVQlOflN+/5NHVpoHyNzZA+5YlJSvHxknDRFMta9xx
 SaeYFhqHCvBpRY2YFDYeC8gwB1dYcedjoi/EgKMvJ549J7FF4r74x7HSzLG0PlyEKLT1
 mWzOUgO6JLm++TOsA2qQ049lkrGUo6hNmKudipKVmnnnf+WkqqEUTWQhpMbqhkNI3ZnB
 tlizhG3iQ1xlLcvZ5k1UNx1rZnqlk5ByzlbTbDP2JbLZVR7JKe0hHpDJrTyY91vu9N0x
 a+aKbxT34Z+zclr+Doy7/akG/WbOhshQ991A6qKS9IAlNOebFgIk0QSWf1LxBlGnz9uY
 KIcg==
X-Received: by 10.15.45.135 with SMTP id b7mr8408407eew.88.1388076415906;
 Thu, 26 Dec 2013 08:46:55 -0800 (PST)
Received: from [192.168.0.101]
 (host143-4-dynamic.5-87-r.retail.telecomitalia.it. [87.5.4.143])
 by mx.google.com with ESMTPSA id b41sm74201656eef.16.2013.12.26.08.46.51
 for <multiple recipients>
 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
 Thu, 26 Dec 2013 08:46:55 -0800 (PST)
Message-ID: <52BC5D78.6000006@HIDDEN>
Date: Thu, 26 Dec 2013 17:46:48 +0100
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
MIME-Version: 1.0
To: 12620 <at> debbugs.gnu.org
Subject: Re: Parallel tests vs fast tests (and beyond)
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: control
Cc: Reuben Thomas <rrt@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)

severity 12620 wishlist
stop

Reference:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12620>

Marking this as a "wishlist" rather than a bug, since the current
implementation of parallel tests is working correctly and is mostly
"good enough", and your proposal is thus geared towards improving an
already decent situation rather than fixing errors or misbehaviors
(this doesn't mean I'm denying or belittling the merits of your
proposal, of course).

Just one nit:

Reuben Thomas wrote:
>
> The implication of the 1.12 release notes is that serial-tests
> will be dropped at some point post-1.13.
>
I've been convinced that any idea I had in such a direction was
misguided; the old serial harness is now marked merely as
"discouraged", but there no plans to deprecate pr remove it in
the foreseeable future.

Thanks,
  Stefano

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.