Karl Berry <karl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Karl Berry <karl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Karl Berry <karl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Karl Berry <karl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Received: (at 7868) by debbugs.gnu.org; 15 Nov 2020 02:20:14 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Nov 14 21:20:14 2020 Received: from localhost ([127.0.0.1]:52319 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1ke7eH-0005Ak-PG for submit <at> debbugs.gnu.org; Sat, 14 Nov 2020 21:20:14 -0500 Received: from freefriends.org ([96.88.95.60]:57508) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <karl@HIDDEN>) id 1ke7eG-0005Ac-Tv for 7868 <at> debbugs.gnu.org; Sat, 14 Nov 2020 21:20:13 -0500 X-Envelope-From: karl@HIDDEN X-Envelope-To: <7868 <at> debbugs.gnu.org> Received: from freefriends.org (freefriends.org [96.88.95.60]) by freefriends.org (8.14.7/8.14.7) with ESMTP id 0AF2KBQp005260 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for <7868 <at> debbugs.gnu.org>; Sat, 14 Nov 2020 19:20:12 -0700 Received: (from apache@localhost) by freefriends.org (8.14.7/8.14.7/Submit) id 0AF2KBgO005257; Sat, 14 Nov 2020 19:20:11 -0700 Date: Sat, 14 Nov 2020 19:20:11 -0700 Message-Id: <202011150220.0AF2KBgO005257@HIDDEN> From: Karl Berry <karl@HIDDEN> To: 7868 <at> debbugs.gnu.org Subject: Re: bug#7868: automake 1.11.3 check-TESTS and command line length In-Reply-To: <4F4BE205.1070207@HIDDEN> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 7868 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) It seems something more needs to be done with this bug, but I don't know what. I hope someone with more energy, expertise, and/or time can look into it and find a resolution. Marking it as severity normal (can't be that important after 8 years) and with the "help" tag.
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at 7868) by debbugs.gnu.org; 27 Feb 2012 20:08:24 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 27 15:08:24 2012 Received: from localhost ([127.0.0.1]:60200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1S26rv-00086v-65 for submit <at> debbugs.gnu.org; Mon, 27 Feb 2012 15:08:24 -0500 Received: from mx1.redhat.com ([209.132.183.28]:27441) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from <eblake@HIDDEN>) id 1S26rs-00086m-9u for 7868 <at> debbugs.gnu.org; Mon, 27 Feb 2012 15:08:21 -0500 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q1RK5QSE001945 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 27 Feb 2012 15:05:26 -0500 Received: from [10.3.113.13] ([10.3.113.13]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q1RK5P7M008735; Mon, 27 Feb 2012 15:05:26 -0500 Message-ID: <4F4BE205.1070207@HIDDEN> Date: Mon, 27 Feb 2012 13:05:25 -0700 From: Eric Blake <eblake@HIDDEN> Organization: Red Hat User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120216 Thunderbird/10.0.1 MIME-Version: 1.0 To: Stefano Lattarini <stefano.lattarini@HIDDEN> Subject: Re: automake 1.11.3 check-TESTS and command line length References: <alpine.GSO.2.01.1202191252190.6378@HIDDEN> <4F44B95F.1080005@HIDDEN> <alpine.GSO.2.01.1202220848270.6378@HIDDEN> <4F45481D.10307@HIDDEN> <alpine.GSO.2.01.1202221411280.28275@HIDDEN> <4F455613.1010605@HIDDEN> <4F4B65CA.3050804@HIDDEN> <4F4BDF1A.2080901@HIDDEN> In-Reply-To: <4F4BDF1A.2080901@HIDDEN> X-Enigmail-Version: 1.3.5 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigC76729A104CBC82366CD63DF" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Spam-Score: -6.9 (------) X-Debbugs-Envelope-To: 7868 Cc: Peter Rosin <peda@HIDDEN>, 7868 <at> debbugs.gnu.org, automake-patches@HIDDEN, automake@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -6.9 (------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigC76729A104CBC82366CD63DF Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 02/27/2012 12:52 PM, Stefano Lattarini wrote: > [re-adding the relevant automake bug in CC] >=20 > Hi Peter, thanks for chiming in. >=20 > On 02/27/2012 12:15 PM, Peter Rosin wrote: >> >> I *think* the environment and the command line shares space (approx 64= kB, I can confirm this, based on my testing of Windows process spawning. > So, basically, on MSYS, a: >=20 > $ make TESTS=3D"..." >=20 > invocation reduces the size available for the command lines of the make= > recipes, because the value of TESTS that gets exported in the environme= nt > eats away space that could be used by those command lines? Oh joy ... Yes. Unlike on *ix systems, where argv and environ are independent entities, Windows makes you deal with both limits at the same time, so increasing the size of environ reduces the size permitted in argv. --=20 Eric Blake eblake@HIDDEN +1-919-301-3266 Libvirt virtualization library http://libvirt.org --------------enigC76729A104CBC82366CD63DF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJPS+IFAAoJEKeha0olJ0Nqr1sH/i/kQneuSIQUpe1EAacCPo1u oQ+SIoiTIZKOqz728NbHaQ857lLTZZbq8sf89br+Dix2KrknhaHxm7HXv9SdMk4z b/M00tGl0stzvL7ad9n9bMGXWkPySoC/8L/5YkeAwPgN/EGxJHkFK6syF1GnQWZE ZG5n4cYfSWUdjXu5ZK1C7iTuSiCbm0DXdQt8CTSRinKQKR+9GS6X4BpSwEXoow/t R7ldkwweGEvVavW1c5nnksxbqUCzDcIQVyROZbaWTcgO1LLIBrTemy8VGrEeZEH6 K8nse1zunIByo1M9TwIiOrSA5SEt8rZy6AIk/qCCX8VzGIjJXtTYPDIZoxKNfd4= =DWzs -----END PGP SIGNATURE----- --------------enigC76729A104CBC82366CD63DF--
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at 7868) by debbugs.gnu.org; 27 Feb 2012 19:56:00 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Feb 27 14:56:00 2012 Received: from localhost ([127.0.0.1]:60194 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1S26fv-0007pK-7S for submit <at> debbugs.gnu.org; Mon, 27 Feb 2012 14:55:59 -0500 Received: from mail-wi0-f172.google.com ([209.85.212.172]:36394) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from <stefano.lattarini@HIDDEN>) id 1S26fs-0007pC-5y for 7868 <at> debbugs.gnu.org; Mon, 27 Feb 2012 14:55:57 -0500 Received: by wicr5 with SMTP id r5so968334wic.3 for <7868 <at> debbugs.gnu.org>; Mon, 27 Feb 2012 11:53:02 -0800 (PST) Received-SPF: pass (google.com: domain of stefano.lattarini@HIDDEN designates 10.180.93.232 as permitted sender) client-ip=10.180.93.232; Authentication-Results: mr.google.com; spf=pass (google.com: domain of stefano.lattarini@HIDDEN designates 10.180.93.232 as permitted sender) smtp.mail=stefano.lattarini@HIDDEN; dkim=pass header.i=stefano.lattarini@HIDDEN Received: from mr.google.com ([10.180.93.232]) by 10.180.93.232 with SMTP id cx8mr89383wib.14.1330372382983 (num_hops = 1); Mon, 27 Feb 2012 11:53:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:mime-version:to:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=/6sayDVjcseEmdTbM5RUNJJYy/FRASwhAe2ugBlYJ9Y=; b=CkwiURTtSO4/wCIdRQH4He3LFWTm1Bkzmxi4sa6ncqzVmTHNSYzNzebOMkQpLsVic0 HBDb0hCyMZ4NXKXneEUx8HiD2qu/+eAYVk/eyrvSu8MSqEE9vpIWfSeguCKlRbRHXQP9 aHNYdSMdYObrOsTKroKNZYytGAmfNr784D+RI= Received: by 10.180.93.232 with SMTP id cx8mr70376wib.14.1330372382920; Mon, 27 Feb 2012 11:53:02 -0800 (PST) Received: from [79.21.97.71] (host71-97-dynamic.21-79-r.retail.telecomitalia.it. [79.21.97.71]) by mx.google.com with ESMTPS id df3sm24101653wib.1.2012.02.27.11.53.00 (version=SSLv3 cipher=OTHER); Mon, 27 Feb 2012 11:53:01 -0800 (PST) Message-ID: <4F4BDF1A.2080901@HIDDEN> Date: Mon, 27 Feb 2012 20:52:58 +0100 From: Stefano Lattarini <stefano.lattarini@HIDDEN> MIME-Version: 1.0 To: Peter Rosin <peda@HIDDEN> Subject: Re: automake 1.11.3 check-TESTS and command line length References: <alpine.GSO.2.01.1202191252190.6378@HIDDEN> <4F44B95F.1080005@HIDDEN> <alpine.GSO.2.01.1202220848270.6378@HIDDEN> <4F45481D.10307@HIDDEN> <alpine.GSO.2.01.1202221411280.28275@HIDDEN> <4F455613.1010605@HIDDEN> <4F4B65CA.3050804@HIDDEN> In-Reply-To: <4F4B65CA.3050804@HIDDEN> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 7868 Cc: automake@HIDDEN, 7868 <at> debbugs.gnu.org, automake-patches@HIDDEN, Bob Friesenhahn <bfriesen@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -2.6 (--) [re-adding the relevant automake bug in CC] Hi Peter, thanks for chiming in. On 02/27/2012 12:15 PM, Peter Rosin wrote: > > I *think* the environment and the command line shares space (approx 64kB, > I repeat *think* here, I don't know the details off the top of my head, Cygwin > isn't affected since it uses the cygwin DLL to communicate this stuff between > cygwin processes using normal ipc mechanisms). If I'm right, reducing the > command line length for a process might sufficiently elevate the limit, even > if it isn't eliminated. > So, basically, on MSYS, a: $ make TESTS="..." invocation reduces the size available for the command lines of the make recipes, because the value of TESTS that gets exported in the environment eats away space that could be used by those command lines? Oh joy ... Anyway, if we now reintroduce Ralf's patch, we will have a good explanation of why it actually works. Thanks, Stefano
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Stefano Lattarini <stefano.lattarini@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Stefano Lattarini <stefano.lattarini@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 30 Dec 2011 16:54:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 30 11:54:52 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1RgfjH-00022z-Od
for submit <at> debbugs.gnu.org; Fri, 30 Dec 2011 11:54:52 -0500
Received: from mail-ee0-f44.google.com ([74.125.83.44])
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <stefano.lattarini@HIDDEN>) id 1RgfjF-00022n-6S
for 7868 <at> debbugs.gnu.org; Fri, 30 Dec 2011 11:54:50 -0500
Received: by eekc14 with SMTP id c14so14555465eek.3
for <7868 <at> debbugs.gnu.org>; Fri, 30 Dec 2011 08:51:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=message-id:date:from:mime-version:to:cc:subject:references
:in-reply-to:content-type:content-transfer-encoding;
bh=/so6jkSFJrN5RaTUI/4dYv1xTW9LtC4ORpQGv04zKTk=;
b=fewH3XDeR6qQpY0u8Bj6KoOg1XaCKDmiFg50vszsjTE9KBpaIU72ftKiJsu8A5fjcs
cEIWBJs4i7yO2f2naZ5F3bN8W3QUq87D9+72zwjkwbOdhWjwxfKh7sS5BmLPvxrxh7re
QTyWWa98eBKcOtc0WM+XgoueMtHNv96h6l60s=
Received: by 10.213.28.11 with SMTP id k11mr8242523ebc.101.1325263911764;
Fri, 30 Dec 2011 08:51:51 -0800 (PST)
Received: from [87.0.95.73] (host73-95-dynamic.0-87-r.retail.telecomitalia.it.
[87.0.95.73])
by mx.google.com with ESMTPS id z54sm150021759eeh.5.2011.12.30.08.51.50
(version=SSLv3 cipher=OTHER); Fri, 30 Dec 2011 08:51:51 -0800 (PST)
Message-ID: <4EFDEC1A.3090808@HIDDEN>
Date: Fri, 30 Dec 2011 17:51:38 +0100
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
MIME-Version: 1.0
To: Bob Friesenhahn <bfriesen@HIDDEN>
Subject: Re: bug#7868: parallel-tests: avoid command-line length limit issue.
References: <alpine.GSO.2.01.1009051927440.12887@HIDDEN> <20100906193953.GO16188@HIDDEN>
<20100906200743.GP16188@HIDDEN> <201009070315.55743.stefano.lattarini@HIDDEN> <20100907031038.GC3444@HIDDEN>
<20101002165332.GC11911@HIDDEN> <alpine.GSO.2.01.1112281322500.15666@HIDDEN> <4EFBA597.8060107@HIDDEN>
<4EFDAC7F.6060005@HIDDEN>
<alpine.GSO.2.01.1112301003000.15666@HIDDEN>
In-Reply-To: <alpine.GSO.2.01.1112301003000.15666@HIDDEN>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Spam-Score: -3.9 (---)
X-Debbugs-Envelope-To: 7868
Cc: 7868 <at> debbugs.gnu.org, automake-patches@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.9 (---)
Hi Bob.
On 12/30/2011 05:10 PM, Bob Friesenhahn wrote:
> On Fri, 30 Dec 2011, Stefano Lattarini wrote:
>>>
>> OK, I hope I've finally managed to partially fix this incredibly annoying bug
>> -- I say "partially" because the fix is sadly *for GNU make*. Even more sadly,
>> the solution is pretty hacky and somewhat brittle. Still, it should cause no
>> regression with the non-GNU makes (as they will continue to use the old
>> implementation), and I have verified that it works with at least all the GNU
>> make versions >= 3.78.
>
> I see that there is a new test which tests for GNU make. How solid is it?
>
Pretty solid IMHO. In fact, it correctly exposes the problem at hand in the
following setups:
* Solaris 10 with /usr/ccs/bin/make
* Solaris 10 with GNU make 3.82
* Debian unstable with GNU make 3.81, freebsd-make (8.x) and
pmake (port of NetBSDmake)
* NetBSD 5.1 with the system make
On Cygwin 1.5, it exposes a probably-related problem (a segfault in make).
> For example, the GNU make I use under MinGW is called 'csmake' (for
> "Case Sensitive make") because the default MinGW GNU case-insensitive
> make is horrendously slow... [SNIP] ... As a result, whenever I would
> normally type 'make' I type 'csmake' instead.
>
Then, to be 100% correct, you'll have to call configure as:
./configure MAKE=csmake
Still, if you only have GNU make installed on you system, the correct
code paths in the generated Makefile will be correctly activated even
if you don't specify the $MAKE override at configure time, so you will
later be able to safely run the generated Makefiles with other versions
of GNU make.
Regards,
Stefano
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at 7868) by debbugs.gnu.org; 30 Dec 2011 16:13:11 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 30 11:13:11 2011 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1Rgf4w-00010S-Go for submit <at> debbugs.gnu.org; Fri, 30 Dec 2011 11:13:11 -0500 Received: from blade.simplesystems.org ([65.66.246.74]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <bfriesen@HIDDEN>) id 1Rgf4u-00010J-1P for 7868 <at> debbugs.gnu.org; Fri, 30 Dec 2011 11:13:09 -0500 Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id pBUGA9Bh023832; Fri, 30 Dec 2011 10:10:09 -0600 (CST) Date: Fri, 30 Dec 2011 10:10:09 -0600 (CST) From: Bob Friesenhahn <bfriesen@HIDDEN> X-X-Sender: bfriesen@HIDDEN To: Stefano Lattarini <stefano.lattarini@HIDDEN> Subject: Re: parallel-tests: avoid command-line length limit issue. In-Reply-To: <4EFDAC7F.6060005@HIDDEN> Message-ID: <alpine.GSO.2.01.1112301003000.15666@HIDDEN> References: <alpine.GSO.2.01.1009051927440.12887@HIDDEN> <20100906193953.GO16188@HIDDEN> <20100906200743.GP16188@HIDDEN> <201009070315.55743.stefano.lattarini@HIDDEN> <20100907031038.GC3444@HIDDEN> <20101002165332.GC11911@HIDDEN> <alpine.GSO.2.01.1112281322500.15666@HIDDEN> <4EFBA597.8060107@HIDDEN> <4EFDAC7F.6060005@HIDDEN> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Fri, 30 Dec 2011 10:10:10 -0600 (CST) X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: 7868 Cc: 7868 <at> debbugs.gnu.org, automake-patches@HIDDEN X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -3.8 (---) On Fri, 30 Dec 2011, Stefano Lattarini wrote: >> > OK, I hope I've finally managed to partially fix this incredibly annoying bug > -- I say "partially" because the fix is sadly *for GNU make*. Even more sadly, > the solution is pretty hacky and somewhat brittle. Still, it should cause no > regression with the non-GNU makes (as they will continue to use the old > implementation), and I have verified that it works with at least all the GNU > make versions >= 3.78. I see that there is a new test which tests for GNU make. How solid is it? For example, the GNU make I use under MinGW is called 'csmake' (for "Case Sensitive make") because the default MinGW GNU case-insensitive make is horrendously slow, taking a couple of minutes before it does anything at all. It seems that someone decided that MinGW make needs to be case-insensitive and it operates via brute-force, trying every case permutation. If a network is involved, this becomes frightfully slow. As a result, whever I would normally type 'make' I type 'csmake' instead. Bob -- Bob Friesenhahn bfriesen@HIDDEN, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 30 Dec 2011 12:23:31 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 30 07:23:31 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1RgbUg-00020z-11
for submit <at> debbugs.gnu.org; Fri, 30 Dec 2011 07:23:30 -0500
Received: from mail-wi0-f172.google.com ([209.85.212.172])
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <stefano.lattarini@HIDDEN>) id 1RgbUc-00020p-1D
for 7868 <at> debbugs.gnu.org; Fri, 30 Dec 2011 07:23:28 -0500
Received: by wibhj6 with SMTP id hj6so7491445wib.3
for <7868 <at> debbugs.gnu.org>; Fri, 30 Dec 2011 04:20:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=message-id:date:from:mime-version:to:cc:subject:references
:in-reply-to:content-type;
bh=d5Etj9DI4vN4Nexkoq17hoRV8/vips6t4PuUeY8Yn2A=;
b=QjJhuq4ZtL/Xa+yXFsWj29lCBLoNX6UA8SftdMuV1UDMl6vgxnVLjzUjbRGdeewZOr
PTNE3HLXQmUFCuDOinTguOCS8N3M9kpm2m8Tl6o0G0bFuJg9FcVonJhrE88Y872N07QZ
zHFGLGy5296GXNP8cfRFxolG7qDu3h8i478zo=
Received: by 10.216.137.155 with SMTP id y27mr26758536wei.53.1325247629784;
Fri, 30 Dec 2011 04:20:29 -0800 (PST)
Received: from [87.2.95.175]
(host175-95-dynamic.2-87-r.retail.telecomitalia.it. [87.2.95.175])
by mx.google.com with ESMTPS id fg15sm39378857wbb.7.2011.12.30.04.20.27
(version=SSLv3 cipher=OTHER); Fri, 30 Dec 2011 04:20:28 -0800 (PST)
Message-ID: <4EFDAC7F.6060005@HIDDEN>
Date: Fri, 30 Dec 2011 13:20:15 +0100
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
MIME-Version: 1.0
To: Bob Friesenhahn <bfriesen@HIDDEN>
Subject: Re: parallel-tests: avoid command-line length limit issue.
References: <alpine.GSO.2.01.1009051927440.12887@HIDDEN>
<20100906193953.GO16188@HIDDEN> <20100906200743.GP16188@HIDDEN>
<201009070315.55743.stefano.lattarini@HIDDEN>
<20100907031038.GC3444@HIDDEN> <20101002165332.GC11911@HIDDEN>
<alpine.GSO.2.01.1112281322500.15666@HIDDEN>
<4EFBA597.8060107@HIDDEN>
In-Reply-To: <4EFBA597.8060107@HIDDEN>
Content-Type: multipart/mixed; boundary="------------000009080302050202090008"
X-Spam-Score: -3.9 (---)
X-Debbugs-Envelope-To: 7868
Cc: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>, 7868 <at> debbugs.gnu.org,
automake-patches@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.9 (---)
This is a multi-part message in MIME format.
--------------000009080302050202090008
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
On 12/29/2011 12:26 AM, Stefano Lattarini wrote:
> On 12/28/2011 08:27 PM, Bob Friesenhahn wrote:
>> Sorry for top-posting. I don't want to lose any text of the original mail.
>>
>> I totally forgot that I had applied Ralf's patch to my 1.11.1 install. Now
>> that I have updated GraphicsMagick to using Automake 1.11.2, the patch is no
>> longer present and so 'make check' on MinGW/MSYS leads to this:
>>
>> /usr/bin/csmake check-TESTS check-local
>> csmake[2]: Entering directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[3]: Entering directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[3]: execvp: /bin/sh: Invalid argument
>> csmake[3]: *** [tests/constitute_char_bgr.log] Error 127
>> csmake[3]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[2]: *** [check-TESTS] Error 2
>> csmake[2]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake[1]: *** [check-am] Error 2
>> csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
>> csmake: *** [check] Error 2
>>
>> Today I lost more hair because I had totally forgotten about this issue and
>> the problem was first noticed after making some changes to configure.ac.
>>
>> Hopefully this problem can be resolved before Automake 1.12 is released.
>>
> Let's start by exposing the problem once and for all.
>
> [SNIP]
>
OK, I hope I've finally managed to partially fix this incredibly annoying bug
-- I say "partially" because the fix is sadly *for GNU make*. Even more sadly,
the solution is pretty hacky and somewhat brittle. Still, it should cause no
regression with the non-GNU makes (as they will continue to use the old
implementation), and I have verified that it works with at least all the GNU
make versions >= 3.78.
I'm posting the patches here for early feedback (note that the second one still
needs a commit message and possibly some polishing), with the hope that someone
can suggest a less hacky solution.
Regards,
Stefano
--------------000009080302050202090008
Content-Type: text/x-diff;
name="0001-coverage-expose-automake-bug-7868.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0001-coverage-expose-automake-bug-7868.patch"
From e9d7d23529b21511e4d3d7c71bd239a45288e4bc Mon Sep 17 00:00:00 2001
Message-Id: <e9d7d23529b21511e4d3d7c71bd239a45288e4bc.1325246573.git.stefano.lattarini@HIDDEN>
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
Date: Wed, 28 Dec 2011 22:37:44 +0100
Subject: [PATCH 1/2] coverage: expose automake bug#7868
Expose the command-line length limit issue that can affect the
Automake-generated parallel-tests harness, especially on systems
where this limit is smaller (e.g., MinGW/MSYS).
Suggestion by Bob Friesenhahn.
* tests/parallel-tests-many.test: New test. We have verified that
it actually exposes the bug#7868, as it passes when we opportunely
reduce the number of test cases in $(TESTS). Checked on NetBSD 5.1
64bit, Debian unstable 32bit, Solaris 10 64bit and Cygwin 1.5 32bit.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Add it.
---
tests/Makefile.am | 2 +
tests/parallel-tests-many.test | 176 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 178 insertions(+), 0 deletions(-)
create mode 100755 tests/parallel-tests-many.test
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 6ce7c53..e14d57e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -26,6 +26,7 @@ gcj6.test \
java-nobase.test \
pr8365-remake-timing.test \
lex-subobj-nodep.test \
+parallel-tests-many.test \
remake-am-pr10111.test \
remake-m4-pr10111.test \
vala-vpath.test \
@@ -648,6 +649,7 @@ parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
parallel-tests-log-compiler-example.test \
+parallel-tests-many.test \
test-extensions.test \
test-extensions-cond.test \
parse.test \
diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test
new file mode 100755
index 0000000..ef01db1
--- /dev/null
+++ b/tests/parallel-tests-many.test
@@ -0,0 +1,176 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the parallel-tests harness does not hit errors due to
+# an exceeded command line length when there are many tests.
+# For automake bug#7868. This test is currently expected to fail.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+# Sanity check that the $(TESTS) is going to exceed the system
+# command line length.
+# Extra quoting and indirections below are required to ensure the
+# various make implementations (e.g, GNU make or Sun Distributed Make)
+# will truly spawn a shell to execute this command, instead of relying
+# on optimizations that might mask the "Argument list too long" error
+# we expect.
+this-will-fail:
+ @":" && ":" $(TEST_LOGS)
+TEST_LOG_COMPILER = true
+include list-of-tests.am
+# So that we won't have to create a ton of dummy test cases.
+$(TESTS):
+END
+
+# The real instance will be dynamically created later.
+echo TESTS = foo.test > list-of-tests.am
+
+$ACLOCAL && $AUTOCONF && $AUTOMAKE -a \
+ || framework_failure_ "unexpected autotools failure"
+./configure \
+ || framework_failure_ "unexpected configure failure"
+
+# We want to hit the system command-line length limit without hitting
+# the filename length limit or the PATHMAX limit; so we use longish
+# (but not too long) names for the testcase, and place them in a nested
+# (but not too deeply) directory.
+# We also prefer to use the minimal(ish) number of test cases that can
+# make us hit the command-line length limit, since the more the test
+# cases are, the more time "automake" and "make check" will take to run
+# (especially on Cygwin and MinGW/MSYS).
+
+tname="wow-this-is-a-very-long-name-for-a-simple-dummy-test-case"
+dname="and-this-too-is-a-very-long-name-for-a-dummy-directory"
+
+deepdir=.
+depth=0
+for i in 1 2 3 4 5 6 7 8 9 10 12 13 14 15 16 17 18 19 29 21 22 23 24; do
+ new_deepdir=$deepdir/$dname.d$i
+ mkdir $new_deepdir || break
+ tmpfile=$new_deepdir/$tname-some-more-chars-for-good-measure
+ if touch $tmpfile; then
+ rm -f $tmpfile
+ else
+ rmdir $new_deepdir
+ fi
+ deepdir=$new_deepdir
+ unset tmpfile new_deepdir
+ depth=$i
+done
+
+cat <<END
+*********************************************************************
+Our tests will be in the following directory (depth = $depth)
+*********************************************************************
+$deepdir
+*********************************************************************
+END
+
+setup_data ()
+{
+ awk "
+ BEGIN {
+ print \"TESTS = \\\\\"
+ for (i = 1; i < $count; i = i + 1)
+ print \" $deepdir/$tname-\" i \".test \\\\\"
+ print \" $deepdir/$tname-\" i \".test\"
+ }
+ " > list-of-tests.am || Exit 99
+ $AUTOMAKE Makefile \
+ || framework_failure_ "unexpected automake failure"
+ ./config.status Makefile \
+ || framework_failure_ "unexpected config.status failure"
+}
+
+for count in 1 2 4 8 12 16 20 24 28 32 48 64 96 128 E_HUGE; do
+ test $count = E_HUGE && break
+ count=`expr $count '*' 100`
+ setup_data
+ if $MAKE this-will-fail; then
+ continue
+ else
+ # We have managed to find a number of test cases large enough to
+ # hit the system command-line limits; we can stop. But first, for
+ # good measure, increase the number of tests of some 20%, to be
+ # "even more sure" of really tickling command line length limits.
+ count=`expr '(' $count '*' 12 ')' / 10` || Exit 99
+ setup_data
+ break
+ fi
+done
+
+if test $count = E_HUGE; then
+ framework_failure_ "system has a too-high limit on command line length"
+else
+ cat <<END
+*********************************************************************
+ Number of tests we will use: $count
+*********************************************************************
+END
+fi
+
+env TESTS=$deepdir/$tname-1.test $MAKE -e check \
+ && test -f $deepdir/$tname-1.log \
+ || framework_failure_ "\"make check\" with one single tests"
+
+rm -f $deepdir/* || Exit 99
+
+$MAKE check > stdout || { cat stdout; Exit 1; }
+cat stdout
+
+grep "All $count tests" stdout
+
+grep "^PASS: .*$tname-[0-9][0-9]*\.test" stdout > grp
+ls -1 $deepdir | grep '\.log$' > lst
+
+sed 20q lst # For debugging.
+sed 20q grp # Likewise.
+
+test `cat <grp | wc -l` -eq $count
+test `cat <lst | wc -l` -eq $count
+
+# We need to simulate a failure of two tests.
+st=0
+env TESTS="$deepdir/$tname-1.test $deepdir/$tname-2.test" \
+ TEST_LOG_COMPILER=false $MAKE -e check > stdout && st=1
+cat stdout
+test `grep -c '^FAIL:' stdout` -eq 2 || st=1
+test $st -eq 0 || fatal_ "couldn't simulate failure of two tests"
+unset st
+
+$MAKE recheck > stdout || { cat stdout; Exit 1; }
+cat stdout
+grep "^PASS: .*$tname-1\.test" stdout
+grep "^PASS: .*$tname-2\.test" stdout
+test `LC_ALL=C grep -c "^[A-Z][A-Z]*:" stdout` -eq 2
+grep "All 2 tests" stdout
+
+# "make clean" might ignore some failures, so we prefer to also grep its
+# output to ensure that no "Argument list too long" error was encountered.
+$MAKE clean >output 2>&1 || { cat output; Exit 1; }
+cat output
+grep -i 'list.* too long' output && Exit 1
+ls -1 $deepdir | grep '\.log$' && Exit 1
+
+:
--
1.7.7.3
--------------000009080302050202090008
Content-Type: text/x-diff;
name="0002-fix-the-bug-for-GNU-make.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-fix-the-bug-for-GNU-make.patch"
From ca6000f9cf6c84c516230dc86eba8b9d920dfb20 Mon Sep 17 00:00:00 2001
Message-Id: <ca6000f9cf6c84c516230dc86eba8b9d920dfb20.1325246574.git.stefano.lattarini@HIDDEN>
In-Reply-To: <e9d7d23529b21511e4d3d7c71bd239a45288e4bc.1325246573.git.stefano.lattarini@HIDDEN>
References: <e9d7d23529b21511e4d3d7c71bd239a45288e4bc.1325246573.git.stefano.lattarini@HIDDEN>
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
Date: Thu, 29 Dec 2011 10:46:48 +0100
Subject: [PATCH 2/2] fix the bug for GNU make
---
NEWS | 6 +++
automake.in | 22 +++++-----
lib/am/check.am | 86 ++++++++++++++++++++++++++++++++++------
m4/init.m4 | 17 ++++++++
tests/Makefile.am | 1 -
tests/parallel-tests-many.test | 4 +-
6 files changed, 111 insertions(+), 25 deletions(-)
diff --git a/NEWS b/NEWS
index f6eb29f..34948f3 100644
--- a/NEWS
+++ b/NEWS
@@ -73,6 +73,12 @@ Bugs fixed in 1.11.0a:
* Bugs introduced by 1.11:
+ - When GNU make is used, the parallel testsuite harness can now work
+ around problems due to exceeded command-line length limits,
+ problems that that could be encountered when issuing "make check"
+ and "make clean". This is especially important for system with
+ low command-line length limits, like MinGW/MSYS.
+
- The `parallel-tests' test driver works around a GNU make 3.80 bug with
trailing white space in the test list (`TESTS = foo $(EMPTY)'), and
does not report spurious successes when used with concurrent FreeBSD
diff --git a/automake.in b/automake.in
index 309eade..a25c5c9 100644
--- a/automake.in
+++ b/automake.in
@@ -4933,9 +4933,13 @@ sub is_valid_test_extension ($)
return 0;
}
+# handle_tests ($MAKEFILE)
+# ------------------------
# Handle TESTS variable and other checks.
-sub handle_tests
+sub handle_tests ($)
{
+ my ($makefile) = @_;
+
if (option 'dejagnu')
{
&handle_tests_dejagnu;
@@ -4954,7 +4958,8 @@ sub handle_tests
push (@check_tests, 'check-TESTS');
$output_rules .= &file_contents ('check', new Automake::Location,
COLOR => !! option 'color-tests',
- PARALLEL_TESTS => !! option 'parallel-tests');
+ PARALLEL_TESTS => !! option 'parallel-tests',
+ MAKEFILE => basename $makefile);
# Tests that are known programs should have $(EXEEXT) appended.
# For matching purposes, we need to adjust XFAIL_TESTS as well.
@@ -5046,7 +5051,7 @@ sub handle_tests
{
if ($test_suffix eq $last_suffix)
{
- $cur = 'TEST_LOGS';
+ $cur = 'am__test_logs';
}
else
{
@@ -5075,13 +5080,8 @@ sub handle_tests
am__EXEEXT => $am_exeext);
}
}
-
- define_variable ('TEST_LOGS_TMP', '$(TEST_LOGS:.log=.log-t)', INTERNAL);
-
- $clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
- $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
- $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
- $clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN;
+ # This needs to be further postprocessed in lib:am/check.am.
+ define_variable ('am__test_logs', "\$($cur)", INTERNAL);
}
}
}
@@ -8269,7 +8269,7 @@ sub generate_makefile ($$)
handle_tags;
handle_minor_options;
# Must come after handle_programs so that %known_programs is up-to-date.
- handle_tests;
+ handle_tests ($makefile);
# This must come after most other rules.
handle_dist;
diff --git a/lib/am/check.am b/lib/am/check.am
index 3d0188d..82d9fe2 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -148,9 +148,46 @@ echo "$$res: $$f (exit: $$estatus)" | \
cat $@-t >>$@; \
rm -f $@-t
+if am__MAKE_IS_GNU
+## FIXME: This is heineous. There must be a better way to obtain what we
+## want from GNU make! But still, this at least works for the moment.
+am__start = <--am--start--here-->
+am__end= <--am--end--here-->
+am__get_list_from_var = \
+ { echo 'am--nil:' \
+ && echo '.PHONY: am--nil' \
+## We use $(warning ...), not $(info ...), since the latter has only been
+## introduced in GNU make 3.81.
+ && echo '$$(warning $(am__start) $$('$$am__varname') $(am__end))'; } \
+ | $(MAKE) $(AM_MAKEFLAGS) -f %MAKEFILE% -f - am--nil 2>&1 \
+ | tr ' ' '\012\012' | grep . \
+ | sed -n '/^$(am__start)$$/,/^$(am__end)$$/p' | sed -e '1d' -e '$$d'
+am__delete_files_in_var = \
+ $(am__get_list_from_var) | $(am__base_list) | ( \
+ st=0; \
+ while read files; do \
+ test -z "$$files" || rm -f $$files || st=1; \
+ done; \
+ exit $$st)
+am__get_test_logs_list = \
+ am__varname=TEST_LOGS && list=`$(am__get_list_from_var)` || exit 1
+else ! am__MAKE_IS_GNU
+am__get_test_logs_list = list=' $(TEST_LOGS) '
+endif
+
+# Trailing whitespace in "TESTS = foo.test $(empty)" causes at least
+# GNU make 3.80 to erroneously expand $(TESTS_LOGS) to "foo.log .log".
+# Work around this bug.
+if am__MAKE_IS_GNU
+TEST_LOGS = @am__dollar@(filter-out .log, $(am__test_logs))
+else
+TEST_LOGS = $(am__test_logs)
+endif
+TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); \
- list='$(TEST_LOGS)'; \
+ $(am__get_test_logs_list); \
results=`for f in $$list; do \
test -r $$f && read line < $$f && echo "$$line" \
|| echo FAIL; \
@@ -235,8 +272,11 @@ RECHECK_LOGS = $(TEST_LOGS)
# Run all the tests.
check-TESTS:
-## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
+if am__MAKE_IS_GNU
+ @am__varname=RECHECK_LOGS && $(am__delete_files_in_var)
+else
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+endif
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
## or a prior run plus reruns all happen within the same timestamp
@@ -245,16 +285,15 @@ check-TESTS:
## OTOH, this means that, in the rule for `$(TEST_SUITE_LOG)', we
## cannot use `$?' to compute the set of lazily rerun tests, lest
## we rely on .PHONY to work portably.
-##
-## Trailing whitespace in `TESTS = foo.test $(empty)' causes GNU make
-## 3.80 to erroneously expand $(TESTS_LOGS) to `foo.log .log'.
-## Work around this bug.
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
- @list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test .log = $$f || echo $$f; \
- done | tr '\012\015' ' '`; \
- $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
+if am__MAKE_IS_GNU
+## No need to pass on '$(TEST_LOGS)' explicitly. GNU make will take
+## care of that automatically, even if the user has overridden $(TESTS)
+## or $(TEST_LOGS).
+ $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG)
+else
+ $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS='$(TEST_LOGS)'
+endif
AM_RECURSIVE_TARGETS += check
@@ -299,7 +338,7 @@ AM_RECURSIVE_TARGETS += check-html
## Rerun all FAILed or XPASSed tests.
recheck recheck-html:
@target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
+ $(am__get_test_logs_list); \
list=`for f in $$list; do \
test -f $$f || continue; \
if test -r $$f && read line < $$f; then \
@@ -309,6 +348,8 @@ recheck recheck-html:
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
list=`echo "$$list" | sed 's/ *$$//'`; \
+## FIXME: this might fail on systems with low command-line length limits
+## FIXME: in case there are many failed tests. Is it worth worrying about?
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'
.PHONY: recheck recheck-html
@@ -316,6 +357,27 @@ recheck recheck-html:
AM_RECURSIVE_TARGETS += recheck recheck-html
+## ------------- ##
+## Cleaning up. ##
+## ------------- ##
+
+## FIXME: the support for variables with contents exceeding the
+## FIXME: command line length limit should be propagated to
+## FIXME: `lib/am/clean.am'.
+mostlyclean-am: am--parallel-tests-mostlyclean
+am--parallel-tests-mostlyclean:
+ @echo ' rm -f $$(TEST_LOGS)'
+if am__MAKE_IS_GNU
+ @am__varname=TEST_LOGS_TMP && $(am__delete_files_in_var)
+ @am__varname=TEST_LOGS && $(am__delete_files_in_var)
+else
+ @list='$(TEST_LOGS_TMP)'; test -z "$$list" || rm -f $$list
+ @list='$(TEST_LOGS)'; test -z "$$list" || rm -f $$list
+endif
+ @echo " rm -f $(TEST_SUITE_LOG) $(TEST_SUITE_HTML)"
+ @rm -f $(TEST_SUITE_LOG) $(TEST_SUITE_HTML)
+.PHONY: am--parallel-tests-mostlyclean
+
else !%?PARALLEL_TESTS%
check-TESTS: $(TESTS)
diff --git a/m4/init.m4 b/m4/init.m4
index 365c9ac..84ada72 100644
--- a/m4/init.m4
+++ b/m4/init.m4
@@ -83,6 +83,23 @@ AC_REQUIRE([AM_PROG_MKDIR_P])dnl
AC_REQUIRE([AC_PROG_AWK])dnl
AC_REQUIRE([AC_PROG_MAKE_SET])dnl
AC_REQUIRE([AM_SET_LEADING_DOT])dnl
+dnl
+dnl FIXME: is it worth to extract this in a new macro of its own?
+am_make=${MAKE-make}
+AC_MSG_CHECKING([whether $am_make is GNU make])
+if ($am_make -n -f /dev/null --version) >/dev/null 2>&1; then
+ am_using_gmake=yes
+else
+ am_using_gmake=no
+fi
+AC_MSG_RESULT([$am_using_gmake])
+AM_CONDITIONAL([am__MAKE_IS_GNU], [test $am_using_gmake = yes])dnl
+dnl
+dnl Hack useful to write Makefile fragments that might otherwise
+dnl trigger spurious portability warning w.r.t. non-POSIX variable
+dnl names.
+AC_SUBST([am__dollar], [$])_AM_SUBST_NOTMAKE([am__dollar])dnl
+dnl
_AM_IF_OPTION([tar-ustar], [_AM_PROG_TAR([ustar])],
[_AM_IF_OPTION([tar-pax], [_AM_PROG_TAR([pax])],
[_AM_PROG_TAR([v7])])])
diff --git a/tests/Makefile.am b/tests/Makefile.am
index e14d57e..27305d0 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -26,7 +26,6 @@ gcj6.test \
java-nobase.test \
pr8365-remake-timing.test \
lex-subobj-nodep.test \
-parallel-tests-many.test \
remake-am-pr10111.test \
remake-m4-pr10111.test \
vala-vpath.test \
diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test
index ef01db1..c470745 100755
--- a/tests/parallel-tests-many.test
+++ b/tests/parallel-tests-many.test
@@ -16,9 +16,11 @@
# Check that the parallel-tests harness does not hit errors due to
# an exceeded command line length when there are many tests.
-# For automake bug#7868. This test is currently expected to fail.
+# For automake bug#7868. This test is currently expected to pass
+# only with GNU make.
parallel_tests=yes
+required=GNUmake
. ./defs || Exit 1
set -e
--
1.7.7.3
--------------000009080302050202090008--
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 28 Dec 2011 23:29:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Dec 28 18:29:10 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1Rg2vl-000534-GC
for submit <at> debbugs.gnu.org; Wed, 28 Dec 2011 18:29:09 -0500
Received: from mail-ee0-f44.google.com ([74.125.83.44])
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <stefano.lattarini@HIDDEN>) id 1Rg2vi-00052w-Ss
for 7868 <at> debbugs.gnu.org; Wed, 28 Dec 2011 18:29:08 -0500
Received: by eekc14 with SMTP id c14so13379445eek.3
for <7868 <at> debbugs.gnu.org>; Wed, 28 Dec 2011 15:26:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=message-id:date:from:mime-version:to:cc:subject:references
:in-reply-to:content-type;
bh=Wcft3NSIFK0uOc4H3rZOQCDpno9A3nYLiUJjq1XXujo=;
b=NAM4MSy909VNn6ezsiCOOMjmU3lzk8C2kfpF8PLqVZHQLxJjbfKSPhx5cLJ+0rEWzf
lmXejscgPmFim4jCiRxE3cKhAOQ5IL3zeckeCKJuHWS4Bs/03H5GVTTsy8xVs9tD1AMA
Z8kLBO7xGZ7OY9NLOky0Jq8mwHfG54q9DGqDM=
Received: by 10.213.113.2 with SMTP id y2mr6059754ebp.147.1325114779546;
Wed, 28 Dec 2011 15:26:19 -0800 (PST)
Received: from [87.2.102.244]
(host244-102-dynamic.2-87-r.retail.telecomitalia.it. [87.2.102.244])
by mx.google.com with ESMTPS id
y12sm125585683eeb.11.2011.12.28.15.26.18
(version=SSLv3 cipher=OTHER); Wed, 28 Dec 2011 15:26:18 -0800 (PST)
Message-ID: <4EFBA597.8060107@HIDDEN>
Date: Thu, 29 Dec 2011 00:26:15 +0100
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
MIME-Version: 1.0
To: Bob Friesenhahn <bfriesen@HIDDEN>
Subject: Re: parallel-tests: avoid command-line length limit issue.
References: <alpine.GSO.2.01.1009051927440.12887@HIDDEN>
<20100906193953.GO16188@HIDDEN> <20100906200743.GP16188@HIDDEN>
<201009070315.55743.stefano.lattarini@HIDDEN>
<20100907031038.GC3444@HIDDEN> <20101002165332.GC11911@HIDDEN>
<alpine.GSO.2.01.1112281322500.15666@HIDDEN>
In-Reply-To: <alpine.GSO.2.01.1112281322500.15666@HIDDEN>
Content-Type: multipart/mixed; boundary="------------010701040009080909040001"
X-Spam-Score: -3.9 (---)
X-Debbugs-Envelope-To: 7868
Cc: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>, 7868 <at> debbugs.gnu.org,
automake-patches@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.9 (---)
This is a multi-part message in MIME format.
--------------010701040009080909040001
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
On 12/28/2011 08:27 PM, Bob Friesenhahn wrote:
> Sorry for top-posting. I don't want to lose any text of the original mail.
>
> I totally forgot that I had applied Ralf's patch to my 1.11.1 install. Now
> that I have updated GraphicsMagick to using Automake 1.11.2, the patch is no
> longer present and so 'make check' on MinGW/MSYS leads to this:
>
> /usr/bin/csmake check-TESTS check-local
> csmake[2]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: Entering directory `/home/bfriesen/mingw/GM-16-static'
> csmake[3]: execvp: /bin/sh: Invalid argument
> csmake[3]: *** [tests/constitute_char_bgr.log] Error 127
> csmake[3]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[2]: *** [check-TESTS] Error 2
> csmake[2]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake[1]: *** [check-am] Error 2
> csmake[1]: Leaving directory `/home/bfriesen/mingw/GM-16-static'
> csmake: *** [check] Error 2
>
> Today I lost more hair because I had totally forgotten about this issue and
> the problem was first noticed after making some changes to configure.ac.
>
> Hopefully this problem can be resolved before Automake 1.12 is released.
>
Let's start by exposing the problem once and for all.
What about the attached patch? Tested on Debian unstable, NetBSD 5.1,
Cygwin 1.5.25 and Solaris 10, and it seems to correctly expose the problem
on all those systems.
Thanks,
Stefano
--------------010701040009080909040001
Content-Type: text/x-diff;
name="0001-coverage-expose-automake-bug-7868.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0001-coverage-expose-automake-bug-7868.patch"
From 426d0ba21120eacf296eb0a6c81902d2b24ffa60 Mon Sep 17 00:00:00 2001
Message-Id: <426d0ba21120eacf296eb0a6c81902d2b24ffa60.1325114454.git.stefano.lattarini@HIDDEN>
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
Date: Wed, 28 Dec 2011 22:37:44 +0100
Subject: [PATCH] coverage: expose automake bug#7868
Expose the command-line length limit issue that can affect the
Automake-generated parallel-tests harness, especially on systems
where this limit is smaller (e.g., MinGW/MSYS).
Suggestion by Bob Friesenhahn.
* tests/parallel-tests-many.test: New test. We have verified that
it actually exposes the bug#7868, as it passes when we opportunely
reduce the number of test cases in $(TESTS). Checked on NetBSD 5.1,
Debian unstable, Solaris 10 and Cygwin 1.5.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Add it.
---
tests/Makefile.am | 2 +
tests/parallel-tests-many.test | 92 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 94 insertions(+), 0 deletions(-)
create mode 100755 tests/parallel-tests-many.test
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 6ce7c53..e14d57e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -26,6 +26,7 @@ gcj6.test \
java-nobase.test \
pr8365-remake-timing.test \
lex-subobj-nodep.test \
+parallel-tests-many.test \
remake-am-pr10111.test \
remake-m4-pr10111.test \
vala-vpath.test \
@@ -648,6 +649,7 @@ parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
parallel-tests-log-compiler-example.test \
+parallel-tests-many.test \
test-extensions.test \
test-extensions-cond.test \
parse.test \
diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test
new file mode 100755
index 0000000..f0b4c85
--- /dev/null
+++ b/tests/parallel-tests-many.test
@@ -0,0 +1,92 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the parallel-tests harness does not hit errors due to
+# an exceeded command line length when there are many tests.
+# For automake bug#7868. This test is currently expected to fail.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+basename="wow-this-is-a-very-long-name-for-a-simple-dummy-test-case"
+basename=$basename-$basename
+count=10000
+
+cat > Makefile.am <<'END'
+## Sanity check that the $(TESTS) is going to exceed the system
+## command line length. We can't use echo here, since it might
+## be a built-in, so use our dummy program.
+this-will-fail:
+## Extra quoting and indirections below are required to ensure the
+## various make implementations (e.g, GNU make or Sun Distributed Make)
+## will truly spawn a shell to execute this command, instead of relying
+## on optimizations that might mask the "Argument list too long" error
+## we expect.
+ @sh=sh && "$sh" -c : $(TESTS)
+TEST_LOG_COMPILER = true
+## We will shave *a lot* of automake execution time by extending
+## this by hand, later.
+TESTS = @my_tests@
+END
+
+cat >> configure.in << 'END'
+AC_SUBST([my_tests], ['$(my_tests)'])
+AM_SUBST_NOTMAKE([my_tests])
+AC_OUTPUT
+END
+
+awk "
+ BEGIN {
+ for (i = 1; i <= $count; i = i + 1)
+ print \"$basename-\" i \".test\"
+ }
+" > list-of-tests
+
+$ACLOCAL && $AUTOCONF && $AUTOMAKE -a || Exit 99
+
+echo 'my_tests = \' >> Makefile.in
+sed 's/$/ \\/' list-of-tests >> Makefile.in
+echo ' $(empty)' >> Makefile.in
+# So that we won't have to create a ton of dummy test cases.
+echo '$(TESTS):' >> Makefile.in
+
+tail -n 15 Makefile.in || : # For debugging.
+
+./configure || Exit 99
+
+env TESTS="$basename-1.test" $MAKE -e check && rm -f *.log || Exit 99
+
+if $MAKE this-will-fail; then
+ framework_failure_ "system has a too-high limit on command line length"
+fi
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+grep "All $count tests" stdout
+
+grep "^PASS: $basename" stdout > grp
+ls -1 | grep '\.log' | $EGREP -v '^(config|test-suite)\.log$' > lst
+
+sed 20q lst # For debugging.
+sed 20q grp # Likewise.
+
+test `cat <grp | wc -l` -eq $count
+test `cat <lst | wc -l` -eq $count
+
+:
--
1.7.7.3
--------------010701040009080909040001--
bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Stefano Lattarini <stefano.lattarini@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 22 Jan 2011 09:58:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jan 22 04:58:56 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1PgaFE-0002oF-4s
for submit <at> debbugs.gnu.org; Sat, 22 Jan 2011 04:58:56 -0500
Received: from mailout-de.gmx.net ([213.165.64.23])
by debbugs.gnu.org with smtp (Exim 4.69)
(envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PgaFB-0002nz-Rw
for 7868 <at> debbugs.gnu.org; Sat, 22 Jan 2011 04:58:55 -0500
Received: (qmail invoked by alias); 22 Jan 2011 10:06:49 -0000
Received: from xdsl-89-0-146-100.netcologne.de (EHLO localhost.localdomain)
[89.0.146.100]
by mail.gmx.net (mp023) with SMTP; 22 Jan 2011 11:06:49 +0100
X-Authenticated: #13673931
X-Provags-ID: V01U2FsdGVkX1/OZ6p4LMjO73+47VMUWGV78JmERF6ibJwdiCXHQ9
LqgdNCwFdQsKtV
Received: from ralf by localhost.localdomain with local (Exim 4.69)
(envelope-from <Ralf.Wildenhues@HIDDEN>)
id 1PgaMq-0000GJ-BV; Sat, 22 Jan 2011 11:06:48 +0100
Date: Sat, 22 Jan 2011 11:06:48 +0100
From: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>
To: Stefano Lattarini <stefano.lattarini@HIDDEN>
Subject: Re: bug#7868: splitting up test suites
Message-ID: <20110122100648.GG31878@HIDDEN>
References: <20110119222211.GA14878@HIDDEN>
<201101201238.18228.stefano.lattarini@HIDDEN>
<20110120194108.GG21455@HIDDEN>
<201101202219.25662.stefano.lattarini@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <201101202219.25662.stefano.lattarini@HIDDEN>
Organization: Institute for Numerical Simulation, University of Bonn
User-Agent: Mutt/1.5.20 (2010-08-04)
X-Y-GMX-Trusted: 0
X-Spam-Score: -2.8 (--)
X-Debbugs-Envelope-To: 7868
Cc: 7868 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -2.8 (--)
Hi Stefano,
* Stefano Lattarini wrote on Thu, Jan 20, 2011 at 10:19:24PM CET:
> On Thursday 20 January 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> > > Hmmm... while this feature might be worth having even indipendently
> > > from the issue at hand (but see below for small nits), I still think
> > > that in the long run it would be nicer to transparently work around
> > > such command-line length issues in the test driver, if possible.
> >
> > No, that is not possible with portable make.
> >
> Yes :-( I've found that out while trying to write a a proper test
> 'parallel-tests-long-cmdline.test' (attached, just for reference).
[...]
> But this makes me think. If I substitute the `$MAKE check' call
> in my test with a `$MAKE dist' call, I get:
>
> + make dist
> { test ! -d "parallel-tests-long-cmdline-1.0" || \
> { find "parallel-tests-long-cmdline-1.0" -type d ! -perm -200 -exec chmod u+w {} ';' \
> && rm -fr "parallel-tests-long-cmdline-1.0"; }; }
> test -d "parallel-tests-long-cmdline-1.0" || mkdir "parallel-tests-long-cmdline-1.0"
> make: execvp: /bin/sh: Argument list too long
> make: *** [distdir] Error 127
>
> which means that, with the current implementation of the `dist' target,
> even breaking the testsuite in two or more testsuites *in the same
> directory* won't help at "make dist" time. Or am I missing something?
No, you are correct. This was similarly reported even for GNU/Linux by
Xan Lopez on the automake list over a year ago (and probably before that
too).
I'm not too worried about dist however, for a couple of reasons:
dist is not usually required for downloading, building, testing, and
installing software. As such, even if it is GCS-mandated, it is not
quite as big a problem when it doesn't work everywhere; but of course
it needs to work on developer machines. Furthermore, it is almost
always possible for the developer to restructure the package so that
line lengths are not exceeded, usually by introducing (more) Makefile
recursion. See the `Length Limitations' node in the manual.
That said, I would like to address distdir in the long run as well.
I see two possible strategies:
As far as possible, automake can try to split the copying part of
distdir into several targets that copy parts of the files. This isn't
as easy as it sounds, because usually it's one of the variables SOURCES,
EXTRA_DIST, etc. alone that exceeds the limit. But the splitting can be
done at automake time. (Note that if @substitutions@ are involved, then
a single expansion can currently not exceed the line length limit by
itself, that would already break at configure or config.status time.)
Second, and much simpler: when we have gnu-make support, we can simply
use GNU make-specific code to avoid running into limits. This can be
done transparently, and since in practice virtually all package
maintainers at least have access to GNU make, it should be sufficient
in practice.
I have a half-done patch for the second scheme. I hope to post all the
gnu-make stuff soonish, sorry for the delay.
> But maybe allowing the user to say, e.g.:
>
> TESTS = test1 test2 test3
> EXTRA_TESTS_LIST = @file-with-list-of-test
>
> might still be worth after all; if he prefers to lose some make semantics
> in order not to have to split the testsuite, then he should be allowed to
> do so. It remains to see if that's doable (preferably in an easy way).
Yes, it may be worth it. But if we can avoid it, all the better.
I prefer not breaking our usual semantic scheme.
> > If we can handle the latter via automake's register_language machinery,
> > then putting tests in *_SOURCES variables would be fairly natural.
> >
> > OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
> > final, nonderived files,
> >
> That's not exactly true, as automke allows (as it IMHO should)
> generated files to be places in *_SOURCES. But automake also needs
> to know the contents of a foo_SOURCES variable statically [1] at
> Makefile.in's generation time, while it wants to allow the user to
> specify *at make runtime* the list of TESTS to run.
Yes, that is one of the other complications/semantic differences:
users usually cope with not being able to override *_OBJECTS.
We considered declaring TESTS an internal detail too, but I just
find it too helpful to give up this API.
> [1] OK, it's smart enough to resolve variable indirections and
> conditionals, but won't allow @var@ substitution to be placed
> in a *_SOURCES variable (by design).
FWIW, I regard that a necessary limitation, not a design feature.
> > > # These are all specified by the user:
> > > TEST_SUITES = suite1 suite2 suite3 ..
[...]
> > > # These are then produced by automake:
> > > TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > > TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
> > > suite1_LOGS = $(am__helper_var:.test=.log)
> >
> > Unfortunately, this is error-prone, because some GNU make versions
> > expand
> > empty =
> > TEST_SUITES = foo $(empty)
> > TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> >
> > into `foo.log .log' rather than `foo.log '. This happens in practice
> > if `empty' is set only under some Automake conditional, as in
> > if COND
> > TESTS += bar
> > endif
> >
> And I guess that specifying dummy `.log:' (and maybe `.html:' etc.)
> rules won't work either, right? :-(
Nope.
> > That is the reason the check-TESTS rule is so ugly (and recursive) in
> > the first place. I really would like to avoid more instances of this
> > wart; so specifying files without extension is Not Good(TM).
> >
> OK, noted (and these considerations could IMHO end up somewhere in the
> manual).
Good suggestion. They should be mentioned in the portability section of
autoconf.texi if they aren't already, and as ## comments in check.am.
Let me see about a quick patch.
> > Separate check/recheck targets are of course nice, but IMVHO optional
> > for the first iteration.
> >
> I agree; the only important thing is to devise a design that will allow
> them to be easily added in later refinings.
Don't worry.
Cheers,
Ralf
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 20 Jan 2011 21:13:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 20 16:13:32 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1Pg1ox-0003iW-QN
for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 16:13:32 -0500
Received: from mail-bw0-f44.google.com ([209.85.214.44])
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <stefano.lattarini@HIDDEN>) id 1Pg1ov-0003iK-4p
for 7868 <at> debbugs.gnu.org; Thu, 20 Jan 2011 16:13:30 -0500
Received: by bwz12 with SMTP id 12so970735bwz.3
for <7868 <at> debbugs.gnu.org>; Thu, 20 Jan 2011 13:21:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=domainkey-signature:from:to:subject:date:user-agent:cc:references
:in-reply-to:mime-version:content-type:message-id;
bh=EWEz9BDQbNp387YHS0IOqZdQhLhGidjaPQBp207KIPw=;
b=EPISrjypz2O0s1QZU5eJq+7jSrOVoSNNt3g19x6gZXu85+y7VbvgJAN48Odx3BXr+c
9unsm++bBNa4k2p1ZqROVDNoDzfPmUm/x+O7VDuqTudXxnjFx/YRGaqYvFCv3dRaHAe4
jjObH2X/IdgbRKl7G3UE5NjBtVRi/a9PoUIw8=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
h=from:to:subject:date:user-agent:cc:references:in-reply-to
:mime-version:content-type:message-id;
b=RNB6RW7VO02DC7wji7JSQBkLZFqtg5yZ84ZHNNYuRGM7WVA/e69Bcef9edtb3j+5mu
9z2DOTJIGY2Y4B1aEg2gCmrK0cyQQgsvcI+NEHeHEsgfdobOmQ6OTR6ZZ0dkl9e1sSW7
VJOpmTYcWth2IndSPO25PjSeb1dBc8vtUgiU0=
Received: by 10.204.112.147 with SMTP id w19mr1463769bkp.137.1295558383854;
Thu, 20 Jan 2011 13:19:43 -0800 (PST)
Received: from bigio.localnet
(host176-97-dynamic.30-79-r.retail.telecomitalia.it [79.30.97.176])
by mx.google.com with ESMTPS id x38sm4202514bkj.1.2011.01.20.13.19.41
(version=TLSv1/SSLv3 cipher=RC4-MD5);
Thu, 20 Jan 2011 13:19:42 -0800 (PST)
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
To: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>
Subject: Re: bug#7868: splitting up test suites
Date: Thu, 20 Jan 2011 22:19:24 +0100
User-Agent: KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; )
References: <20110119222211.GA14878@HIDDEN>
<201101201238.18228.stefano.lattarini@HIDDEN>
<20110120194108.GG21455@HIDDEN>
In-Reply-To: <20110120194108.GG21455@HIDDEN>
MIME-Version: 1.0
Content-Type: Multipart/Mixed;
boundary="Boundary-00=_dbKONkyi+FSxmxb"
Message-Id: <201101202219.25662.stefano.lattarini@HIDDEN>
X-Spam-Score: -3.6 (---)
X-Debbugs-Envelope-To: 7868
Cc: 7868 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.6 (---)
--Boundary-00=_dbKONkyi+FSxmxb
Content-Type: Text/Plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
On Thursday 20 January 2011, Ralf Wildenhues wrote:
> Hi Stefano,
>
> * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> > On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > > We've finally reached the point where we have more than 1000
> > > tests, $(TESTS) expands to 15k characters, and where 'make check' will
> > > not work at all any more on MSYS, because it cannot spawn sh any more,
> > > presumably in 'make check TESTS="..."'. (MSYS make doesn't export
> > > macros to the environment of spawned processes even without .NOEXPORT,
> > > presumably otherwise lots of Makefiles would be really unusable here.)
> > > This also clears up the spurious failure of sed a few days ago.
> > >
> > > Here's a preliminary plan for multiple testsuites per Makefile.am.
> > >
> > Hmmm... while this feature might be worth having even indipendently
> > from the issue at hand (but see below for small nits), I still think
> > that in the long run it would be nicer to transparently work around
> > such command-line length issues in the test driver, if possible.
>
> No, that is not possible with portable make.
>
Yes :-( I've found that out while trying to write a a proper test
'parallel-tests-long-cmdline.test' (attached, just for reference).
Relevant excerpt of the log (on Linux 2.6.30 with Bash 4.1 and GNU
make 3.81):
+ automake-1.11 --foreign -Werror -Wall -a
+ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
configure: creating ./config.status
config.status: creating Makefile
+ make check
make[1]: execvp: /bin/sh: Argument list too long
make[1]: *** [check-TESTS] Error 127
make: *** [check-am] Error 2
But this makes me think. If I substitute the `$MAKE check' call
in my test with a `$MAKE dist' call, I get:
+ make dist
{ test ! -d "parallel-tests-long-cmdline-1.0" || \
{ find "parallel-tests-long-cmdline-1.0" -type d ! -perm -200 -exec chmod u+w {} ';' \
&& rm -fr "parallel-tests-long-cmdline-1.0"; }; }
test -d "parallel-tests-long-cmdline-1.0" || mkdir "parallel-tests-long-cmdline-1.0"
make: execvp: /bin/sh: Argument list too long
make: *** [distdir] Error 127
which means that, with the current implementation of the `dist' target,
even breaking the testsuite in two or more testsuites *in the same
directory* won't help at "make dist" time. Or am I missing something?
> > Do you think your patch "parallel-tests: avoid command-line length
> > limit issue" (from commit v1.11-191-g24e3b4e) could be resurrected
> > in some way?
>
> No. It is fundamentally flawed. Here's why: while it may fix things at
> the level of recursion below that, eventually the rule commands which
> were changed in that patch will fail to execute, because they pose too
> long a command line for the shell.
>
Exactly :-(
> There simply is no way to get things to scale except by using GNU make
> specifics, or splitting long lists of files up manually. Well, an
> external list of files to be read in could work,
>
I thought about that too ..
> but then we don't have make semantics easily available.
>
... and I thought about this too. But maybe allowing the user to say,
e.g.:
TESTS = test1 test2 test3
EXTRA_TESTS_LIST = @file-with-list-of-test
might still be worth after all; if he prefers to lose some make semantics
in order not to have to split the testsuite, then he should be allowed to
do so. It remains to see if that's doable (preferably in an easy way).
> In that light, I'm sorry you already wrote the patch you did, because I
> don't see how it can improve things significantly. I didn't try the
> patch or look in detail, please ping me if you still want me to do that.
>
No need to: the approach is really fundamentally flawed (as you said).
> > > It would be nice if this worked somehow:
> > >
> > > # These are all specified by the user:
> > > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
> > > TEST_EXTENSIONS = .test ...
> > > # these undergo $(EXEEXT) autoexpansion internally:
> > > suite1_log_SOURCES = all.test aclocal7.test ...
> > > suite2_log_SOURCES = suffix.test ...
> > >
> > What about using `TESTS' instead of `SOURCES' in these two last variables?
> > That would seems more natural to me, especially considering the API of the
> > current parallel-tests driver.
>
> First off, it would make sense to review the several discussions we had
> about this back when parallel-tests was implemented (more precisely,
> ported from Akim's GNU make-specific implementation). See e.g.,
> http://thread.gmane.org/gmane.comp.sysutils.automake.patches/3225
> http://thread.gmane.org/gmane.comp.sysutils.automake.general/8063
>
Thanks for the pointers, I'll take a look.
> The idea was to treat tests as we treat compiled languages:
> .c files are sources for programs but get turned into .o files.
> Here, .test files are sources for testsuite logs, but the intermediates
> are the per-test .log files. And there's maybe the unification of
> suite*.log's to a final log on top of that.
>
I haven't thought in thie terms before -- nice abstraction! But see
below.
> If we can handle the latter via automake's register_language machinery,
> then putting tests in *_SOURCES variables would be fairly natural.
>
> OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
> final, nonderived files,
>
That's not exactly true, as automke allows (as it IMHO should)
generated files to be places in *_SOURCES. But automake also needs
to know the contents of a foo_SOURCES variable statically [1] at
Makefile.in's generation time, while it wants to allow the user to
specify *at make runtime* the list of TESTS to run.
[1] OK, it's smart enough to resolve variable indirections and
conditionals, but won't allow @var@ substitution to be placed
in a *_SOURCES variable (by design).
> which doesn't fit the picture here. So maybe suite1_log_TESTS
> is more appropriate after all.
> > OTOH, would that maybe make the implementation more difficult?
>
> Not if it's like the current implementation, no.
>
> > I think it would also be nice to generate separate check/recheck targets
> > for each testsuite; for example, "make check-suite1" could run all the
> > tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run
> > all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the
> > previous run.
> >
> > Hmm... No, wait, it would be even nicer to allow the user choose which
> > testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable:
> >
> > make check TEST_SUITE_LOGS='suite1.log suite2.log'
> >
> > Which makes me think that, perhaps, a variable like $(TEST_SUITES) would
> > be preferable:
> >
> > make check TEST_SUITES='suite1 suite2'
> >
> > but then the API should be changed to something like:
> >
> > # These are all specified by the user:
> > TEST_SUITES = suite1 suite2 suite3 ..
> > TEST_EXTENSIONS = .test ...
> > # these undergo $(EXEEXT) autoexpansion internally:
> > suite1_TESTS = all.test aclocal7.test ...
> > suite2_TESTS = suffix.test ...
> >
> > # These are then produced by automake:
> > TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
> > suite1_LOGS = $(am__helper_var:.test=.log)
>
> Unfortunately, this is error-prone, because some GNU make versions
> expand
> empty =
> TEST_SUITES = foo $(empty)
> TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
>
> into `foo.log .log' rather than `foo.log '. This happens in practice
> if `empty' is set only under some Automake conditional, as in
> if COND
> TESTS += bar
> endif
>
And I guess that specifying dummy `.log:' (and maybe `.html:' etc.)
rules won't work either, right? :-(
> That is the reason the check-TESTS rule is so ugly (and recursive) in
> the first place. I really would like to avoid more instances of this
> wart; so specifying files without extension is Not Good(TM).
>
OK, noted (and these considerations could IMHO end up somewhere in the
manual).
> Separate check/recheck targets are of course nice, but IMVHO optional
> for the first iteration.
>
I agree; the only important thing is to devise a design that will allow
them to be easily added in later refinings.
> > > In the <suite>_SOURCES, $(EXEEXT) transformation should take place
> > > (unless no-exeext is given, of course), just as is currently done for
> > > TESTS and *_PROGRAMS. Hmm, alternatively we could also require all
> > > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
> > > this variable as well, at the cost of some specification redundance.
> > >
> >
> > > Open questions: how to produce nice results, both on stdout and in suite
> > > log files. One way is to merge all logs into a final TEST_SUITE_LOG
> > > (that way we could also reuse that variable). Another is to try to even
> > > hide the summaries of the individual suites, iff a final suite is being
> > > made.
> > >
> > These both sound sensible. Another problem is how to avoid that, in a
> > parallel make run, the summary from a testsuite gets mixed and garbled
> > with the output/summary from another testsuite, as in e.g.:
>
> Yes. We definitely want to be able to run tests from different
> testsuites in parallel. So some form of partial summary hiding is
> prudent; I hope we can achieve that without yet another make
> indirection, but I'm not optimistic.
>
> Thanks for the feedback,
> Ralf
>
Regards,
Stefano
--Boundary-00=_dbKONkyi+FSxmxb
Content-Type: application/x-shellscript;
name="parallel-tests-long-cmdline.test"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="parallel-tests-long-cmdline.test"
#! /bin/sh
# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Check that the parallel-test testsuite driver behaves well with
# huge testsuites.
parallel_tests=yes
. ./defs || Exit 1
set -e
cat >> configure.in << 'END'
AC_OUTPUT
END
# We'll have 2**13 = 8192 tests, each with a name that is 31 bytes long.
# In truth, we'll only use a single file repeated 2**13 in $(TESTS), to
# avoid ridiculously cluttering up the test directory.
tst=abcdefghijklmnopqrstuvwxyz.test # 26 + 1 + 4 = 31 bytes
echo dummy > $tst || Exit 77
# At the end of each iteration, file `t' will contain `2**$i' lines,
# where `$i' is the value tested at the beginning of the while loop.
# In the last iteration, `$i' is 13, so `t' will finally contain
# 2**13 = 8192 lines.
i=2
(echo x && echo x) > t
while test $i -le 13; do
cat t t > t2
mv -f t2 t
i=`expr $i + 1`
done
# Sanity check.
test `wc -l <t` = 8192 || Exit 99
cat > Makefile.am <<'END'
# Avoid thousands of forks -- literally!
TEST_LOG_COMPILER = :
EXTRA_DIST = $(TESTS)
END
echo 'TESTS = \' >> Makefile.am
sed -e "s/.*/$tst \\\\/" -e '$s/ *\\$//' t >> Makefile.am
# Another sanity check.
test `wc -l <Makefile.am` -gt 8000 || Exit 99
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
./configure
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
grep '^[Aa]ll 8192 tests passed' stdout
$MAKE distcheck
:
--Boundary-00=_dbKONkyi+FSxmxb--
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at 7868) by debbugs.gnu.org; 20 Jan 2011 19:33:29 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 20 14:33:29 2011 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1Pg0G8-0001Ya-JM for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 14:33:29 -0500 Received: from mailout-de.gmx.net ([213.165.64.22]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1Pg0G4-0001YK-Sj for 7868 <at> debbugs.gnu.org; Thu, 20 Jan 2011 14:33:26 -0500 Received: (qmail invoked by alias); 20 Jan 2011 19:41:14 -0000 Received: from xdsl-89-0-99-113.netcologne.de (EHLO localhost.localdomain) [89.0.99.113] by mail.gmx.net (mp062) with SMTP; 20 Jan 2011 20:41:14 +0100 X-Authenticated: #13673931 X-Provags-ID: V01U2FsdGVkX18XM1oYAfUXB05VGS/LjR9ZtPhxS5893OXTuNF8P8 vv3fwz+ZyykDe4 Received: from ralf by localhost.localdomain with local (Exim 4.69) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1Pg0Na-0005oj-9F; Thu, 20 Jan 2011 20:41:10 +0100 Date: Thu, 20 Jan 2011 20:41:10 +0100 From: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN> To: Stefano Lattarini <stefano.lattarini@HIDDEN> Subject: Re: bug#7868: splitting up test suites Message-ID: <20110120194108.GG21455@HIDDEN> References: <20110119222211.GA14878@HIDDEN> <201101201238.18228.stefano.lattarini@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201101201238.18228.stefano.lattarini@HIDDEN> Organization: Institute for Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.20 (2010-08-04) X-Y-GMX-Trusted: 0 X-Spam-Score: -2.8 (--) X-Debbugs-Envelope-To: 7868 Cc: 7868 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -2.8 (--) Hi Stefano, * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET: > On Wednesday 19 January 2011, Ralf Wildenhues wrote: > > We've finally reached the point where we have more than 1000 > > tests, $(TESTS) expands to 15k characters, and where 'make check' will > > not work at all any more on MSYS, because it cannot spawn sh any more, > > presumably in 'make check TESTS="..."'. (MSYS make doesn't export > > macros to the environment of spawned processes even without .NOEXPORT, > > presumably otherwise lots of Makefiles would be really unusable here.) > > This also clears up the spurious failure of sed a few days ago. > > > > Here's a preliminary plan for multiple testsuites per Makefile.am. > > > Hmmm... while this feature might be worth having even indipendently > from the issue at hand (but see below for small nits), I still think > that in the long run it would be nicer to transparently work around > such command-line length issues in the test driver, if possible. No, that is not possible with portable make. > Do > you think your patch "parallel-tests: avoid command-line length limit > issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some > way? No. It is fundamentally flawed. Here's why: while it may fix things at the level of recursion below that, eventually the rule commands which were changed in that patch will fail to execute, because they pose too long a command line for the shell. There simply is no way to get things to scale except by using GNU make specifics, or splitting long lists of files up manually. Well, an external list of files to be read in could work, but then we don't have make semantics easily available. In that light, I'm sorry you already wrote the patch you did, because I don't see how it can improve things significantly. I didn't try the patch or look in detail, please ping me if you still want me to do that. > > It would be nice if this worked somehow: > > > > # These are all specified by the user: > > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log .. > > TEST_EXTENSIONS = .test ... > > # these undergo $(EXEEXT) autoexpansion internally: > > suite1_log_SOURCES = all.test aclocal7.test ... > > suite2_log_SOURCES = suffix.test ... > > > What about using `TESTS' instead of `SOURCES' in these two last variables? > That would seems more natural to me, especially considering the API of the > current parallel-tests driver. First off, it would make sense to review the several discussions we had about this back when parallel-tests was implemented (more precisely, ported from Akim's GNU make-specific implementation). See e.g., http://thread.gmane.org/gmane.comp.sysutils.automake.patches/3225 http://thread.gmane.org/gmane.comp.sysutils.automake.general/8063 The idea was to treat tests as we treat compiled languages: .c files are sources for programs but get turned into .o files. Here, .test files are sources for testsuite logs, but the intermediates are the per-test .log files. And there's maybe the unification of suite*.log's to a final log on top of that. If we can handle the latter via automake's register_language machinery, then putting tests in *_SOURCES variables would be fairly natural. OTOH, you are right in that elsewhere, _SOURCES are usually meant to be final, nonderived files, which doesn't fit the picture here. So maybe suite1_log_TESTS is more appropriate after all. > OTOH, would that maybe make the implementation more difficult? Not if it's like the current implementation, no. > I think it would also be nice to generate separate check/recheck targets > for each testsuite; for example, "make check-suite1" could run all the > tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run > all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the > previous run. > > Hmm... No, wait, it would be even nicer to allow the user choose which > testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable: > > make check TEST_SUITE_LOGS='suite1.log suite2.log' > > Which makes me think that, perhaps, a variable like $(TEST_SUITES) would > be preferable: > > make check TEST_SUITES='suite1 suite2' > > but then the API should be changed to something like: > > # These are all specified by the user: > TEST_SUITES = suite1 suite2 suite3 .. > TEST_EXTENSIONS = .test ... > # these undergo $(EXEEXT) autoexpansion internally: > suite1_TESTS = all.test aclocal7.test ... > suite2_TESTS = suffix.test ... > > # These are then produced by automake: > TEST_SUITE_LOGS = $(TEST_SUITES:=.log) > TEST_SUITE_HTMLS = $(TEST_SUITES:=.html) > suite1_LOGS = $(am__helper_var:.test=.log) Unfortunately, this is error-prone, because some GNU make versions expand empty = TEST_SUITES = foo $(empty) TEST_SUITE_LOGS = $(TEST_SUITES:=.log) into `foo.log .log' rather than `foo.log '. This happens in practice if `empty' is set only under some Automake conditional, as in if COND TESTS += bar endif That is the reason the check-TESTS rule is so ugly (and recursive) in the first place. I really would like to avoid more instances of this wart; so specifying files without extension is Not Good(TM). Separate check/recheck targets are of course nice, but IMVHO optional for the first iteration. > > In the <suite>_SOURCES, $(EXEEXT) transformation should take place > > (unless no-exeext is given, of course), just as is currently done for > > TESTS and *_PROGRAMS. Hmm, alternatively we could also require all > > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of > > this variable as well, at the cost of some specification redundance. > > > > > Open questions: how to produce nice results, both on stdout and in suite > > log files. One way is to merge all logs into a final TEST_SUITE_LOG > > (that way we could also reuse that variable). Another is to try to even > > hide the summaries of the individual suites, iff a final suite is being > > made. > > > These both sound sensible. Another problem is how to avoid that, in a > parallel make run, the summary from a testsuite gets mixed and garbled > with the output/summary from another testsuite, as in e.g.: Yes. We definitely want to be able to run tests from different testsuites in parallel. So some form of partial summary hiding is prudent; I hope we can achieve that without yet another make indirection, but I'm not optimistic. Thanks for the feedback, Ralf
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.
Received: (at 7868) by debbugs.gnu.org; 20 Jan 2011 17:03:35 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 20 12:03:35 2011
Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1Pfxv4-0005ZM-Ax
for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 12:03:35 -0500
Received: from mail-bw0-f44.google.com ([209.85.214.44])
by debbugs.gnu.org with esmtp (Exim 4.69)
(envelope-from <stefano.lattarini@HIDDEN>) id 1Pfxv0-0005Z8-Sc
for 7868 <at> debbugs.gnu.org; Thu, 20 Jan 2011 12:03:32 -0500
Received: by bwz12 with SMTP id 12so766409bwz.3
for <7868 <at> debbugs.gnu.org>; Thu, 20 Jan 2011 09:11:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
h=domainkey-signature:from:to:subject:date:user-agent:cc:references
:in-reply-to:mime-version:content-type:message-id;
bh=CcZ4n+RImHorwy6hmxgSQFwQVjYBlhJrzhB2vnp95Vo=;
b=fJasTJ41YR0nZiTN2C8W8wbHMa506rNYo/BsbrxPSS3WD5ST1RkgiMBlEfddCbo/7K
qcS9aLsdLCAOgPaO7HSWLX4WVJ9UidmarLvFLtchbxmPSytDNqG1CfMCQg/8iOMG7wOP
knseJ62RXI8xyJgITrYIpuY931YxsgcvGOlJo=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
h=from:to:subject:date:user-agent:cc:references:in-reply-to
:mime-version:content-type:message-id;
b=dG4r06gckHpaMw5B7P6PtdFlc6WSWzwf66zputWomD8kVHQYskuxLb7WV92yOV23re
ciNnXItf1bdvkhTqCfU36SDSVrH3SVKbM5DQpHT6npxaJQyA3paIYVM9FTdPLKR82GCw
c0/cFv/79fW/LkSSNiirvIAWcfU6nW6lnVG/U=
Received: by 10.204.85.11 with SMTP id m11mr2231148bkl.115.1295543477850;
Thu, 20 Jan 2011 09:11:17 -0800 (PST)
Received: from bigio.localnet
(host162-93-dynamic.10-79-r.retail.telecomitalia.it [79.10.93.162])
by mx.google.com with ESMTPS id b6sm4046378bkb.10.2011.01.20.09.11.15
(version=TLSv1/SSLv3 cipher=RC4-MD5);
Thu, 20 Jan 2011 09:11:16 -0800 (PST)
From: Stefano Lattarini <stefano.lattarini@HIDDEN>
To: automake-patches@HIDDEN
Subject: [RFC] parallel-tests: avoid command-line length limit issue (was: Re:
bug#7868: splitting up test suites)
Date: Thu, 20 Jan 2011 18:11:00 +0100
User-Agent: KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; )
References: <20110119222211.GA14878@HIDDEN>
<201101201238.18228.stefano.lattarini@HIDDEN>
In-Reply-To: <201101201238.18228.stefano.lattarini@HIDDEN>
MIME-Version: 1.0
Content-Type: Multipart/Mixed;
boundary="Boundary-00=_lyGONBFucI6mNRv"
Message-Id: <201101201811.01721.stefano.lattarini@HIDDEN>
X-Spam-Score: -3.9 (---)
X-Debbugs-Envelope-To: 7868
Cc: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>, 7868 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.11
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Sender: debbugs-submit-bounces <at> debbugs.gnu.org
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
X-Spam-Score: -3.9 (---)
--Boundary-00=_lyGONBFucI6mNRv
Content-Type: Text/Plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
On Thursday 20 January 2011, Stefano Lattarini wrote:
> Hello Ralf.
>
> On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > The testsuite is too large for MSYS.
> >
> Ouch.
>
> > We've finally reached the point where we have more than 1000
> > tests, $(TESTS) expands to 15k characters, and where 'make check' will
> > not work at all any more on MSYS, because it cannot spawn sh any more,
> > presumably in 'make check TESTS="..."'. (MSYS make doesn't export
> > macros to the environment of spawned processes even without .NOEXPORT,
> > presumably otherwise lots of Makefiles would be really unusable here.)
> > This also clears up the spurious failure of sed a few days ago.
> >
> > Here's a preliminary plan for multiple testsuites per Makefile.am.
> >
> Hmmm... while this feature might be worth having even indipendently
> from the issue at hand (but see below for small nits), I still think
> that in the long run it would be nicer to transparently work around
> such command-line length issues in the test driver, if possible. Do
> you think your patch "parallel-tests: avoid command-line length limit
> issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some
> way?
>
OK, I've done my homework and come up with the attached patch.
It's not yet very well polished, and certainly needs more testsuite
exposure [1] on various systems before being "ok to apply", but it
seems quite promising to me.
[1] I'm posting it now anyway because I'm out of time for today.
Sorry.
Also, I'd appreciate if anyone could test it on MSYS (and Cygwin?),
since I don't have access to those systems.
Thanks,
Stefano
--Boundary-00=_lyGONBFucI6mNRv
Content-Type: text/x-patch;
charset="us-ascii";
name="0001-parallel-tests-avoid-command-line-length-limit-issue.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline;
filename="0001-parallel-tests-avoid-command-line-length-limit-issue.patch"
=46rom f6f4dc5d2e6e3d174f696409fa5c07e207d377a4 Mon Sep 17 00:00:00 2001
=46rom: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>
Date: Tue, 7 Sep 2010 04:38:08 +0200
Subject: [PATCH] parallel-tests: avoid command-line length limit issue.
* automake.in (handle_tests): New argument $makefile, new
substitution %MAKEFILE%.
(generate_makefile): Adjust.
* lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Use a
temporary makefile to sanitize TEST_LOGS value passed to the
recursive $(MAKE) invocation, to avoid exceeding the command
line limit on w32 (MSYS). Extend comments.
* tests/parallel-tests-linewrap.test: New test.
* tests/parallel-tests-cleanup.test: Likewise.
* tests/parallel-tests-gnumakefile.test: Likewise.
* tests/parallel-tests-long-cmdline.test: Likewise.
* tests/Makefile.am (TESTS): Updated.
* NEWS: Update.
Report by Bob Friesenhahn.
=2D--
ChangeLog | 19 +++++++
NEWS | 3 +
automake.in | 11 +++-
lib/Automake/tests/Makefile.in | 21 ++++++--
lib/am/check.am | 35 +++++++++++--
tests/Makefile.am | 3 +
tests/Makefile.in | 24 +++++++--
tests/parallel-tests-cleanup.test | 91 +++++++++++++++++++++++++++++=
++++
tests/parallel-tests-gnumakefile.test | 49 ++++++++++++++++++
tests/parallel-tests-linewrap.test | 63 +++++++++++++++++++++++
10 files changed, 301 insertions(+), 18 deletions(-)
create mode 100755 tests/parallel-tests-cleanup.test
create mode 100755 tests/parallel-tests-gnumakefile.test
create mode 100755 tests/parallel-tests-linewrap.test
diff --git a/ChangeLog b/ChangeLog
index 31ff009..4a0e7e1 100644
=2D-- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@
+2011-01-20 Stefano Lattarini <stefano.lattarini@HIDDEN>
+ Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>
+
+ parallel-tests: avoid command-line length limit issue.
+ * automake.in (handle_tests): New argument $makefile, new
+ substitution %MAKEFILE%.
+ (generate_makefile): Adjust.
+ * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Use a
+ temporary makefile to sanitize TEST_LOGS value passed to the
+ recursive $(MAKE) invocation, to avoid exceeding the command
+ line limit on w32 (MSYS). Extend comments.
+ * tests/parallel-tests-linewrap.test: New test.
+ * tests/parallel-tests-cleanup.test: Likewise.
+ * tests/parallel-tests-gnumakefile.test: Likewise.
+ * tests/parallel-tests-long-cmdline.test: Likewise.
+ * tests/Makefile.am (TESTS): Updated.
+ * NEWS: Update.
+ Report by Bob Friesenhahn.
+
2011-01-19 Stefano Lattarini <stefano.lattarini@HIDDEN>
Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>
=20
diff --git a/NEWS b/NEWS
index b5cb6e9..c5f28b5 100644
=2D-- a/NEWS
+++ b/NEWS
@@ -22,6 +22,9 @@ Bugs fixed in 1.11.0a:
- The AM_COND_IF macro also works if the shell expression for the condit=
ional
is no longer valid for the condition.
=20
+ - The `parallel-tests' driver works around a problem with command-line
+ length limits with `make check' on w32 (MSYS).
+
* Long standing bugs:
=20
- On Darwin 9, `pythondir' and `pyexecdir' pointed below `/Library/Pytho=
n'
diff --git a/automake.in b/automake.in
index d56fbf7..bd9f453 100755
=2D-- a/automake.in
+++ b/automake.in
@@ -4919,9 +4919,13 @@ sub handle_tests_dejagnu
}
=20
=20
+# handle_tests ($MAKEFILE)
+# ------------------------
# Handle TESTS variable and other checks.
=2Dsub handle_tests
+sub handle_tests ($)
{
+ my ($makefile) =3D @_;
+
if (option 'dejagnu')
{
&handle_tests_dejagnu;
@@ -4940,7 +4944,8 @@ sub handle_tests
push (@check_tests, 'check-TESTS');
$output_rules .=3D &file_contents ('check', new Automake::Location,
COLOR =3D> !! option 'color-tests',
=2D PARALLEL_TESTS =3D> !! option 'parallel-tests');
+ PARALLEL_TESTS =3D> !! option 'parallel-tests',
+ MAKEFILE =3D> basename $makefile);
=20
# Tests that are known programs should have $(EXEEXT) appended.
# For matching purposes, we need to adjust XFAIL_TESTS as well.
@@ -8220,7 +8225,7 @@ sub generate_makefile ($$)
handle_tags;
handle_minor_options;
# Must come after handle_programs so that %known_programs is up-to-date.
=2D handle_tests;
+ handle_tests ($makefile);
=20
# This must come after most other rules.
handle_dist;
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index b4940db..d3846aa 100644
=2D-- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -403,11 +403,22 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
check-TESTS:
@list=3D'$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
=2D @list=3D'$(TEST_LOGS)'; \
=2D list=3D`for f in $$list; do \
=2D test .log =3D $$f || echo $$f; \
=2D done | tr '\012\015' ' '`; \
=2D $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS=3D"$$list"
+ @am__tmpdir=3Dam-tmp$$$$; \
+ am__mkfile=3D$$am__tmpdir/Makefile; \
+ am__trap=3D'rm -rf $$am__tmpdir; (exit $$st); exit $$st'; \
+ trap "st=3D129; $$am__trap" 1; trap "st=3D130; $$am__trap" 2; \
+ trap "st=3D141; $$am__trap" 13; trap "st=3D143; $$am__trap" 15; \
+ st=3D0; \
+ mkdir $$am__tmpdir \
+ && { echo "TEST_LOGS =3D \\"; \
+ list=3D'$(TEST_LOGS)'; for f in $$list; do \
+ test .log =3D $$f || echo "$$f \\"; \
+ done; \
+ } | sed '$$s/ *\\$$//' > $$am__mkfile \
+ && sed '/^TEST_LOGS =3D/d' Makefile >> $$am__mkfile \
+ && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) \
+ || st=3D$$?; \
+ rm -rf $$am__tmpdir; exit $$st
=20
.log.html:
@list=3D'$(RST2HTML) $$RST2HTML rst2html rst2html.py'; \
diff --git a/lib/am/check.am b/lib/am/check.am
index 5728081..837f1b4 100644
=2D-- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -236,11 +236,36 @@ check-TESTS:
## cannot use `$?' to compute the set of lazily rerun tests, lest
## we rely on .PHONY to work portably.
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
=2D @list=3D'$(TEST_LOGS)'; \
=2D list=3D`for f in $$list; do \
=2D test .log =3D $$f || echo $$f; \
=2D done | tr '\012\015' ' '`; \
=2D $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS=3D"$$list"
+## We'll need a temporary file (see comments below for why), and the safer
+## way to create it portably is to use a temporary directory (as the mkdir
+## utility should be truly atomic everywhere).
+ @am__tmpdir=3Dam-tmp$$$$; \
+ am__mkfile=3D$$am__tmpdir/Makefile; \
+ am__trap=3D'rm -rf $$am__tmpdir; (exit $$st); exit $$st'; \
+ trap "st=3D129; $$am__trap" 1; trap "st=3D130; $$am__trap" 2; \
+ trap "st=3D141; $$am__trap" 13; trap "st=3D143; $$am__trap" 15; \
+ st=3D0; \
+ mkdir $$am__tmpdir \
+## Yes, this is hacky, but we cannot simply override the $(TEST_LOGS)
+## definition by appending to Makefile, since at that point it's original
+## value has been already used in a dependency declaration, i.e.
+## $(TEST_SUITE_LOG): $(TEST_LOGS)
+## (see above in this same *.am fragment).
+ && { echo "TEST_LOGS =3D \\"; \
+ list=3D'$(TEST_LOGS)'; for f in $$list; do \
+## A bug in GNU make 3.80 can lead to bare `.log' occurrences.
+## Strip them out.
+ test .log =3D $$f || echo "$$f \\"; \
+ done; \
+ } | sed '$$s/ *\\$$//' > $$am__mkfile \
+ && sed '/^TEST_LOGS =3D/d' %MAKEFILE% >> $$am__mkfile \
+## It would have been nice to be able to use something like:
+## $(POST_PROCESSING_COMMAND) Makefile.in | $(MAKE) -f -
+## instead of a temporary file here, but unfortunately that doesn't
+## work with at least Solaris 10 dmake.
+ && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) \
+ || st=3D$$?; \
+ rm -rf $$am__tmpdir; exit $$st
=20
AM_RECURSIVE_TARGETS +=3D check
=20
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 713dd92..f263764 100644
=2D-- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -567,6 +567,9 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-cleanup.test \
+parallel-tests-gnumakefile.test \
+parallel-tests-linewrap.test \
parallel-tests-unreadable-log.test \
parse.test \
percent.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 45adb04..7491f05 100644
=2D-- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -834,6 +834,9 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-cleanup.test \
+parallel-tests-gnumakefile.test \
+parallel-tests-linewrap.test \
parallel-tests-unreadable-log.test \
parse.test \
percent.test \
@@ -1222,11 +1225,22 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
check-TESTS:
@list=3D'$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
=2D @list=3D'$(TEST_LOGS)'; \
=2D list=3D`for f in $$list; do \
=2D test .log =3D $$f || echo $$f; \
=2D done | tr '\012\015' ' '`; \
=2D $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS=3D"$$list"
+ @am__tmpdir=3Dam-tmp$$$$; \
+ am__mkfile=3D$$am__tmpdir/Makefile; \
+ am__trap=3D'rm -rf $$am__tmpdir; (exit $$st); exit $$st'; \
+ trap "st=3D129; $$am__trap" 1; trap "st=3D130; $$am__trap" 2; \
+ trap "st=3D141; $$am__trap" 13; trap "st=3D143; $$am__trap" 15; \
+ st=3D0; \
+ mkdir $$am__tmpdir \
+ && { echo "TEST_LOGS =3D \\"; \
+ list=3D'$(TEST_LOGS)'; for f in $$list; do \
+ test .log =3D $$f || echo "$$f \\"; \
+ done; \
+ } | sed '$$s/ *\\$$//' > $$am__mkfile \
+ && sed '/^TEST_LOGS =3D/d' Makefile >> $$am__mkfile \
+ && $(MAKE) -f $$am__mkfile $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) \
+ || st=3D$$?; \
+ rm -rf $$am__tmpdir; exit $$st
=20
.log.html:
@list=3D'$(RST2HTML) $$RST2HTML rst2html rst2html.py'; \
diff --git a/tests/parallel-tests-cleanup.test b/tests/parallel-tests-clean=
up.test
new file mode 100755
index 0000000..26e9d5c
=2D-- /dev/null
+++ b/tests/parallel-tests-cleanup.test
@@ -0,0 +1,91 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the temporary files/directories used by the Makefile
+# post-processing code in the parallel-tests testsuite driver are
+# duly cleaned up on success, on failure, and when well-known signals
+# are received.
+
+parallel_tests=3Dyes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS =3D .sh
+TESTS =3D foo.sh
+SH_LOG_COMPILER =3D sh
+EXTRA_DIST =3D foo.sh
+END
+
+check_filelist ()
+{
+ expect_filelist=3D$1
+ got_filelist=3D`ls`
+ if test x"$expect_filelist" !=3D x"$got_filelist"; then
+ # Display the differences in a more user-friendly way.
+ # Useful for debugging and failure analysis.
+ set +x # Don't be overly verbose.
+ for f in $expect_filelist; do echo $f; done > exp
+ for f in $got_filelist; do echo $f; done > got
+ set -x
+ diff exp got || :
+ return 1
+ else
+ return 0
+ fi
+}
+
+$ACLOCAL
+$AUTOMAKE -a
+$AUTOCONF
+
+./configure
+
+aborted=3D`(ls && echo foo.sh ) | sort`
+completed=3D`(echo "$aborted" && echo test-suite.log && echo foo.log) | so=
rt`
+
+echo 'exit 0' > foo.sh
+$MAKE check
+check_filelist "$completed"
+$MAKE clean
+
+echo 'exit 1' > foo.sh
+$MAKE check && Exit 1
+check_filelist "$completed"
+$MAKE clean
+
+echo 'sleep 10' > foo.sh
+
+# Yes, all the "sleeps" below sucks, but here is better to play "dumb
+# and safer" than having stray I/O or leaving zombies around.
+for signum in 1 2 13 15; do=20
+ failed=3Dfalse
+ $MAKE check &
+ kill -$signum $!
+ check_filelist "$aborted" || failed=3D:
+ sleep 12 # Wait for all the children to complete.
+ $failed && Exit 1
+ $MAKE clean
+done
+
+$MAKE distcheck
+
+:
diff --git a/tests/parallel-tests-gnumakefile.test b/tests/parallel-tests-g=
numakefile.test
new file mode 100755
index 0000000..60d6fd7
=2D-- /dev/null
+++ b/tests/parallel-tests-gnumakefile.test
@@ -0,0 +1,49 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that recursive make calls triggered by parallel-tests testsuite
+# driver work also if the makefile is named GNUmakefile.
+
+required=3DGNUmake
+parallel_tests=3Dyes
+. ./defs || Exit 1
+set -e
+
+cat configure.in - > t << 'END'
+AC_OUTPUT
+END
+sed '/^AC_CONFIG_FILES/s|Makefile|GNUmakefile|' t > configure.in
+rm -f t
+
+cat > GNUmakefile.am <<'END'
+TESTS =3D foo.test
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+exit 0
+END
+chmod +x foo.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check
+
+:
diff --git a/tests/parallel-tests-linewrap.test b/tests/parallel-tests-line=
wrap.test
new file mode 100755
index 0000000..b3969f0
=2D-- /dev/null
+++ b/tests/parallel-tests-linewrap.test
@@ -0,0 +1,63 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the definition of TEST_LOGS is not broken on multiple lines,
+# even if the list of tests, the list of test extensions, the test names
+# and the test extensions are all very long.
+# We need to be sure of this in order for the Makefile post-processing
+# code in the parallel-tests testsuite driver to work.
+
+parallel_tests=3Dyes
+. ./defs || Exit 1
+
+set -e
+
+# Avoid forks if possible
+i=3D1
+if (i=3D$(($i+1)) && test $i -eq 2); then
+ incr_i() { i=3D$(($i+1)); }
+else
+ incr_i() { i=3D`expr $i + 1`; }
+fi
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS =3D
+TESTS =3D
+END
+
+lc_ext=3Da_very_very_very_long_test_extension_indeed_it_is_slightly_longer=
_than_81_characters
+uc_ext=3DA_VERY_VERY_VERY_LONG_TEST_EXTENSION_INDEED_IT_IS_SLIGHTLY_LONGER=
_THAN_81_CHARACTERS
+
+while test $i -lt 200; do
+ echo TEST_EXTENSIONS +=3D .${lc_ext}_${i}
+ echo ${uc_ext}_${i}_LOG_COMPILER =3D sh
+ echo TESTS +=3D wow-this-definitely-is-a-very-long-name-for-a-dummy-test=
case.${lc_ext}_${i}
+ incr_i
+done >> Makefile.am
+
+$ACLOCAL
+$AUTOMAKE -a
+
+# For debugging.
+$FGREP 'TEST_LOGS' Makefile.in
+
+grep '^TEST_LOGS *=3D.*\\$' Makefile.in && Exit 1
+
+:
=2D-=20
1.7.2.3
--Boundary-00=_lyGONBFucI6mNRv--
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at 7868) by debbugs.gnu.org; 20 Jan 2011 11:31:12 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 20 06:31:12 2011 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1PfsjP-0005Zv-Ey for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:31:11 -0500 Received: from mail-bw0-f44.google.com ([209.85.214.44]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <stefano.lattarini@HIDDEN>) id 1PfsjM-0005Zg-2O for 7868 <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:31:08 -0500 Received: by bwz12 with SMTP id 12so437088bwz.3 for <7868 <at> debbugs.gnu.org>; Thu, 20 Jan 2011 03:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:subject:date:user-agent:cc:references :in-reply-to:mime-version:content-type:content-transfer-encoding :message-id; bh=HFRStOffXkWXoZLjKXiTQN6shJAyp32+uH54f0P3cBk=; b=l7TrvKzrXffJTV6GRCtyfI0k9aeE/1TqM0W38BjFLiOkZAn0JkgeDZRY3C3yTN4eDA R4Kjfzo5JaLv8dg3LWp+lldoJrKGxzXe6WvJI43vtE52k3Fq0Sq9pPGwi+GXvPcTJIyT q4ya8/Lxc4cHSWBftwQryjkRszQFYw9odpCMY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:cc:references:in-reply-to :mime-version:content-type:content-transfer-encoding:message-id; b=XE0NCuihNvob4EY1sciFs4GBVeIAjXb7vhPZZEi7+atyLg8Z2MiMcr/TGjc6fBrSCe ijoLp3ftfxBKqnc0+Od9swACV8Mue6QQy0xKD2jkYyzTopQTK4CH507hIfvkj5OyOoVx tv5oVh2sKCSo5qOp1SelnIGPhcBiEIiyFIJEg= Received: by 10.204.64.74 with SMTP id d10mr1798062bki.7.1295523538441; Thu, 20 Jan 2011 03:38:58 -0800 (PST) Received: from bigio.localnet (host162-93-dynamic.10-79-r.retail.telecomitalia.it [79.10.93.162]) by mx.google.com with ESMTPS id f20sm3848794bkf.4.2011.01.20.03.38.56 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 20 Jan 2011 03:38:56 -0800 (PST) From: Stefano Lattarini <stefano.lattarini@HIDDEN> To: bug-automake@HIDDEN Subject: Re: bug#7868: splitting up test suites Date: Thu, 20 Jan 2011 12:38:17 +0100 User-Agent: KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) References: <20110119222211.GA14878@HIDDEN> In-Reply-To: <20110119222211.GA14878@HIDDEN> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201101201238.18228.stefano.lattarini@HIDDEN> X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: 7868 Cc: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>, 7868 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -3.8 (---) Hello Ralf. On Wednesday 19 January 2011, Ralf Wildenhues wrote: > The testsuite is too large for MSYS. > Ouch. > We've finally reached the point where we have more than 1000 > tests, $(TESTS) expands to 15k characters, and where 'make check' will > not work at all any more on MSYS, because it cannot spawn sh any more, > presumably in 'make check TESTS="..."'. (MSYS make doesn't export > macros to the environment of spawned processes even without .NOEXPORT, > presumably otherwise lots of Makefiles would be really unusable here.) > This also clears up the spurious failure of sed a few days ago. > > Here's a preliminary plan for multiple testsuites per Makefile.am. > Hmmm... while this feature might be worth having even indipendently from the issue at hand (but see below for small nits), I still think that in the long run it would be nicer to transparently work around such command-line length issues in the test driver, if possible. Do you think your patch "parallel-tests: avoid command-line length limit issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some way? > It would be nice if this worked somehow: > > # These are all specified by the user: > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log .. > TEST_EXTENSIONS = .test ... > # these undergo $(EXEEXT) autoexpansion internally: > suite1_log_SOURCES = all.test aclocal7.test ... > suite2_log_SOURCES = suffix.test ... > What about using `TESTS' instead of `SOURCES' in these two last variables? That would seems more natural to me, especially considering the API of the current parallel-tests driver. OTOH, would that maybe make the implementation more difficult? > # These are then produced by automake: > TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html) > suite1_log_LOGS = $(am__helper_var:.test=.log) > > mostlyclean-generic: > -test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS) > -test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS) > I think it would also be nice to generate separate check/recheck targets for each testsuite; for example, "make check-suite1" could run all the tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the previous run. Hmm... No, wait, it would be even nicer to allow the user choose which testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable: make check TEST_SUITE_LOGS='suite1.log suite2.log' Which makes me think that, perhaps, a variable like $(TEST_SUITES) would be preferable: make check TEST_SUITES='suite1 suite2' but then the API should be changed to something like: # These are all specified by the user: TEST_SUITES = suite1 suite2 suite3 .. TEST_EXTENSIONS = .test ... # these undergo $(EXEEXT) autoexpansion internally: suite1_TESTS = all.test aclocal7.test ... suite2_TESTS = suffix.test ... # These are then produced by automake: TEST_SUITE_LOGS = $(TEST_SUITES:=.log) TEST_SUITE_HTMLS = $(TEST_SUITES:=.html) suite1_LOGS = $(am__helper_var:.test=.log) > In the <suite>_SOURCES, $(EXEEXT) transformation should take place > (unless no-exeext is given, of course), just as is currently done for > TESTS and *_PROGRAMS. Hmm, alternatively we could also require all > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of > this variable as well, at the cost of some specification redundance. > > Open questions: how to produce nice results, both on stdout and in suite > log files. One way is to merge all logs into a final TEST_SUITE_LOG > (that way we could also reuse that variable). Another is to try to even > hide the summaries of the individual suites, iff a final suite is being > made. > These both sound sensible. Another problem is how to avoid that, in a parallel make run, the summary from a testsuite gets mixed and garbled with the output/summary from another testsuite, as in e.g.: suite1_log_SOURCES = all.test aclocal7.test suite2_log_SOURCES = suffix.test yacc-dist-nobuild.test $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: aclocal7.test ==================================================== PASS: suffix.test All 2 tests behaved as expected (1 expected failure) FAIL: yacc-dist-nobuild.test ===================================== 1 of 2 tests failed ==================================================== See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== (yuck!), where we'd want at least: $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: suffix.test PASS: aclocal7.test ==================================================== All 2 tests behaved as expected (1 expected failure) ==================================================== FAIL: yacc-dist-nobuild.test ===================================== 1 of 2 tests failed See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== or better again: $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: suffix.test PASS: aclocal7.test FAIL: yacc-dist-nobuild.test ========================================================== suite1: All tests behaved as expected (1 expected failure) ========================================================== ===================================== suite2: 1 of 2 tests failed See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== > Internally, it would be nice if the register_language could be exploited > (but this is not a requirement for this, as some features with tests are > distinctly different from other languages). > > Cheers, > Ralf > Thanks for tackling this! Regards, Stefano
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at submit) by debbugs.gnu.org; 20 Jan 2011 11:31:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 20 06:31:17 2011 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1PfsjV-0005a9-C4 for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:31:17 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <stefano.lattarini@HIDDEN>) id 1PfsjT-0005Zu-Rr for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:31:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <stefano.lattarini@HIDDEN>) id 1Pfsr3-0008B3-S1 for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:39:07 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, T_DKIM_INVALID, T_TO_NO_BRKTS_FREEMAIL autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:45088) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <stefano.lattarini@HIDDEN>) id 1Pfsr3-0008Az-P9 for submit <at> debbugs.gnu.org; Thu, 20 Jan 2011 06:39:05 -0500 Received: from [140.186.70.92] (port=50870 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Pfsr2-0006Rq-CQ for bug-automake@HIDDEN; Thu, 20 Jan 2011 06:39:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <stefano.lattarini@HIDDEN>) id 1Pfsr1-0008AU-2f for bug-automake@HIDDEN; Thu, 20 Jan 2011 06:39:04 -0500 Received: from mail-bw0-f41.google.com ([209.85.214.41]:61767) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <stefano.lattarini@HIDDEN>) id 1Pfsr0-0008A7-QG for bug-automake@HIDDEN; Thu, 20 Jan 2011 06:39:03 -0500 Received: by bwz16 with SMTP id 16so402368bwz.0 for <bug-automake@HIDDEN>; Thu, 20 Jan 2011 03:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:subject:date:user-agent:cc:references :in-reply-to:mime-version:content-type:content-transfer-encoding :message-id; bh=HFRStOffXkWXoZLjKXiTQN6shJAyp32+uH54f0P3cBk=; b=l7TrvKzrXffJTV6GRCtyfI0k9aeE/1TqM0W38BjFLiOkZAn0JkgeDZRY3C3yTN4eDA R4Kjfzo5JaLv8dg3LWp+lldoJrKGxzXe6WvJI43vtE52k3Fq0Sq9pPGwi+GXvPcTJIyT q4ya8/Lxc4cHSWBftwQryjkRszQFYw9odpCMY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:cc:references:in-reply-to :mime-version:content-type:content-transfer-encoding:message-id; b=XE0NCuihNvob4EY1sciFs4GBVeIAjXb7vhPZZEi7+atyLg8Z2MiMcr/TGjc6fBrSCe ijoLp3ftfxBKqnc0+Od9swACV8Mue6QQy0xKD2jkYyzTopQTK4CH507hIfvkj5OyOoVx tv5oVh2sKCSo5qOp1SelnIGPhcBiEIiyFIJEg= Received: by 10.204.64.74 with SMTP id d10mr1798062bki.7.1295523538441; Thu, 20 Jan 2011 03:38:58 -0800 (PST) Received: from bigio.localnet (host162-93-dynamic.10-79-r.retail.telecomitalia.it [79.10.93.162]) by mx.google.com with ESMTPS id f20sm3848794bkf.4.2011.01.20.03.38.56 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 20 Jan 2011 03:38:56 -0800 (PST) From: Stefano Lattarini <stefano.lattarini@HIDDEN> To: bug-automake@HIDDEN Subject: Re: bug#7868: splitting up test suites Date: Thu, 20 Jan 2011 12:38:17 +0100 User-Agent: KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) References: <20110119222211.GA14878@HIDDEN> In-Reply-To: <20110119222211.GA14878@HIDDEN> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201101201238.18228.stefano.lattarini@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -4.9 (----) X-Debbugs-Envelope-To: submit Cc: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>, 7868 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -5.0 (-----) Hello Ralf. On Wednesday 19 January 2011, Ralf Wildenhues wrote: > The testsuite is too large for MSYS. > Ouch. > We've finally reached the point where we have more than 1000 > tests, $(TESTS) expands to 15k characters, and where 'make check' will > not work at all any more on MSYS, because it cannot spawn sh any more, > presumably in 'make check TESTS="..."'. (MSYS make doesn't export > macros to the environment of spawned processes even without .NOEXPORT, > presumably otherwise lots of Makefiles would be really unusable here.) > This also clears up the spurious failure of sed a few days ago. > > Here's a preliminary plan for multiple testsuites per Makefile.am. > Hmmm... while this feature might be worth having even indipendently from the issue at hand (but see below for small nits), I still think that in the long run it would be nicer to transparently work around such command-line length issues in the test driver, if possible. Do you think your patch "parallel-tests: avoid command-line length limit issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some way? > It would be nice if this worked somehow: > > # These are all specified by the user: > TEST_SUITE_LOGS = suite1.log suite2.log suite3.log .. > TEST_EXTENSIONS = .test ... > # these undergo $(EXEEXT) autoexpansion internally: > suite1_log_SOURCES = all.test aclocal7.test ... > suite2_log_SOURCES = suffix.test ... > What about using `TESTS' instead of `SOURCES' in these two last variables? That would seems more natural to me, especially considering the API of the current parallel-tests driver. OTOH, would that maybe make the implementation more difficult? > # These are then produced by automake: > TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html) > suite1_log_LOGS = $(am__helper_var:.test=.log) > > mostlyclean-generic: > -test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS) > -test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS) > I think it would also be nice to generate separate check/recheck targets for each testsuite; for example, "make check-suite1" could run all the tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the previous run. Hmm... No, wait, it would be even nicer to allow the user choose which testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable: make check TEST_SUITE_LOGS='suite1.log suite2.log' Which makes me think that, perhaps, a variable like $(TEST_SUITES) would be preferable: make check TEST_SUITES='suite1 suite2' but then the API should be changed to something like: # These are all specified by the user: TEST_SUITES = suite1 suite2 suite3 .. TEST_EXTENSIONS = .test ... # these undergo $(EXEEXT) autoexpansion internally: suite1_TESTS = all.test aclocal7.test ... suite2_TESTS = suffix.test ... # These are then produced by automake: TEST_SUITE_LOGS = $(TEST_SUITES:=.log) TEST_SUITE_HTMLS = $(TEST_SUITES:=.html) suite1_LOGS = $(am__helper_var:.test=.log) > In the <suite>_SOURCES, $(EXEEXT) transformation should take place > (unless no-exeext is given, of course), just as is currently done for > TESTS and *_PROGRAMS. Hmm, alternatively we could also require all > <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of > this variable as well, at the cost of some specification redundance. > > Open questions: how to produce nice results, both on stdout and in suite > log files. One way is to merge all logs into a final TEST_SUITE_LOG > (that way we could also reuse that variable). Another is to try to even > hide the summaries of the individual suites, iff a final suite is being > made. > These both sound sensible. Another problem is how to avoid that, in a parallel make run, the summary from a testsuite gets mixed and garbled with the output/summary from another testsuite, as in e.g.: suite1_log_SOURCES = all.test aclocal7.test suite2_log_SOURCES = suffix.test yacc-dist-nobuild.test $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: aclocal7.test ==================================================== PASS: suffix.test All 2 tests behaved as expected (1 expected failure) FAIL: yacc-dist-nobuild.test ===================================== 1 of 2 tests failed ==================================================== See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== (yuck!), where we'd want at least: $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: suffix.test PASS: aclocal7.test ==================================================== All 2 tests behaved as expected (1 expected failure) ==================================================== FAIL: yacc-dist-nobuild.test ===================================== 1 of 2 tests failed See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== or better again: $ make check TEST_SUITE_LOGS='suite1.log suite2.log' XFAIL: all.test PASS: suffix.test PASS: aclocal7.test FAIL: yacc-dist-nobuild.test ========================================================== suite1: All tests behaved as expected (1 expected failure) ========================================================== ===================================== suite2: 1 of 2 tests failed See tests/suite2.log Please report to bug-automake@HIDDEN ===================================== > Internally, it would be nice if the register_language could be exploited > (but this is not a requirement for this, as some features with tests are > distinctly different from other languages). > > Cheers, > Ralf > Thanks for tackling this! Regards, Stefano
owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.Received: (at submit) by debbugs.gnu.org; 19 Jan 2011 22:18:39 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jan 19 17:18:38 2011 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1PfgMQ-0004eI-43 for submit <at> debbugs.gnu.org; Wed, 19 Jan 2011 17:18:38 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgMN-0004e1-Vj for submit <at> debbugs.gnu.org; Wed, 19 Jan 2011 17:18:37 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgTw-0008HI-Ok for submit <at> debbugs.gnu.org; Wed, 19 Jan 2011 17:26:25 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, T_TO_NO_BRKTS_FREEMAIL autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40782) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgTr-0007lg-LB for submit <at> debbugs.gnu.org; Wed, 19 Jan 2011 17:26:24 -0500 Received: from [140.186.70.92] (port=48668 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PfgR0-0006BY-Fi for bug-automake@HIDDEN; Wed, 19 Jan 2011 17:23:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgQ4-0007Ec-55 for bug-automake@HIDDEN; Wed, 19 Jan 2011 17:22:25 -0500 Received: from mailout-de.gmx.net ([213.165.64.22]:49302) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgQ3-0007Dl-PS for bug-automake@HIDDEN; Wed, 19 Jan 2011 17:22:24 -0500 Received: (qmail invoked by alias); 19 Jan 2011 22:22:20 -0000 Received: from xdsl-89-0-138-36.netcologne.de (EHLO localhost.localdomain) [89.0.138.36] by mail.gmx.net (mp018) with SMTP; 19 Jan 2011 23:22:20 +0100 X-Authenticated: #13673931 X-Provags-ID: V01U2FsdGVkX19eEu4rq39ezgtj40mLm+yBCb/vceHx/4JFJ+y2HG 1pGlG5i86Pgda3 Received: from ralf by localhost.localdomain with local (Exim 4.69) (envelope-from <Ralf.Wildenhues@HIDDEN>) id 1PfgPz-0003sX-6L for bug-automake@HIDDEN; Wed, 19 Jan 2011 23:22:19 +0100 Date: Wed, 19 Jan 2011 23:22:19 +0100 From: Ralf Wildenhues <Ralf.Wildenhues@HIDDEN> To: bug-automake@HIDDEN Subject: splitting up test suites Message-ID: <20110119222211.GA14878@HIDDEN> Mail-Followup-To: bug-automake@HIDDEN MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: Institute for Numerical Simulation, University of Bonn User-Agent: Mutt/1.5.20 (2010-08-04) X-Y-GMX-Trusted: 0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -4.4 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <http://debbugs.gnu.org/pipermail/debbugs-submit> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Sender: debbugs-submit-bounces <at> debbugs.gnu.org Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org X-Spam-Score: -4.5 (----) The testsuite is too large for MSYS. We've finally reached the point where we have more than 1000 tests, $(TESTS) expands to 15k characters, and where 'make check' will not work at all any more on MSYS, because it cannot spawn sh any more, presumably in 'make check TESTS="..."'. (MSYS make doesn't export macros to the environment of spawned processes even without .NOEXPORT, presumably otherwise lots of Makefiles would be really unusable here.) This also clears up the spurious failure of sed a few days ago. Here's a preliminary plan for multiple testsuites per Makefile.am. It would be nice if this worked somehow: # These are all specified by the user: TEST_SUITE_LOGS = suite1.log suite2.log suite3.log .. TEST_EXTENSIONS = .test ... # these undergo $(EXEEXT) autoexpansion internally: suite1_log_SOURCES = all.test aclocal7.test ... suite2_log_SOURCES = suffix.test ... # These are then produced by automake: TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html) suite1_log_LOGS = $(am__helper_var:.test=.log) mostlyclean-generic: -test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS) -test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS) In the <suite>_SOURCES, $(EXEEXT) transformation should take place (unless no-exeext is given, of course), just as is currently done for TESTS and *_PROGRAMS. Hmm, alternatively we could also require all <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of this variable as well, at the cost of some specification redundance. Open questions: how to produce nice results, both on stdout and in suite log files. One way is to merge all logs into a final TEST_SUITE_LOG (that way we could also reuse that variable). Another is to try to even hide the summaries of the individual suites, iff a final suite is being made. Internally, it would be nice if the register_language could be exploited (but this is not a requirement for this, as some features with tests are distinctly different from other languages). Cheers, Ralf
Ralf Wildenhues <Ralf.Wildenhues@HIDDEN>:bug-automake@HIDDEN.
Full text available.owner <at> debbugs.gnu.org, bug-automake@HIDDEN:bug#7868; Package automake.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.