Karl Berry <karl@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at 42883) by debbugs.gnu.org; 25 Aug 2020 01:39:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 24 21:39:33 2020
Received: from localhost ([127.0.0.1]:59672 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1kANvx-0004k7-I9
for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:33 -0400
Received: from freefriends.org ([96.88.95.60]:52784)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <karl@HIDDEN>) id 1kANvr-0004jv-MN
for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:31 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
by freefriends.org (8.14.7/8.14.7) with ESMTP id 07P1dQfR027448
(version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Mon, 24 Aug 2020 19:39:26 -0600
Received: (from apache@localhost)
by freefriends.org (8.14.7/8.14.7/Submit) id 07P1dPl9027447;
Mon, 24 Aug 2020 19:39:25 -0600
Date: Mon, 24 Aug 2020 19:39:25 -0600
Message-Id: <202008250139.07P1dPl9027447@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
unnecessary failures
In-Reply-To: <20200824105448.2f33d717@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hi Glenn,
The example is meant to be such that the test X in root and SUBDIR
are tests that would be run when not explicitly specifying tests.
Yep, I got that.
So its to be able to run any test that would be run when not
explicitly specifying tests.
Clearly yes, it would be desirable in principle to be able to name tests
anything you want, in any directory you want, and then be able to
specify exactly the individual tests you want to run, regardless of
names or (sub)directories. I'm just not sure how feasible it is to
implement.
I think the way to support this is to have TESTS be a list of test
paths relative to the root makefile.
Isn't it already that way?
So in the example above, I would
specify TESTS="X SUBDIR/X".
To run a specific test in automake now, what I came up with is
make check TESTS=t/sometest.sh
Although the manual says it can be overridden in the environment
(Parallel Test Harness node):
env TESTS="foo.test bar.test" make -e check
this has never worked for me. (Neither foo.sh or t/foo.sh.) I have not
tried to discern what I was doing wrong. (I feel pretty sure it was me
doing something wrong and not Automake, at least.)
I realize the case you're concerned about is when both ./sometest.sh and
t/sometest.sh are present. My gut reaction is that it would be more
reliable to detect that case and complain about it than try to support
it. But ok, never mind, proceeding along the ideal lines ...
Before running the recursive make in SUBDIR, the TESTS variable
would be filtered to only include tests with a relative path of the
sub directory. Also, the root makefile would not run tests that
have a directory component to the path.
It sounds sensible to me, but regrettably, I just don't have a great
overall grip on the code, having come to the implementation side of
Automake rather late. Jim or Paul would have better things to say.
Unfortunately I think they are busy with other things (which is why I'm
here in the first place ... anyway ...).
I would like to get a rough idea of if this would be an approved of
approach so that I or someone else could take a stab at implementing it
with the hopes of the changes being accepted.
In general, the top priority for any change in Automake is to not break
existing practice. If you can devise a change that, for starters, still
passes all the existing Automake tests, then I surmise there's a good
shot.
As you may infer from my TESTS= stuff above, exactly what invariants
have to be maintained to keep compatibility in this case is not clear to
me :(.
(Also, as one would expect, the change has to be comprehensible and
maintainable, but that's nothing special to Automake.)
Regarding your previous patch, it actually seemed pretty plausible to me
(modulo EXEEXT and other such, as you noted; not a big deal), except
that it's a shame to lose the typo checking. There are over a thousand
tests in Automake, and no doubt that many or more in other packages. So
typos are inevitable. Seems like some sort of separate check for them
would need to be available, going down that road.
Thanks for following up. I hope you are sufficiently interested to spend
some more time on it. --best, karl.
bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.Received: (at 42883) by debbugs.gnu.org; 24 Aug 2020 15:54:59 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 24 11:54:59 2020 Received: from localhost ([127.0.0.1]:58981 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1kAEoF-000230-EZ for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:59 -0400 Received: from mail-qt1-f170.google.com ([209.85.160.170]:46105) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <development@HIDDEN>) id 1kAEoD-00022i-Qi for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:58 -0400 Received: by mail-qt1-f170.google.com with SMTP id o22so6461504qtt.13 for <42883 <at> debbugs.gnu.org>; Mon, 24 Aug 2020 08:54:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=; b=vx7Tlg/lpCMapybAPMn8VWTYNmY53bHghqDZqd3/KVSAXkKcsTzAflBLYksyxGX1// MOo8006GiiwXZewcpAMzh3Flxd9MYg52Om5vDKetVG7ZLtW3pQcxQpGx5WUc1eCW/ePw /glUPHCFJ9fyj+82fjdCiNycPfkEdgfSlBXcZMZ9cJX4bKFQkU1l284166AdLQ5yGPUo JQcCGL5Oki+aXkLVzB1T6G6j2QLTuB6TEY3BJLRIU7zBf9BGfFvqZWBssArRClRn4ojr HIENLxdd8M2wrjaK+VOIWgHNUIKMBN6qbuq0El6jbnc9DEloODQGNTvDXmmc7SYDtUkz LpBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=; b=T4TQ9S+VhrRnrZCSJrJzV9l3JFz0DzfiOW+bsbA90OzEtWrDqKlLDGHJijEpPqDDYL LUbY8EVmRpjOfm73Z+wRB+emI/FqG7WYurWh4HEqK6fXuzCk6cXg5ISZl5AaO5cMsd+O 4VugKacgAJVDUTOt/nli/TX6HtPL6Vc6iVzIKYEl01j5hpzqiKuK3TVcpFlyyOgR9p1j XZclADQd+YVBu5xv5ZuRb1z9XUsfwVyf3DWsqVS5Z4FJ8l0qg/EJ3CFrGbQ+EpvmrTGk aLznTVkih2IB4yaram9wuuqE5Xl93cAuskqmk8k2a3SWKYy06jzUkQ9Xc0QdQqZyM2mz WDlw== X-Gm-Message-State: AOAM532cjar+t/gH5rXQ6v661/GyBHAbFrDOErefZG1QUHW8Y1kTI7e0 +23raAd8Y32/WUfT9Umw5YAiEg7lI/QSpQ== X-Google-Smtp-Source: ABdhPJzX441otFAVTZZkkCoD39saIqplyaRj54GpkjLXUv8rdd0K187Io9W83JimZl0VHHIJjdjOJg== X-Received: by 2002:ac8:6647:: with SMTP id j7mr5165230qtp.335.1598284492134; Mon, 24 Aug 2020 08:54:52 -0700 (PDT) Received: from crass-HP-ZBook-15-G2 ([2605:a601:aaec:9b00:5122:8a61:cec2:28f4]) by smtp.gmail.com with ESMTPSA id h6sm2856366qtk.47.2020.08.24.08.54.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Aug 2020 08:54:51 -0700 (PDT) Date: Mon, 24 Aug 2020 10:54:48 -0500 From: Glenn Washburn <development@HIDDEN> To: Karl Berry <karl@HIDDEN> Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures Message-ID: <20200824105448.2f33d717@crass-HP-ZBook-15-G2> In-Reply-To: <202008212205.07LM5OlR013325@HIDDEN> References: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2> <202008212205.07LM5OlR013325@HIDDEN> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42883 Cc: 42883 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: development@HIDDEN Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) On Fri, 21 Aug 2020 16:05:24 -0600 Karl Berry <karl@HIDDEN> wrote: >> Another issue is that if there is a test X in the root makefile >> and a test X in SUBDIR > > Isn't that fundamentally not supported, or supportable? I admit I > don't actually know if there's any history about it, but using the > same name for two tests in different places just seems like a basic > source of confusion. As far as I know, the TESTS mechanism just isn't > set up to be able to specify arbitrary test names in arbitrary > directories. I'm not sure if I was sufficiently clear in my example. The example is meant to be such that the test X in root and SUBDIR are tests that would be run when not explicitly specifying tests. So its to be able to run any test that would be run when not explicitly specifying tests. I think the way to support this is to have TESTS be a list of test paths relative to the root makefile. So in the example above, I would specify TESTS="X SUBDIR/X". Before running the recursive make in SUBDIR, the TESTS variable would be filtered to only include tests with a relative path of the sub directory. Also, the root makefile would not run tests that have a directory component to the path. I would like to get a rough idea of if this would be an approved of approach so that I or someone else could take a stab at implementing it with the hopes of the changes being accepted. Glenn
bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.
Received: (at 42883) by debbugs.gnu.org; 21 Aug 2020 22:05:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 21 18:05:28 2020
Received: from localhost ([127.0.0.1]:48454 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1k9FA8-0008H6-N4
for submit <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:28 -0400
Received: from freefriends.org ([96.88.95.60]:35636)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <karl@HIDDEN>) id 1k9FA6-0008Gw-31
for 42883 <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:27 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
by freefriends.org (8.14.7/8.14.7) with ESMTP id 07LM5OA1013326
(version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Fri, 21 Aug 2020 16:05:25 -0600
Received: (from apache@localhost)
by freefriends.org (8.14.7/8.14.7/Submit) id 07LM5OlR013325;
Fri, 21 Aug 2020 16:05:24 -0600
Date: Fri, 21 Aug 2020 16:05:24 -0600
Message-Id: <202008212205.07LM5OlR013325@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
unnecessary failures
In-Reply-To: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Yes, I agree. Contemplating.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.
I think I agree again, but ... not sure ...
Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR
Isn't that fundamentally not supported, or supportable? I admit I don't
actually know if there's any history about it, but using the same name
for two tests in different places just seems like a basic source of
confusion. As far as I know, the TESTS mechanism just isn't set up to
be able to specify arbitrary test names in arbitrary directories.
Admittedly if it is not supported, it could at least be detected and
complained about. On the other hand, I suppose it works when not
explicitly specifying tests to run, which is 99% of the time. So, once
again ... I don't know. Nothing is ever simple :).
Jim, Paul, anyone? --thanks, karl.
bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.Received: (at 42883) by debbugs.gnu.org; 21 Aug 2020 08:07:33 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 21 04:07:33 2020 Received: from localhost ([127.0.0.1]:44956 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1k925F-0002CG-HM for submit <at> debbugs.gnu.org; Fri, 21 Aug 2020 04:07:33 -0400 Received: from mail-qt1-f178.google.com ([209.85.160.178]:33163) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <development@HIDDEN>) id 1k925C-0002C1-8h for 42883 <at> debbugs.gnu.org; Fri, 21 Aug 2020 04:07:31 -0400 Received: by mail-qt1-f178.google.com with SMTP id 6so677508qtt.0 for <42883 <at> debbugs.gnu.org>; Fri, 21 Aug 2020 01:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h+2/FI4J8433ssAW3suBntFPvhO2IFeDJY8bvNHw5RE=; b=T6FF5+2ooVnxBHBOtwwVLBf57V9JXKccSe8SOxzdmS3HOTAXsaUNwWHtPs9FVawII0 6n9lgSvhK+DyUI2GUK0b0F+653Uwq69ZPQX2XWrXZ1RiTSilBT3U3S8kTWjedHbWdZlJ g1KMtFcBWS8F5M27lvEwe2wlMIysmnsfKrHxKWcr1duGnLKQuyO/wqNLnh2J4EmLEkah Qri5WBEiE3t1gj//fSscXZO+dRSsGmr5RI4otQdiSKsSBapiTsqdCRq2wpHMpgyfYfP6 iER1F5qaGev2buFZSt5VbhMIIJDz28d0B/nk2gS6u4Mg55hwfhN9pWMeH/5pcUsmqmf8 MHKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h+2/FI4J8433ssAW3suBntFPvhO2IFeDJY8bvNHw5RE=; b=hcgBVBko1rpBL6Wi0/zFREgTBb0JyEstx2P77tZw64Hnp5nmsTt/paraFdOjzke5Xp c3gA/Xm3i3wjqepcBn6G1z5QfNrDNt386aLyh3BYxXH/ptgBOKgbU3PEFTZlszH7vGIk 1o38O4TFk+xUZBmbXXIUoPiV/IB4NPrxWe/Slq894bj4eVlUKg05iQ7cO+pquS/tSkaJ W+iWq5v/1c8HK3o7cuCxa++gijiGH6KIwE+uBaQculu1i7aeEYaldzlRmJS4t9Ane1+u 4a9JYwmmnwl/Thm99jmMyQHry566GnOOnD1MgQOxvC7P88NDN0YGP0DQdXoXJ3dJV2kZ gOqQ== X-Gm-Message-State: AOAM531/NqjZtdXVV8vN5zgTGZ+53kcMP482JdW4JKtmd0sA0/1p2Y9e MFWEWv3EKN2WIFcs7YyD9oz7frSDO7Roaw== X-Google-Smtp-Source: ABdhPJw9nusPrGbIEbAadLl0bc38zuyRJw93hVkm93vP/FGBBunPJ8KHlaomM3134B1xjzWefb5Imw== X-Received: by 2002:ac8:1e95:: with SMTP id c21mr1582499qtm.306.1597997244674; Fri, 21 Aug 2020 01:07:24 -0700 (PDT) Received: from crass-HP-ZBook-15-G2 ([2605:a601:aaec:9b00:6c92:654e:5b7c:82b2]) by smtp.gmail.com with ESMTPSA id y46sm1334996qth.78.2020.08.21.01.07.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 01:07:24 -0700 (PDT) Date: Fri, 21 Aug 2020 03:07:21 -0500 From: Glenn Washburn <development@HIDDEN> To: Karl Berry <karl@HIDDEN> Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures Message-ID: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2> In-Reply-To: <202008200117.07K1HDu4021525@HIDDEN> References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2> <202008200117.07K1HDu4021525@HIDDEN> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42883 Cc: development@HIDDEN, 42883 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hi Karl, On Wed, 19 Aug 2020 19:17:13 -0600 Karl Berry <karl@HIDDEN> wrote: > Hi Glenn, > > I noticed that when specifying a subset of tests to run via the > TESTS environment variable, I could have all tests complete > successfully > > Thanks for the report. That does seem suboptimal, but on the other > hand, subsets of tests are a complication. Does it work to override > SUBDIRS along with TESTS? As in (thanks to Jim): > env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS= Yes this does work in the special case of grub, thanks for the suggestion. I think at a minimum this behavior should be documented with the test subsets documentation and this work around mentioned. Ideally, the subset tests feature design should be updated to play nice with makefile recursion. Setting SUBDIRS= is just disabling the makefile recusion. If I had a test in grub-core/unittest1 and I wanted to run just the test subset of unittest1 and grub_cmd_echo, how would I do that such that make returns success? If I run `TESTS="unittest1 grub_cmd_echo" make -e check', make will fail even if both test individually succeed because the recursive make check in grub-core will try to run grub_cmd_echo (in addition to unittest1) and fail. Another issue is that if there is a test X in the root makefile and a test X in SUBDIR, how does one choose to run the X in SUBDIR and not the X in root? Right now TESTS=X will run both tests. Sure one could do `TESTS=X make -e -C SUBDIR check', but then again you're just bypassing the makefile recursion. And thanks Karl for helping to maintain this venerable and essential, yet not super glamorous project. Glenn
bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.
Received: (at 42883) by debbugs.gnu.org; 20 Aug 2020 01:17:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 19 21:17:17 2020
Received: from localhost ([127.0.0.1]:41338 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1k8ZCf-0000XN-1v
for submit <at> debbugs.gnu.org; Wed, 19 Aug 2020 21:17:17 -0400
Received: from freefriends.org ([96.88.95.60]:60268)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <karl@HIDDEN>) id 1k8ZCd-0000Vt-Be
for 42883 <at> debbugs.gnu.org; Wed, 19 Aug 2020 21:17:16 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
by freefriends.org (8.14.7/8.14.7) with ESMTP id 07K1HD1w021526
(version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Wed, 19 Aug 2020 19:17:14 -0600
Received: (from apache@localhost)
by freefriends.org (8.14.7/8.14.7/Submit) id 07K1HDu4021525;
Wed, 19 Aug 2020 19:17:13 -0600
Date: Wed, 19 Aug 2020 19:17:13 -0600
Message-Id: <202008200117.07K1HDu4021525@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
unnecessary failures
In-Reply-To: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hi Glenn,
I noticed that when specifying a subset of tests to run via the TESTS
environment variable, I could have all tests complete successfully
Thanks for the report. That does seem suboptimal, but on the other hand,
subsets of tests are a complication. Does it work to override SUBDIRS
along with TESTS? As in (thanks to Jim):
env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=
Thanks,
Karl
bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.Received: (at submit) by debbugs.gnu.org; 16 Aug 2020 06:23:48 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 16 02:23:47 2020 Received: from localhost ([127.0.0.1]:56704 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1k7C55-0001y9-0m for submit <at> debbugs.gnu.org; Sun, 16 Aug 2020 02:23:47 -0400 Received: from lists.gnu.org ([209.51.188.17]:42370) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <development@HIDDEN>) id 1k7BTV-000760-4d for submit <at> debbugs.gnu.org; Sun, 16 Aug 2020 01:44:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34678) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <development@HIDDEN>) id 1k7BTU-0001Ua-Mg for bug-automake@HIDDEN; Sun, 16 Aug 2020 01:44:56 -0400 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]:35119) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from <development@HIDDEN>) id 1k7BTS-0003WM-Ax for bug-automake@HIDDEN; Sun, 16 Aug 2020 01:44:56 -0400 Received: by mail-pf1-x434.google.com with SMTP id d188so6551638pfd.2 for <bug-automake@HIDDEN>; Sat, 15 Aug 2020 22:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficientek-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:reply-to:mime-version; bh=qo1Uila4Xs/Bmm/qx2thvf90eI7/bDrYIYOhwHKRGRc=; b=LXNqmt+Cpta+m1zcdxlMqSqPvq08rJG+ig78mwoqtq7BQLLQ8ALXDIMsZYWF9eHh0i OUqC+0UbtMU64lpIB1ICh3mpkbVIs8sM/v7LtyYmIyTWnUgvKf0Cc2ulypq+wPdaZZOT 18TvSh4BTFfTv46xWroVajsnh5AgyjCwUATYXurTqGTXyMpNpZxfeD8cCMZWXjjSO/R9 80ebf3j8SAQ1iDcqeNxxfApP5BdWAq+CRwJa1ufY9nGHHQCW3J1O3wDJ3iZ0Ce4Kg/8E Z9PkfMXVnuilwPKGVEzaZeq2faHvvQ0mJhCwf2SkuaRgt2IwUqFRlFGmsTzdFUqt8HP1 DlsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:reply-to :mime-version; bh=qo1Uila4Xs/Bmm/qx2thvf90eI7/bDrYIYOhwHKRGRc=; b=YtF6wuo+lEehaZ2yLXR+YFq/cj8+yfFuZTManT2z3U3sLmfY8uhMiq5n9vtvrGHTn3 h5D4ypLV9ZNqaazi1MUheWT0vW6TqriIZorouuDA7P0KgiSUQxChvLiF9GL7Y0UXVMs2 +iDWWNMtoO0wv/G8yAuAek8J5g0yEQvb6XXV/ysoHhJIgD7/H4oBwXGI5VvFIVWGAaL7 2MM8+cuGJIV1y/TthM4WPlb1uU9vxu9jHM7z5al8QGAPDTNihIpWV8V0vxPHV1oA2StZ S5c13x0cZhZPcUKrsD+HLTW7uAD/Z4jyfdQ63qh//nJg2plYqdyX9L8vq7sw8j+K76Lm yiHw== X-Gm-Message-State: AOAM532FP0Jf++TSihKe/H+j1UPtMEPve2+mrSrc/Juu8/JZ2daOca0a jycLrnwdhRVC7o6KQAN6xn9ynZAr1rfMhg== X-Google-Smtp-Source: ABdhPJxGLSKN6gl1ESFE+3FWkrZlAsMjGyqHSIACAyp4ETIQrWqRQrVC1WvgpovkaJH+PIUSzAcC1g== X-Received: by 2002:a63:7542:: with SMTP id f2mr6190400pgn.151.1597556691937; Sat, 15 Aug 2020 22:44:51 -0700 (PDT) Received: from crass-HP-ZBook-15-G2 ([136.49.44.103]) by smtp.gmail.com with ESMTPSA id 27sm12383250pgk.89.2020.08.15.22.44.50 for <bug-automake@HIDDEN> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Aug 2020 22:44:51 -0700 (PDT) Date: Sun, 16 Aug 2020 00:44:41 -0500 From: Glenn Washburn <development@HIDDEN> To: bug-automake@HIDDEN Subject: Selecting a subset of tests in recursive make causes unnecessary failures Message-ID: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/sT4As_qzOipHo49O6kH/coI" Received-SPF: pass client-ip=2607:f8b0:4864:20::434; envelope-from=development@HIDDEN; helo=mail-pf1-x434.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Sun, 16 Aug 2020 02:23:45 -0400 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: development@HIDDEN Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.3 (--) --MP_/sT4As_qzOipHo49O6kH/coI Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi, I've been working on some automake tests for grub, and ran in to a bug. I noticed that when specifying a subset of tests to run via the TESTS environment variable, I could have all tests complete successfully and yet still have make return an error. Here's a sequence of commands that should reproduce the error: git clone https://git.savannah.gnu.org/git/grub.git cd grub ./bootstrap ./configure make env TESTS=grub_cmd_echo make -e check VERBOSE=yes It turns out that make check is being recursively run in the grub-core directory, as expected. The TESTS environment variable is the same in both invocations, also as expected. However, the test program is not in the grub-core subdirectory. So the test is run successfully in the root make, and not found in the subdirectory, thus generating an error. In fact, grub has no tests in the grub-core subdirectory, but the harness is run because its part of automake, even though I do not need or want it to. I've attached a patch which fixes this issue by ignoring tests which do not exist. I'm not sure this patch takes care of everything needed, for instance EXEEXT, to check for the existence of the test program. Grub does not use any TEST_EXTENSIONS, so this works. This will change make it so TESTS which should exist, but do not, will not cause an error, whereas currently it will. I'm not sure how this situation would arise, because make should require the existence as a dependency. If there is an error in the TESTS variable (eg. a typo in a test name), this will not cause an error either, which I deem undesirable. However, I think this is an acceptable trade-off. Perhaps a better solution would be to not recursively run make check if TESTS is defined and run all given tests from the root make. Tests which are run in recursive make invocations when TESTS is not defined can be run via TESTS by specifying a path component (eg. TESTS="grub-core/test-program"). This could be a problem if test programs assume they are run from the directory in which they reside (in which case, we could chdir to the directory and run the program). Any thoughts? Glenn --MP_/sT4As_qzOipHo49O6kH/coI Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=fix-recursive-subset-test.patch diff --git a/lib/am/check.am b/lib/am/check.am index 449708742..94021775e 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -416,8 +416,8 @@ check-TESTS: %CHECK_DEPS% ## we rely on .PHONY to work portably. @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) @set +e; $(am__set_TESTS_bases); \ - log_list=`for i in $$bases; do echo $$i.log; done`; \ - trs_list=`for i in $$bases; do echo $$i.trs; done`; \ + log_list=`for i in $$bases; do test -x $$i && echo $$i.log; done`; \ + trs_list=`for i in $$bases; do test -x $$i && echo $$i.trs; done`; \ ## Remove newlines and normalize whitespace. Trailing (and possibly ## leading) whitespace is known to cause segmentation faults on ## Solaris 10 XPG4 make. --MP_/sT4As_qzOipHo49O6kH/coI Content-Type: text/plain Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=grub-tests-error.txt make[3]: Leaving directory '/home/user/grub.git' make check-TESTS make[3]: Entering directory '/home/user/grub.git' make[4]: Entering directory '/home/user/grub.git' PASS: grub_cmd_echo ============================================================================ Testsuite summary for GRUB 2.05 ============================================================================ # TOTAL: 1 # PASS: 1 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ make[4]: Leaving directory '/home/user/grub.git' make[3]: Leaving directory '/home/user/grub.git' make[2]: Leaving directory '/home/user/grub.git' Making check in grub-core make[2]: Entering directory '/home/user/grub.git/grub-core' make check-am make[3]: Entering directory '/home/user/grub.git/grub-core' make make[4]: Entering directory '/home/user/grub.git/grub-core' make all-am make[5]: Entering directory '/home/user/grub.git/grub-core' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/user/grub.git/grub-core' make[4]: Leaving directory '/home/user/grub.git/grub-core' make check-TESTS make[4]: Entering directory '/home/user/grub.git/grub-core' make[5]: Entering directory '/home/user/grub.git/grub-core' make[6]: Entering directory '/home/user/grub.git/grub-core' make[6]: Nothing to be done for 'grub_cmd_echo.log'. make[6]: Leaving directory '/home/user/grub.git/grub-core' fatal: making test-suite.log: failed to create grub_cmd_echo.trs fatal: making test-suite.log: failed to create grub_cmd_echo.log make[5]: *** [Makefile:43107: test-suite.log] Error 1 make[5]: Leaving directory '/home/user/grub.git/grub-core' make[4]: *** [Makefile:43216: check-TESTS] Error 2 make[4]: Leaving directory '/home/user/grub.git/grub-core' make[3]: *** [Makefile:43283: check-am] Error 2 make[3]: Leaving directory '/home/user/grub.git/grub-core' make[2]: *** [Makefile:43285: check] Error 2 make[2]: Leaving directory '/home/user/grub.git/grub-core' make[1]: *** [Makefile:11920: check-recursive] Error 1 make[1]: Leaving directory '/home/user/grub.git' make: *** [Makefile:12944: check] Error 2 --MP_/sT4As_qzOipHo49O6kH/coI--
development@HIDDEN:bug-automake@HIDDEN.
Full text available.bug-automake@HIDDEN:bug#42883; Package automake.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.