GNU bug report logs - #42883
Selecting a subset of tests in recursive make causes unnecessary failures

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: automake; Reported by: development@HIDDEN; Keywords: help confirmed; dated Sun, 16 Aug 2020 06:24:01 UTC; Maintainer for automake is bug-automake@HIDDEN.
Added tag(s) help and confirmed. Request was from Karl Berry <karl@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 42883 <at> debbugs.gnu.org:


Received: (at 42883) by debbugs.gnu.org; 25 Aug 2020 01:39:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 24 21:39:33 2020
Received: from localhost ([127.0.0.1]:59672 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kANvx-0004k7-I9
	for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:33 -0400
Received: from freefriends.org ([96.88.95.60]:52784)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1kANvr-0004jv-MN
 for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:31 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 07P1dQfR027448
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); 
 Mon, 24 Aug 2020 19:39:26 -0600
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 07P1dPl9027447;
 Mon, 24 Aug 2020 19:39:25 -0600
Date: Mon, 24 Aug 2020 19:39:25 -0600
Message-Id: <202008250139.07P1dPl9027447@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
In-Reply-To: <20200824105448.2f33d717@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Glenn,

    The example is meant to be such that the test X in root and SUBDIR
    are tests that would be run when not explicitly specifying tests.

Yep, I got that.

    So its to be able to run any test that would be run when not
    explicitly specifying tests.

Clearly yes, it would be desirable in principle to be able to name tests
anything you want, in any directory you want, and then be able to
specify exactly the individual tests you want to run, regardless of
names or (sub)directories. I'm just not sure how feasible it is to
implement.

    I think the way to support this is to have TESTS be a list of test
    paths relative to the root makefile.  

Isn't it already that way?

    So in the example above, I would
    specify TESTS="X SUBDIR/X".

To run a specific test in automake now, what I came up with is 
make check TESTS=t/sometest.sh

Although the manual says it can be overridden in the environment
(Parallel Test Harness node):
  env TESTS="foo.test bar.test" make -e check
this has never worked for me. (Neither foo.sh or t/foo.sh.) I have not
tried to discern what I was doing wrong. (I feel pretty sure it was me
doing something wrong and not Automake, at least.)

I realize the case you're concerned about is when both ./sometest.sh and
t/sometest.sh are present. My gut reaction is that it would be more
reliable to detect that case and complain about it than try to support
it. But ok, never mind, proceeding along the ideal lines ...

    Before running the recursive make in SUBDIR, the TESTS variable
    would be filtered to only include tests with a relative path of the
    sub directory.  Also, the root makefile would not run tests that
    have a directory component to the path.

It sounds sensible to me, but regrettably, I just don't have a great
overall grip on the code, having come to the implementation side of
Automake rather late. Jim or Paul would have better things to say.
Unfortunately I think they are busy with other things (which is why I'm
here in the first place ... anyway ...).

    I would like to get a rough idea of if this would be an approved of
    approach so that I or someone else could take a stab at implementing it
    with the hopes of the changes being accepted.

In general, the top priority for any change in Automake is to not break
existing practice. If you can devise a change that, for starters, still
passes all the existing Automake tests, then I surmise there's a good
shot.

As you may infer from my TESTS= stuff above, exactly what invariants
have to be maintained to keep compatibility in this case is not clear to
me :(.

(Also, as one would expect, the change has to be comprehensible and
maintainable, but that's nothing special to Automake.)

Regarding your previous patch, it actually seemed pretty plausible to me
(modulo EXEEXT and other such, as you noted; not a big deal), except
that it's a shame to lose the typo checking. There are over a thousand
tests in Automake, and no doubt that many or more in other packages. So
typos are inevitable. Seems like some sort of separate check for them
would need to be available, going down that road.

Thanks for following up. I hope you are sufficiently interested to spend
some more time on it. --best, karl.




Information forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.

Message received at 42883 <at> debbugs.gnu.org:


Received: (at 42883) by debbugs.gnu.org; 24 Aug 2020 15:54:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Aug 24 11:54:59 2020
Received: from localhost ([127.0.0.1]:58981 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kAEoF-000230-EZ
	for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:59 -0400
Received: from mail-qt1-f170.google.com ([209.85.160.170]:46105)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <development@HIDDEN>) id 1kAEoD-00022i-Qi
 for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:58 -0400
Received: by mail-qt1-f170.google.com with SMTP id o22so6461504qtt.13
 for <42883 <at> debbugs.gnu.org>; Mon, 24 Aug 2020 08:54:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=efficientek-com.20150623.gappssmtp.com; s=20150623;
 h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to
 :mime-version:content-transfer-encoding;
 bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=;
 b=vx7Tlg/lpCMapybAPMn8VWTYNmY53bHghqDZqd3/KVSAXkKcsTzAflBLYksyxGX1//
 MOo8006GiiwXZewcpAMzh3Flxd9MYg52Om5vDKetVG7ZLtW3pQcxQpGx5WUc1eCW/ePw
 /glUPHCFJ9fyj+82fjdCiNycPfkEdgfSlBXcZMZ9cJX4bKFQkU1l284166AdLQ5yGPUo
 JQcCGL5Oki+aXkLVzB1T6G6j2QLTuB6TEY3BJLRIU7zBf9BGfFvqZWBssArRClRn4ojr
 HIENLxdd8M2wrjaK+VOIWgHNUIKMBN6qbuq0El6jbnc9DEloODQGNTvDXmmc7SYDtUkz
 LpBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
 :references:reply-to:mime-version:content-transfer-encoding;
 bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=;
 b=T4TQ9S+VhrRnrZCSJrJzV9l3JFz0DzfiOW+bsbA90OzEtWrDqKlLDGHJijEpPqDDYL
 LUbY8EVmRpjOfm73Z+wRB+emI/FqG7WYurWh4HEqK6fXuzCk6cXg5ISZl5AaO5cMsd+O
 4VugKacgAJVDUTOt/nli/TX6HtPL6Vc6iVzIKYEl01j5hpzqiKuK3TVcpFlyyOgR9p1j
 XZclADQd+YVBu5xv5ZuRb1z9XUsfwVyf3DWsqVS5Z4FJ8l0qg/EJ3CFrGbQ+EpvmrTGk
 aLznTVkih2IB4yaram9wuuqE5Xl93cAuskqmk8k2a3SWKYy06jzUkQ9Xc0QdQqZyM2mz
 WDlw==
X-Gm-Message-State: AOAM532cjar+t/gH5rXQ6v661/GyBHAbFrDOErefZG1QUHW8Y1kTI7e0
 +23raAd8Y32/WUfT9Umw5YAiEg7lI/QSpQ==
X-Google-Smtp-Source: ABdhPJzX441otFAVTZZkkCoD39saIqplyaRj54GpkjLXUv8rdd0K187Io9W83JimZl0VHHIJjdjOJg==
X-Received: by 2002:ac8:6647:: with SMTP id j7mr5165230qtp.335.1598284492134; 
 Mon, 24 Aug 2020 08:54:52 -0700 (PDT)
Received: from crass-HP-ZBook-15-G2 ([2605:a601:aaec:9b00:5122:8a61:cec2:28f4])
 by smtp.gmail.com with ESMTPSA id h6sm2856366qtk.47.2020.08.24.08.54.51
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 24 Aug 2020 08:54:51 -0700 (PDT)
Date: Mon, 24 Aug 2020 10:54:48 -0500
From: Glenn Washburn <development@HIDDEN>
To: Karl Berry <karl@HIDDEN>
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Message-ID: <20200824105448.2f33d717@crass-HP-ZBook-15-G2>
In-Reply-To: <202008212205.07LM5OlR013325@HIDDEN>
References: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
 <202008212205.07LM5OlR013325@HIDDEN>
X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: development@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, 21 Aug 2020 16:05:24 -0600
Karl Berry <karl@HIDDEN> wrote:

>> Another issue is that if there is a test X in the root makefile
>> and a test X in SUBDIR
> 
> Isn't that fundamentally not supported, or supportable? I admit I
> don't actually know if there's any history about it, but using the
> same name for two tests in different places just seems like a basic
> source of confusion. As far as I know, the TESTS mechanism just isn't
> set up to be able to specify arbitrary test names in arbitrary
> directories.

I'm not sure if I was sufficiently clear in my example. The example is
meant to be such that the test X in root and SUBDIR are tests that
would be run when not explicitly specifying tests.  So its to be able
to run any test that would be run when not explicitly specifying tests.

I think the way to support this is to have TESTS be a list of test
paths relative to the root makefile.  So in the example above, I would
specify TESTS="X SUBDIR/X".  Before running the recursive make in
SUBDIR, the TESTS variable would be filtered to only include tests with
a relative path of the sub directory.  Also, the root makefile would
not run tests that have a directory component to the path.

I would like to get a rough idea of if this would be an approved of
approach so that I or someone else could take a stab at implementing it
with the hopes of the changes being accepted.

Glenn




Information forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.

Message received at 42883 <at> debbugs.gnu.org:


Received: (at 42883) by debbugs.gnu.org; 21 Aug 2020 22:05:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 21 18:05:28 2020
Received: from localhost ([127.0.0.1]:48454 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k9FA8-0008H6-N4
	for submit <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:28 -0400
Received: from freefriends.org ([96.88.95.60]:35636)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1k9FA6-0008Gw-31
 for 42883 <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:27 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 07LM5OA1013326
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); 
 Fri, 21 Aug 2020 16:05:25 -0600
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 07LM5OlR013325;
 Fri, 21 Aug 2020 16:05:24 -0600
Date: Fri, 21 Aug 2020 16:05:24 -0600
Message-Id: <202008212205.07LM5OlR013325@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
In-Reply-To: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

    at a minimum this behavior should be documented
    with the test subsets documentation and this work around mentioned.

Yes, I agree. Contemplating.

    Ideally, the subset tests feature design should be updated to play nice
    with makefile recursion.

I think I agree again, but ... not sure ...

    Another issue is that if there is a test X in the root makefile and a
    test X in SUBDIR

Isn't that fundamentally not supported, or supportable? I admit I don't
actually know if there's any history about it, but using the same name
for two tests in different places just seems like a basic source of
confusion. As far as I know, the TESTS mechanism just isn't set up to
be able to specify arbitrary test names in arbitrary directories.

Admittedly if it is not supported, it could at least be detected and
complained about. On the other hand, I suppose it works when not
explicitly specifying tests to run, which is 99% of the time. So, once
again ... I don't know. Nothing is ever simple :).

Jim, Paul, anyone? --thanks, karl.




Information forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.

Message received at 42883 <at> debbugs.gnu.org:


Received: (at 42883) by debbugs.gnu.org; 21 Aug 2020 08:07:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 21 04:07:33 2020
Received: from localhost ([127.0.0.1]:44956 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k925F-0002CG-HM
	for submit <at> debbugs.gnu.org; Fri, 21 Aug 2020 04:07:33 -0400
Received: from mail-qt1-f178.google.com ([209.85.160.178]:33163)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <development@HIDDEN>) id 1k925C-0002C1-8h
 for 42883 <at> debbugs.gnu.org; Fri, 21 Aug 2020 04:07:31 -0400
Received: by mail-qt1-f178.google.com with SMTP id 6so677508qtt.0
 for <42883 <at> debbugs.gnu.org>; Fri, 21 Aug 2020 01:07:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=efficientek-com.20150623.gappssmtp.com; s=20150623;
 h=date:from:to:cc:subject:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding;
 bh=h+2/FI4J8433ssAW3suBntFPvhO2IFeDJY8bvNHw5RE=;
 b=T6FF5+2ooVnxBHBOtwwVLBf57V9JXKccSe8SOxzdmS3HOTAXsaUNwWHtPs9FVawII0
 6n9lgSvhK+DyUI2GUK0b0F+653Uwq69ZPQX2XWrXZ1RiTSilBT3U3S8kTWjedHbWdZlJ
 g1KMtFcBWS8F5M27lvEwe2wlMIysmnsfKrHxKWcr1duGnLKQuyO/wqNLnh2J4EmLEkah
 Qri5WBEiE3t1gj//fSscXZO+dRSsGmr5RI4otQdiSKsSBapiTsqdCRq2wpHMpgyfYfP6
 iER1F5qaGev2buFZSt5VbhMIIJDz28d0B/nk2gS6u4Mg55hwfhN9pWMeH/5pcUsmqmf8
 MHKQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
 :references:mime-version:content-transfer-encoding;
 bh=h+2/FI4J8433ssAW3suBntFPvhO2IFeDJY8bvNHw5RE=;
 b=hcgBVBko1rpBL6Wi0/zFREgTBb0JyEstx2P77tZw64Hnp5nmsTt/paraFdOjzke5Xp
 c3gA/Xm3i3wjqepcBn6G1z5QfNrDNt386aLyh3BYxXH/ptgBOKgbU3PEFTZlszH7vGIk
 1o38O4TFk+xUZBmbXXIUoPiV/IB4NPrxWe/Slq894bj4eVlUKg05iQ7cO+pquS/tSkaJ
 W+iWq5v/1c8HK3o7cuCxa++gijiGH6KIwE+uBaQculu1i7aeEYaldzlRmJS4t9Ane1+u
 4a9JYwmmnwl/Thm99jmMyQHry566GnOOnD1MgQOxvC7P88NDN0YGP0DQdXoXJ3dJV2kZ
 gOqQ==
X-Gm-Message-State: AOAM531/NqjZtdXVV8vN5zgTGZ+53kcMP482JdW4JKtmd0sA0/1p2Y9e
 MFWEWv3EKN2WIFcs7YyD9oz7frSDO7Roaw==
X-Google-Smtp-Source: ABdhPJw9nusPrGbIEbAadLl0bc38zuyRJw93hVkm93vP/FGBBunPJ8KHlaomM3134B1xjzWefb5Imw==
X-Received: by 2002:ac8:1e95:: with SMTP id c21mr1582499qtm.306.1597997244674; 
 Fri, 21 Aug 2020 01:07:24 -0700 (PDT)
Received: from crass-HP-ZBook-15-G2 ([2605:a601:aaec:9b00:6c92:654e:5b7c:82b2])
 by smtp.gmail.com with ESMTPSA id y46sm1334996qth.78.2020.08.21.01.07.24
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 21 Aug 2020 01:07:24 -0700 (PDT)
Date: Fri, 21 Aug 2020 03:07:21 -0500
From: Glenn Washburn <development@HIDDEN>
To: Karl Berry <karl@HIDDEN>
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Message-ID: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
In-Reply-To: <202008200117.07K1HDu4021525@HIDDEN>
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
 <202008200117.07K1HDu4021525@HIDDEN>
X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 42883
Cc: development@HIDDEN, 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Karl,

On Wed, 19 Aug 2020 19:17:13 -0600
Karl Berry <karl@HIDDEN> wrote:

> Hi Glenn,
> 
>     I noticed that when specifying a subset of tests to run via the
> TESTS environment variable, I could have all tests complete
> successfully
> 
> Thanks for the report. That does seem suboptimal, but on the other
> hand, subsets of tests are a complication. Does it work to override
> SUBDIRS along with TESTS? As in (thanks to Jim):
>   env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Yes this does work in the special case of grub, thanks for the
suggestion. I think at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.  Setting SUBDIRS= is just disabling the
makefile recusion.

If I had a test in grub-core/unittest1 and I wanted to run just the
test subset of unittest1 and grub_cmd_echo, how would I do that such
that make returns success?  If I run `TESTS="unittest1 grub_cmd_echo"
make -e check', make will fail even if both test individually succeed
because the recursive make check in grub-core will try to run
grub_cmd_echo (in addition to unittest1) and fail.

Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR, how does one choose to run the X in SUBDIR and not the
X in root?  Right now TESTS=X will run both tests.  Sure one could do
`TESTS=X make -e -C SUBDIR check', but then again you're just bypassing
the makefile recursion.

And thanks Karl for helping to maintain this venerable and essential,
yet not super glamorous project.

Glenn





Information forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.

Message received at 42883 <at> debbugs.gnu.org:


Received: (at 42883) by debbugs.gnu.org; 20 Aug 2020 01:17:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Aug 19 21:17:17 2020
Received: from localhost ([127.0.0.1]:41338 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k8ZCf-0000XN-1v
	for submit <at> debbugs.gnu.org; Wed, 19 Aug 2020 21:17:17 -0400
Received: from freefriends.org ([96.88.95.60]:60268)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1k8ZCd-0000Vt-Be
 for 42883 <at> debbugs.gnu.org; Wed, 19 Aug 2020 21:17:16 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 07K1HD1w021526
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); 
 Wed, 19 Aug 2020 19:17:14 -0600
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 07K1HDu4021525;
 Wed, 19 Aug 2020 19:17:13 -0600
Date: Wed, 19 Aug 2020 19:17:13 -0600
Message-Id: <202008200117.07K1HDu4021525@HIDDEN>
From: Karl Berry <karl@HIDDEN>
To: development@HIDDEN
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
In-Reply-To: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 42883
Cc: 42883 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Glenn,

    I noticed that when specifying a subset of tests to run via the TESTS
    environment variable, I could have all tests complete successfully

Thanks for the report. That does seem suboptimal, but on the other hand,
subsets of tests are a complication. Does it work to override SUBDIRS
along with TESTS? As in (thanks to Jim):
  env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Thanks,
Karl




Information forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 16 Aug 2020 06:23:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Aug 16 02:23:47 2020
Received: from localhost ([127.0.0.1]:56704 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k7C55-0001y9-0m
	for submit <at> debbugs.gnu.org; Sun, 16 Aug 2020 02:23:47 -0400
Received: from lists.gnu.org ([209.51.188.17]:42370)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <development@HIDDEN>) id 1k7BTV-000760-4d
 for submit <at> debbugs.gnu.org; Sun, 16 Aug 2020 01:44:57 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34678)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <development@HIDDEN>)
 id 1k7BTU-0001Ua-Mg
 for bug-automake@HIDDEN; Sun, 16 Aug 2020 01:44:56 -0400
Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]:35119)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <development@HIDDEN>)
 id 1k7BTS-0003WM-Ax
 for bug-automake@HIDDEN; Sun, 16 Aug 2020 01:44:56 -0400
Received: by mail-pf1-x434.google.com with SMTP id d188so6551638pfd.2
 for <bug-automake@HIDDEN>; Sat, 15 Aug 2020 22:44:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=efficientek-com.20150623.gappssmtp.com; s=20150623;
 h=date:from:to:subject:message-id:reply-to:mime-version;
 bh=qo1Uila4Xs/Bmm/qx2thvf90eI7/bDrYIYOhwHKRGRc=;
 b=LXNqmt+Cpta+m1zcdxlMqSqPvq08rJG+ig78mwoqtq7BQLLQ8ALXDIMsZYWF9eHh0i
 OUqC+0UbtMU64lpIB1ICh3mpkbVIs8sM/v7LtyYmIyTWnUgvKf0Cc2ulypq+wPdaZZOT
 18TvSh4BTFfTv46xWroVajsnh5AgyjCwUATYXurTqGTXyMpNpZxfeD8cCMZWXjjSO/R9
 80ebf3j8SAQ1iDcqeNxxfApP5BdWAq+CRwJa1ufY9nGHHQCW3J1O3wDJ3iZ0Ce4Kg/8E
 Z9PkfMXVnuilwPKGVEzaZeq2faHvvQ0mJhCwf2SkuaRgt2IwUqFRlFGmsTzdFUqt8HP1
 DlsQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:subject:message-id:reply-to
 :mime-version;
 bh=qo1Uila4Xs/Bmm/qx2thvf90eI7/bDrYIYOhwHKRGRc=;
 b=YtF6wuo+lEehaZ2yLXR+YFq/cj8+yfFuZTManT2z3U3sLmfY8uhMiq5n9vtvrGHTn3
 h5D4ypLV9ZNqaazi1MUheWT0vW6TqriIZorouuDA7P0KgiSUQxChvLiF9GL7Y0UXVMs2
 +iDWWNMtoO0wv/G8yAuAek8J5g0yEQvb6XXV/ysoHhJIgD7/H4oBwXGI5VvFIVWGAaL7
 2MM8+cuGJIV1y/TthM4WPlb1uU9vxu9jHM7z5al8QGAPDTNihIpWV8V0vxPHV1oA2StZ
 S5c13x0cZhZPcUKrsD+HLTW7uAD/Z4jyfdQ63qh//nJg2plYqdyX9L8vq7sw8j+K76Lm
 yiHw==
X-Gm-Message-State: AOAM532FP0Jf++TSihKe/H+j1UPtMEPve2+mrSrc/Juu8/JZ2daOca0a
 jycLrnwdhRVC7o6KQAN6xn9ynZAr1rfMhg==
X-Google-Smtp-Source: ABdhPJxGLSKN6gl1ESFE+3FWkrZlAsMjGyqHSIACAyp4ETIQrWqRQrVC1WvgpovkaJH+PIUSzAcC1g==
X-Received: by 2002:a63:7542:: with SMTP id f2mr6190400pgn.151.1597556691937; 
 Sat, 15 Aug 2020 22:44:51 -0700 (PDT)
Received: from crass-HP-ZBook-15-G2 ([136.49.44.103])
 by smtp.gmail.com with ESMTPSA id 27sm12383250pgk.89.2020.08.15.22.44.50
 for <bug-automake@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 15 Aug 2020 22:44:51 -0700 (PDT)
Date: Sun, 16 Aug 2020 00:44:41 -0500
From: Glenn Washburn <development@HIDDEN>
To: bug-automake@HIDDEN
Subject: Selecting a subset of tests in recursive make causes unnecessary
 failures
Message-ID: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MP_/sT4As_qzOipHo49O6kH/coI"
Received-SPF: pass client-ip=2607:f8b0:4864:20::434;
 envelope-from=development@HIDDEN; helo=mail-pf1-x434.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Sun, 16 Aug 2020 02:23:45 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: development@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,

I've been working on some automake tests for grub, and ran in to a bug.
I noticed that when specifying a subset of tests to run via the TESTS
environment variable, I could have all tests complete successfully and
yet still have make return an error.

Here's a sequence of commands that should reproduce the error:
   git clone https://git.savannah.gnu.org/git/grub.git
   cd grub
   ./bootstrap
   ./configure
   make
   env TESTS=grub_cmd_echo make -e check VERBOSE=yes

It turns out that make check is being recursively run in the grub-core
directory, as expected.  The TESTS environment variable is the same in
both invocations, also as expected.  However, the test program is not in
the grub-core subdirectory.  So the test is run successfully in the root
make, and not found in the subdirectory, thus generating an error. In
fact, grub has no tests in the grub-core subdirectory, but the harness
is run because its part of automake, even though I do not need or want
it to.

I've attached a patch which fixes this issue by ignoring tests which do
not exist. I'm not sure this patch takes care of everything needed, for
instance EXEEXT, to check for the existence of the test program. Grub
does not use any TEST_EXTENSIONS, so this works.

This will change make it so TESTS which should exist, but do not, will
not cause an error, whereas currently it will.  I'm not sure how this
situation would arise, because make should require the existence as a
dependency.  If there is an error in the TESTS variable (eg. a typo in
a test name), this will not cause an error either, which I deem
undesirable. However, I think this is an acceptable trade-off.

Perhaps a better solution would be to not recursively run make check if
TESTS is defined and run all given tests from the root make. Tests
which are run in recursive make invocations when TESTS is not defined
can be run via TESTS by specifying a path component (eg.
TESTS="grub-core/test-program").  This could be a problem if test
programs assume they are run from the directory in which they reside
(in which case, we could chdir to the directory and run the program).

Any thoughts?
Glenn


--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=fix-recursive-subset-test.patch

diff --git a/lib/am/check.am b/lib/am/check.am
index 449708742..94021775e 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -416,8 +416,8 @@ check-TESTS: %CHECK_DEPS%
 ## we rely on .PHONY to work portably.
 	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
 	@set +e; $(am__set_TESTS_bases); \
-	log_list=`for i in $$bases; do echo $$i.log; done`; \
-	trs_list=`for i in $$bases; do echo $$i.trs; done`; \
+	log_list=`for i in $$bases; do test -x $$i && echo $$i.log; done`; \
+	trs_list=`for i in $$bases; do test -x $$i && echo $$i.trs; done`; \
 ## Remove newlines and normalize whitespace.  Trailing (and possibly
 ## leading) whitespace is known to cause segmentation faults on
 ## Solaris 10 XPG4 make.

--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=grub-tests-error.txt

make[3]: Leaving directory '/home/user/grub.git'
make  check-TESTS
make[3]: Entering directory '/home/user/grub.git'
make[4]: Entering directory '/home/user/grub.git'
PASS: grub_cmd_echo
============================================================================
Testsuite summary for GRUB 2.05
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/user/grub.git'
make[3]: Leaving directory '/home/user/grub.git'
make[2]: Leaving directory '/home/user/grub.git'
Making check in grub-core
make[2]: Entering directory '/home/user/grub.git/grub-core'
make  check-am
make[3]: Entering directory '/home/user/grub.git/grub-core'
make
make[4]: Entering directory '/home/user/grub.git/grub-core'
make  all-am
make[5]: Entering directory '/home/user/grub.git/grub-core'
make[5]: Nothing to be done for 'all-am'.
make[5]: Leaving directory '/home/user/grub.git/grub-core'
make[4]: Leaving directory '/home/user/grub.git/grub-core'
make  check-TESTS
make[4]: Entering directory '/home/user/grub.git/grub-core'
make[5]: Entering directory '/home/user/grub.git/grub-core'
make[6]: Entering directory '/home/user/grub.git/grub-core'
make[6]: Nothing to be done for 'grub_cmd_echo.log'.
make[6]: Leaving directory '/home/user/grub.git/grub-core'
fatal: making test-suite.log: failed to create grub_cmd_echo.trs
fatal: making test-suite.log: failed to create grub_cmd_echo.log
make[5]: *** [Makefile:43107: test-suite.log] Error 1
make[5]: Leaving directory '/home/user/grub.git/grub-core'
make[4]: *** [Makefile:43216: check-TESTS] Error 2
make[4]: Leaving directory '/home/user/grub.git/grub-core'
make[3]: *** [Makefile:43283: check-am] Error 2
make[3]: Leaving directory '/home/user/grub.git/grub-core'
make[2]: *** [Makefile:43285: check] Error 2
make[2]: Leaving directory '/home/user/grub.git/grub-core'
make[1]: *** [Makefile:11920: check-recursive] Error 1
make[1]: Leaving directory '/home/user/grub.git'
make: *** [Makefile:12944: check] Error 2

--MP_/sT4As_qzOipHo49O6kH/coI--




Acknowledgement sent to development@HIDDEN:
New bug report received and forwarded. Copy sent to bug-automake@HIDDEN. Full text available.
Report forwarded to bug-automake@HIDDEN:
bug#42883; Package automake. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sun, 22 Nov 2020 02:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.