GNU logs - #42883, boring messages

Message sent to bug-automake@HIDDEN:

Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
Resent-From: Glenn Washburn <development@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Sun, 16 Aug 2020 06:24:01 +0000
Resent-Message-ID: <handler.42883.B.15975590287578 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 42883 <at> debbugs.gnu.org
Reply-To: development@HIDDEN
Date: Sun, 16 Aug 2020 00:44:41 -0500
From: Glenn Washburn <development@HIDDEN>
Message-ID: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MP_/sT4As_qzOipHo49O6kH/coI"
Received-SPF: pass client-ip=2607:f8b0:4864:20::434;
 envelope-from=development@HIDDEN; helo=mail-pf1-x434.google.com
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Mailman-Approved-At: Sun, 16 Aug 2020 02:23:45 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,

I've been working on some automake tests for grub, and ran in to a bug.
I noticed that when specifying a subset of tests to run via the TESTS
environment variable, I could have all tests complete successfully and
yet still have make return an error.

Here's a sequence of commands that should reproduce the error:
   git clone https://git.savannah.gnu.org/git/grub.git
   cd grub
   ./bootstrap
   ./configure
   make
   env TESTS=grub_cmd_echo make -e check VERBOSE=yes

It turns out that make check is being recursively run in the grub-core
directory, as expected.  The TESTS environment variable is the same in
both invocations, also as expected.  However, the test program is not in
the grub-core subdirectory.  So the test is run successfully in the root
make, and not found in the subdirectory, thus generating an error. In
fact, grub has no tests in the grub-core subdirectory, but the harness
is run because its part of automake, even though I do not need or want
it to.

I've attached a patch which fixes this issue by ignoring tests which do
not exist. I'm not sure this patch takes care of everything needed, for
instance EXEEXT, to check for the existence of the test program. Grub
does not use any TEST_EXTENSIONS, so this works.

This will change make it so TESTS which should exist, but do not, will
not cause an error, whereas currently it will.  I'm not sure how this
situation would arise, because make should require the existence as a
dependency.  If there is an error in the TESTS variable (eg. a typo in
a test name), this will not cause an error either, which I deem
undesirable. However, I think this is an acceptable trade-off.

Perhaps a better solution would be to not recursively run make check if
TESTS is defined and run all given tests from the root make. Tests
which are run in recursive make invocations when TESTS is not defined
can be run via TESTS by specifying a path component (eg.
TESTS="grub-core/test-program").  This could be a problem if test
programs assume they are run from the directory in which they reside
(in which case, we could chdir to the directory and run the program).

Any thoughts?
Glenn


--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=fix-recursive-subset-test.patch

diff --git a/lib/am/check.am b/lib/am/check.am
index 449708742..94021775e 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -416,8 +416,8 @@ check-TESTS: %CHECK_DEPS%
 ## we rely on .PHONY to work portably.
 	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
 	@set +e; $(am__set_TESTS_bases); \
-	log_list=`for i in $$bases; do echo $$i.log; done`; \
-	trs_list=`for i in $$bases; do echo $$i.trs; done`; \
+	log_list=`for i in $$bases; do test -x $$i && echo $$i.log; done`; \
+	trs_list=`for i in $$bases; do test -x $$i && echo $$i.trs; done`; \
 ## Remove newlines and normalize whitespace.  Trailing (and possibly
 ## leading) whitespace is known to cause segmentation faults on
 ## Solaris 10 XPG4 make.

--MP_/sT4As_qzOipHo49O6kH/coI
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=grub-tests-error.txt

make[3]: Leaving directory '/home/user/grub.git'
make  check-TESTS
make[3]: Entering directory '/home/user/grub.git'
make[4]: Entering directory '/home/user/grub.git'
PASS: grub_cmd_echo
============================================================================
Testsuite summary for GRUB 2.05
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[4]: Leaving directory '/home/user/grub.git'
make[3]: Leaving directory '/home/user/grub.git'
make[2]: Leaving directory '/home/user/grub.git'
Making check in grub-core
make[2]: Entering directory '/home/user/grub.git/grub-core'
make  check-am
make[3]: Entering directory '/home/user/grub.git/grub-core'
make
make[4]: Entering directory '/home/user/grub.git/grub-core'
make  all-am
make[5]: Entering directory '/home/user/grub.git/grub-core'
make[5]: Nothing to be done for 'all-am'.
make[5]: Leaving directory '/home/user/grub.git/grub-core'
make[4]: Leaving directory '/home/user/grub.git/grub-core'
make  check-TESTS
make[4]: Entering directory '/home/user/grub.git/grub-core'
make[5]: Entering directory '/home/user/grub.git/grub-core'
make[6]: Entering directory '/home/user/grub.git/grub-core'
make[6]: Nothing to be done for 'grub_cmd_echo.log'.
make[6]: Leaving directory '/home/user/grub.git/grub-core'
fatal: making test-suite.log: failed to create grub_cmd_echo.trs
fatal: making test-suite.log: failed to create grub_cmd_echo.log
make[5]: *** [Makefile:43107: test-suite.log] Error 1
make[5]: Leaving directory '/home/user/grub.git/grub-core'
make[4]: *** [Makefile:43216: check-TESTS] Error 2
make[4]: Leaving directory '/home/user/grub.git/grub-core'
make[3]: *** [Makefile:43283: check-am] Error 2
make[3]: Leaving directory '/home/user/grub.git/grub-core'
make[2]: *** [Makefile:43285: check] Error 2
make[2]: Leaving directory '/home/user/grub.git/grub-core'
make[1]: *** [Makefile:11920: check-recursive] Error 1
make[1]: Leaving directory '/home/user/grub.git'
make: *** [Makefile:12944: check] Error 2

--MP_/sT4As_qzOipHo49O6kH/coI--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: development@HIDDEN
Subject: bug#42883: Acknowledgement (Selecting a subset of tests in
 recursive make causes unnecessary failures)
Message-ID: <handler.42883.B.15975590287578.ack <at> debbugs.gnu.org>
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
X-Gnu-PR-Message: ack 42883
X-Gnu-PR-Package: automake
Reply-To: 42883 <at> debbugs.gnu.org
Date: Sun, 16 Aug 2020 06:24:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-automake@HIDDEN

If you wish to submit further information on this problem, please
send it to 42883 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
42883: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D42883
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-automake@HIDDEN:

Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
Resent-From: Karl Berry <karl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Thu, 20 Aug 2020 01:18:02 +0000
Resent-Message-ID: <handler.42883.B42883.15978862372087 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: development@HIDDEN
Cc: 42883 <at> debbugs.gnu.org
Date: Wed, 19 Aug 2020 19:17:13 -0600
Message-Id: <202008200117.07K1HDu4021525@HIDDEN>
From: Karl Berry <karl@HIDDEN>
In-Reply-To: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Glenn,

    I noticed that when specifying a subset of tests to run via the TESTS
    environment variable, I could have all tests complete successfully

Thanks for the report. That does seem suboptimal, but on the other hand,
subsets of tests are a complication. Does it work to override SUBDIRS
along with TESTS? As in (thanks to Jim):
  env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Thanks,
Karl

Message sent to bug-automake@HIDDEN:

Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
Resent-From: Glenn Washburn <development@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 21 Aug 2020 08:08:01 +0000
Resent-Message-ID: <handler.42883.B42883.15979972538452 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Karl Berry <karl@HIDDEN>
Cc: development@HIDDEN, 42883 <at> debbugs.gnu.org
Date: Fri, 21 Aug 2020 03:07:21 -0500
From: Glenn Washburn <development@HIDDEN>
Message-ID: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
In-Reply-To: <202008200117.07K1HDu4021525@HIDDEN>
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
 <202008200117.07K1HDu4021525@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Karl,

On Wed, 19 Aug 2020 19:17:13 -0600
Karl Berry <karl@HIDDEN> wrote:

> Hi Glenn,
> 
>     I noticed that when specifying a subset of tests to run via the
> TESTS environment variable, I could have all tests complete
> successfully
> 
> Thanks for the report. That does seem suboptimal, but on the other
> hand, subsets of tests are a complication. Does it work to override
> SUBDIRS along with TESTS? As in (thanks to Jim):
>   env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Yes this does work in the special case of grub, thanks for the
suggestion. I think at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.  Setting SUBDIRS= is just disabling the
makefile recusion.

If I had a test in grub-core/unittest1 and I wanted to run just the
test subset of unittest1 and grub_cmd_echo, how would I do that such
that make returns success?  If I run `TESTS="unittest1 grub_cmd_echo"
make -e check', make will fail even if both test individually succeed
because the recursive make check in grub-core will try to run
grub_cmd_echo (in addition to unittest1) and fail.

Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR, how does one choose to run the X in SUBDIR and not the
X in root?  Right now TESTS=X will run both tests.  Sure one could do
`TESTS=X make -e -C SUBDIR check', but then again you're just bypassing
the makefile recursion.

And thanks Karl for helping to maintain this venerable and essential,
yet not super glamorous project.

Glenn

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
Resent-From: Karl Berry <karl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Fri, 21 Aug 2020 22:06:01 +0000
Resent-Message-ID: <handler.42883.B42883.159804752831816 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 42883
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: development@HIDDEN
Cc: 42883 <at> debbugs.gnu.org
Received: via spool by 42883-submit <at> debbugs.gnu.org id=B42883.159804752831816
          (code B ref 42883); Fri, 21 Aug 2020 22:06:01 +0000
Received: (at 42883) by debbugs.gnu.org; 21 Aug 2020 22:05:28 +0000
Received: from localhost ([127.0.0.1]:48454 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1k9FA8-0008H6-N4
	for submit <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:28 -0400
Received: from freefriends.org ([96.88.95.60]:35636)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1k9FA6-0008Gw-31
 for 42883 <at> debbugs.gnu.org; Fri, 21 Aug 2020 18:05:27 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 07LM5OA1013326
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); 
 Fri, 21 Aug 2020 16:05:25 -0600
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 07LM5OlR013325;
 Fri, 21 Aug 2020 16:05:24 -0600
Date: Fri, 21 Aug 2020 16:05:24 -0600
Message-Id: <202008212205.07LM5OlR013325@HIDDEN>
From: Karl Berry <karl@HIDDEN>
In-Reply-To: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

    at a minimum this behavior should be documented
    with the test subsets documentation and this work around mentioned.

Yes, I agree. Contemplating.

    Ideally, the subset tests feature design should be updated to play nice
    with makefile recursion.

I think I agree again, but ... not sure ...

    Another issue is that if there is a test X in the root makefile and a
    test X in SUBDIR

Isn't that fundamentally not supported, or supportable? I admit I don't
actually know if there's any history about it, but using the same name
for two tests in different places just seems like a basic source of
confusion. As far as I know, the TESTS mechanism just isn't set up to
be able to specify arbitrary test names in arbitrary directories.

Admittedly if it is not supported, it could at least be detected and
complained about. On the other hand, I suppose it works when not
explicitly specifying tests to run, which is 99% of the time. So, once
again ... I don't know. Nothing is ever simple :).

Jim, Paul, anyone? --thanks, karl.

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
Resent-From: Glenn Washburn <development@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Mon, 24 Aug 2020 15:55:02 +0000
Resent-Message-ID: <handler.42883.B42883.15982844997878 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 42883
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: Karl Berry <karl@HIDDEN>
Cc: 42883 <at> debbugs.gnu.org
Reply-To: development@HIDDEN
Received: via spool by 42883-submit <at> debbugs.gnu.org id=B42883.15982844997878
          (code B ref 42883); Mon, 24 Aug 2020 15:55:02 +0000
Received: (at 42883) by debbugs.gnu.org; 24 Aug 2020 15:54:59 +0000
Received: from localhost ([127.0.0.1]:58981 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kAEoF-000230-EZ
	for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:59 -0400
Received: from mail-qt1-f170.google.com ([209.85.160.170]:46105)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <development@HIDDEN>) id 1kAEoD-00022i-Qi
 for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 11:54:58 -0400
Received: by mail-qt1-f170.google.com with SMTP id o22so6461504qtt.13
 for <42883 <at> debbugs.gnu.org>; Mon, 24 Aug 2020 08:54:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=efficientek-com.20150623.gappssmtp.com; s=20150623;
 h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to
 :mime-version:content-transfer-encoding;
 bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=;
 b=vx7Tlg/lpCMapybAPMn8VWTYNmY53bHghqDZqd3/KVSAXkKcsTzAflBLYksyxGX1//
 MOo8006GiiwXZewcpAMzh3Flxd9MYg52Om5vDKetVG7ZLtW3pQcxQpGx5WUc1eCW/ePw
 /glUPHCFJ9fyj+82fjdCiNycPfkEdgfSlBXcZMZ9cJX4bKFQkU1l284166AdLQ5yGPUo
 JQcCGL5Oki+aXkLVzB1T6G6j2QLTuB6TEY3BJLRIU7zBf9BGfFvqZWBssArRClRn4ojr
 HIENLxdd8M2wrjaK+VOIWgHNUIKMBN6qbuq0El6jbnc9DEloODQGNTvDXmmc7SYDtUkz
 LpBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
 :references:reply-to:mime-version:content-transfer-encoding;
 bh=x6sbUWVmr0xHdZ9E1fd3z3vs+7LhWuR+8MPf1FyANdU=;
 b=T4TQ9S+VhrRnrZCSJrJzV9l3JFz0DzfiOW+bsbA90OzEtWrDqKlLDGHJijEpPqDDYL
 LUbY8EVmRpjOfm73Z+wRB+emI/FqG7WYurWh4HEqK6fXuzCk6cXg5ISZl5AaO5cMsd+O
 4VugKacgAJVDUTOt/nli/TX6HtPL6Vc6iVzIKYEl01j5hpzqiKuK3TVcpFlyyOgR9p1j
 XZclADQd+YVBu5xv5ZuRb1z9XUsfwVyf3DWsqVS5Z4FJ8l0qg/EJ3CFrGbQ+EpvmrTGk
 aLznTVkih2IB4yaram9wuuqE5Xl93cAuskqmk8k2a3SWKYy06jzUkQ9Xc0QdQqZyM2mz
 WDlw==
X-Gm-Message-State: AOAM532cjar+t/gH5rXQ6v661/GyBHAbFrDOErefZG1QUHW8Y1kTI7e0
 +23raAd8Y32/WUfT9Umw5YAiEg7lI/QSpQ==
X-Google-Smtp-Source: ABdhPJzX441otFAVTZZkkCoD39saIqplyaRj54GpkjLXUv8rdd0K187Io9W83JimZl0VHHIJjdjOJg==
X-Received: by 2002:ac8:6647:: with SMTP id j7mr5165230qtp.335.1598284492134; 
 Mon, 24 Aug 2020 08:54:52 -0700 (PDT)
Received: from crass-HP-ZBook-15-G2 ([2605:a601:aaec:9b00:5122:8a61:cec2:28f4])
 by smtp.gmail.com with ESMTPSA id h6sm2856366qtk.47.2020.08.24.08.54.51
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 24 Aug 2020 08:54:51 -0700 (PDT)
Date: Mon, 24 Aug 2020 10:54:48 -0500
From: Glenn Washburn <development@HIDDEN>
Message-ID: <20200824105448.2f33d717@crass-HP-ZBook-15-G2>
In-Reply-To: <202008212205.07LM5OlR013325@HIDDEN>
References: <20200821030721.63ffbbf2@crass-HP-ZBook-15-G2>
 <202008212205.07LM5OlR013325@HIDDEN>
X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Fri, 21 Aug 2020 16:05:24 -0600
Karl Berry <karl@HIDDEN> wrote:

>> Another issue is that if there is a test X in the root makefile
>> and a test X in SUBDIR
> 
> Isn't that fundamentally not supported, or supportable? I admit I
> don't actually know if there's any history about it, but using the
> same name for two tests in different places just seems like a basic
> source of confusion. As far as I know, the TESTS mechanism just isn't
> set up to be able to specify arbitrary test names in arbitrary
> directories.

I'm not sure if I was sufficiently clear in my example. The example is
meant to be such that the test X in root and SUBDIR are tests that
would be run when not explicitly specifying tests.  So its to be able
to run any test that would be run when not explicitly specifying tests.

I think the way to support this is to have TESTS be a list of test
paths relative to the root makefile.  So in the example above, I would
specify TESTS="X SUBDIR/X".  Before running the recursive make in
SUBDIR, the TESTS variable would be filtered to only include tests with
a relative path of the sub directory.  Also, the root makefile would
not run tests that have a directory component to the path.

I would like to get a rough idea of if this would be an approved of
approach so that I or someone else could take a stab at implementing it
with the hopes of the changes being accepted.

Glenn

Message sent to bug-automake@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures
References: <20200816004441.4ca4b667@crass-HP-ZBook-15-G2>
Resent-From: Karl Berry <karl@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-automake@HIDDEN
Resent-Date: Tue, 25 Aug 2020 01:40:02 +0000
Resent-Message-ID: <handler.42883.B42883.159831957318241 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 42883
X-GNU-PR-Package: automake
X-GNU-PR-Keywords: 
To: development@HIDDEN
Cc: 42883 <at> debbugs.gnu.org
Received: via spool by 42883-submit <at> debbugs.gnu.org id=B42883.159831957318241
          (code B ref 42883); Tue, 25 Aug 2020 01:40:02 +0000
Received: (at 42883) by debbugs.gnu.org; 25 Aug 2020 01:39:33 +0000
Received: from localhost ([127.0.0.1]:59672 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kANvx-0004k7-I9
	for submit <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:33 -0400
Received: from freefriends.org ([96.88.95.60]:52784)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1kANvr-0004jv-MN
 for 42883 <at> debbugs.gnu.org; Mon, 24 Aug 2020 21:39:31 -0400
X-Envelope-From: karl@HIDDEN
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 07P1dQfR027448
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); 
 Mon, 24 Aug 2020 19:39:26 -0600
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 07P1dPl9027447;
 Mon, 24 Aug 2020 19:39:25 -0600
Date: Mon, 24 Aug 2020 19:39:25 -0600
Message-Id: <202008250139.07P1dPl9027447@HIDDEN>
From: Karl Berry <karl@HIDDEN>
In-Reply-To: <20200824105448.2f33d717@crass-HP-ZBook-15-G2>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Glenn,

    The example is meant to be such that the test X in root and SUBDIR
    are tests that would be run when not explicitly specifying tests.

Yep, I got that.

    So its to be able to run any test that would be run when not
    explicitly specifying tests.

Clearly yes, it would be desirable in principle to be able to name tests
anything you want, in any directory you want, and then be able to
specify exactly the individual tests you want to run, regardless of
names or (sub)directories. I'm just not sure how feasible it is to
implement.

    I think the way to support this is to have TESTS be a list of test
    paths relative to the root makefile.  

Isn't it already that way?

    So in the example above, I would
    specify TESTS="X SUBDIR/X".

To run a specific test in automake now, what I came up with is 
make check TESTS=t/sometest.sh

Although the manual says it can be overridden in the environment
(Parallel Test Harness node):
  env TESTS="foo.test bar.test" make -e check
this has never worked for me. (Neither foo.sh or t/foo.sh.) I have not
tried to discern what I was doing wrong. (I feel pretty sure it was me
doing something wrong and not Automake, at least.)

I realize the case you're concerned about is when both ./sometest.sh and
t/sometest.sh are present. My gut reaction is that it would be more
reliable to detect that case and complain about it than try to support
it. But ok, never mind, proceeding along the ideal lines ...

    Before running the recursive make in SUBDIR, the TESTS variable
    would be filtered to only include tests with a relative path of the
    sub directory.  Also, the root makefile would not run tests that
    have a directory component to the path.

It sounds sensible to me, but regrettably, I just don't have a great
overall grip on the code, having come to the implementation side of
Automake rather late. Jim or Paul would have better things to say.
Unfortunately I think they are busy with other things (which is why I'm
here in the first place ... anyway ...).

    I would like to get a rough idea of if this would be an approved of
    approach so that I or someone else could take a stab at implementing it
    with the hopes of the changes being accepted.

In general, the top priority for any change in Automake is to not break
existing practice. If you can devise a change that, for starters, still
passes all the existing Automake tests, then I surmise there's a good
shot.

As you may infer from my TESTS= stuff above, exactly what invariants
have to be maintained to keep compatibility in this case is not clear to
me :(.

(Also, as one would expect, the change has to be comprehensible and
maintainable, but that's nothing special to Automake.)

Regarding your previous patch, it actually seemed pretty plausible to me
(modulo EXEEXT and other such, as you noted; not a big deal), except
that it's a shame to lose the typo checking. There are over a thousand
tests in Automake, and no doubt that many or more in other packages. So
typos are inevitable. Seems like some sort of separate check for them
would need to be available, going down that road.

Thanks for following up. I hope you are sufficiently interested to spend
some more time on it. --best, karl.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 22 Nov 2020 02:09:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Nov 21 21:09:10 2020
Received: from localhost ([127.0.0.1]:46717 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kgeoQ-0000G7-Gb
	for submit <at> debbugs.gnu.org; Sat, 21 Nov 2020 21:09:10 -0500
Received: from freefriends.org ([96.88.95.60]:33368)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <karl@HIDDEN>) id 1kgeoH-0000F6-3R
 for control <at> debbugs.gnu.org; Sat, 21 Nov 2020 21:09:01 -0500
X-Envelope-From: karl@HIDDEN
X-Envelope-To: <control <at> debbugs.gnu.org>
Received: from freefriends.org (freefriends.org [96.88.95.60])
 by freefriends.org (8.14.7/8.14.7) with ESMTP id 0AM28x9Q003742
 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT)
 for <control <at> debbugs.gnu.org>; Sat, 21 Nov 2020 19:09:00 -0700
Received: (from apache@localhost)
 by freefriends.org (8.14.7/8.14.7/Submit) id 0AM28xIm003741;
 Sat, 21 Nov 2020 19:08:59 -0700
Message-Id: <202011220208.0AM28xIm003741@HIDDEN>
Date: Thu, 19 Nov 2020 18:05:54 -0800
From: Karl Berry <karl@HIDDEN>
To: control <at> debbugs.gnu.org
Subject: 
X-Spam-Score: -0.3 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.3 (-)

tags 42883 + help confirmed

Last modified: Sun, 22 Nov 2020 02:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.