GNU bug report logs - #11287
Various issues with the test suite framework

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11287 in the body.
You can then email your comments to 11287 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Automake Bugs <bug-automake <at> gnu.org>
Subject: Various issues with the test suite framework
Date: Fri, 20 Apr 2012 13:13:58 +0200

Hi!

I have seen that check-html will be removed, or rather moved
into the contrib part, but there are a few issues:

- the target is not declared recursive, so one has to write
 the bouncing target herself.

- because of that, the "naive" implementation of check-html
 that just bounces the right directories will miss the
 dependencies that check features (in other words, make
 check properly recursively builds before, while check-html
 would just go into the typical tests/ directory, and then
 bounce to "check", so it completely missed the recursion
 in the other directories and their dependencies).

- RECURSIVE_TARGETS cannot be highjacked by the users, that's
 sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
 RECURSIVE_TARGETS so that we can add our own targets that
 traverse the whole package?  Actually maybe this should
 be an Automake option to add recursive targets, and
 then TARGET-am and TARGET-recursive would be generated and
 bound to the proper TARGET.

- the .log.html recipe features $$RST2HTML, but in '', so it is
 useless.  And anyway $(RST2HTML) already does the job.

- "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.

- the test compiler does not appear as a dependency for
 the test logs that use it.  This is most useful when
 RECHECK_LOG =.

- it would be equally useful to be able to add dependencies
 to a category of test drivers.  Say, with the documented
 example:

    TESTS = foo.pl bar.py baz
    TEST_EXTENSIONS = .pl .py
    PL_LOG_COMPILER = $(PERL)
    AM_PL_LOG_FLAGS = -w
    PY_LOG_COMPILER = $(PYTHON)
    AM_PY_LOG_FLAGS = -v
    LOG_COMPILER = ./wrapper-script
    AM_LOG_FLAGS = -d

 be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
 LOG_DEPENDENCIES.  I don't think EXTRA dependencies would
 be useful.

Cheers!

Message #8 received at 11287 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 11287 <at> debbugs.gnu.org
Subject: Re: bug#11287: Various issues with the test suite framework
Date: Tue, 24 Apr 2012 11:42:05 +0200

On 04/20/2012 01:13 PM, Akim Demaille wrote:
> Hi!
>
Hi Akim, sorry for the delay.

> I have seen that check-html will be removed, or rather moved
> into the contrib part, but there are a few issues:
>
> - the target is not declared recursive, so one has to write
>  the bouncing target herself.
> 
> - because of that, the "naive" implementation of check-html
>  that just bounces the right directories will miss the
>  dependencies that check features (in other words, make
>  check properly recursively builds before, while check-html
>  would just go into the typical tests/ directory, and then
>  bounce to "check", so it completely missed the recursion
>  in the other directories and their dependencies).
>
Wasn't this issue already present in Automake 1.11.1?  (So, not a
regression, unless I'm mistaken).  In any case, a test case
exposing the issue would be appreciated, if you have one at hand.

> - the .log.html recipe features $$RST2HTML, but in '', so it is
>  useless.  And anyway $(RST2HTML) already does the job.
>
> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>
None of these are covered by the Automake test suite sadly, and I
see them as really low low priority anyway, so I'm not going to
tackle them right now.  But I'll gladly accept patches by anyone
who has more knowledge or interest than me in this area ;-)

> - RECURSIVE_TARGETS cannot be highjacked by the users, that's
>  sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
>  RECURSIVE_TARGETS so that we can add our own targets that
>  traverse the whole package?
>
There had been an attempt to offer an interface for user-defined
recursive rules already:

  <http://thread.gmane.org/gmane.comp.sysutils.automake.general/11890>

But unfortunately the discussion stagnated and died out, and I don't
have time to resurrect it nor the relevant patches ATM.  Still, it
would be a cool feature to have, so any help would be appreciated.

>  Actually maybe this should
>  be an Automake option to add recursive targets, and
>  then TARGET-am and TARGET-recursive would be generated and
>  bound to the proper TARGET.
> 

> - the test compiler does not appear as a dependency for
>   the test logs that use it.
>
This makes sense, since the test compiler is not required to be an
in-project script, and not even to be specified as an absolute path
-- it can be a command like "sh" or "/usr/bin/env perl".

> - it would be equally useful to be able to add dependencies
>   to a category of test drivers.  Say, with the documented
>   example:
> 
>     TESTS = foo.pl bar.py baz
>     TEST_EXTENSIONS = .pl .py
>     PL_LOG_COMPILER = $(PERL)
>     AM_PL_LOG_FLAGS = -w
>     PY_LOG_COMPILER = $(PYTHON)
>     AM_PY_LOG_FLAGS = -v
>     LOG_COMPILER = ./wrapper-script
>     AM_LOG_FLAGS = -d
> 
>  be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>  LOG_DEPENDENCIES.
>
This seems like a cool feature, but not trivial to implement in the current
Automake code base.  Still, I have some half-finished patches to rework
the parallel-tests harness in Automake-NG (reviews welcome, BTW ;-), and
once those are in place, implementing your proposed feature in Automake-NG
might be very easy.

> I don't think EXTRA dependencies would be useful.
>
Right.  If we'll ever need automatically computed dependencies, we should
name them AM_PL_LOG_DEPENDENCIES etc., without impinging on the user
namespace.

Thanks,
  Stefano

Message #11 received at 11287 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 11287 <at> debbugs.gnu.org
Subject: Re: bug#11287: Various issues with the test suite framework
Date: Tue, 24 Apr 2012 13:58:40 +0200

Le 24 avr. 2012 à 11:42, Stefano Lattarini a écrit :

> On 04/20/2012 01:13 PM, Akim Demaille wrote:
>> Hi!
>> 
> Hi Akim, sorry for the delay.

Hi Stefano,

No worries.

>> I have seen that check-html will be removed, or rather moved
>> into the contrib part, but there are a few issues:
>> 
>> - the target is not declared recursive, so one has to write
>> the bouncing target herself.
>> 
>> - because of that, the "naive" implementation of check-html
>> that just bounces the right directories will miss the
>> dependencies that check features (in other words, make
>> check properly recursively builds before, while check-html
>> would just go into the typical tests/ directory, and then
>> bounce to "check", so it completely missed the recursion
>> in the other directories and their dependencies).
>> 
> Wasn't this issue already present in Automake 1.11.1?  (So, not a
> regression, unless I'm mistaken).  In any case, a test case
> exposing the issue would be appreciated, if you have one at hand.

No, I don't think this is a regression.  It is a problem though.
I don't know (yet) what contribs/ are in Automake, but this shows
that some real assistance from Automake would really help, it's
not just a matter of a single Make snippet in a single Makefile,
but something more pervasive to the whole package.

>> - the .log.html recipe features $$RST2HTML, but in '', so it is
>> useless.  And anyway $(RST2HTML) already does the job.
>> 
>> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>> 
> None of these are covered by the Automake test suite sadly, and I
> see them as really low low priority anyway, so I'm not going to
> tackle them right now.  But I'll gladly accept patches by anyone
> who has more knowledge or interest than me in this area ;-)

I'll look for someone matching these specs :)


>> - RECURSIVE_TARGETS cannot be highjacked by the users, that's
>> sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
>> RECURSIVE_TARGETS so that we can add our own targets that
>> traverse the whole package?
>> 
> There had been an attempt to offer an interface for user-defined
> recursive rules already:
> 
>  <http://thread.gmane.org/gmane.comp.sysutils.automake.general/11890>
> 
> But unfortunately the discussion stagnated and died out, and I don't
> have time to resurrect it nor the relevant patches ATM.  Still, it
> would be a cool feature to have, so any help would be appreciated.

Yes, definitely.

>> - the test compiler does not appear as a dependency for
>>  the test logs that use it.
>> 
> This makes sense, since the test compiler is not required to be an
> in-project script,

Which is not a problem.  I am very happy that my dependencies
include headers from /usr and not just those in the package.

> and not even to be specified as an absolute path
> -- it can be a command like "sh" or "/usr/bin/env perl".

which is one, indeed.

>> - it would be equally useful to be able to add dependencies
>>  to a category of test drivers.  Say, with the documented
>>  example:
>> 
>>    TESTS = foo.pl bar.py baz
>>    TEST_EXTENSIONS = .pl .py
>>    PL_LOG_COMPILER = $(PERL)
>>    AM_PL_LOG_FLAGS = -w
>>    PY_LOG_COMPILER = $(PYTHON)
>>    AM_PY_LOG_FLAGS = -v
>>    LOG_COMPILER = ./wrapper-script
>>    AM_LOG_FLAGS = -d
>> 
>> be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>> LOG_DEPENDENCIES.
>> 
> This seems like a cool feature, but not trivial to implement in the current
> Automake code base.  Still, I have some half-finished patches to rework
> the parallel-tests harness in Automake-NG (reviews welcome, BTW ;-),

I'll pull :)

> and
> once those are in place, implementing your proposed feature in Automake-NG
> might be very easy.

Good news.

Message #14 received at 11287 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: automake-patches <at> gnu.org
Cc: 11287 <at> debbugs.gnu.org
Subject: [FYI] {master} coverage: check-html can be used recursively
Date: Mon,  2 Jul 2012 20:34:07 +0200

Addresses the main part of Automake bug#11287.

* contrib/t/parallel-tests-html-recursive.sh: New test.
* contrib/t/local.am: Add it.

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---
 contrib/t/local.am                         |    1 +
 contrib/t/parallel-tests-html-recursive.sh |  163 ++++++++++++++++++++++++++++
 2 files changed, 164 insertions(+)
 create mode 100755 contrib/t/parallel-tests-html-recursive.sh

diff --git a/contrib/t/local.am b/contrib/t/local.am
index 0a5f5c1..3859c70 100644
--- a/contrib/t/local.am
+++ b/contrib/t/local.am
@@ -20,5 +20,6 @@ contrib_testsuite_dir = contrib/t
 
 contrib_TESTS = \
   $(contrib_testsuite_dir)/parallel-tests-html.sh \
+  $(contrib_testsuite_dir)/parallel-tests-html-recursive.sh \
   $(contrib_testsuite_dir)/help-multilib.sh \
   $(contrib_testsuite_dir)/multilib.sh
diff --git a/contrib/t/parallel-tests-html-recursive.sh b/contrib/t/parallel-tests-html-recursive.sh
new file mode 100755
index 0000000..12d967e
--- /dev/null
+++ b/contrib/t/parallel-tests-html-recursive.sh
@@ -0,0 +1,163 @@
+#! /bin/sh
+# Copyright (C) 2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Recursive use of 'check-html'.  See Automake bug#11287.
+
+. ./defs || exit 1
+
+# Try the variants that are tried in check-html.am.
+while :; do
+  for r2h in $RST2HTML rst2html rst2html.py; do
+    echo "$me: running $r2h --version"
+    $r2h --version && break 2
+    : For shells with busted 'set -e'.
+  done
+  skip_all_ "no proper rst2html program found"
+done
+unset r2h
+
+cp "$am_top_srcdir"/contrib/check-html.am . \
+  || fatal_ "cannot fetch 'check-html.am' from contrib"
+
+cat >> configure.ac << 'END'
+AM_EXTRA_RECURSIVE_TARGETS([check-html])
+AC_CONFIG_FILES([sub/Makefile sub/more/Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+SUBDIRS = sub
+EXTRA_DIST = $(TESTS)
+TEST_SUITE_LOG = mylog.log
+TESTS = foo.test bar.sh mu
+XFAIL_TESTS = bar.sh
+check_SCRIPTS = bla
+bla:
+	echo '#!/bin/sh' > $@-t
+	echo 'echo Blah Blah Blah' >> $@-t
+	chmod a+x,a-w $@-t
+	mv -f $@-t $@
+CLEANFILES = bla
+include $(srcdir)/check-html.am
+END
+
+mkdir sub
+echo SUBDIRS = more > sub/Makefile.am
+
+mkdir sub/more
+cat > sub/more/Makefile.am << 'END'
+include $(top_srcdir)/check-html.am
+TEST_EXTENSIONS = .test .sh
+TESTS = sh.sh test.test
+LOG_COMPILER = true
+test.log: sh.log
+nodist_check_DATA = x.txt
+$(nodist_check_DATA):
+	echo $@ > $@
+CLEANFILES = $(nodist_check_DATA)
+EXTRA_DIST = $(TESTS)
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+./bla
+exit 77
+END
+
+cat > bar.sh <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 1
+END
+
+cat > mu <<'END'
+#! /bin/sh
+set -x
+test -f sub/more/test.log
+test -f sub/more/sh.log
+END
+
+cat > sub/more/test.test << 'END'
+#!/bin/sh
+echo "this is $0"
+set -x
+test -f sh.log
+test -f x.txt
+exit 77
+END
+
+cat > sub/more/sh.sh << 'END'
+#!/bin/sh
+set -x
+test ! -f test.log
+test -f x.txt
+END
+
+
+cat > sub/more/mu << 'END'
+#!/bin/sh
+exit 99
+END
+
+chmod a+x foo.test bar.sh mu sub/more/test.test sub/more/sh.sh
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check-html
+grep 'Blah Blah Blah' mylog.html
+grep 'this is .*bar\.sh' mylog.html
+grep 'this is .*test\.test' sub/more/test-suite.html
+# check-html should cause check_SCRIPTS and check_DATA to be created.
+test -f bla
+test -f sub/more/x.txt
+
+# "make clean" should remove HTML files.
+$MAKE clean
+test ! -e mylog.html
+test ! -e sub/more/test-suite.html
+test ! -e bla
+test ! -e sub/more/x.txt
+
+# Create HTML output for individual tests.
+
+$MAKE bla
+$MAKE foo.html bar.sh.html
+grep 'Blah Blah Blah' foo.html
+grep 'this is .*bar\.sh' bar.sh.html
+test ! -e mu.hml
+
+ocwd=$(pwd) || fatal_ "getting current workind directory"
+( cd sub/more \
+    && $MAKE sh.html \
+    && test -f sh.html \
+    && test ! -e test.html \
+    && $MAKE test.html \
+    && grep 'this is .*test\.test' test.html) || exit 1
+
+# HTML output removed by mostlyclean.
+$MAKE check-html
+test -f mylog.html
+test -f sub/more/test-suite.html
+$MAKE mostlyclean
+find . -name '*.html' | grep . && exit 1
+
+$MAKE distcheck
+
+:
-- 
1.7.9.5

Message #17 received at 11287 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: automake-patches <at> gnu.org
Cc: 11287 <at> debbugs.gnu.org
Subject: [FYI] {master} contrib: check-html: pass flags to rst2html invocations
Date: Mon,  2 Jul 2012 21:14:08 +0200

Related to automake bug#11287.

* contrib/t/check-html.am (.log.html): Here, passing $(AM_RST2HTMLFLAGS)
and $(RST2HTMLFLAGS) (in that order).

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---
 contrib/check-html.am |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/contrib/check-html.am b/contrib/check-html.am
index 47e4f47..36a5dbe 100644
--- a/contrib/check-html.am
+++ b/contrib/check-html.am
@@ -38,7 +38,8 @@ mostlyclean-check-html:
 	  echo "cannot find rst2html, cannot create $@" >&2;		\
 	  exit 2;							\
 	done;								\
-	$$r2h $< >$@-t && mv -f $@-t $@
+	$$r2h $(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS) $< >$@-t		\
+	  && mv -f $@-t $@
 
 # Be sure to run check first, and then to convert the result.
 # Beware of concurrent executions.  Run "check" not "check-TESTS", as
-- 
1.7.9.5

Message #20 received at 11287 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: 11287 <at> debbugs.gnu.org
Cc: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Subject: [PATCH] contrib: simple improvements to check-html
Date: Mon,  2 Jul 2012 21:14:47 +0200

Related to automake bug#11287.

* contrib/t/check-html.am (.log.html): Do not look for $RST2HTML in
the environment, looking for $(RST2HTML) should be enough (also, the
pre-existing code was broken, because it single-quoted $RST2HTML).
Fix the loop-and-search implementation to be more similar to the
one in 'contrib/t/parallel-tests-html.sh'.  Prefer 'mv -f' over bare
'mv'.  Do not break the recipe in

Signed-off-by: Stefano Lattarini <stefano.lattarini <at> gmail.com>
---
 contrib/check-html.am |   20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/contrib/check-html.am b/contrib/check-html.am
index 4495ac0..47e4f47 100644
--- a/contrib/check-html.am
+++ b/contrib/check-html.am
@@ -29,18 +29,16 @@ mostlyclean-check-html:
 	rm -f $(TEST_SUITE_HTML)
 
 .log.html:
-	@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py';		\
-	for r2h in $$list; do						\
-	  if ($$r2h --version) >/dev/null 2>&1; then			\
-	    R2H=$$r2h;							\
-	  fi;								\
-	done;								\
-	if test -z "$$R2H"; then					\
-	  echo >&2 "cannot find rst2html, cannot create $@";		\
+	@list='$(RST2HTML) rst2html rst2html.py';			\
+	while :; do							\
+	  for r2h in $$list; do						\
+	    if ($$r2h --version) >/dev/null 2>&1; then break 2;		\
+	    else :; fi;							\
+	  done;								\
+	  echo "cannot find rst2html, cannot create $@" >&2;		\
 	  exit 2;							\
-	fi;								\
-	$$R2H $< >$@.tmp
-	@mv $@.tmp $@
+	done;								\
+	$$r2h $< >$@-t && mv -f $@-t $@
 
 # Be sure to run check first, and then to convert the result.
 # Beware of concurrent executions.  Run "check" not "check-TESTS", as
-- 
1.7.9.5

Message #25 received at 11287-done <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 11287-done <at> debbugs.gnu.org
Subject: Re: bug#11287: Various issues with the test suite framework
Date: Mon, 02 Jul 2012 21:27:22 +0200

Reference: <http://debbugs.gnu.org/11287>

To sum up the actions related to this bug report:

On 04/20/2012 01:13 PM, Akim Demaille wrote:
> Hi!
> 
> I have seen that check-html will be removed, or rather moved
> into the contrib part, but there are a few issues:
> 
> - the target is not declared recursive, so one has to write
>  the bouncing target herself.
>
This isn't a big deal anymore, now that the user can define extra
recursive targets through the macro AM_EXTRA_RECURSIVE_TARGETS
(features introduced in commit v1.12.1-163-g3da1457); the test
'contrib/t/parallel-tests-html-recursive.sh' (introduced by commit
v1.12.1-191-g53d4984) should prove this assertion.

> - because of that, the "naive" implementation of check-html
>  that just bounces the right directories will miss the
>  dependencies that check features (in other words, make
>  check properly recursively builds before, while check-html
>  would just go into the typical tests/ directory, and then
>  bounce to "check", so it completely missed the recursion
>  in the other directories and their dependencies).
>
Adding 'check-html' to AM_EXTRA_RECURSIVE_TARGETS should take
care of this issue as well.

> - RECURSIVE_TARGETS cannot be highjacked by the users, that's
>  sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
>  RECURSIVE_TARGETS so that we can add our own targets that
>  traverse the whole package?  Actually maybe this should
>  be an Automake option to add recursive targets, and
>  then TARGET-am and TARGET-recursive would be generated and
>  bound to the proper TARGET.
>
Addressed by commit v1.12.1-163-g3da1457 (with a different design
 but comparable functionalities); see this thread for more info:
<http://lists.gnu.org/archive/html/automake-patches/2012-06/msg00195.html>

> - the .log.html recipe features $$RST2HTML, but in '', so it is
>  useless.  And anyway $(RST2HTML) already does the job.
>
Agreed.  Fixed by commit v1.12.1-192-g18dac90.

> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>
Fixed by commit v1.12.1-193-g67abae4.

> - the test compiler does not appear as a dependency for
>  the test logs that use it.
>
Reporting what I've stated in an earlier reply:

    This makes sense, since the test compiler is not required to be
    an in-project script, and not even to be specified as an absolute
    path -- it can be a command like "sh" or "/usr/bin/env perl".

> (This is most useful when RECHECK_LOG =.)

> - it would be equally useful to be able to add dependencies
>  to a category of test drivers.  Say, with the documented
>  example:
> 
>     TESTS = foo.pl bar.py baz
>     TEST_EXTENSIONS = .pl .py
>     PL_LOG_COMPILER = $(PERL)
>     AM_PL_LOG_FLAGS = -w
>     PY_LOG_COMPILER = $(PYTHON)
>     AM_PY_LOG_FLAGS = -v
>     LOG_COMPILER = ./wrapper-script
>     AM_LOG_FLAGS = -d
> 
>  be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>  LOG_DEPENDENCIES.  I don't think EXTRA dependencies would
>  be useful.
>
This would be tricky to implement in the current Automake code base, and
IMHO for a too little benefit; so I'm going to label this part of the
report as a "wontfix".  However, it was easy to implement in Automake-NG:
<http://lists.gnu.org/archive/html/automake-ng/2012-05/msg00022.html>

-*-*-*-

All that considered, I'm closing this bug report now.

Regards,
  Stefano

Message #26 received at 11287-done <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 11287-done <at> debbugs.gnu.org
Subject: Re: bug#11287: Various issues with the test suite framework
Date: Wed, 4 Jul 2012 09:30:55 +0200

Hi Stefano,

Le 2 juil. 2012 à 21:27, Stefano Lattarini a écrit :

>> - the target is not declared recursive, so one has to write
>> the bouncing target herself.
>> 
> This isn't a big deal anymore, now that the user can define extra
> recursive targets through the macro AM_EXTRA_RECURSIVE_TARGETS
> (features introduced in commit v1.12.1-163-g3da1457); the test
> 'contrib/t/parallel-tests-html-recursive.sh' (introduced by commit
> v1.12.1-191-g53d4984) should prove this assertion.

Great!

>> - because of that, the "naive" implementation of check-html
>> that just bounces the right directories will miss the
>> dependencies that check features (in other words, make
>> check properly recursively builds before, while check-html
>> would just go into the typical tests/ directory, and then
>> bounce to "check", so it completely missed the recursion
>> in the other directories and their dependencies).
>> 
> Adding 'check-html' to AM_EXTRA_RECURSIVE_TARGETS should take
> care of this issue as well.

Good.

> Addressed by commit v1.12.1-163-g3da1457 (with a different design
> but comparable functionalities); see this thread for more info:
> <http://lists.gnu.org/archive/html/automake-patches/2012-06/msg00195.html>

Great news.  FWIW, I'm not sure I understand why this is appears
in configure.ac, there might be technical issues I have not understood,
but when possible, I think it's good to avoid having to fire
autoconf again.  Just $0.02, not a change request.

>> - the .log.html recipe features $$RST2HTML, but in '', so it is
>> useless.  And anyway $(RST2HTML) already does the job.
>> 
> Agreed.  Fixed by commit v1.12.1-192-g18dac90.
> 
>> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>> 
> Fixed by commit v1.12.1-193-g67abae4.

Excellent.

>> - it would be equally useful to be able to add dependencies
>> to a category of test drivers.  Say, with the documented
>> example:
>> 
>>    TESTS = foo.pl bar.py baz
>>    TEST_EXTENSIONS = .pl .py
>>    PL_LOG_COMPILER = $(PERL)
>>    AM_PL_LOG_FLAGS = -w
>>    PY_LOG_COMPILER = $(PYTHON)
>>    AM_PY_LOG_FLAGS = -v
>>    LOG_COMPILER = ./wrapper-script
>>    AM_LOG_FLAGS = -d
>> 
>> be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>> LOG_DEPENDENCIES.  I don't think EXTRA dependencies would
>> be useful.
>> 
> This would be tricky to implement in the current Automake code base, and
> IMHO for a too little benefit; so I'm going to label this part of the
> report as a "wontfix".  However, it was easy to implement in Automake-NG:
> <http://lists.gnu.org/archive/html/automake-ng/2012-05/msg00022.html>

Good enough, thanks!

Message #27 received at 11287-done <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 11287-done <at> debbugs.gnu.org
Subject: Re: bug#11287: Various issues with the test suite framework
Date: Wed, 04 Jul 2012 09:55:36 +0200

On 07/04/2012 09:30 AM, Akim Demaille wrote:
> Hi Stefano,
> 
> Le 2 juil. 2012 à 21:27, Stefano Lattarini a écrit :
>> Addressed by commit v1.12.1-163-g3da1457 (with a different design
>> but comparable functionalities); see this thread for more info:
>> <http://lists.gnu.org/archive/html/automake-patches/2012-06/msg00195.html>
> 
> Great news.  FWIW, I'm not sure I understand why this is appears
> in configure.ac,
>
Basically because Automake has to know what the user-defined recursive rules
are during the processing of any its Makefile.am files (because it has to
generate proper stub rules in all of them), and the simpler way to ensure
this happens is to have the extra recursive rules declared in configure.ac,
which is always analyzed by any Automake run.

> there might be technical issues I have not understood,
> but when possible, I think it's good to avoid having to fire
> autoconf again.  Just $0.02, not a change request.
>
In the long term, a better approach might be to add a "configuration file"
to automake, where to declare stuff that (like extra user-defined recursive
rules) only affects the generated Makefile.in files, not the generated
configure.  This is overkill right now IMHO, but might be worth considering
in the future.

Thanks,
  Stefano

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.