GNU bug report logs - #68855
wording for user on test suite failure?

Previous Next

To reply to this bug, email your comments to 68855 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: automake-patches <at> gnu.org
Subject: wording for user on test suite failure?
Date: Wed, 31 Jan 2024 11:59:14 -0700

I was thinking about making the message when the test suite failure more
explicit about what to do, as in below. If any comments or criticisms,
let me know.

I was actually inspired to do this not so much to tweak these words, but
because I was thinking of adding yet more instructions about how to help
debug the subsecond mtime stuff (for automake only, of course), i.e.,
setting am_cv_sleep_fractional_seconds=false. But with the new
possibility that checking make's behavior will be enough to obviate
those random failures, that doesn't seem necessary/desirable now.

But having gone to all the trouble of figuring out how to change the
text and which tests need to be changed to keep them in sync, I guess I
still hope people think it's worth it, or at least ok, to make this
change. The output would look like this:

============================================================================
See tests/test-suite.log for debugging.
Some test(s) failed.  Please report this to <address>,
together with the test-suite.log file (gzipped) and your system
information.  Thanks.
============================================================================

Hopefully I did not break the colorization, but I don't have a good way
to test that. --thanks, karl.

-----------------------------------------------------------------------------
test: make bug-reporting message on test failure more explicit.

* NEWS: mention this.
* lib/am/check.am ($(TEST_SUITE_LOG)) <please_report>: new fn,
new wording.
* t/ax/test-defs.in (failure_footer_text): new wording.
(failure_footer_text_colorized): add red.
* t/ax/tap-summary-aux.sh (failure_footer): use failure_footer_text*.
* t/tap-doc2.sh: change grep.
* t/testsuite-summary-color.sh: likewise.
* t/testsuite-summary-count-many.sh: likewise.
* t/testsuite-summary-count.sh: likewise.
* t/testsuite-summary-reference-log.sh: likewise.
diff --git a/NEWS b/NEWS
index 153cb7733..964d82e8c 100644
--- a/NEWS
+++ b/NEWS
@@ -44,6 +44,10 @@ New in 1.17:
   - The missing script also supports autoreconf, autogen, and perl.
     (https://lists.gnu.org/archive/html/automake-patches/2015-08/msg00000.html)

+  - test-suite.log now contains basic system information, and the
+    console message about bug reporting on failure has a bit more detail.
+    (bug#68746)
+
 * Bugs fixed

   - Generated file timestamp checks handle filesystems with sub-second
diff --git a/lib/am/check.am b/lib/am/check.am
index d210172a6..b6fa3aaad 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -246,6 +246,7 @@ am--force-recheck:
 ## Exists only to be overridden.  See bug#11745.
 AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING)'

+## emacs-page
 $(TEST_SUITE_LOG): $(TEST_LOGS)
 	@$(am__set_TESTS_bases); \
 ## Helper shell function, tells whether a path refers to an existing,
@@ -360,7 +361,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  result_count $$1 "ERROR:" $$error "$$mgn"; \
 	}; \
 ##
-## Write some basic system info in the log. 
+## Function to write some basic system info in the log. 
 	output_system_information () \
 	{ \
 ## Omit the hostname for privacy.  In practice it's a single word?
@@ -372,6 +373,16 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	    cat /etc/issue;                        \
 	  fi;                                      \
 	}; \
+##
+## Function to output the "please report bugs" message.
+	please_report () \
+	{ \
+echo "Some test(s) failed.  Please report this to $(PACKAGE_BUGREPORT),";    \
+echo "together with the test-suite.log file (gzipped) and your system";      \
+echo "information.  Thanks.";                                                \
+	}; \
+##
+## emacs-page
 ## Write "global" testsuite log.
 	{								\
 	  echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" |	\
@@ -379,8 +390,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	  create_testsuite_report --no-color;				\
 	  output_system_information;                                    \
 	  echo;								\
-## What is this line for? It seems to be intended for machine parsing,
-## but I can't find any other reference to it. --karl, 27jan24.
+## What is this line for? It's a Sphinx reStructuredText directive that
+## would insert a table of contents that goes two levels deep on headings.
+## But why is it in test-suite.log? No other references to it are evident.
+## Please inform automake <at> gnu.org.
 	  echo ".. contents:: :depth: 2";				\
 	  echo;								\
 	  for b in $$bases; do echo $$b; done				\
@@ -403,9 +416,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
 	create_testsuite_report --maybe-color;				\
 	echo "$$col$$br$$std";						\
 	if $$success; then :; else					\
-	  echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}";		\
+	  echo "$${col}See $(subdir)/$(TEST_SUITE_LOG) for debugging.$${std}";\
 	  if test -n "$(PACKAGE_BUGREPORT)"; then			\
-	    echo "$${col}Please report to $(PACKAGE_BUGREPORT)$${std}";	\
+	    please_report | sed -e "s/^/$${col}/" -e s/'$$'/"$${std}"/; \
 	  fi;								\
 	  echo "$$col$$br$$std";					\
 	fi;								\
diff --git a/t/ax/tap-summary-aux.sh b/t/ax/tap-summary-aux.sh
index d1e2888c3..a709f1ef3 100644
--- a/t/ax/tap-summary-aux.sh
+++ b/t/ax/tap-summary-aux.sh
@@ -87,8 +87,17 @@ if test $use_colors = yes; then
   mgn="$esc[0;35m"
   brg="$esc[1m"
   std="$esc[m"
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text_colorized
+${red}${br}${std}"
+
 else
   red= grn= lgn= blu= mgn= brg= std=
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text
+${red}${br}${std}"
 fi

   success_header="\
@@ -101,12 +110,6 @@ ${grn}${br}${std}"
   failure_header="\
 ${red}${br}${std}
 ${red}Testsuite summary for GNU AutoTAP 5.12${std}
-${red}${br}${std}"
-
-  failure_footer="\
-${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to bug-automake <at> gnu.org${std}
 ${red}${br}${std}"

 $ACLOCAL
diff --git a/t/ax/test-defs.in b/t/ax/test-defs.in
index 8ea842355..c1ead418e 100644
--- a/t/ax/test-defs.in
+++ b/t/ax/test-defs.in
@@ -215,6 +215,20 @@ sleep="sleep $MTIME_RESOLUTION"
 # in UTC; see <https://www.gnu.org/gnu/initial-announcement.html>.
 old_timestamp=198309271735.59

+# We need this in multiple tests and the wording is subject to change.
+# Don't include the ${br} lines since br is not defined at this point.
+#
+failure_footer_text="\
+See ./test-suite.log for debugging.
+Some test(s) failed.  Please report this to bug-automake <at> gnu.org,
+together with the test-suite.log file (gzipped) and your system
+information.  Thanks."
+#
+# And with color.
+red=''
+std='';
+failure_footer_text_colorized="`echo \"$failure_footer_text\" | sed -e "s/^/${red}/" -e s/'$'/"${std}"/`"
+
 # Make our wrapper script (or installed scripts, if running under
 # "installcheck") accessible by default.  And avoid uselessly
 # extending $PATH multiple times if this file is sourced multiple times.
diff --git a/t/tap-doc2.sh b/t/tap-doc2.sh
index fdca66ad8..54c69b631 100644
--- a/t/tap-doc2.sh
+++ b/t/tap-doc2.sh
@@ -102,7 +102,7 @@ cat exp
 cat got
 diff exp got

-grep '^Please report to bug-automake <at> gnu\.org$' stdout
+grep 'Please report this to bug-automake <at> gnu\.org' stdout

 run_make -O check \
   TESTS='foo.test baz.test' \
diff --git a/t/testsuite-summary-color.sh b/t/testsuite-summary-color.sh
index 7b6254f61..ffbe5c158 100644
--- a/t/testsuite-summary-color.sh
+++ b/t/testsuite-summary-color.sh
@@ -46,8 +46,7 @@ ${red}${br}${std}"

 failure_footer="\
 ${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to bug-automake <at> gnu.org${std}
+$failure_footer_text_colorized
 ${red}${br}${std}"

 do_check '' <<END
diff --git a/t/testsuite-summary-count-many.sh b/t/testsuite-summary-count-many.sh
index 2560ef82c..3cb5a500c 100644
--- a/t/testsuite-summary-count-many.sh
+++ b/t/testsuite-summary-count-many.sh
@@ -34,7 +34,7 @@ ${br}"

 footer="\
 ${br}
-See ./test-suite.log
+See ./test-suite.log for debugging.
 ${br}"

 echo AC_OUTPUT >> configure.ac
diff --git a/t/testsuite-summary-count.sh b/t/testsuite-summary-count.sh
index c8f15389d..9cd2fa58e 100644
--- a/t/testsuite-summary-count.sh
+++ b/t/testsuite-summary-count.sh
@@ -32,8 +32,7 @@ success_footer=${br}

 failure_footer="\
 ${br}
-See ./test-suite.log
-Please report to bug-automake <at> gnu.org
+$failure_footer_text
 ${br}"

 # Corner cases.
diff --git a/t/testsuite-summary-reference-log.sh b/t/testsuite-summary-reference-log.sh
index 64cc4eb6c..f27e6fc39 100644
--- a/t/testsuite-summary-reference-log.sh
+++ b/t/testsuite-summary-reference-log.sh
@@ -46,11 +46,11 @@ cd build
 ../configure

 run_make -O -e FAIL check
-grep '^See \./my_test_suite\.log$' stdout
+grep '^See \./my_test_suite\.log for debugging\.$' stdout

 mkdir bar
 run_make -O -e FAIL TEST_SUITE_LOG=bar/bar.log check
-grep '^See \./bar/bar\.log$' stdout
+grep '^See \./bar/bar\.log for debugging\.$' stdout

 cd ..

@@ -70,13 +70,13 @@ $AUTOMAKE

 ./configure
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd sub
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd ..

 run_make -O -e FAIL TEST_SUITE_LOG=foo.log check
-grep '^See sub/foo\.log$' stdout
+grep '^See sub/foo\.log for debugging\.$' stdout

 :

compile finished at Wed Jan 31 10:56:25 2024

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.