GNU bug report logs - #41824
Dejagnu's unknown proc aborts testsuite run when triggered in test-case

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: dejagnu; Reported by: Tom de Vries <tdevries@HIDDEN>; Owned by: jcb62281@HIDDEN; dated Fri, 12 Jun 2020 08:36:01 UTC; Maintainer for dejagnu is bug-dejagnu@HIDDEN.
Owner recorded as jcb62281@HIDDEN Request was from Jacob Bachmeyer <jcb62281@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 41824 <at> debbugs.gnu.org:


Received: (at 41824) by debbugs.gnu.org; 16 Jun 2020 15:54:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 16 11:54:03 2020
Received: from localhost ([127.0.0.1]:49462 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1jlDuU-0004N3-V9
	for submit <at> debbugs.gnu.org; Tue, 16 Jun 2020 11:54:03 -0400
Received: from mx2.suse.de ([195.135.220.15]:44934)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tdevries@HIDDEN>) id 1jlDuS-0004MZ-KD
 for 41824 <at> debbugs.gnu.org; Tue, 16 Jun 2020 11:54:01 -0400
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (unknown [195.135.220.254])
 by mx2.suse.de (Postfix) with ESMTP id DD204ADF7;
 Tue, 16 Jun 2020 15:53:58 +0000 (UTC)
Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when
 triggered in test-case
To: jcb62281@HIDDEN
References: <e5f42a45-5773-8819-03d0-854015a925a0@HIDDEN>
 <5EE8450D.9080609@HIDDEN>
From: Tom de Vries <tdevries@HIDDEN>
Autocrypt: addr=tdevries@HIDDEN; keydata=
 xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7
 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc
 u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l
 CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P
 TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB
 AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL
 MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh
 CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo
 eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ
 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ
 UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn
 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827
 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B
 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W
 pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ
 Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z
 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9
 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats
 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3
 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7
 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC
 q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U
 yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS
 DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ==
Message-ID: <ef5fac05-3947-f262-fd58-c1ab24521761@HIDDEN>
Date: Tue, 16 Jun 2020 17:53:54 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <5EE8450D.9080609@HIDDEN>
Content-Type: multipart/mixed; boundary="------------F860BC22FFD7297290218B0F"
Content-Language: en-US
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 41824
Cc: 41824 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

This is a multi-part message in MIME format.
--------------F860BC22FFD7297290218B0F
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

On 6/16/20 6:05 AM, Jacob Bachmeyer wrote:
> Tom de Vries wrote:
>> Hi,
>>
>> the gdb testsuite uses the dejagnu framework.
>>   
> 
> Apologies for the delay in responding; I took a few days to think about
> this issue because it requires a careful balance, as silently continuing
> after aborting a testcase could cause errors to go unnoticed in large
> testsuites.
> 
>> If I add a call to foobar at the end of a test-case in that testsuite,
>> and I do a run of the testsuite, then the test-case aborts, and
>> subsequently the testsuite run aborts, that is, no further tests are run.
>>
>> While it is as expected that the test-case aborts, it's not desirable
>> that the testsuite run aborts.
>>   
> 
> The main problem I have with this is that calling an unknown procedure
> is presumably a very serious error.  Aborting the entire run is a good
> way to ensure that the problem is noticed, but perhaps with a testsuite
> as large (and long-running) as the GDB testsuite has become, continuing
> may also be reasonable to salvage at least some results.
> 
> Where in the GDB testsuite is this a problem?  When is it appropriate to
> continue after such a serious error?

It's not a problem in a specific test-case, but a generic problem. If
somebody makes a typo in a test-case and checks it in, there's no need
to abort the test run.

> How to ensure that the error is
> not buried in the logs amidst the normal output from other tests and is
> actually noticed?
> 

I monitor ERROR and WARNING in my test runs, so I didn't think of this,
but indeed, those are not tracked in the summary, so that could be a
problem, agreed.

>> We've installed a hack in ${tool}_init/${tool}_finish to workaround this
>> in the gdb testsuite (
>> https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3
>>
>> ).
>>
>> It would preferable though if this problem was addressed in dejagnu,
>> such that we can eventually remove the hack.
>>   
> 
> That hack (if not reverted entirely) needs to be made conditional on the
> actual existence of ::tcl_unknown as soon as possible -- the existence
> of ::tcl_unknown (kept to allow Tcl autoloading to work) is very much an
> internal implementation detail, and it *will* be moved out of the global
> namespace and eventually *will* cease to exist entirely in the
> interpreters that run testcases.  Really, tcl_unknown is not supposed to
> be there and relying on it introduces a latent bug into the GDB testsuite.
> 
> I am sorry, and I will seek to work with you to fix these problems with
> a minimum of breakage, but I have to put my foot down on this:  I
> *cannot* treat every internal symbol as long-term stable API.  That hack
> will *not* be supported long-term.  Any GDB releases that include it
> *will* break with some as-yet-undetermined future DejaGnu release.  I
> have to draw a line somewhere or I may as well declare DejaGnu
> unmaintained and frozen at 1.6.2 forever.  I have no inclination to do
> the latter.
> 

Ack, makes sense.

>> A possible solution to this problem could be to make the exiting on
>> error (which is done in dejagnu's version of proc unknown in
>> framework.exp) optional.
>>   
> 
> This is obviously a good solution, but leads to the obvious question of
> how (command-line option?  (Make normally aborts on error, but has a
> --keep-going option.)  configuration variable?) that should be
> determined, or if exit-on-unknown-procedure should ever be a default. 
> Could simply reporting an UNRESOLVED test (indicating that the testcase
> script aborted due to calling an undefined command) be a better option? 
> It would show up in the summary report at the end.
> 
> At first, I was planning to schedule this for the 1.7 series, but I
> checked and runtest.exp:runtest already catches Tcl errors, so fixing
> this does not require a significant change to the normal control flow. 
> Would simply reporting an "UNRESOLVED:  testcase $file aborted on
> unknown command '$what'" result, then propagating the Tcl error in the
> ::unknown proc be a workable solution?

I think so, yes.  I've tried that out in attached gdb patch, which also
solves the reliance on ::tcl_unknown.

Then when running into the unknown command in the test-case, we have in
gdb.sum:
...
Running
/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp
...
PASS: gdb.ada/O2_float_param.exp: compilation foo.adb
PASS: gdb.ada/O2_float_param.exp: frame
UNRESOLVED: gdb.ada/O2_float_param.exp: testcase aborted on unknown
command 'foo'
ERROR: tcl error sourcing
/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp.
ERROR: invalid command name "foo"
    while executing
"::gdb_unknown foo"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 ::gdb_unknown $args"
    (procedure "::unknown" line 4)
    invoked from within
"foo"
    (file
"/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp"
line 33)
    invoked from within
"source
/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 source
/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp"
    invoked from within
"catch "uplevel #0 source $test_file_name""

                === gdb Summary ===

# of expected passes            2
# of unresolved testcases       1
...

I get similar results if I disable the hack in the gdb testsuite, and
use attached runtest.exp demonstrator patch.

Note btw that in both cases I didn't put $file in the unresolved
message, because that's already present in pf_prefix, but that might not
always be true.

Thanks,
- Tom

--------------F860BC22FFD7297290218B0F
Content-Type: text/x-patch; charset=UTF-8;
 name="0001-fix.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="0001-fix.patch"

fix

---
 gdb/testsuite/lib/gdb.exp | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index f502eb157d..5cb9e7907c 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -5177,6 +5177,13 @@ proc gdb_cleanup_globals {} {
     }
 }
 
+# Create gdb_unknown, a copy tcl's ::unknown.
+set temp [interp create]
+set old_args [interp eval $temp "info args ::unknown"]
+set old_body [interp eval $temp "info body ::unknown"]
+interp delete $temp
+eval proc gdb_unknown {$old_args} {$old_body}
+
 proc gdb_init { test_file_name } {
     # Reset the timeout value to the default.  This way, any testcase
     # that changes the timeout value without resetting it cannot affect
@@ -5285,12 +5292,13 @@ proc gdb_init { test_file_name } {
 
     # Dejagnu overrides proc unknown.  The dejagnu version may trigger in a
     # test-case but abort the entire test run.  To fix this, we install a
-    # local version here, which reverts dejagnu's override, and restore
-    # dejagnu's version in gdb_finish.
+    # local version here, which instead calls unresolved, and then propagates the
+    # error to tcl's unknown. In gdb_finish, we restore dejagnu's version.
     rename ::unknown ::dejagnu_unknown
     proc unknown { args } {
-	# Dejagnu saves the original version in ::tcl_unknown, use it.
-	return [uplevel 1 ::tcl_unknown $args]
+	# Restore ::unknown.
+	unresolved "testcase aborted on unknown command '$args'"
+	return [uplevel 1 ::gdb_unknown $args]
     }
 
     return $res

--------------F860BC22FFD7297290218B0F
Content-Type: text/x-patch; charset=UTF-8;
 name="runtest-exp.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="runtest-exp.patch"

--- runtest.exp.orig	2020-06-16 17:36:15.038208916 +0200
+++ runtest.exp	2020-06-16 17:38:19.189251033 +0200
@@ -1431,6 +1431,11 @@
     return $found
 }
 
+proc test_case_unknown { args } {
+    unresolved "testcase aborted on unknown command '$args'"
+    return [uplevel 1 ::tcl_unknown $args]
+}
+
 #
 # Source the testcase in TEST_FILE_NAME.
 #
@@ -1457,6 +1462,9 @@
 	    }
 	}
 
+	rename ::unknown dejagnu_unknown
+	rename ::test_case_unknown ::unknown
+	
 	if { [catch "uplevel #0 source $test_file_name"] == 1 } {
 	    # If we have a Tcl error, propagate the exit status so
 	    # that 'make' (if it invokes runtest) notices the error.
@@ -1476,6 +1484,9 @@
 	    }
 	}
 
+	rename ::unknown ::test_case_unknown 
+	rename dejagnu_unknown ::unknown 
+
 	if {[info exists tool]} {
 	    if { [info procs "${tool}_finish"] != "" } {
 		${tool}_finish

--------------F860BC22FFD7297290218B0F--




Information forwarded to bug-dejagnu@HIDDEN:
bug#41824; Package dejagnu. Full text available.

Message received at 41824 <at> debbugs.gnu.org:


Received: (at 41824) by debbugs.gnu.org; 16 Jun 2020 04:05:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 16 00:05:46 2020
Received: from localhost ([127.0.0.1]:47911 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1jl2r3-0001M2-Rc
	for submit <at> debbugs.gnu.org; Tue, 16 Jun 2020 00:05:46 -0400
Received: from mail-oi1-f180.google.com ([209.85.167.180]:34431)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jcb62281@HIDDEN>) id 1jl2r1-0001Lo-E0
 for 41824 <at> debbugs.gnu.org; Tue, 16 Jun 2020 00:05:44 -0400
Received: by mail-oi1-f180.google.com with SMTP id b8so18111073oic.1
 for <41824 <at> debbugs.gnu.org>; Mon, 15 Jun 2020 21:05:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject
 :references:in-reply-to:content-transfer-encoding;
 bh=YgAWT3lNIINJ9dR5W9I9J9PvZ8iLWCCDz6nRf4wfTSI=;
 b=mMoniFR+ggE8lgneSs1EdgxPLGYNYGrjbBB37QbT1HvApCBR6EVFcx2f6IZex/gI1q
 UcHtGIc+R7gNrYFWrPequdrup7zGp2ePrTxnfl4bGo8NQzHzUCA+jJidGSQbGbmuwL3d
 YQDWW1cK0nmzPKU6Z2bQeFZK9FGSD79ijnLRLje2YIWKrxeMmKALn4Lamv5l0zxEjVVX
 NeeVAKGxDpZNH1yzfQqaKscHHnTsy2vCLL2OW0D986kacHlsE9Fp8tysHCV0vBg7rGQw
 n0fSHi7zbX5Ximvo4on2Nm+8BlpPoATu0RA/87bINs9m3gsRKS7sX7+qlQRFrXjD7aCq
 i7Dw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:message-id:date:from:reply-to:user-agent
 :mime-version:to:cc:subject:references:in-reply-to
 :content-transfer-encoding;
 bh=YgAWT3lNIINJ9dR5W9I9J9PvZ8iLWCCDz6nRf4wfTSI=;
 b=MIq7JrUKbONeC3O7LEoevT+BDAROWtg42/p0M1npOAp7k4sznjIbRhoO71i5VJyf+M
 Kxrlctp+vywem67CArJtIfiPMMK2mohTxvakMJoGIQ8QK91Q8n5qARY8m/Q5iV70Cwbq
 I8HqjSXZS6kiMODYb1+/Em6Q4z8mtd/9OtBS5me/8Q7JkN4o6Bfi2rNO4l5pziSXtfSU
 ABy/i4Hs93lCkgxdavVH4294Ow2cZc1ByIGxERSRkfUenPKMMwS/DuymrG4K/+ACFALX
 FBVDuTV+jRsYDW4ooz3vzu1NHkpbQwYlxD5N9ZTBVRnNMoJS2embd2Z+joHKbnql/zI1
 OOqA==
X-Gm-Message-State: AOAM532PUmESho1CukkmVTgRVmGoF/s0TNMbaTho7QsI+hQZNCVNqNs2
 4y8ssmIFNGt+pp/FHfQc6Rs=
X-Google-Smtp-Source: ABdhPJyPHp2BN6RMniZfYcr4MVPyN/hkAIPMG5XOYW8FDjGYyHy4aAF3EAri2Qv1BNlJ6j0uVacVsA==
X-Received: by 2002:aca:1308:: with SMTP id e8mr1856181oii.119.1592280337495; 
 Mon, 15 Jun 2020 21:05:37 -0700 (PDT)
Received: from [192.168.2.42] (adsl-70-133-146-38.dsl.ablntx.sbcglobal.net.
 [70.133.146.38])
 by smtp.gmail.com with ESMTPSA id p18sm15325oto.11.2020.06.15.21.05.35
 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
 Mon, 15 Jun 2020 21:05:36 -0700 (PDT)
Message-ID: <5EE8450D.9080609@HIDDEN>
Date: Mon, 15 Jun 2020 23:05:33 -0500
From: Jacob Bachmeyer <jcb62281@HIDDEN>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17
 Mnenhy/0.7.6.0
MIME-Version: 1.0
To: Tom de Vries <tdevries@HIDDEN>
Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when
 triggered in test-case
References: <e5f42a45-5773-8819-03d0-854015a925a0@HIDDEN>
In-Reply-To: <e5f42a45-5773-8819-03d0-854015a925a0@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.5 (/)
X-Debbugs-Envelope-To: 41824
Cc: 41824 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Reply-To: jcb62281@HIDDEN
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.5 (/)

Tom de Vries wrote:
> Hi,
>
> the gdb testsuite uses the dejagnu framework.
>   

Apologies for the delay in responding; I took a few days to think about 
this issue because it requires a careful balance, as silently continuing 
after aborting a testcase could cause errors to go unnoticed in large 
testsuites.

> If I add a call to foobar at the end of a test-case in that testsuite,
> and I do a run of the testsuite, then the test-case aborts, and
> subsequently the testsuite run aborts, that is, no further tests are run.
>
> While it is as expected that the test-case aborts, it's not desirable
> that the testsuite run aborts.
>   

The main problem I have with this is that calling an unknown procedure 
is presumably a very serious error.  Aborting the entire run is a good 
way to ensure that the problem is noticed, but perhaps with a testsuite 
as large (and long-running) as the GDB testsuite has become, continuing 
may also be reasonable to salvage at least some results.

Where in the GDB testsuite is this a problem?  When is it appropriate to 
continue after such a serious error?  How to ensure that the error is 
not buried in the logs amidst the normal output from other tests and is 
actually noticed?

> We've installed a hack in ${tool}_init/${tool}_finish to workaround this
> in the gdb testsuite (
> https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3
> ).
>
> It would preferable though if this problem was addressed in dejagnu,
> such that we can eventually remove the hack.
>   

That hack (if not reverted entirely) needs to be made conditional on the 
actual existence of ::tcl_unknown as soon as possible -- the existence 
of ::tcl_unknown (kept to allow Tcl autoloading to work) is very much an 
internal implementation detail, and it *will* be moved out of the global 
namespace and eventually *will* cease to exist entirely in the 
interpreters that run testcases.  Really, tcl_unknown is not supposed to 
be there and relying on it introduces a latent bug into the GDB testsuite.

I am sorry, and I will seek to work with you to fix these problems with 
a minimum of breakage, but I have to put my foot down on this:  I 
*cannot* treat every internal symbol as long-term stable API.  That hack 
will *not* be supported long-term.  Any GDB releases that include it 
*will* break with some as-yet-undetermined future DejaGnu release.  I 
have to draw a line somewhere or I may as well declare DejaGnu 
unmaintained and frozen at 1.6.2 forever.  I have no inclination to do 
the latter.

> A possible solution to this problem could be to make the exiting on
> error (which is done in dejagnu's version of proc unknown in
> framework.exp) optional.
>   

This is obviously a good solution, but leads to the obvious question of 
how (command-line option?  (Make normally aborts on error, but has a 
--keep-going option.)  configuration variable?) that should be 
determined, or if exit-on-unknown-procedure should ever be a default.  
Could simply reporting an UNRESOLVED test (indicating that the testcase 
script aborted due to calling an undefined command) be a better option?  
It would show up in the summary report at the end.

At first, I was planning to schedule this for the 1.7 series, but I 
checked and runtest.exp:runtest already catches Tcl errors, so fixing 
this does not require a significant change to the normal control flow.  
Would simply reporting an "UNRESOLVED:  testcase $file aborted on 
unknown command '$what'" result, then propagating the Tcl error in the 
::unknown proc be a workable solution?


-- Jacob




Information forwarded to bug-dejagnu@HIDDEN:
bug#41824; Package dejagnu. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 12 Jun 2020 08:35:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jun 12 04:35:52 2020
Received: from localhost ([127.0.0.1]:38887 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1jjfAF-0004LL-QC
	for submit <at> debbugs.gnu.org; Fri, 12 Jun 2020 04:35:52 -0400
Received: from lists.gnu.org ([209.51.188.17]:52910)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tdevries@HIDDEN>) id 1jjfAE-0004LE-Ny
 for submit <at> debbugs.gnu.org; Fri, 12 Jun 2020 04:35:51 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:45826)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <tdevries@HIDDEN>) id 1jjfAE-00047j-Jy
 for bug-dejagnu@HIDDEN; Fri, 12 Jun 2020 04:35:50 -0400
Received: from mx2.suse.de ([195.135.220.15]:45488)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <tdevries@HIDDEN>) id 1jjfAC-0003rl-Pj
 for bug-dejagnu@HIDDEN; Fri, 12 Jun 2020 04:35:50 -0400
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (unknown [195.135.220.254])
 by mx2.suse.de (Postfix) with ESMTP id 52CC4AC79
 for <bug-dejagnu@HIDDEN>; Fri, 12 Jun 2020 08:35:51 +0000 (UTC)
To: bug-dejagnu@HIDDEN
From: Tom de Vries <tdevries@HIDDEN>
Subject: Dejagnu's unknown proc aborts testsuite run when triggered in
 test-case
Autocrypt: addr=tdevries@HIDDEN; keydata=
 xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7
 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc
 u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l
 CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P
 TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB
 AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL
 MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh
 CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo
 eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ
 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ
 UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn
 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827
 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B
 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W
 pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ
 Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z
 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9
 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats
 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3
 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7
 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC
 q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U
 yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS
 DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ==
Message-ID: <e5f42a45-5773-8819-03d0-854015a925a0@HIDDEN>
Date: Fri, 12 Jun 2020 10:35:43 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=195.135.220.15; envelope-from=tdevries@HIDDEN;
 helo=mx2.suse.de
X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 23:44:17
X-ACL-Warn: Detected OS   = Linux 2.2.x-3.x (no timestamps) [generic]
X-Spam_score_int: -41
X-Spam_score: -4.2
X-Spam_bar: ----
X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3,
 RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=_AUTOLEARN
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hi,

the gdb testsuite uses the dejagnu framework.

If I add a call to foobar at the end of a test-case in that testsuite,
and I do a run of the testsuite, then the test-case aborts, and
subsequently the testsuite run aborts, that is, no further tests are run.

While it is as expected that the test-case aborts, it's not desirable
that the testsuite run aborts.

We've installed a hack in ${tool}_init/${tool}_finish to workaround this
in the gdb testsuite (
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3
).

It would preferable though if this problem was addressed in dejagnu,
such that we can eventually remove the hack.

A possible solution to this problem could be to make the exiting on
error (which is done in dejagnu's version of proc unknown in
framework.exp) optional.

Thanks,
- Tom




Acknowledgement sent to Tom de Vries <tdevries@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-dejagnu@HIDDEN. Full text available.
Report forwarded to bug-dejagnu@HIDDEN:
bug#41824; Package dejagnu. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 17 Jun 2020 02:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.