GNU bug report logs - #79077
host_execute and non-zero exit status

Previous Next

To reply to this bug, email your comments to 79077 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: bug-dejagnu <at> gnu.org
Subject: host_execute and non-zero exit status
Date: Wed, 23 Jul 2025 09:31:21 +0200

Hi,

The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
check the exit status of the executed test programs. A test program
that simply aborts (e.g. using the C function of the same name) won't
cause any testsuite failures. This seems brittle and like a
misfeature.

What is the supposed way to deal with this?

Thanks,

Marc

--

[1] https://www.gnu.org/software/dejagnu/manual/Running-unit-tests.html

Message #8 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Wed, 23 Jul 2025 22:08:43 -0500

On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
> Hi,
>
> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> check the exit status of the executed test programs. A test program
> that simply aborts (e.g. using the C function of the same name) won't
> cause any testsuite failures. This seems brittle and like a
> misfeature.
>
> What is the supposed way to deal with this?

The host_execute procedure is intended for running unit test programs 
that speak a special DejaGnu protocol and ignoring exit codes from unit 
test programs is intended.  The DejaGnu unit testing protocol does not 
depend on the exit code of the unit test program at all, because that 
exit code might not be available in all environments.

Instead, DejaGnu expects an explicit "END" token from the unit test 
program to indicate that the program has reached its intended 
completion.  A warning is produced if this token is not observed; 
perhaps a future version of DejaGnu should insert an UNRESOLVED result 
like we currently do when a Tcl test script aborts?


-- Jacob

Message #11 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: jcb62281 <at> gmail.com
Cc: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Thu, 24 Jul 2025 07:57:31 +0200

Hi Jacob,

Thank you for your quick response!

Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <jcb62281 <at> gmail.com>:
>
> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
> > Hi,
> >
> > The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> > check the exit status of the executed test programs. A test program
> > that simply aborts (e.g. using the C function of the same name) won't
> > cause any testsuite failures. This seems brittle and like a
> > misfeature.
> >
> > What is the supposed way to deal with this?
>
> The host_execute procedure is intended for running unit test programs
> that speak a special DejaGnu protocol and ignoring exit codes from unit
> test programs is intended.  The DejaGnu unit testing protocol does not
> depend on the exit code of the unit test program at all, because that
> exit code might not be available in all environments.

I am sorry if I wasn't clear enough in my previous email. I didn't
mean that the exit code should be part of the actual protocol, only
that running the testcase should be considered unsuccessful if the
program exits with a failure code (in a POSIX system; in some other
environment, there may be other indications for failure of execution).

> Instead, DejaGnu expects an explicit "END" token from the unit test
> program to indicate that the program has reached its intended
> completion.  A warning is produced if this token is not observed;
> perhaps a future version of DejaGnu should insert an UNRESOLVED result
> like we currently do when a Tcl test script aborts?

I think inserting an UNRESOLVED result would be more appropriate.
Otherwise, there would be no formal failure result if a unit test
breaks due to, say, a segfault.

libgccjit uses a heavily patched version of host_execute ([1]) that
also handles tests run under Valgrind; perhaps some of the ideas from
that version can be incorporated into DejaGnu proper.

-- Marc

[1] https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89

>
>
> -- Jacob
>
>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.