GNU bug report logs - #62406
“! failing-command” pattern in shell tests is wrong

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 62406 in the body.
You can then email your comments to 62406 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: “! failing-command” pattern in shell
 tests is wrong
Date: Thu, 23 Mar 2023 17:00:14 +0100

In <https://issues.guix.gnu.org/43616> (commit
d8934360d2453a403b5433e71d09188e4ed23b57), we changed:

  if command that should fail; then false; else true; fi

to:

  ! command that should fail

I had reservations back then, and now I know why: :-)

--8<---------------cut here---------------start------------->8---
$ bash -xe -c '! true; true'
+ true
+ true
$ echo $?
0
$ bash -xe -c '! false; true'
+ false
+ true
$ echo $?
0
--8<---------------cut here---------------end--------------->8---

Whether or not the command following the exclamation mark succeeds, the
statement succeeds.  Bummer.

The Bash manual (info "(bash) Pipelines") reads:

  If the reserved word '!' precedes the pipeline, the exit status is the
  logical negation of the exit status as described above.  The shell
  waits for all commands in the pipeline to terminate before returning a
  value.

To me, that means it should work as we thought, but it’s a fact that it
doesn’t.

Thoughts?

Ludo’.

Message #8 received at 62406 <at> debbugs.gnu.org (full text, mbox):

From: Eric Bavier <bavier <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>, 62406 <at> debbugs.gnu.org
Subject: Re: bug#62406: “! failing-command” pattern in shell tests is wrong
Date: Thu, 23 Mar 2023 18:53:47 +0000

[Message part 1 (text/plain, inline)]

On Thu, 2023-03-23 at 17:00 +0100, Ludovic Courtès wrote:
> In <https://issues.guix.gnu.org/43616> (commit
> d8934360d2453a403b5433e71d09188e4ed23b57), we changed:
> 
>   if command that should fail; then false; else true; fi
> 
> to:
> 
>   ! command that should fail
> 
> I had reservations back then, and now I know why: :-)
> 
> --8<---------------cut here---------------start------------->8---
> $ bash -xe -c '! true; true'
> + true
> + true
> $ echo $?
> 0
> $ bash -xe -c '! false; true'
> + false
> + true
> $ echo $?
> 0
> --8<---------------cut here---------------end--------------->8---
> 
> Whether or not the command following the exclamation mark succeeds, the
> statement succeeds.  Bummer.

I think it's maybe not that the statement succeeds regardless.  But that 'set
-e' doesn't consider it a "failure".  From "The Set Builtin":

    '-e'
          Exit immediately if a pipeline (*note Pipelines::)... returns a
          non-zero status.  The shell does not exit if the command that
          fails is ..., or if the command's return status is being
          inverted with '!'.

So in each of your examples, execution continues to the second 'true'
statement and the overall exit status is 0.  This is not the behavior we want
in our tests.

The purpose of d89343 was to ease visual parsing of the tests.  I mentioned
having used the '!' syntax in my own shell tests, but I realize now that I
was not relying on `set -e` like guix is.

I'll consider a few options.  Do we have a known issue where this is causing
a test to not to catch a failure?

Thanks for bringing this up,
`~Eric

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 62406 <at> debbugs.gnu.org (full text, mbox):

From: Martin Castillo <castilma <at> uni-bremen.de>
To: Ludovic Courtès <ludo <at> gnu.org>, 62406 <at> debbugs.gnu.org
Cc: Eric Bavier <bavier <at> posteo.net>
Subject: Re: bug#62406: “! failing-command” pattern in shell tests is wrong
Date: Sun, 26 Mar 2023 20:16:38 +0200

Hi,

Am 23.03.23 um 17:00 schrieb Ludovic Courtès:> In 
<https://issues.guix.gnu.org/43616> (commit
> d8934360d2453a403b5433e71d09188e4ed23b57), we changed:
>
>    if command that should fail; then false; else true; fi
>
> to:
>
>    ! command that should fail
>
> I had reservations back then, and now I know why: :-)
>
> --8<---------------cut here---------------start------------->8---
> $ bash -xe -c '! true; true'
> + true
> + true
> $ echo $?
> 0
> $ bash -xe -c '! false; true'
> + false
> + true
> $ echo $?
> 0
> --8<---------------cut here---------------end--------------->8---
>
> Whether or not the command following the exclamation mark succeeds, the
> statement succeeds.  Bummer.
>
> The Bash manual (info "(bash) Pipelines") reads:
>
>    If the reserved word '!' precedes the pipeline, the exit status is the
>    logical negation of the exit status as described above.  The shell
>    waits for all commands in the pipeline to terminate before returning a
>    value.
>
> To me, that means it should work as we thought, but it’s a fact that it
> doesn’t.
the documentation on `-e` says:

-e      Exit  immediately  if a pipeline (which may consist of a
        single simple command), a list, or  a  compound  command
        (see SHELL GRAMMAR above), exits with a non-zero status.
        The shell does not exit [...] if the
        command's return value is being inverted with !.

I have no idea what might be the rationale for that.

Message #16 received at 62406 <at> debbugs.gnu.org (full text, mbox):

From: Martin Castillo <castilma <at> uni-bremen.de>
To: Ludovic Courtès <ludo <at> gnu.org>, 62406 <at> debbugs.gnu.org
Cc: Eric Bavier <bavier <at> posteo.net>
Subject: Re: bug#62406: “! failing-command” pattern in shell tests is wrong
Date: Sun, 26 Mar 2023 20:21:49 +0200

Sorry for my answer. I overlooked that Eric already answered.

Message #19 received at 62406 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Eric Bavier <bavier <at> posteo.net>
Cc: 62406 <at> debbugs.gnu.org
Subject: Re: bug#62406: “! failing-command”
 pattern in shell tests is wrong
Date: Tue, 28 Mar 2023 18:21:12 +0200

Hi Eric,

Eric Bavier <bavier <at> posteo.net> skribis:

> The purpose of d89343 was to ease visual parsing of the tests.  I mentioned
> having used the '!' syntax in my own shell tests, but I realize now that I
> was not relying on `set -e` like guix is.
>
> I'll consider a few options.

Neat.  I guess we could have a ‘lib.sh’ with an ‘expect_fail’ function
or something.

> Do we have a known issue where this is causing a test to not to catch
> a failure?

No; I noticed it while writing a new test that I expected to fail.

Thanks for your feedback!  Shell semantics are definitely weird.  :-)

Ludo’.

Message #22 received at 62406 <at> debbugs.gnu.org (full text, mbox):

From: Eric Bavier <bavier <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 62406 <at> debbugs.gnu.org
Subject: Re: bug#62406: “! failing-command” pattern in shell tests is wrong
Date: Thu, 20 Apr 2023 05:48:55 +0000

[Message part 1 (text/plain, inline)]

On Tue, 2023-03-28 at 18:21 +0200, Ludovic Courtès wrote:
> Hi Eric,
> 
> Eric Bavier <bavier <at> posteo.net> skribis:
> 
> > The purpose of d89343 was to ease visual parsing of the tests.  I mentioned
> > having used the '!' syntax in my own shell tests, but I realize now that I
> > was not relying on `set -e` like guix is.
> > 
> > I'll consider a few options.
> 
> Neat.  I guess we could have a ‘lib.sh’ with an ‘expect_fail’ function
> or something.
> 

Instead of a shared 'lib.sh', the attached patch uses 'cmd && false', which
has the desired semantics under 'set -e' and is no more verbose than a
wrapping function call.

If 'cmd' fails, the return status is ignored by 'set -e', which considers
only the return status of a command following the final '&&' or '||'.  And
because 'cmd' failed the statement short-circuits without executing the
'false.  Otherwise, if 'cmd' succeeds, the 'false' is executed and the shell
exits immediately.

In other places the '! test ...' pattern is replaced with 'test ! ...'.

There was some small amount of fall-out.  I fixed a couple issues where I
could:

  - tests/guix-archive.sh: added '--export' to command

  - tests/guix-style.sh: added an escape to a sed pattern

But a couple others have failures I'm not as confident in fixing myself:

  - tests/guix-refresh.sh: 'guix refresh' seems to not exit with a failure
status if a warning is issued, but the tests seem to think it should.

  - tests/guix-git-authenticate.sh: A general failure to authenticate a
particular commit.  This could be an issue with my test environment.

This patch should probably not be applied until those tests are fixed.  I
would appreciate any help with that.

`~Eric
> 

[0001-tests-Fix-checks-for-expected-failures.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.