GNU bug report logs - #28105
Inconsistent exit code with --files-without-match

Previous Next

Package: grep;

Reported by: Anthony Sottile <asottile <at> umich.edu>

Date: Tue, 15 Aug 2017 22:24:01 UTC

Severity: normal

Done: Paul Eggert <eggert <at> cs.ucla.edu>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28105 in the body.
You can then email your comments to 28105 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Tue, 15 Aug 2017 22:24:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Anthony Sottile <asottile <at> umich.edu>:
New bug report received and forwarded. Copy sent to bug-grep <at> gnu.org. (Tue, 15 Aug 2017 22:24:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Anthony Sottile <asottile <at> umich.edu>
To: bug-grep <at> gnu.org
Subject: Inconsistent exit code with --files-without-match
Date: Tue, 15 Aug 2017 15:01:13 -0700
Given the exit code of `grep` vs. `grep -v`, I expect similar results
when comparing `grep -l` and `grep -L` (`--files-with-matches` /
`--files-without-match`)

For at least `grep` / `grep -v` / `grep -l` when the "search" is
successful, it exits `0` and when the search is unsuccessful it exits
`1`.  `grep -L` (`--files-without-match`) does not follow this pattern
which I believe to be an oversight / bug.

Consider the following:

$ echo hello > f

# Search is for "hello", it matches so exit code is 0
$ grep hello -- f; echo $?
hello
0
# Search is for "hi", it fails to match so the exit code is 1
$ grep hi -- f; echo $?
1

# Search is for not "hi", it matches so exit code is 0
$ grep -v hi -- f; echo $?
hello
0
# Search is for not "hello", it fails to match so the exit code is 1
$ grep -v hello -- f; echo $?
1

# Search is for filenames containing hello, it matches so exit code is 0
$ grep -l hello -- f; echo $?
f
0
# Search is for filenames containing hi, it fails to match so exit code is 1
$ grep -l hi -- f; echo $?
1

# Search is for filenames not containing hi, this search is successful
**but it exits 1**
$ grep -L hi -- f; echo $?
f
1
# Search is for filenames not containing hello, this search fails
**but it exits 0**
$ grep -L hello -- f; echo $?
0

Anthony




Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Wed, 16 Aug 2017 05:16:01 GMT) Full text and rfc822 format available.

Message #8 received at 28105 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Anthony Sottile <asottile <at> umich.edu>, 28105 <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Tue, 15 Aug 2017 22:15:03 -0700
Anthony Sottile wrote:
> # Search is for filenames not containing hi, this search is successful
> **but it exits 1**
> $ grep -L hi -- f; echo $?
> f
> 1
> # Search is for filenames not containing hello, this search fails
> **but it exits 0**
> $ grep -L hello -- f; echo $?
> 0

The grep documentation says exit status depends on whether lines (not files) are 
selected, so grep is conforming to its documentation here. Perhaps grep's 
documentation and behavior could be changed, though I worry that existing uses 
of grep might be adversely affected. What is the use case that justifies such a 
change?




Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Wed, 16 Aug 2017 22:05:02 GMT) Full text and rfc822 format available.

Message #11 received at 28105 <at> debbugs.gnu.org (full text, mbox):

From: Anthony Sottile <asottile <at> umich.edu>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 28105 <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Wed, 16 Aug 2017 15:04:25 -0700
On Tue, Aug 15, 2017 at 10:15 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> Anthony Sottile wrote:
>>
>> # Search is for filenames not containing hi, this search is successful
>> **but it exits 1**
>> $ grep -L hi -- f; echo $?
>> f
>> 1
>> # Search is for filenames not containing hello, this search fails
>> **but it exits 0**
>> $ grep -L hello -- f; echo $?
>> 0
>
>
> The grep documentation says exit status depends on whether lines (not files)
> are selected, so grep is conforming to its documentation here. Perhaps
> grep's documentation and behavior could be changed, though I worry that
> existing uses of grep might be adversely affected. What is the use case that
> justifies such a change?

Admittedly, I'm down a rabbit hole at this point but I'll try and
explain the full pathway :)

My initial goal is I am writing a tool to manage multiple git repositories:
- https://github.com/asottile/all-repos

One of the goals of this tool is to be able to grep across all of the
repositories, mapping `git grep` over each repository fits this goal
pretty well.

`all-repos-grep` supports two modes:
- normal mode:
    - for each repository run `git grep $args`
    - if that `git grep` command returns `0`, reproduce the output
with the repository name prefixing each line.
- --repos-with-matches:
    - for each repository run `git grep --quiet $args`
    - if that `git grep` command returns `0`, include this repository
name in the output

While trying some queries, I noticed an oddity (with git grep, not
with gnu grep) in that `git grep -L ...` and `git grep --quiet -L`
were returning different exit codes.

I then reported a bug to the git mailing list about this discrepancy
between return codes when `--quiet` was passed on the cli.  I also
submitted a patch to fix this discrepancy.

It was at this point that a git maintainer noticed that `git grep -L`
and `grep -L` disagreed in exit codes (deferring my patch until the
correct way forward on what `git grep`'s exit codes should be).

The current `git grep -L` exits 1 when no files fit the search (that
is, they *don't* contain the pattern), while `grep -L` exits 0 (the
current `git grep -L` behaviour is the one I'm suggesting for `grep
-L`).

I essentially have two options:
- grep documents / adjust behaviour to be more consistent with "return
0 when successfully finding things", `git grep -L` behaves the same as
today with exit codes.
- adjust my tool to *ignore* exit codes and instead look at output.
This would also mean dropping the `--quiet` optimization.

My main argument is that the inconsistency between grep with/without
`-L` hurts scriptability due to inconsistency.

Anthony




Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Thu, 17 Aug 2017 00:05:02 GMT) Full text and rfc822 format available.

Message #14 received at 28105 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Anthony Sottile <asottile <at> umich.edu>
Cc: 28105 <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Wed, 16 Aug 2017 17:04:13 -0700
On 08/16/2017 03:04 PM, Anthony Sottile wrote:
> It was at this point that a git maintainer noticed that `git grep -L`
> and `grep -L` disagreed in exit codes (deferring my patch until the
> correct way forward on what `git grep`'s exit codes should be).
>
Thanks for the summary. Do you have a URL for the git bug report and 
maintainer response? That'd be helpful here. You do make a case that the 
exit status for grep -L should differ from what it is now.





Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Thu, 17 Aug 2017 04:18:02 GMT) Full text and rfc822 format available.

Message #17 received at 28105 <at> debbugs.gnu.org (full text, mbox):

From: Anthony Sottile <asottile <at> umich.edu>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 28105 <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Wed, 16 Aug 2017 21:17:48 -0700
On Wed, Aug 16, 2017 at 5:04 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> On 08/16/2017 03:04 PM, Anthony Sottile wrote:
>>
>> It was at this point that a git maintainer noticed that `git grep -L`
>> and `grep -L` disagreed in exit codes (deferring my patch until the
>> correct way forward on what `git grep`'s exit codes should be).
>>
> Thanks for the summary. Do you have a URL for the git bug report and
> maintainer response? That'd be helpful here. You do make a case that the
> exit status for grep -L should differ from what it is now.
>

I'm not sure what the "official" mirror of the mailing list is but
here's a mirror that I found that works:

- Initial bugreport about `--quiet`:
http://marc.info/?l=git&m=150281283027760&w=2
- Patch and discussion: http://marc.info/?l=git&m=150283014132406&w=2




Reply sent to Paul Eggert <eggert <at> cs.ucla.edu>:
You have taken responsibility. (Thu, 17 Aug 2017 21:23:02 GMT) Full text and rfc822 format available.

Notification sent to Anthony Sottile <asottile <at> umich.edu>:
bug acknowledged by developer. (Thu, 17 Aug 2017 21:23:02 GMT) Full text and rfc822 format available.

Message #22 received at 28105-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Anthony Sottile <asottile <at> umich.edu>
Cc: 28105-done <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Thu, 17 Aug 2017 14:21:59 -0700
[Message part 1 (text/plain, inline)]
Thanks for the pointer. It looks to me like git-grep's behavior is 
better than grep's, so I installed the attached.


[0001-grep-L-exits-with-status-0-if-a-file-is-selected.patch (text/x-patch, attachment)]

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Fri, 15 Sep 2017 11:24:04 GMT) Full text and rfc822 format available.

bug unarchived. Request was from Antonio Diaz Diaz <antonio <at> gnu.org> to control <at> debbugs.gnu.org. (Thu, 06 Aug 2020 17:40:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Fri, 07 Aug 2020 11:13:02 GMT) Full text and rfc822 format available.

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Antonio Diaz Diaz <antonio <at> gnu.org>
To: bug-grep <at> gnu.org
Subject: bug#28105: Inconsistent exit code with --files-without-match
Date: Fri, 07 Aug 2020 13:14:05 +0200
Sorry for being late in the conversation, but I didn't know about this 
change in grep's exit status until now.

On Tue, 15 Aug 2017 22:16:58 -0700 Paul Eggert wrote:
> The grep documentation says exit status depends on whether lines (not
> files) are selected, so grep is conforming to its documentation here.

Conforming to its documentation and to POSIX:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html
EXIT STATUS
    The following exit values shall be returned:
     0  One or more lines were selected.
     1  No lines were selected.
    >1  An error occurred.

Inverting the exit status of 'grep -L' also departs from the exit status one 
could expect from the etymology of grep itself. (The ed command 'g/RE/p' 
fails if no lines match).


> Perhaps grep's documentation and behavior could be changed, though I
> worry that existing uses of grep might be adversely affected.

I have recently received a report[1] that the change[2] in exit status of 
GNU grep when using -L has in fact affected zgrep adversely.
[1] http://www.mail-archive.com/zutils-bug <at> nongnu.org/msg00120.html
[2] http://www.mail-archive.com/bug-grep <at> gnu.org/msg06886.html

This change in grep breaks zgrep (from zutils) and gzgrep (from gzip) (and 
possibly other *zgrep tools) because they both feed the decompressed 
contents of the file to grep through stdin and must replace the string 
"(standard input)" printed by grep with the name of the matching file, for 
which they depend on the exit status of grep which has now been changed.

What worries me most is that I have been unable to find a reliable way to 
make zgrep work with both old and new versions of GNU grep.

This change may also be incompatible with other tools like grep-dctrl:
http://manpages.debian.org/stretch/dctrl-tools/grep-dctrl.1.en.html#DIAGNOSTICS

This change also breaks the internal logic of the -q option. Until GNU grep 
3.1 the -q (quiet) option just suppressed the output, but didn't change the 
exit status in absence of errors. But in GNU grep 3.2 to 3.4 -q inverts the 
exit status when -L is also used. Now 'grep -L > /dev/null' and 'grep -Lq' 
return opposite values.

Note that the original reporter who requested this change in GNU grep also 
reported the inconsistency of -Lq to git-grep:
http://marc.info/?l=git&m=150281283027760&w=2
Inconsistent exit code for `git grep --files-without-match` with `--quiet`
  $ ./git-grep --files-without-match nope -- blob.c; echo $?
  blob.c
  0
  $ ./git-grep --files-without-match --quiet nope -- blob.c; echo $?
  1
  I expect both to exit 0

I propose to revert this change to remain consistent with POSIX. If this is 
not possible or desirable, then I propose to limit the function of -q to 
quieten standard output but without reversing the exit status of -L, and 
modify the POSIX standard to reflect this change.

Best regards,
Antonio.




Information forwarded to bug-grep <at> gnu.org:
bug#28105; Package grep. (Sat, 22 Aug 2020 21:12:01 GMT) Full text and rfc822 format available.

Message #32 received at 28105-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Antonio Diaz Diaz <antonio <at> gnu.org>
Cc: 28105-done <at> debbugs.gnu.org
Subject: Re: bug#28105: Inconsistent exit code with --files-without-match
Date: Sat, 22 Aug 2020 14:11:47 -0700
[Message part 1 (text/plain, inline)]
On 8/7/20 4:14 AM, Antonio Diaz Diaz wrote:
> I propose to revert this change to remain consistent with POSIX.

It's not a POSIX issue, since POSIX doesn't specify -L, which means grep can do 
whatever it wants if you specify -L.

> But in GNU grep 3.2 to 3.4 -q inverts the exit status when -L is also used. Now 'grep -L > /dev/null' and 'grep -Lq' return opposite values.

That is indeed a bug.

I looked into this a bit further, and it turns out that when the Git folks were 
told that git-grep behaved differently from GNU grep 3.1 and earlier, they 
changed git-grep to match. This was around the same time we changed GNU grep to 
behave more like the git-grep at the time. What a comedy of errors, huh?

Anyway, given all these issues it does seem like it's wise to revert the change. 
I installed the attached patch into GNU grep master, with the goal of something 
like this appearing in the next grep release. Thanks for reporting the problem.
[0001-Revert-L-exit-status-change-introduced-in-grep-3.2.patch (text/x-patch, attachment)]

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sun, 20 Sep 2020 11:24:03 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 190 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.