GNU bug report logs - #31493
please add null-terminated line output for sha256sum, sha1sum, etc

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31493 in the body.
You can then email your comments to 31493 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Daniel Kahn Gillmor <dkg <at> fifthhorseman.net>
To: bug-coreutils <at> gnu.org
Subject: please add null-terminated line output for sha256sum, sha1sum, etc
Date: Fri, 18 May 2018 10:49:28 -0400

[Message part 1 (text/plain, inline)]

Hi GNU coreutils developers!

This is a feature request for the hashing/digesting tools in coreutils,
like sha256sum, sha1sum, md5sum, etc.

currently, those tools emit line-oriented output about the files that
they hash.  if a file has a newline in the name, those tools emit output
with a leading /, like so:

0 dkg <at> alice:~$ sha256sum a*
348df4eb47f9230bfe89637afe7409bec883424d822257b6cbbce93ee780d992  a
98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4  a	b
\98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4  a\nb\n
0 dkg <at> alice:~$ 

most users of sha256sum (etc) in shell scripts probably are unaware of
this behavior and just cross their fingers and hope no file has a
newline in its filename.

However, other common tools have processing options that use
NUL-terminated records. For example, sort (-z) and find (-print0) and
xargs (-0)

It would be nice if sha256sum (etc) had a comparable functionality to be
able to fit better into pipelines with these other tools.

So i'm proposing a flag -z, --zero-terminated for these tools that
alters the output (and input) format, to use NUL chars instead of
newlines.  presumably when that flag is used, the emitted data should
*not* be backslash-escaped.

      --dkg

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 31493 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Daniel Kahn Gillmor <dkg <at> fifthhorseman.net>, 31493 <at> debbugs.gnu.org
Subject: Re: bug#31493: please add null-terminated line output for sha256sum, 
 sha1sum, etc
Date: Fri, 18 May 2018 10:09:30 -0700

On 18/05/18 07:49, Daniel Kahn Gillmor wrote:
> So i'm proposing a flag -z, --zero-terminated for these tools that
> alters the output (and input) format, to use NUL chars instead of
> newlines.  presumably when that flag is used, the emitted data should
> *not* be backslash-escaped.

This is a good suggestion I think.
`...sum | sort` is a common operation for example.
I'll do this for the upcoming release.

cheers,
Pádraig

Message #11 received at 31493 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Daniel Kahn Gillmor <dkg <at> fifthhorseman.net>, 31493 <at> debbugs.gnu.org
Subject: Re: bug#31493: please add null-terminated line output for sha256sum, 
 sha1sum, etc
Date: Sun, 10 Jun 2018 18:02:02 -0700

[Message part 1 (text/plain, inline)]

On 18/05/18 07:49, Daniel Kahn Gillmor wrote:
> Hi GNU coreutils developers!
> 
> This is a feature request for the hashing/digesting tools in coreutils,
> like sha256sum, sha1sum, md5sum, etc.
> 
> currently, those tools emit line-oriented output about the files that
> they hash.  if a file has a newline in the name, those tools emit output
> with a leading /, like so:
> 
> 0 dkg <at> alice:~$ sha256sum a*
> 348df4eb47f9230bfe89637afe7409bec883424d822257b6cbbce93ee780d992  a
> 98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4  a	b
> \98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4  a\nb\n
> 0 dkg <at> alice:~$ 
> 
> most users of sha256sum (etc) in shell scripts probably are unaware of
> this behavior and just cross their fingers and hope no file has a
> newline in its filename.
> 
> However, other common tools have processing options that use
> NUL-terminated records. For example, sort (-z) and find (-print0) and
> xargs (-0)
> 
> It would be nice if sha256sum (etc) had a comparable functionality to be
> able to fit better into pipelines with these other tools.
> 
> So i'm proposing a flag -z, --zero-terminated for these tools that
> alters the output (and input) format, to use NUL chars instead of
> newlines.  presumably when that flag is used, the emitted data should
> *not* be backslash-escaped.

I used --zero rather than --zero-terminated like other tools that
only change _output_ format with -z, rather than supporting it also
as an _input_ format. It would complicate the processing to support
--check of input with --zero, with no added functionality.
Having --check support this format, would also an incompat with
older versions of these tools.

As an example of processing now possible, the following would
robustly pass duplicate files to xargs for processing:

  $ md5sum src/md5sum.{c,c} -z |
    sort -z | uniq -z -d -w32 | cut -z -c35- |
    xargs -r0

    src/md5sum.c

cheers,
Pádraig

[md5sum--zero.patch (text/x-patch, attachment)]

Message #14 received at 31493 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Kahn Gillmor <dkg <at> fifthhorseman.net>
To: Pádraig Brady <P <at> draigBrady.com>, 31493 <at> debbugs.gnu.org
Subject: Re: bug#31493: please add null-terminated line output for sha256sum,
 sha1sum, etc
Date: Mon, 11 Jun 2018 16:17:35 -0400

On Sun 2018-06-10 18:02:02 -0700, Pádraig Brady wrote:
> I used --zero rather than --zero-terminated like other tools that
> only change _output_ format with -z, rather than supporting it also
> as an _input_ format. It would complicate the processing to support
> --check of input with --zero, with no added functionality.
> Having --check support this format, would also an incompat with
> older versions of these tools.

this looks like a fine first step to me, and i hope it will be merged.
thanks for working on it, Pádraig!

I'm not convinced yet that adding --zero for the --check functionality
is a guaranteed incompatibility.  Clearly the same flags would need to
be used on both sides, but i don't see why having it available in
--check would be a problem.

        --dkg

Message #17 received at 31493 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 31493 <at> debbugs.gnu.org
Subject: Re: bug#31493: please add null-terminated line output for sha256sum, 
 sha1sum, etc
Date: Mon, 29 Oct 2018 21:29:17 -0600

tags 31493 fixed
close 31493
stop

Pushed here:

https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=c744c65f50e03846e037361d613a5d2b2e334b61

So closing as "fixed".

-assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.