GNU bug report logs - #72232
"make dist" is not reproducible

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 72232 in the body.
You can then email your comments to 72232 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-coreutils <at> gnu.org
Cc: Simon Josefsson <simon <at> josefsson.org>
Subject: "make dist" is not reproducible
Date: Sun, 21 Jul 2024 18:18:06 +0200

Hi,

Subsequent runs of "make dist" in the same environment produce tarballs with
different contents.

How to reproduce:

In a git checkout of coreutils, do:

$ echo snapshot > .tarball-version
$ ./configure; make -k maintainer-clean
$ ./bootstrap
$ ./configure; make dist
$ mkdir 1; (cd 1 && tar xf ../*.tar.gz)
$ make; make dist
$ mkdir 2; (cd 2 && tar xf ../*.tar.gz)
$ diff -r -q 1 2

The last command produces a difference:

$ diff -r -q 1 2
Files 1/coreutils-snapshot/.timestamp and 2/coreutils-snapshot/.timestamp differ
$ cat 1/coreutils-snapshot/.timestamp
1721577765
$ cat 2/coreutils-snapshot/.timestamp
1721577832

Can this be avoided? Can the contents of '.timestamp' always be the same?
(Wouldn't it be enough to give it a different modification time, each time?)

Bruno

Message #10 received at 72232-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>, 72232-done <at> debbugs.gnu.org
Cc: Simon Josefsson <simon <at> josefsson.org>
Subject: Re: bug#72232: "make dist" is not reproducible
Date: Sun, 21 Jul 2024 18:44:04 +0100

[Message part 1 (text/plain, inline)]

On 21/07/2024 17:18, Bruno Haible wrote:
> Hi,
> 
> Subsequent runs of "make dist" in the same environment produce tarballs with
> different contents.
> 
> How to reproduce:
> 
> In a git checkout of coreutils, do:
> 
> $ echo snapshot > .tarball-version
> $ ./configure; make -k maintainer-clean
> $ ./bootstrap
> $ ./configure; make dist
> $ mkdir 1; (cd 1 && tar xf ../*.tar.gz)
> $ make; make dist
> $ mkdir 2; (cd 2 && tar xf ../*.tar.gz)
> $ diff -r -q 1 2
> 
> The last command produces a difference:
> 
> $ diff -r -q 1 2
> Files 1/coreutils-snapshot/.timestamp and 2/coreutils-snapshot/.timestamp differ
> $ cat 1/coreutils-snapshot/.timestamp
> 1721577765
> $ cat 2/coreutils-snapshot/.timestamp
> 1721577832
> 
> Can this be avoided? Can the contents of '.timestamp' always be the same?
> (Wouldn't it be enough to give it a different modification time, each time?)

Right, we should be able to adjust this.
The .timestamp file was added to support reproducible _builds_:
https://github.com/coreutils/coreutils/commit/c1b3d6587
https://reproducible-builds.org/docs/source-date-epoch/

We can just rely on the timestamp of the .tarball-version
to support reproducible _tarballs_.
That's done in the attached, which I'll apply later.

Marking this as done.

thanks,
Pádraig

[coreutils-repro-dist.patch (text/x-patch, attachment)]

Message #13 received at 72232 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 72232 <at> debbugs.gnu.org, P <at> draigBrady.com, bruno <at> clisp.org
Subject: Re: bug#72232: "make dist" is not reproducible
Date: Sun, 21 Jul 2024 10:58:30 -0700

On 2024-07-21 10:44, Pádraig Brady wrote:
> We can just rely on the timestamp of the .tarball-version
> to support reproducible _tarballs_.

Although this makes the distributed file contents reproducible, the 
tarballs themselves are still not reproducible, since they contain 
random timestamps and (I imagine) other metadata. I vaguely recall Bruno 
having a general solution for that sort of problem but don't recall 
where it got put. (I do something more complicated in TZDB.)

Message #16 received at 72232 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 72232 <at> debbugs.gnu.org, bruno <at> clisp.org
Subject: Re: bug#72232: "make dist" is not reproducible
Date: Sun, 21 Jul 2024 19:06:55 +0100

On 21/07/2024 18:58, Paul Eggert wrote:
> On 2024-07-21 10:44, Pádraig Brady wrote:
>> We can just rely on the timestamp of the .tarball-version
>> to support reproducible _tarballs_.
> 
> Although this makes the distributed file contents reproducible, the
> tarballs themselves are still not reproducible, since they contain
> random timestamps and (I imagine) other metadata. I vaguely recall Bruno
> having a general solution for that sort of problem but don't recall
> where it got put. (I do something more complicated in TZDB.)

Fair enough.  That issue is discussed at:
https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html

For now at least I'll adjust the summary
to say "tarball contents" rather than "tarballs".

thanks,
Pádraig

Message #19 received at 72232 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: 72232 <at> debbugs.gnu.org, P <at> draigbrady.com, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#72232: "make dist" is not reproducible
Date: Sun, 21 Jul 2024 20:28:06 +0200

Paul Eggert wrote:
> the tarballs themselves are still not reproducible, since they contain 
> random timestamps and (I imagine) other metadata. I vaguely recall Bruno 
> having a general solution for that sort of problem but don't recall 
> where it got put.

It was at the end of
<https://lists.gnu.org/archive/html/automake/2024-06/msg00070.html>.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.