GNU bug report logs - #48933
[PATCH] build: Make outputs of node-build-system reproducible.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 48933 in the body.
You can then email your comments to 48933 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: guix-patches <at> gnu.org
Cc: jlicht <at> fsfe.org
Subject: [PATCH] build: Make outputs of node-build-system reproducible.
Date: Wed, 9 Jun 2021 14:56:58 +0200

package.json records two hashes of package.tgz, which change for each
build, resulting in non-reproducible builds.

* guix/build/node-build-system.scm (repack): Add reproducibility options
to tar command.
---
 guix/build/node-build-system.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
index a55cab237c..9b3de43e24 100644
--- a/guix/build/node-build-system.scm
+++ b/guix/build/node-build-system.scm
@@ -120,7 +120,15 @@
   #t)
 
 (define* (repack #:key inputs #:allow-other-keys)
-  (invoke "tar" "-czf" "../package.tgz" ".")
+  (invoke "tar"
+          ;; Add options suggested by https://reproducible-builds.org/docs/archives/
+          "--sort=name"
+          (string-append "--mtime=" (getenv "SOURCE_DATE_EPOCH"))
+          "--owner=0"
+          "--group=0"
+          "--numeric-owner"
+          "--pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime"
+          "-czf" "../package.tgz" ".")
   #t)
 
 (define* (install #:key outputs inputs #:allow-other-keys)
-- 
2.31.1

Message #8 received at 48933 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 48933 <at> debbugs.gnu.org, jlicht <at> fsfe.org
Subject: Re: bug#48933: [PATCH] build: Make outputs of node-build-system
 reproducible.
Date: Wed, 16 Jun 2021 22:51:59 +0200

Hi,

Lars-Dominik Braun <lars <at> 6xq.net> skribis:

> package.json records two hashes of package.tgz, which change for each
> build, resulting in non-reproducible builds.
>
> * guix/build/node-build-system.scm (repack): Add reproducibility options
> to tar command.

Yay!

>  (define* (repack #:key inputs #:allow-other-keys)
> -  (invoke "tar" "-czf" "../package.tgz" ".")
> +  (invoke "tar"
> +          ;; Add options suggested by https://reproducible-builds.org/docs/archives/
> +          "--sort=name"
> +          (string-append "--mtime=" (getenv "SOURCE_DATE_EPOCH"))

I think it should be "--mtime=@".

> +          "--owner=0"
> +          "--group=0"
> +          "--numeric-owner"
> +          "--pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime"
> +          "-czf" "../package.tgz" ".")

I didn’t know about this ‘--pax-option’ trick; since it’s only useful
when POSIXLY_CORRECT is set, perhaps we can remove it?

(guix docker) does this:

--8<---------------cut here---------------start------------->8---
(define %tar-determinism-options
  ;; GNU tar options to produce archives deterministically.
  '("--sort=name" "--mtime=@1"
    "--owner=root:0" "--group=root:0"

    ;; When 'build-docker-image' is passed store items, the 'nlink' of the
    ;; files therein leads tar to store hard links instead of actual copies.
    ;; However, the 'nlink' count depends on deduplication in the store; it's
    ;; an "implicit input" to the build process.  '--hard-dereference'
    ;; eliminates it.
    "--hard-dereference"))
--8<---------------cut here---------------end--------------->8---

and (guix packages) does something similar.

So ‘--sort=name’ seems to be missing.

HTH,
Ludo’.

Message #11 received at 48933 <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48933 <at> debbugs.gnu.org, jlicht <at> fsfe.org
Subject: Re: bug#48933: [PATCH] build: Make outputs of node-build-system
 reproducible.
Date: Thu, 17 Jun 2021 14:12:18 +0200

[Message part 1 (text/plain, inline)]

Hi Ludo,

> >  (define* (repack #:key inputs #:allow-other-keys)
> > -  (invoke "tar" "-czf" "../package.tgz" ".")
> > +  (invoke "tar"
> > +          ;; Add options suggested by https://reproducible-builds.org/docs/archives/
*
> > +          "--sort=name"
> > +          (string-append "--mtime=" (getenv "SOURCE_DATE_EPOCH"))
> 
> I think it should be "--mtime=@".
you’re right, fixed.

> I didn’t know about this ‘--pax-option’ trick; since it’s only useful
> when POSIXLY_CORRECT is set, perhaps we can remove it?
True, removed.

> (guix docker) does this:
> and (guix packages) does something similar.
Hm, maybe it would make sense to export a set of options, so
build systems/packages can share them? Or create a package that wraps
tar with the proper options?

> So ‘--sort=name’ seems to be missing.
It’s present, see above ↑*

Updated patch attached. I’ll push it if there are no further comments.

Cheers,
Lars

[0001-build-Make-outputs-of-node-build-system-reproducible.patch (text/x-diff, attachment)]

Message #14 received at 48933 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: 48933 <at> debbugs.gnu.org, jlicht <at> fsfe.org
Subject: Re: bug#48933: [PATCH] build: Make outputs of node-build-system
 reproducible.
Date: Sun, 20 Jun 2021 22:58:31 +0200

Hi,

Lars-Dominik Braun <lars <at> 6xq.net> skribis:

[...]

>> (guix docker) does this:
>> and (guix packages) does something similar.
> Hm, maybe it would make sense to export a set of options, so
> build systems/packages can share them? Or create a package that wraps
> tar with the proper options?

Would be nice.

Even better would be to use a custom tar implementation (similar to
(guix cpio)) that would make it easier and less clunky to ensure
reproducibility.

>> So ‘--sort=name’ seems to be missing.
> It’s present, see above ↑*

Indeed.  :-)

> Updated patch attached. I’ll push it if there are no further comments.

Perfect, thanks!

Ludo’.

Message #19 received at 48933-done <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 48933-done <at> debbugs.gnu.org, jlicht <at> fsfe.org
Subject: Re: bug#48933: [PATCH] build: Make outputs of node-build-system
 reproducible.
Date: Thu, 24 Jun 2021 14:12:08 +0200

Hi,

> > Updated patch attached. I’ll push it if there are no further comments.
> 
> Perfect, thanks!
pushed as 9c93573d15e90232de0effb4c28332c454dbc290.

Cheers,
Lars

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.