GNU bug report logs - #44835
gnu/ci.go: Embeds build path, breaking reproducible builds

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 44835 in the body.
You can then email your comments to 44835 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> reproducible-builds.org>
To: bug-guix <at> gnu.org
Subject: gnu/ci.go: Embeds build path, breaking reproducible builds
Date: Mon, 23 Nov 2020 20:19:54 -0800

[Message part 1 (text/plain, inline)]

Something in gnu/ci.scm is embedding the build path when compiled into
gnu/ci.go, as can be seen:

  https://tests.reproducible-builds.org/debian/rb-pkg/experimental/amd64/diffoscope-results/guix.html

  ./usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/gnu/ci.go
  strings --all --bytes=8 {}
  Offset 85, 15 lines modified	Offset 85, 15 lines modified
  ...
  92 	/build/1st/guix-1.2.0~rc2/gnu/ci.scm	92 	/build/2/guix-1.2.0~rc2/2nd/gnu/ci.scm

While guix builds of guix are typically built with a consistent build
path, it would be nice to fix this issue for other environments where
the build path may vary between builds.


My *wild* guess is it maybe has something to do with the use of
canonicalize-path:

  (define (find-current-checkout arguments)
  "Find the first checkout of ARGUMENTS that provided the current file.
Return #f if no such checkout is found."
  (let ((current-root
         (canonicalize-path
          (string-append (dirname (current-filename)) "/.."))))
    (find (lambda (argument)
            (and=> (assq-ref argument 'file-name)
                   (lambda (name)
                     (string=? name current-root)))) arguments)))

Either directly or indirectly... does canonicalize-path resolve at build
time? run time? somewhere in-between? Or is this a red herring, and
something else entirely is responsible? I'll let a competent schemer
ponder this! :)


Thanks for all your help!


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 44835 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Vagrant Cascadian <vagrant <at> reproducible-builds.org>
Cc: Mathieu Othacehe <othacehe <at> gnu.org>, 44835 <at> debbugs.gnu.org
Subject: Re: bug#44835: gnu/ci.go: Embeds build path, breaking reproducible
 builds
Date: Thu, 26 Nov 2020 22:39:08 +0100

[Message part 1 (text/plain, inline)]

Hi!

Vagrant Cascadian <vagrant <at> reproducible-builds.org> skribis:

> My *wild* guess is it maybe has something to do with the use of
> canonicalize-path:
>
>   (define (find-current-checkout arguments)
>   "Find the first checkout of ARGUMENTS that provided the current file.
> Return #f if no such checkout is found."
>   (let ((current-root
>          (canonicalize-path
>           (string-append (dirname (current-filename)) "/.."))))
>     (find (lambda (argument)
>             (and=> (assq-ref argument 'file-name)
>                    (lambda (name)
>                      (string=? name current-root)))) arguments)))

‘canonicalize-path’ is called at run time, so that’s fine.  However,
‘current-filename’ is a macro that captures the source file name at
build time, so it’s the likely culprit here.

I was going to go with something like:

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/ci.scm b/gnu/ci.scm
index 5548d9560e..0bacfbe025 100644
--- a/gnu/ci.scm
+++ b/gnu/ci.scm
@@ -488,7 +488,8 @@ valid."
 Return #f if no such checkout is found."
   (let ((current-root
          (canonicalize-path
-          (string-append (dirname (current-filename)) "/.."))))
+          (string-append (dirname (search-path %load-path "gnu/ci.scm"))
+                         "/.."))))
     (find (lambda (argument)
             (and=> (assq-ref argument 'file-name)
                    (lambda (name)

[Message part 3 (text/plain, inline)]

… but I don’t think we can assume that the checkout is in the
‘%load-path’ when this code is executed.  WDYT, Mathieu?

Looking at f71b0a0012d46bd30ead1a14ed58fd59647415e2, which introduced
this, there might be other options too.

Thanks,
Ludo’.

Message #11 received at 44835 <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> reproducible-builds.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Mathieu Othacehe <othacehe <at> gnu.org>, 44835 <at> debbugs.gnu.org
Subject: Re: bug#44835: gnu/ci.go: Embeds build path,
 breaking reproducible builds
Date: Fri, 27 Nov 2020 12:25:05 -0800

[Message part 1 (text/plain, inline)]

On 2020-11-26, Ludovic Courtès wrote:
> Vagrant Cascadian <vagrant <at> reproducible-builds.org> skribis:
>> My *wild* guess is it maybe has something to do with the use of
>> canonicalize-path:
>>
>>   (define (find-current-checkout arguments)
>>   "Find the first checkout of ARGUMENTS that provided the current file.
>> Return #f if no such checkout is found."
>>   (let ((current-root
>>          (canonicalize-path
>>           (string-append (dirname (current-filename)) "/.."))))
>>     (find (lambda (argument)
>>             (and=> (assq-ref argument 'file-name)
>>                    (lambda (name)
>>                      (string=? name current-root)))) arguments)))
>
> ‘canonicalize-path’ is called at run time, so that’s fine.  However,
> ‘current-filename’ is a macro that captures the source file name at
> build time, so it’s the likely culprit here.
>
> I was going to go with something like:
>
> diff --git a/gnu/ci.scm b/gnu/ci.scm
> index 5548d9560e..0bacfbe025 100644
> --- a/gnu/ci.scm
> +++ b/gnu/ci.scm
> @@ -488,7 +488,8 @@ valid."
>  Return #f if no such checkout is found."
>    (let ((current-root
>           (canonicalize-path
> -          (string-append (dirname (current-filename)) "/.."))))
> +          (string-append (dirname (search-path %load-path "gnu/ci.scm"))
> +                         "/.."))))
>      (find (lambda (argument)
>              (and=> (assq-ref argument 'file-name)
>                     (lambda (name)
>
> … but I don’t think we can assume that the checkout is in the
> ‘%load-path’ when this code is executed.  WDYT, Mathieu?
>
> Looking at f71b0a0012d46bd30ead1a14ed58fd59647415e2, which introduced
> this, there might be other options too.

This does resolve the reproducibility issues after some "quick" testing;
no idea if it breaks anything, though I didn't see any new test suite
failures (though my builds are patched to skip many tests)...


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 44835 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44835 <at> debbugs.gnu.org, Vagrant Cascadian <vagrant <at> reproducible-builds.org>
Subject: Re: bug#44835: gnu/ci.go: Embeds build path, breaking reproducible
 builds
Date: Tue, 01 Dec 2020 10:41:00 +0100

Hey,

> … but I don’t think we can assume that the checkout is in the
> ‘%load-path’ when this code is executed.  WDYT, Mathieu?

Cuirass happens to add checkouts to the %load-path just before loading
this file.

I tested your path, it works fine. I think it is acceptable as this (gnu
ci) interface needs a big rework anyway.

I can apply your patch if it's ok for you.

Thanks,

Mathieu

Message #17 received at 44835 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 44835 <at> debbugs.gnu.org, Vagrant Cascadian <vagrant <at> reproducible-builds.org>
Subject: Re: bug#44835: gnu/ci.go: Embeds build path, breaking reproducible
 builds
Date: Thu, 03 Dec 2020 14:23:52 +0100

Hi,

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

>> … but I don’t think we can assume that the checkout is in the
>> ‘%load-path’ when this code is executed.  WDYT, Mathieu?
>
> Cuirass happens to add checkouts to the %load-path just before loading
> this file.

Is that systematic?  Isn’t it only when ‘load_path_inputs’ is non-empty?

> I tested your path, it works fine. I think it is acceptable as this (gnu
> ci) interface needs a big rework anyway.
>
> I can apply your patch if it's ok for you.

Sure if you’re confident you can go ahead.  :-)

Thanks,
Ludo’.

Message #22 received at 44835-done <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> reproducible-builds.org>
To: Ludovic Courtès <ludo <at> gnu.org>, Mathieu Othacehe
 <othacehe <at> gnu.org>
Cc: 44835-done <at> debbugs.gnu.org
Subject: Re: bug#44835: gnu/ci.go: Embeds build path, breaking reproducible
 builds
Date: Thu, 07 Mar 2024 13:46:01 -0800

[Message part 1 (text/plain, inline)]

On 2020-12-03, Ludovic Courtès wrote:
> Mathieu Othacehe <othacehe <at> gnu.org> skribis:
>
>>> … but I don’t think we can assume that the checkout is in the
>>> ‘%load-path’ when this code is executed.  WDYT, Mathieu?
>>
>> Cuirass happens to add checkouts to the %load-path just before loading
>> this file.
>
> Is that systematic?  Isn’t it only when ‘load_path_inputs’ is non-empty?
>
>> I tested your path, it works fine. I think it is acceptable as this (gnu
>> ci) interface needs a big rework anyway.
>>
>> I can apply your patch if it's ok for you.
>
> Sure if you’re confident you can go ahead.  :-)

This looks to have been fixed some time ago in:

  76bea3f8bcd951ded88dfb7f8cad5bc3e5a1701f ci: Remove hydra support.

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.