GNU bug report logs - #50597
different fixed-output when using ’--with-source’ transformation

Previous Next

To reply to this bug, email your comments to 50597 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Bug Guix <bug-guix <at> gnu.org>
Subject: different fixed-output when using ’--with-source’ transformation
Date: Wed, 15 Sep 2021 07:44:36 +0200

Hi,

If the package source tarball is provided to ’--with-source’, then the
hash differs.  I would expect a fixed point.

Let investigate:

--8<---------------cut here---------------start------------->8---
$ guix build hello <at> 2.10 --with-source=hello=https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz
/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10

$ guix build hello <at> 2.10 
/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
--8<---------------cut here---------------end--------------->8---

Where https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz is an
official mirror.  I expect the same store item.

Why are they different?  It is because the derivations are different.
The first call contains the source tarball as an “input”; that’s fine.
The second call contains the source tarball as a derivation; again
that’s fine.  The corresponding first and second derivations are:

--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10","","")]
 ,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])

[...]

   ,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
 ,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
 "/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz",
 "/gnu/store/rndq9g8877l29ha41dvsl3aj1z0gw0ng-hello-2.10-guile-builder"]

[...]

--8<---------------cut here---------------end--------------->8---

and

--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10","","")]
 ,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])
   ,("/gnu/store/1a7xfcqcxj0pqi4f81x1agcxa46v2bbm-hello-2.10.tar.gz.drv",["out"])

[...]

   ,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
 ,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
 "/gnu/store/kql8b2hbsabcmany4m3hfm3wzdiymliy-hello-2.10-guile-builder"]

[...]

--8<---------------cut here---------------end--------------->8---

Therefore, I understand why the two store items have different hashes.
Although, it should be better that the derivations would be the same;
another story. :-)


What puzzled me are the corresponding ’guile-builder’s:

--8<---------------cut here---------------start------------->8---
  (define %build-inputs
    (quote
     (("source" . "/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz")
--8<---------------cut here---------------end--------------->8---

and

--8<---------------cut here---------------start------------->8---
  (define %build-inputs
    (quote
     (("source" . "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz")
--8<---------------cut here---------------end--------------->8---

but these 2 items have the same hash:

--8<---------------cut here---------------start------------->8---
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---

Why?  It is fixed-output so I was expecting the same thing.  And the
same as:

--8<---------------cut here---------------start------------->8---
$ guix download https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz

Starting download of /tmp/guix-file.qkhxkp
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirrors.sarata.com/gnu/hello/hello-2.10.tar.gz'...
 …10.tar.gz  709KiB                                           549KiB/s 00:01 [##################]
100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---

Why the store item of source tarball is it different when applying the
transformation?

Maxime provided a clue [1].

All the best,
simon

1: <https://yhetil.org/guix/bd48bb84243bb6b6a3a7f6e1350767d7c1015238.camel <at> telenet.be>

Message #8 received at 50597 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 50597 <at> debbugs.gnu.org
Subject: Re: bug#50597: different fixed-output when using
 ’--with-source’ transformation
Date: Sat, 18 Sep 2021 18:06:38 +0200

Hello!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Why the store item of source tarball is it different when applying the
> transformation?
>
> Maxime provided a clue [1].

Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
whereas it’s usually #f for a ‘url-fetch’ kind of URI.

What we could do is make it #t if and only if the URL denotes a local
file and that file is a directory.  I think that would achieve what you
want without any observable regression in practice.

WDYT?

Thanks,
Ludo’.

Message #11 received at 50597 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Ludovic Courtès <ludo <at> gnu.org>, zimoun
 <zimon.toutoune <at> gmail.com>
Cc: 50597 <at> debbugs.gnu.org
Subject: Re: bug#50597: different fixed-output when using
 ’--with-source’ transformation
Date: Sat, 18 Sep 2021 19:35:12 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès schreef op za 18-09-2021 om 18:06 [+0200]:
> Hello!
> 
> zimoun <zimon.toutoune <at> gmail.com> skribis:
> 
> > Why the store item of source tarball is it different when applying the
> > transformation?
> > 
> > Maxime provided a clue [1].
> 
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
> 
> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory.  I think that would achieve what you
> want without any observable regression in practice.
>
> WDYT?

This seems reasonable to me.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 50597 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 50597 <at> debbugs.gnu.org
Subject: Re: bug#50597: different fixed-output when using
 ’--with-source’ transformation
Date: Tue, 21 Sep 2021 09:40:36 +0200

Hi Ludo,

On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo <at> gnu.org> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
>> Why the store item of source tarball is it different when applying the
>> transformation?
>>
>> Maxime provided a clue [1].
>
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.

Thanks for confirming.  Why is it different?


> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory.  I think that would achieve what you
> want without any observable regression in practice.

What I expect is that 2 (fixed-output) files with the same hash
(0ssi1w…) lives at the same store location…

--8<---------------cut here---------------start------------->8---
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---

…whatever if these (fixed-output) files come from “guix download”, “guix
build -S” or “--with-source=https://” or etc.

The mapping from hash (0ssi1w…) to store location should be 1:1 for
fixed-output, IMHO.

Cheers,
simon

Message #17 received at 50597 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 50597 <at> debbugs.gnu.org, Maxime Devos <maximedevos <at> telenet.be>
Subject: Re: bug#50597: different fixed-output when using
 ’--with-source’ transformation
Date: Wed, 22 Sep 2021 12:25:12 +0200

[Message part 1 (text/plain, inline)]

Hi,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo <at> gnu.org> wrote:
>> zimoun <zimon.toutoune <at> gmail.com> skribis:
>>
>>> Why the store item of source tarball is it different when applying the
>>> transformation?
>>>
>>> Maxime provided a clue [1].
>>
>> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
>> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
>
> Thanks for confirming.  Why is it different?

Because we want ‘--with-source’ to also work with directories (see
comment in ‘package-with-source’).

>> What we could do is make it #t if and only if the URL denotes a local
>> file and that file is a directory.  I think that would achieve what you
>> want without any observable regression in practice.
>
> What I expect is that 2 (fixed-output) files with the same hash
> (0ssi1w…) lives at the same store location…
>
> $ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
>
> $ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

The difference here is that one is marked as “recursive” and the other
is not.

The patch below does one I suggested earlier: passing #:recursive? #t if
and only if we know the source is a directory.

However it’s not enough to get the same derivation: we’d also need
<downloaded-file> to return a fixed-output derivation rather than a
plain store file name.  (See also <https://issues.guix.gnu.org/49696>.)

To be continued…

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/transformations.scm b/guix/transformations.scm
index 5ae1977cb2..e0f5579c28 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -46,6 +46,7 @@
   #:use-module (srfi srfi-37)
   #:use-module (ice-9 match)
   #:use-module (ice-9 vlist)
+  #:autoload   (web uri) (string->uri uri-scheme uri-path)
   #:export (options->transformation
             manifest-entry-with-transformations
 
@@ -106,15 +107,25 @@ extensions."
 (define* (package-with-source p uri #:optional version)
   "Return a package based on P but with its source taken from URI.  Extract
 the new package's version number from URI."
-  (let ((base (tarball-base-name (basename uri))))
+  (let ((base (tarball-base-name (basename uri)))
+        (file (match (string->uri uri)
+                (#f uri)
+                (uri (and (eq? 'file (uri-scheme uri))
+                          (uri-path uri))))))
     (let-values (((_ version*)
                   (hyphen-package-name->name+version base)))
       (package (inherit p)
                (version (or version version*
                             (package-version p)))
 
-               ;; Use #:recursive? #t to allow for directories.
-               (source (downloaded-file uri #t))))))
+               ;; Default to #:recursive? #f to match what 'url-fetch' does,
+               ;; but use #t when URI denotes a directory.
+               (source (let ((recursive?
+                              (and file
+                                   (match (stat file #f)
+                                     (#f #t)
+                                     (st (eq? 'directory (stat:type st)))))))
+                         (downloaded-file uri recursive?)))))))
 
 
 ;;;

Message #20 received at 50597 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 50597 <at> debbugs.gnu.org, Maxime Devos <maximedevos <at> telenet.be>
Subject: Re: bug#50597: different fixed-output when using
 ’--with-source’ transformation
Date: Wed, 13 Oct 2021 11:31:37 +0200

Hi,

On Wed, 22 Sep 2021 at 12:25, Ludovic Courtès <ludo <at> gnu.org> wrote:

> However it’s not enough to get the same derivation: we’d also need
> <downloaded-file> to return a fixed-output derivation rather than a
> plain store file name.  (See also
> <https://issues.guix.gnu.org/49696>.)

Thanks for the explanations and the pointer.  Indeed, to be continued…


> diff --git a/guix/transformations.scm b/guix/transformations.scm
> index 5ae1977cb2..e0f5579c28 100644
> --- a/guix/transformations.scm
> +++ b/guix/transformations.scm
> @@ -46,6 +46,7 @@
>    #:use-module (srfi srfi-37)
>    #:use-module (ice-9 match)
>    #:use-module (ice-9 vlist)
> +  #:autoload   (web uri) (string->uri uri-scheme uri-path)
>    #:export (options->transformation
>              manifest-entry-with-transformations
>  
> @@ -106,15 +107,25 @@ extensions."
>  (define* (package-with-source p uri #:optional version)
>    "Return a package based on P but with its source taken from URI.  Extract
>  the new package's version number from URI."
> -  (let ((base (tarball-base-name (basename uri))))
> +  (let ((base (tarball-base-name (basename uri)))
> +        (file (match (string->uri uri)
> +                (#f uri)
> +                (uri (and (eq? 'file (uri-scheme uri))
> +                          (uri-path uri))))))
>      (let-values (((_ version*)
>                    (hyphen-package-name->name+version base)))
>        (package (inherit p)
>                 (version (or version version*
>                              (package-version p)))
>  
> -               ;; Use #:recursive? #t to allow for directories.
> -               (source (downloaded-file uri #t))))))
> +               ;; Default to #:recursive? #f to match what 'url-fetch' does,
> +               ;; but use #t when URI denotes a directory.
> +               (source (let ((recursive?
> +                              (and file
> +                                   (match (stat file #f)
> +                                     (#f #t)
> +                                     (st (eq? 'directory (stat:type st)))))))
> +                         (downloaded-file uri recursive?)))))))

LGTM!

Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.