GNU bug report logs - #42009
package cache build is not deterministic

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 42009 in the body.
You can then email your comments to 42009 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marinus <marinus.savoritias <at> disroot.org>
To: bug-guix <at> gnu.org
Subject: Determinism problem with guix pull
Date: Mon, 22 Jun 2020 19:07:42 +0200

Hi,

Run into a determinism problem today with guix pull.
I run guix pull --rounds=3 but guix ended in error that the result
wasn't the same.

The error was this:
building package cache...
|output
‘/gnu/store/277s1r2kxw9pfw1g6wg3vf6wrkksj57y-guix-package-cache’ of
‘/gnu/store/m64b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv’
differs from previous round build of
/gnu/store/m64b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv
failed View build log at
'/var/log/guix/drvs/m6/4b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv.bz2'.
cannot build derivation
`/gnu/store/bcpjm4yvslpf4858lx4pj89xj279z5nv-profile.drv': 1
dependencies couldn't be built guix pull: error: build of
`/gnu/store/bcpjm4yvslpf4858lx4pj89xj279z5nv-profile.drv' failed

The files are indeed different size with one being larger than the
other.

My system is this:
guix describe
Generation 7	Jun 18 2020 13:55:17	(current)
  guix e418c3d
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: e418c3d076ec301a2deda42568035d75f5ed174d


Marinus

Message #8 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Marinus <marinus.savoritias <at> disroot.org>, 42009 <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Wed, 24 Jun 2020 00:46:04 +0200

Dear,

Thank you for the bug report.  It is something already noticed [1] but
without a clean bug report to track it. :-)

1: http://issues.guix.gnu.org/issue/39258#86


On Mon, 22 Jun 2020 at 19:07, Marinus <marinus.savoritias <at> disroot.org> wrote:

> Run into a determinism problem today with guix pull.
> I run guix pull --rounds=3 but guix ended in error that the result
> wasn't the same.

For reproducing, the simplest is:

--8<---------------cut here---------------start------------->8---
$ guix build --check --no-grafts -K \
   $(guix gc --derivers \
      $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
The following profile hooks will be built:
   /gnu/store/l50sinckbl1y0fz2y4yk4vvfdvay9c8l-guix-package-cache.drv
   /gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv
building package cache...
(repl-version 0 1 1)
Generating package cache for '/gnu/store/67zi87xwv2d90kx8pzxsbw2q7qkh11ns-profile'...
(values (value "/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache"))
guix build: error: derivation `/gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv' may not be deterministic: output `/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache' differs
--8<---------------cut here---------------end--------------->8---

Then the usual "diffoscope":

--8<---------------cut here---------------start------------->8---
diffoscope \
/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache \
/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache\
  | head -n50
--8<---------------cut here---------------end--------------->8---

outputs something like:

--8<---------------cut here---------------start------------->8---
--- /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache
+++ /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache
├── readelf --wide --file-header {}
│ @@ -6,15 +6,15 @@
│    OS/ABI:                            <unknown: ff>
│    ABI Version:                       0
│    Type:                              DYN (Shared object file)
│    Machine:                           None
│    Version:                           0x1
│    Entry point address:               0x0
│    Start of program headers:          64 (bytes into file)
│ -  Start of section headers:          4900296 (bytes into file)
│ +  Start of section headers:          4900456 (bytes into file)
│    Flags:                             0x0
│    Size of this header:               64 (bytes)
│    Size of program headers:           56 (bytes)
│    Number of program headers:         3
│    Size of section headers:           64 (bytes)
│    Number of section headers:         20
│    Section header string table index: 17
├── readelf --wide --program-header {}
│ @@ -1,16 +1,16 @@
│  
│  Elf file type is DYN (Shared object file)
│  Entry point 0x0
│  There are 3 program headers, starting at offset 64
│  
│  Program Headers:
│    Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
│ -  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x286a68 0x286a68 R   0x10000
│ -  LOAD           0x290000 0x0000000000290000 0x0000000000290000 0x21c5c8 0x21c5c8 RW  0x10000
│ -  DYNAMIC        0x286a08 0x0000000000286a08 0x0000000000286a08 0x000060 0x000060 R   0x8
│ +  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x286b78 0x286b78 R   0x10000
│ +  LOAD           0x290000 0x0000000000290000 0x0000000000290000 0x21c668 0x21c668 RW  0x10000
│ +  DYNAMIC        0x286b18 0x0000000000286b18 0x0000000000286b18 0x000060 0x000060 R   0x8
│  
│   Section to Segment mapping:
│    Segment Sections...
│     00     .rodata .rtl-text .dynamic  
│     01     .data 
│     02     .dynamic
├── readelf --wide --sections {}
│┄ stderr from `readelf --wide --sections {}`:
│┄ readelf: Warning: [ 5]: Link field (0) should index a string section.
│ @@ -1,29 +1,29 @@
│ -There are 20 section headers, starting at offset 0x4ac5c8:
│ +There are 20 section headers, starting at offset 0x4ac668:
--8<---------------cut here---------------end--------------->8---

Well, I do not know what should the next step.  I mean this
"package.cache" file is created by the function
gnu/packages.scm:(generate-package-cache) which reads:

--8<---------------cut here---------------start------------->8---
      ;; Store the cache as a '.go' file.  This makes loading fast and reduces
      ;; heap usage since some of the static data is directly mmapped.
      (put-bytevector port
                      (compile `'(,@exp)
                               #:to 'bytecode
                               #:opts '(#:to-file? #t)))))
--8<---------------cut here---------------end--------------->8---

Then it is on the Guile side, isn't it? 


All the best,
simon

Message #11 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: Msavoritias <marinus.savoritias <at> disroot.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42009 <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Wed, 24 Jun 2020 20:59:17 +0200

[Message part 1 (text/plain, inline)]

Hi,

Yeah seems like it. Although I have to admit I'm pretty newbie in a lot 
of this stuff.
Do you suggest this bug should be filed against Guile?

Marinus

On Wed, Jun 24, 2020 at 00:46, zimoun <zimon.toutoune <at> gmail.com> wrote:
> Dear,
> 
> Thank you for the bug report.  It is something already noticed [1] but
> without a clean bug report to track it. :-)
> 
> 1: <http://issues.guix.gnu.org/issue/39258#86>
> 
> 
> On Mon, 22 Jun 2020 at 19:07, Marinus <marinus.savoritias <at> disroot.org 
> <mailto:marinus.savoritias <at> disroot.org>> wrote:
> 
>>  Run into a determinism problem today with guix pull.
>>  I run guix pull --rounds=3 but guix ended in error that the result
>>  wasn't the same.
> 
> For reproducing, the simplest is:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build --check --no-grafts -K \
>    $(guix gc --derivers \
>       $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
> The following profile hooks will be built:
>    /gnu/store/l50sinckbl1y0fz2y4yk4vvfdvay9c8l-guix-package-cache.drv
>    /gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv
> building package cache...
> (repl-version 0 1 1)
> Generating package cache for 
> '/gnu/store/67zi87xwv2d90kx8pzxsbw2q7qkh11ns-profile'...
> (values (value 
> "/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache"))
> guix build: error: derivation 
> `/gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv' 
> may not be deterministic: output 
> `/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache' 
> differs
> --8<---------------cut here---------------end--------------->8---
> 
> Then the usual "diffoscope":
> 
> --8<---------------cut here---------------start------------->8---
> diffoscope \
> /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache 
> \
> /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache\
>   | head -n50
> --8<---------------cut here---------------end--------------->8---
> 
> outputs something like:
> 
> --8<---------------cut here---------------start------------->8---
> --- 
> /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache
> +++ 
> /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache
> ├── readelf --wide --file-header {}
> │ @@ -6,15 +6,15 @@
> │    OS/ABI:                            <unknown: ff>
> │    ABI Version:                       0
> │    Type:                              DYN (Shared object file)
> │    Machine:                           None
> │    Version:                           0x1
> │    Entry point address:               0x0
> │    Start of program headers:          64 (bytes into file)
> │ -  Start of section headers:          4900296 (bytes into file)
> │ +  Start of section headers:          4900456 (bytes into file)
> │    Flags:                             0x0
> │    Size of this header:               64 (bytes)
> │    Size of program headers:           56 (bytes)
> │    Number of program headers:         3
> │    Size of section headers:           64 (bytes)
> │    Number of section headers:         20
> │    Section header string table index: 17
> ├── readelf --wide --program-header {}
> │ @@ -1,16 +1,16 @@
> │
> │  Elf file type is DYN (Shared object file)
> │  Entry point 0x0
> │  There are 3 program headers, starting at offset 64
> │
> │  Program Headers:
> │    Type           Offset   VirtAddr           PhysAddr           
> FileSiz  MemSiz   Flg Align
> │ -  LOAD           0x000000 0x0000000000000000 0x0000000000000000 
> 0x286a68 0x286a68 R   0x10000
> │ -  LOAD           0x290000 0x0000000000290000 0x0000000000290000 
> 0x21c5c8 0x21c5c8 RW  0x10000
> │ -  DYNAMIC        0x286a08 0x0000000000286a08 0x0000000000286a08 
> 0x000060 0x000060 R   0x8
> │ +  LOAD           0x000000 0x0000000000000000 0x0000000000000000 
> 0x286b78 0x286b78 R   0x10000
> │ +  LOAD           0x290000 0x0000000000290000 0x0000000000290000 
> 0x21c668 0x21c668 RW  0x10000
> │ +  DYNAMIC        0x286b18 0x0000000000286b18 0x0000000000286b18 
> 0x000060 0x000060 R   0x8
> │
> │   Section to Segment mapping:
> │    Segment Sections...
> │     00     .rodata .rtl-text .dynamic
> │     01     .data
> │     02     .dynamic
> ├── readelf --wide --sections {}
> │┄ stderr from `readelf --wide --sections {}`:
> │┄ readelf: Warning: [ 5]: Link field (0) should index a string 
> section.
> │ @@ -1,29 +1,29 @@
> │ -There are 20 section headers, starting at offset 0x4ac5c8:
> │ +There are 20 section headers, starting at offset 0x4ac668:
> --8<---------------cut here---------------end--------------->8---
> 
> Well, I do not know what should the next step.  I mean this
> "package.cache" file is created by the function
> gnu/packages.scm:(generate-package-cache) which reads:
> 
> --8<---------------cut here---------------start------------->8---
>       ;; Store the cache as a '.go' file.  This makes loading fast 
> and reduces
>       ;; heap usage since some of the static data is directly mmapped.
>       (put-bytevector port
>                       (compile `'(,@exp)
>                                #:to 'bytecode
>                                #:opts '(#:to-file? #t)))))
> --8<---------------cut here---------------end--------------->8---
> 
> Then it is on the Guile side, isn't it?
> 
> 
> All the best,
> simon

[Message part 2 (text/html, inline)]

Message #14 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Msavoritias <marinus.savoritias <at> disroot.org>
Cc: 42009 <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Thu, 25 Jun 2020 11:04:48 +0200

Dear,

On Wed, 24 Jun 2020 at 20:59, Msavoritias <marinus.savoritias <at> disroot.org> wrote:

> Do you suggest this bug should be filed against Guile?

Well, first the origin of the bug should be confirmed.  :-)
And maybe there is an option for 'compile'.  I do not know.

All the best,
simon

Message #17 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: Msavoritias <marinus.savoritias <at> disroot.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42009 <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Thu, 25 Jun 2020 16:53:54 +0200

[Message part 1 (text/plain, inline)]

Is there any tests or any more information I can do on my side?

Marinus

On Thu, Jun 25, 2020 at 11:04, zimoun <zimon.toutoune <at> gmail.com> wrote:
> Dear,
> 
> On Wed, 24 Jun 2020 at 20:59, Msavoritias 
> <marinus.savoritias <at> disroot.org 
> <mailto:marinus.savoritias <at> disroot.org>> wrote:
> 
>>  Do you suggest this bug should be filed against Guile?
> 
> Well, first the origin of the bug should be confirmed.  :-)
> And maybe there is an option for 'compile'.  I do not know.
> 
> All the best,
> simon

[Message part 2 (text/html, inline)]

Message #20 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: Marinus <marinus.savoritias <at> disroot.org>
Cc: 42009 <at> debbugs.gnu.org, zimoun <zimon.toutoune <at> gmail.com>
Subject: Re: bug#42009: package.cache not deterministic
Date: Fri, 26 Jun 2020 00:33:00 +0200

[Message part 1 (text/plain, inline)]

Quick and lazy reply before the sleep comes,

Seems like a ‘simple’ case of the packages not being ordered 
(enough): 
https://www.tobias.gr/guix-package.cache.diffoscope.html/#lib---guix---package.cache---strings---all---

(Sorry for the ad-hoc URL, I'll keep it up until this bug report 
is closed!)

Kind regards,

T G-R

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Marinus <marinus.savoritias <at> disroot.org>
Cc: 42009 <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Fri, 26 Jun 2020 01:19:00 +0200

Hi Tobias,

On Fri, 26 Jun 2020 at 00:33, Tobias Geerinckx-Rice <me <at> tobias.gr> wrote:

> Seems like a ‘simple’ case of the packages not being ordered 
> (enough):

So one culprit would be 'scandir*' from 'scheme-files' used by
'scheme-modules' used by 'all-modules' used by 'generate-package-cache'.
The docstring says: "The returned list is sorted in alphabetical order."

Well, I will try to investigate more.

> https://www.tobias.gr/guix-package.cache.diffoscope.html/#lib---guix---package.cache---strings---all---

Interesting...

Good night,
simon

Message #26 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42009 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Marinus <marinus.savoritias <at> disroot.org>
Subject: Re: bug#42009: package.cache not deterministic
Date: Sun, 28 Jun 2020 22:29:49 +0200

Hi,

Most likely the problem with non-reproducible .go files is that the fix
for <https://bugs.gnu.org/20272> was incomplete.  In particular, I think
that gensyms are not reproducible when building things in parallel,
because the gensym depends on what’s loaded vs. interpreted.

Example of a simpler discrepancy:

--8<---------------cut here---------------start------------->8---
$ guix challenge guile-gcrypt
/gnu/store/i6zzgsjjlqyfn9zmanb55hqdackhc0mf-guile-gcrypt-0.3.0 contents differ:
  local hash: 0bkxim5532bd234iqxvxd31ybp6v7rmbrvyj3jrgpmgxbsw7hmxj
  https://ci.guix.gnu.org/nar/lzip/i6zzgsjjlqyfn9zmanb55hqdackhc0mf-guile-gcrypt-0.3.0: 00zsccxn1485bnvxvssgq50i2w69i38m9hnrk8z1sji9qcmmaxcs
  differing files:
    /lib/guile/3.0/site-ccache/gcrypt/pk-crypto.go
    /lib/guile/3.0/site-ccache/gcrypt/mac.go
--8<---------------cut here---------------end--------------->8---

HTH!

Ludo’.

Message #29 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 42009 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Marinus <marinus.savoritias <at> disroot.org>
Subject: Re: bug#42009: package.cache not deterministic
Date: Mon, 29 Jun 2020 12:03:22 +0200

Hi Ludo,

On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo <at> gnu.org> wrote:

> Most likely the problem with non-reproducible .go files is that the fix
> for <https://bugs.gnu.org/20272> was incomplete.  In particular, I think
> that gensyms are not reproducible when building things in parallel,
> because the gensym depends on what’s loaded vs. interpreted.

Thank you for the pointer.

How can I test the "hypothesis" of "building things in parallel"?

--8<---------------cut here---------------start------------->8---
guix describe -f channels > /tmp/chan.scm
guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1

guix build --check --no-grafts --cores=1 \
     $(guix gc --derivers \
            $(readlink -f /tmp/repull1/lib/guix/package.cache))
The following profile hook will be built:
   /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv
building package cache...
(repl-version 0 1 1)
Generating package cache for '/gnu/store/bgqy3mfpzbpyz3pysqxzkpch39q98yv3-profile'...
(values (value "/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache"))
guix build: error: derivation `/gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv' may not be deterministic: output `/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache' differs
--8<---------------cut here---------------end--------------->8---

BTW, I do not understand why the derivations have different hashes,
containing derivations with different hashes and more importantly, why
it is not the same order.

--8<---------------cut here---------------start------------->8---
guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)
/gnu/store/0pmc85ni7zsd5jrflb0prrj7bhvn1m1y-guix-package-cache.drv

cat $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
Derive
([("out","/gnu/store/pfpbh4v1m2dgn9dwiz6rsbqgx8lmd3ms-guix-package-cache","","")]
 ,[("/gnu/store/3pkfaqkdkaqy8khsfbsl0si3r9mydygl-profile.drv",["out"])
   ,("/gnu/store/nih4g42d2da8p2b5dmxqb081bbpv9ax4-inferior-script.scm.drv",["out"])
   ,("/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv",["out"])]
 ,["/gnu/store/50h7d8cx9k28gdbdzc9y615d1564m8ia-guix-package-cache-builder"]
 ,"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","/gnu/store/50h7d8cx9k28gdbdzc9y615d1564m8ia-guix-package-cache-builder"]
 ,[("guix properties","((type . profile-hook) (hook . package-cache))")
   ,("out","/gnu/store/pfpbh4v1m2dgn9dwiz6rsbqgx8lmd3ms-guix-package-cache")
   ,("preferLocalBuild","1")])
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
guix gc --derivers $(readlink -f /tmp/repull1/lib/guix/package.cache)
/gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv

cat $(guix gc --derivers $(readlink -f /tmp/repull1/lib/guix/package.cache))
Derive
([("out","/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache","","")]
 ,[("/gnu/store/b4dcaccqli2zdfalrn0lc0cz94gd80sk-inferior-script.scm.drv",["out"])
   ,("/gnu/store/hm03mwl234lw43ivx33nsap0j4pjwqjp-profile.drv",["out"])
   ,("/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv",["out"])]
 ,["/gnu/store/251jkjnw9zza2zwr1k45x1049d1axl5q-guix-package-cache-builder"]
 ,"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","/gnu/store/251jkjnw9zza2zwr1k45x1049d1axl5q-guix-package-cache-builder"]
 ,[("guix properties","((type . profile-hook) (hook . package-cache))")
   ,("out","/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache")
   ,("preferLocalBuild","1")])
--8<---------------cut here---------------end--------------->8---

And I am confused because if I repull again with '--cores=2', then,

--8<---------------cut here---------------start------------->8---
/tmp/repull1/bin/guix pull -C /tmp/chan.scm --cores=2 -p /tmp/repull2
md5sum \
  $(readlink -f /tmp/repull2/lib/guix/package.cache) \
  $(readlink -f /tmp/repull2/lib/guix/package.cache)
75f6feb9f52c312cc9cc8f73534926ba  /gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache
75f6feb9f52c312cc9cc8f73534926ba  /gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache
--8<---------------cut here---------------end--------------->8---

But '--check' fails in all cases.  What do I miss?


All the best,
simon

Message #34 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42009 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Marinus <marinus.savoritias <at> disroot.org>
Subject: Re: bug#42009: package.cache not deterministic
Date: Mon, 29 Jun 2020 14:34:40 +0200

[Message part 1 (text/plain, inline)]

Hi Simon,

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo <at> gnu.org> wrote:
>
>> Most likely the problem with non-reproducible .go files is that the fix
>> for <https://bugs.gnu.org/20272> was incomplete.  In particular, I think
>> that gensyms are not reproducible when building things in parallel,
>> because the gensym depends on what’s loaded vs. interpreted.
>
> Thank you for the pointer.
>
> How can I test the "hypothesis" of "building things in parallel"?
>
> guix describe -f channels > /tmp/chan.scm
> guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1
>
> guix build --check --no-grafts --cores=1 \
>      $(guix gc --derivers \
>             $(readlink -f /tmp/repull1/lib/guix/package.cache))
> The following profile hook will be built:
>    /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv

I realize I was a bit off-topic: I was commenting on the more general
issue of .go non-reproducibility.

The problem with the package cache seems to be different.  Sorry for the
confusion!

After --check, we can compare both caches like this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (define a (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache/lib/guix/package.cache"))
scheme@(guile-user)> (define b (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache-check/lib/guix/package.cache"))
scheme@(guile-user)> (length a)
$2 = 13949
scheme@(guile-user)> (length b)
$3 = 13949
scheme@(guile-user)> ,use(srfi srfi-1)
scheme@(guile-user)> (lset= equal? a b)
$4 = #f
scheme@(guile-user)> (car (lset-difference equal? a b))
$5 = #("python2" "2.7.17" (gnu packages python) python-2 ("out" "tk") #t #f "gnu/packages/python.scm" 107 2)
--8<---------------cut here---------------end--------------->8---

So, surprisingly, it’s not just an ordering issue: the caches do contain
different pieces of information.

This patch solves the ordering issue:

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..5154d73deb 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -407,13 +407,25 @@ reducing the memory footprint."
       (_
        result+seen)))
 
+  (define entry-key
+    (match-lambda
+      (#(name version module symbol outputs supported? deprecated?
+              file line column)
+       (string-append name version (or file "")
+                      (if line (number->string line) "")))))
+
+  (define (entry<? a b)
+    (string<? (entry-key a) (entry-key b)))
+
   (define exp
-    (first
-     (fold-module-public-variables* expand-cache
-                                    (cons '() vlist-null)
-                                    (all-modules (%package-module-path)
-                                                 #:warn
-                                                 warn-about-load-error))))
+    (sort
+     (first
+      (fold-module-public-variables* expand-cache
+                                     (cons '() vlist-null)
+                                     (all-modules (%package-module-path)
+                                                  #:warn
+                                                  warn-about-load-error)))
+     entry<?))
 
   (mkdir-p (dirname cache-file))
   (call-with-output-file cache-file

[Message part 3 (text/plain, inline)]

But it’s not quite right because the order in which variables are
traversed is still non-deterministic, so between two runs of
‘generate-package-cache’, caches differ like this:

[Message part 4 (text/x-patch, inline)]

--- a	2020-06-29 14:27:32.291028456 +0200
+++ b	2020-06-29 14:27:37.930993059 +0200
@@ -8581,7 +8581,7 @@
  #("clang-runtime"
    "9.0.1"
    (gnu packages llvm)
-   clang-runtime
+   clang-runtime-9
    ("out")
    #t
    #f
@@ -26511,7 +26511,7 @@
  #("gcc-objc++"
    "7.5.0"
    (gnu packages gcc)
-   gcc-objc++-7
+   gcc-objc++
    ("out" "lib" "debug")
    #t
    #f
@@ -26641,7 +26641,7 @@
  #("gcc-toolchain"
    "7.5.0"
    (gnu packages commencement)
-   gcc-toolchain
+   gcc-toolchain-7
    ("out" "debug" "static")
    #t
    #f
@@ -33311,7 +33311,7 @@
  #("ghc"
    "8.6.5"
    (gnu packages haskell)
-   ghc-8.6
+   ghc-8
    ("out" "doc")
    #t
    #f
@@ -41876,7 +41876,7 @@
  #("icedtea"
    "3.7.0"
    (gnu packages java)
-   icedtea-8
+   icedtea
    ("out" "jdk" "doc")
    #t
    #f
@@ -54376,7 +54376,7 @@
  #("linux-libre-headers"
    "5.4.20"
    (gnu packages linux)
-   linux-libre-headers-5.4.20
+   linux-libre-headers
    ("out")
    #t
    #f
@@ -54636,7 +54636,7 @@
  #("llvm"
    "9.0.1"
    (gnu packages llvm)
-   llvm-9
+   llvm
    ("out" "opt-viewer")
    #t
    #f
@@ -61826,7 +61826,7 @@
  #("ocaml"
    "4.09.0"
    (gnu packages ocaml)
-   ocaml
+   ocaml-4.09
    ("out")
    #t
    #f
@@ -62256,7 +62256,7 @@
  #("opencl-headers"
    "2.2.0-0.e986688"
    (gnu packages opencl)
-   opencl-headers
+   opencl-headers-2.2
    ("out")
    #t
    #f
@@ -92636,7 +92636,7 @@
  #("python2"
    "2.7.17"
    (gnu packages python)
-   python-2
+   python-2.7
    ("out" "tk")
    #t
    #f
@@ -92646,7 +92646,7 @@
  #("python"
    "3.8.2"
    (gnu packages python)
-   python-3
+   python
    ("out" "tk")
    #t
    #f
@@ -123676,7 +123676,7 @@
  #("rust"
    "1.39.0"
    (gnu packages rust)
-   rust-1.39
+   rust
    ("out" "doc" "cargo")
    #t
    #f

[Message part 5 (text/plain, inline)]

The problem has to do with aliases: we don’t always see the same
variable first.  So we have to sort before calling ‘expand-cache’.

To be continued…

Ludo’.

Message #37 received at 42009 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 42009 <at> debbugs.gnu.org, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Marinus <marinus.savoritias <at> disroot.org>
Subject: Re: bug#42009: package.cache not deterministic
Date: Mon, 29 Jun 2020 17:39:42 +0200

Hi Ludo,

On Mon, 29 Jun 2020 at 14:34, Ludovic Courtès <ludo <at> gnu.org> wrote:

> I realize I was a bit off-topic: I was commenting on the more general
> issue of .go non-reproducibility.
>
> The problem with the package cache seems to be different.  Sorry for the
> confusion!

Well, the .go non-reproducibility is nonetheless interesting. :-)

> So, surprisingly, it’s not just an ordering issue: the caches do contain
> different pieces of information.

Interestingly, on my machine with 8ea91d0, I have 12 differences but
none of them is Python.  Instead, I have: "rust", "ocaml",
"mingw-w64-i686", "clang-toolchain", "clang-runtime", "linux-libre",
"icedtea", "gcc-objc++", "gcc-objc", "bdb", "rust-lazy-static",
"gcc-toolchain".  All are aliases and python-2 is too but does not
appear...
On the other hand, these 2 aliases do not appear either in your list.

--8<---------------cut here---------------start------------->8---
(define-public clang clang-9)
(define-public clang-toolchain clang-toolchain-9)
--8<---------------cut here---------------end--------------->8---

The question is: do we need to declare public e.g. clang-9?  Declare
clang is it not enough already?

For example, ocaml-4.09 is not used outside gnu/packages/ocalm.scm.
Idem for ghc-8.6, etc..

> This patch solves the ordering issue

The 'sort' will not help for improving the speed of "guix pull". ;-)

> But it’s not quite right because the order in which variables are
> traversed is still non-deterministic, so between two runs of
> ‘generate-package-cache’, caches differ like this:

It depends on the eddy current in the upper atmosphere. :-)
https://xkcd.com/378/

Well, it finds one or the other first and 'expand-cache' stores the
first considering the second as "seen", isn't it?

> The problem has to do with aliases: we don’t always see the same
> variable first.  So we have to sort before calling ‘expand-cache’.

The question is why it is not always the same variable first.  Well,
IMHO, it could comes from:
 - either 'scandir*' in 'scheme-files' because all the other functions
seem "pure" and this one not;
 - either 'fold-module-public-variables*' where 'seen' is one or the other.
Or if the .go files are not deterministic, especially about 'gensym',
it should explain why one symbol appears sometimes first.

Cheers,
simon

Message #42 received at 42009-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: Marinus <marinus.savoritias <at> disroot.org>,
 Tobias Geerinckx-Rice <me <at> tobias.gr>, 42009-done <at> debbugs.gnu.org
Subject: Re: bug#42009: package.cache not deterministic
Date: Thu, 30 Jul 2020 19:22:47 +0200

Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> But it’s not quite right because the order in which variables are
> traversed is still non-deterministic, so between two runs of
> ‘generate-package-cache’, caches differ like this:
>
> --- a	2020-06-29 14:27:32.291028456 +0200
> +++ b	2020-06-29 14:27:37.930993059 +0200
> @@ -8581,7 +8581,7 @@
>   #("clang-runtime"
>     "9.0.1"
>     (gnu packages llvm)
> -   clang-runtime
> +   clang-runtime-9
>     ("out")
>     #t
>     #f
> @@ -26511,7 +26511,7 @@
>   #("gcc-objc++"
>     "7.5.0"
>     (gnu packages gcc)
> -   gcc-objc++-7
> +   gcc-objc++
>     ("out" "lib" "debug")
>     #t
>     #f
> @@ -26641,7 +26641,7 @@
>   #("gcc-toolchain"
>     "7.5.0"
>     (gnu packages commencement)
> -   gcc-toolchain
> +   gcc-toolchain-7
>     ("out" "debug" "static")
>     #t
>     #f
> @@ -33311,7 +33311,7 @@
>   #("ghc"
>     "8.6.5"
>     (gnu packages haskell)
> -   ghc-8.6
> +   ghc-8
>     ("out" "doc")
>     #t
>     #f
> @@ -41876,7 +41876,7 @@
>   #("icedtea"
>     "3.7.0"
>     (gnu packages java)
> -   icedtea-8
> +   icedtea
>     ("out" "jdk" "doc")
>     #t
>     #f
> @@ -54376,7 +54376,7 @@
>   #("linux-libre-headers"
>     "5.4.20"
>     (gnu packages linux)
> -   linux-libre-headers-5.4.20
> +   linux-libre-headers
>     ("out")
>     #t
>     #f
> @@ -54636,7 +54636,7 @@
>   #("llvm"
>     "9.0.1"
>     (gnu packages llvm)
> -   llvm-9
> +   llvm
>     ("out" "opt-viewer")
>     #t
>     #f
> @@ -61826,7 +61826,7 @@
>   #("ocaml"
>     "4.09.0"
>     (gnu packages ocaml)
> -   ocaml
> +   ocaml-4.09
>     ("out")
>     #t
>     #f
> @@ -62256,7 +62256,7 @@
>   #("opencl-headers"
>     "2.2.0-0.e986688"
>     (gnu packages opencl)
> -   opencl-headers
> +   opencl-headers-2.2
>     ("out")
>     #t
>     #f
> @@ -92636,7 +92636,7 @@
>   #("python2"
>     "2.7.17"
>     (gnu packages python)
> -   python-2
> +   python-2.7
>     ("out" "tk")
>     #t
>     #f
> @@ -92646,7 +92646,7 @@
>   #("python"
>     "3.8.2"
>     (gnu packages python)
> -   python-3
> +   python
>     ("out" "tk")
>     #t
>     #f
> @@ -123676,7 +123676,7 @@
>   #("rust"
>     "1.39.0"
>     (gnu packages rust)
> -   rust-1.39
> +   rust
>     ("out" "doc" "cargo")
>     #t
>     #f
>
>
> The problem has to do with aliases: we don’t always see the same
> variable first.  So we have to sort before calling ‘expand-cache’.

Fixed in a127e52f601ee73f675d5d28eac2388bb1ad11b0!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.