GNU bug report logs - #33066
[PATCH 0/2] Workaround rust reproducibility issues

Previous Next

Package: guix-patches;

Reported by: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>

Date: Tue, 16 Oct 2018 18:12:01 UTC

Severity: normal

Tags: patch

Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 33066 in the body.
You can then email your comments to 33066 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Tue, 16 Oct 2018 18:12:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to Nikolai Merinov <nikolai.merinov <at> member.fsf.org>:
New bug report received and forwarded. Copy sent to guix-patches <at> gnu.org. (Tue, 16 Oct 2018 18:12:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/2] Workaround rust reproducibility issues
Date: Tue, 16 Oct 2018 23:00:14 +0500
Hi,

I prepared patches for Rust packages.

First patch move rust 1.25-1.27 packages back to llvm 3.9.1. I spend much time
tring to find root of non-reproducibility in this releases, but I did not
succeeded. So as easiest solution I suggest just keep old LLVM for this
packages. Drawback of this solution is impossibility to use
"#[target_feature]" will not work for this releases.

Second patch have new 1.28.0, 1.29.1 rust packages. They was reproducible in
my local tests with llvm 6.0.1, so this packages was switched to new LLVM.

Please, note: I tested packages with suggested changes only from
https://github.com/mnd/guix-mnd-pkgs/ repository. Full build chain
reproducibility test with exactly submitted patches currently in progress on
my laptop, but it can take another one or two days.

Regards,
Nikolai

Nikolai Merinov (2):
  gnu: rust: workaround rust 1.25-27 reproducibility issues
  gnu: rust: add rust 1.28 and rust 1.29 packages

 gnu/local.mk                                  |  1 +
 ...ible-builds-by-forcing-window.search.patch | 28 +++++++
 gnu/packages/rust.scm                         | 76 +++++++++++++------
 3 files changed, 81 insertions(+), 24 deletions(-)
 create mode 100644 gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch

-- 
2.19.0





Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Tue, 16 Oct 2018 18:14:01 GMT) Full text and rfc822 format available.

Message #8 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: 33066 <at> debbugs.gnu.org
Subject: [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
Date: Tue, 16 Oct 2018 02:32:11 +0500
* gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
patch that make "searchindex.js" reproducible in rust 1.27 and newer.
* gnu/local.mk (dist_patch_DATA): Add new patch file.
* gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
during cargo build. Note: libgit2 still builded as part of cargo build,
because cargo tests assume specific libgit2 minor release.
(rust-1.23): inherit native-inputs from previous package.
(rust-1.25): switch back to llvm 3.9.1 as workaround for
https://github.com/rust-lang/rust/issues/50556 issue.
(rust-1.27): apply patch to make "searchindex.js" files reproducible.
---
 gnu/local.mk                                  |  1 +
 ...ible-builds-by-forcing-window.search.patch | 28 ++++++++++++++++
 gnu/packages/rust.scm                         | 32 ++++++-------------
 3 files changed, 38 insertions(+), 23 deletions(-)
 create mode 100644 gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1fa2eaba3..2150cbf13 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1130,6 +1130,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-1.25-accept-more-detailed-gdb-lines.patch \
   %D%/packages/patches/rust-bootstrap-stage0-test.patch		\
   %D%/packages/patches/rust-coresimd-doctest.patch		\
+  %D%/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch \
   %D%/packages/patches/rxvt-unicode-escape-sequences.patch	\
   %D%/packages/patches/scheme48-tests.patch			\
   %D%/packages/patches/scotch-test-threading.patch		\
diff --git a/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch b/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
new file mode 100644
index 000000000..916662cbc
--- /dev/null
+++ b/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
@@ -0,0 +1,28 @@
+From b9ca108fcae2b738ca3f0c88c84ae5dc5a6f843f Mon Sep 17 00:00:00 2001
+From: Tim Ryan <id <at> timryan.org>
+Date: Mon, 14 May 2018 06:22:21 -0400
+Subject: [PATCH] Support reproducible builds by forcing window.search to use
+ stable key ordering. (#692)
+
+---
+ src/vendor/mdbook/src/renderer/html_handlebars/search.rs | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+index d49772f8b..1ee66a511 100644
+--- a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
++++ b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+@@ -205,6 +205,10 @@ fn write_to_js(index: Index, search_config: &Search) -> Result<String> {
+         searchoptions,
+         index,
+     };
++
++    // By converting to serde_json::Value as an intermediary, we use a
++    // BTreeMap internally and can force a stable ordering of map keys.
++    let json_contents = serde_json::to_value(&json_contents)?;
+     let json_contents = serde_json::to_string(&json_contents)?;
+ 
+     Ok(format!("window.search = {};", json_contents))
+-- 
+2.19.0
+
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 40633854a..cbfbdccf1 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -480,7 +480,8 @@ test = { path = \"../libtest\" }
      `(("jemalloc" ,jemalloc-4.5.0)
        ("llvm" ,llvm-3.9.1)
        ("openssl" ,openssl)
-       ("libcurl" ,curl))) ; For "cargo"
+       ("libssh2" ,libssh2) ; For "cargo"
+       ("libcurl" ,curl)))  ; For "cargo"
 
     ;; rustc invokes gcc, so we need to set its search paths accordingly.
     ;; Note: duplicate its value here to cope with circular dependencies among
@@ -649,18 +650,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
     (name "rust")
     (version "1.23.0")
     (source (rust-source version "14fb8vhjzsxlbi6yrn1r6fl5dlbdd1m92dn5zj5gmzfwf4w9ar3l"))
+    ;; Use rust-bootstrap <at> 1.22 package to build rust 1.23
     (native-inputs
-     `(("bison" ,bison) ; For the tests
-       ("cmake" ,cmake)
-       ("flex" ,flex) ; For the tests
-       ("gdb" ,gdb)   ; For the tests
-       ("git" ,git)
-       ("procps" ,procps) ; For the tests
-       ("python-2" ,python-2)
-       ("rustc-bootstrap" ,rust-bootstrap)
-       ("cargo-bootstrap" ,rust-bootstrap "cargo")
-       ("pkg-config" ,pkg-config) ; For "cargo"
-       ("which" ,which)))
+     (alist-replace "cargo-bootstrap" (list rust-bootstrap "cargo")
+                    (alist-replace "rustc-bootstrap" (list rust-bootstrap)
+                                   (package-native-inputs rust-1.20))))
     (arguments
      (substitute-keyword-arguments (package-arguments rust-1.20)
        ((#:phases phases)
@@ -699,10 +693,6 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
           #:patches '("rust-1.25-accept-more-detailed-gdb-lines.patch"))))
     (package
       (inherit base-rust)
-      (inputs
-       ;; Use LLVM 6.0
-       (alist-replace "llvm" (list llvm)
-                      (package-inputs base-rust)))
       (arguments
        (substitute-keyword-arguments (package-arguments base-rust)
          ((#:phases phases)
@@ -713,14 +703,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                    ;; This test wants to update the crate index.
                    (("fn no_index_update") "#[ignore]\nfn no_index_update"))
                  #t))
-             (add-after 'configure 'enable-codegen-tests
-               (lambda _
-                 (substitute* "config.toml"
-                   (("codegen-tests = false") ""))
-                 #t))
              ;; FIXME: Re-enable this test if it's indeed supposed to work.
              ;; See <https://github.com/rust-lang/rust/issues/54178>.
-             (add-after 'enable-codegen-tests 'disable-nil-enum-test
+             (add-after 'configure 'disable-nil-enum-test
                (lambda _
                  (substitute* "src/test/debuginfo/nil-enum.rs"
                    (("ignore-lldb") "ignore-gdb"))
@@ -795,7 +780,8 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                                     #:patches
                                     '("rust-coresimd-doctest.patch"
                                       "rust-bootstrap-stage0-test.patch"
-                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"))))
+                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
     (package
       (inherit base-rust)
       (arguments
-- 
2.19.0





Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Tue, 16 Oct 2018 18:14:02 GMT) Full text and rfc822 format available.

Message #11 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: 33066 <at> debbugs.gnu.org
Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
Date: Tue, 16 Oct 2018 02:37:41 +0500
* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Disable test that required network support.
(rust): Package for rust 1.29.1.
---
 gnu/packages/rust.scm | 44 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cbfbdccf1..d61b431e8 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -773,7 +773,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                    (("fn no_index_update") "#[ignore]\nfn no_index_update"))
                  #t)))))))))
 
-(define-public rust
+(define-public rust-1.27
   (let ((base-rust
          (rust-bootstrapped-package rust-1.26 "1.27.2"
                                     "0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -795,3 +795,45 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                  (mkdir-p (assoc-ref outputs "out"))
                  (mkdir-p (assoc-ref outputs "cargo"))
                  #t)))))))))
+
+(define-public rust-1.28
+  (let ((base-rust
+         (rust-bootstrapped-package rust-1.27 "1.28.0"
+                                    "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+                                    #:patches
+                                    '("rust-coresimd-doctest.patch"
+                                      "rust-bootstrap-stage0-test.patch"
+                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+    (package
+      (inherit base-rust)
+      (inputs
+       ;; Use LLVM 6.0
+       (alist-replace "llvm" (list llvm)
+                      (package-inputs base-rust)))
+      (arguments
+       (substitute-keyword-arguments (package-arguments base-rust)
+         ((#:phases phases)
+          `(modify-phases ,phases
+             (add-after 'configure 'enable-codegen-tests
+               ;; codegen tests should work fine with llvm 6
+               (lambda* _
+                 (substitute* "config.toml"
+                   (("codegen-tests = false") ""))
+                 #t))
+             (add-after 'patch-tests 'disable-amd64-avx-test
+               ;; This test will fail on x86_64 machines without avx
+               (lambda* _
+                 (substitute* "src/test/run-pass/issue-44056.rs"
+                   (("only-x86_64") "ignore-test")))))))))))
+
+
+(define-public rust
+  (let ((base-rust
+         (rust-bootstrapped-package rust-1.28 "1.29.1"
+                                    "0jd3c57x3yndizns4pb68nh25si47agfmrdvf9nwwsyfcs5p5c7i"
+                                    #:patches
+                                    '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+    (package
+      (inherit base-rust))))
-- 
2.19.0





Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Wed, 17 Oct 2018 11:15:02 GMT) Full text and rfc822 format available.

Message #14 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Wed, 17 Oct 2018 13:14:06 +0200
[Message part 1 (text/plain, inline)]
Hi Nikolai,

On Tue, 16 Oct 2018 02:32:11 +0500
Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:

> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:

Nitpick: No big "S" (file names are easier to find if they are all lower case).

> patch that make "searchindex.js" reproducible in rust 1.27 and newer.

"New file".

> * gnu/local.mk (dist_patch_DATA): Add new patch file.
> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library

Hmm, I'm not sure about doing this in the same commit.
Is it also related to reproducibility?

> during cargo build. Note: libgit2 still builded as part of cargo build,
> because cargo tests assume specific libgit2 minor release.

What does this mean?  Does it mean "bundled"?

> (rust-1.23): inherit native-inputs from previous package.

Ok.

> (rust-1.25): switch back to llvm 3.9.1 as workaround for
> https://github.com/rust-lang/rust/issues/50556 issue.

Please add the reasoning as a comment inside the source code instead.

> (rust-1.27): apply patch to make "searchindex.js" files reproducible.

Maybe add "[source]".

> -             (add-after 'configure 'enable-codegen-tests
> -               (lambda _
> -                 (substitute* "config.toml"
> -                   (("codegen-tests = false") ""))
> -                 #t))

I think I had reproducibility problems when enabling codegen tests and
parallel tests.  Is that not the case anymore?

>               ;; FIXME: Re-enable this test if it's indeed supposed to work.
>               ;; See <https://github.com/rust-lang/rust/issues/54178>.

Note to myself: I think the issue comments indicate that the newer gdb output
is better - so we should create a patch similar to
rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Wed, 17 Oct 2018 11:16:01 GMT) Full text and rfc822 format available.

Message #17 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29
 packages
Date: Wed, 17 Oct 2018 13:15:36 +0200
[Message part 1 (text/plain, inline)]
> +             (add-after 'patch-tests 'disable-amd64-avx-test
> +               ;; This test will fail on x86_64 machines without avx
> +               (lambda* _
> +                 (substitute* "src/test/run-pass/issue-44056.rs"
> +                   (("only-x86_64") "ignore-test")))))))))))

Please return #t.

Otherwise ok!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Wed, 17 Oct 2018 14:15:02 GMT) Full text and rfc822 format available.

Message #20 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Joe Hillenbrand <joehillen <at> gmail.com>
To: nikolai.merinov <at> member.fsf.org
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29
 packages
Date: Wed, 17 Oct 2018 07:14:06 -0700
On Tue, Oct 16, 2018 at 11:50 AM Nikolai Merinov
<nikolai.merinov <at> member.fsf.org> wrote:
>
> +(define-public rust
> +  (let ((base-rust
> +         (rust-bootstrapped-package rust-1.28 "1.29.1"

The latest version is now 1.29.2




Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Fri, 19 Oct 2018 19:46:01 GMT) Full text and rfc822 format available.

Message #23 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Sat, 20 Oct 2018 00:44:45 +0500
[Message part 1 (text/plain, inline)]
Hi Danny,

I fixed patches according to your comments. Also I updated rust 1.29.1
to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
reproducibility of each package previously only in private my private
repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
slow machine, but currently I already checked reproducibility with
suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.

Both updated patches attached.

Danny Milosavljevic <dannym <at> scratchpost.org> writes:

> Hi Nikolai,
>
> On Tue, 16 Oct 2018 02:32:11 +0500
> Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:
>
>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>
> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>
Fixed.

>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>
> "New file".
Fixed.

>
>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>
> Hmm, I'm not sure about doing this in the same commit.
> Is it also related to reproducibility?

Looks like it should not, but still I started investigation with
non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
related changes at very beginning and never tested wihtout it. I not
sure that it's good idea to remove it now.

>
>> during cargo build. Note: libgit2 still builded as part of cargo build,
>> because cargo tests assume specific libgit2 minor release.
>
> What does this mean?  Does it mean "bundled"?
Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).

>
>> (rust-1.23): inherit native-inputs from previous package.
>
> Ok.
>
>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>> https://github.com/rust-lang/rust/issues/50556 issue.
>
> Please add the reasoning as a comment inside the source code instead.
Added comment for rust-1.25 package.

>
>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>
> Maybe add "[source]".
I repharase this comment and added source URL to patch file itself.

>
>> -             (add-after 'configure 'enable-codegen-tests
>> -               (lambda _
>> -                 (substitute* "config.toml"
>> -                   (("codegen-tests = false") ""))
>> -                 #t))
>
> I think I had reproducibility problems when enabling codegen tests and
> parallel tests.  Is that not the case anymore?
Neither me nor Joe Hillenbrand in
https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
reproducibility issues with this tests.
>
>>               ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>               ;; See <https://github.com/rust-lang/rust/issues/54178>.
>
> Note to myself: I think the issue comments indicate that the newer gdb output
> is better - so we should create a patch similar to
> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.

[0001-gnu-rust-workaround-rust-1.25-27-reproducibility-iss.patch (text/x-patch, attachment)]
[0002-gnu-rust-add-rust-1.28-and-rust-1.29-packages.patch (text/x-patch, attachment)]

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Fri, 19 Oct 2018 21:00:03 GMT) Full text and rfc822 format available.

Message #26 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Sat, 20 Oct 2018 01:58:43 +0500
[Message part 1 (text/plain, inline)]
Hi,

Sorry, ignore my previous mail. There was wrong patch
export. Re-attached patches.

[0001-gnu-rust-workaround-rust-1.25-27-reproducibility-iss.patch (text/x-patch, attachment)]
[0002-gnu-rust-add-rust-1.28-and-rust-1.29-packages.patch (text/x-patch, attachment)]
[Message part 4 (text/plain, inline)]
Regards,
Nikolai

Nikolai Merinov <nikolai.merinov <at> member.fsf.org> writes:

> Hi Danny,
>
> I fixed patches according to your comments. Also I updated rust 1.29.1
> to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
> reproducibility of each package previously only in private my private
> repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
> slow machine, but currently I already checked reproducibility with
> suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.
>
> Both updated patches attached.
>
> Danny Milosavljevic <dannym <at> scratchpost.org> writes:
>
>> Hi Nikolai,
>>
>> On Tue, 16 Oct 2018 02:32:11 +0500
>> Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:
>>
>>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>>
>> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>>
> Fixed.
>
>>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>>
>> "New file".
> Fixed.
>
>>
>>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>>
>> Hmm, I'm not sure about doing this in the same commit.
>> Is it also related to reproducibility?
>
> Looks like it should not, but still I started investigation with
> non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
> related changes at very beginning and never tested wihtout it. I not
> sure that it's good idea to remove it now.
>
>>
>>> during cargo build. Note: libgit2 still builded as part of cargo build,
>>> because cargo tests assume specific libgit2 minor release.
>>
>> What does this mean?  Does it mean "bundled"?
> Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).
>
>>
>>> (rust-1.23): inherit native-inputs from previous package.
>>
>> Ok.
>>
>>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>>> https://github.com/rust-lang/rust/issues/50556 issue.
>>
>> Please add the reasoning as a comment inside the source code instead.
> Added comment for rust-1.25 package.
>
>>
>>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>>
>> Maybe add "[source]".
> I repharase this comment and added source URL to patch file itself.
>
>>
>>> -             (add-after 'configure 'enable-codegen-tests
>>> -               (lambda _
>>> -                 (substitute* "config.toml"
>>> -                   (("codegen-tests = false") ""))
>>> -                 #t))
>>
>> I think I had reproducibility problems when enabling codegen tests and
>> parallel tests.  Is that not the case anymore?
> Neither me nor Joe Hillenbrand in
> https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
> reproducibility issues with this tests.
>>
>>>               ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>>               ;; See <https://github.com/rust-lang/rust/issues/54178>.
>>
>> Note to myself: I think the issue comments indicate that the newer gdb output
>> is better - so we should create a patch similar to
>> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Sun, 21 Oct 2018 07:29:01 GMT) Full text and rfc822 format available.

Message #29 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Sun, 21 Oct 2018 12:28:04 +0500
Hi,

Finally whole rust build chain 1.23-1.29 was built on my machine with next commands:
> guix pull --url=/path/to/guix-f9a8fce10/with/patches --commit=<commit-with-patches>
> for v in 1.2{3..9}; do echo guix build -K --rounds=2 rust@$v || break ; done
and all builds were reproducible.

Regards,
Nikolai

Nikolai Merinov <nikolai.merinov <at> member.fsf.org> writes:

> Hi,
>
> Sorry, ignore my previous mail. There was wrong patch
> export. Re-attached patches.
>
>
>
>
> Regards,
> Nikolai
>
> Nikolai Merinov <nikolai.merinov <at> member.fsf.org> writes:
>
>> Hi Danny,
>>
>> I fixed patches according to your comments. Also I updated rust 1.29.1
>> to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
>> reproducibility of each package previously only in private my private
>> repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
>> slow machine, but currently I already checked reproducibility with
>> suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.
>>
>> Both updated patches attached.
>>
>> Danny Milosavljevic <dannym <at> scratchpost.org> writes:
>>
>>> Hi Nikolai,
>>>
>>> On Tue, 16 Oct 2018 02:32:11 +0500
>>> Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:
>>>
>>>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>>>
>>> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>>>
>> Fixed.
>>
>>>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>>>
>>> "New file".
>> Fixed.
>>
>>>
>>>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>>>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>>>
>>> Hmm, I'm not sure about doing this in the same commit.
>>> Is it also related to reproducibility?
>>
>> Looks like it should not, but still I started investigation with
>> non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
>> related changes at very beginning and never tested wihtout it. I not
>> sure that it's good idea to remove it now.
>>
>>>
>>>> during cargo build. Note: libgit2 still builded as part of cargo build,
>>>> because cargo tests assume specific libgit2 minor release.
>>>
>>> What does this mean?  Does it mean "bundled"?
>> Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).
>>
>>>
>>>> (rust-1.23): inherit native-inputs from previous package.
>>>
>>> Ok.
>>>
>>>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>>>> https://github.com/rust-lang/rust/issues/50556 issue.
>>>
>>> Please add the reasoning as a comment inside the source code instead.
>> Added comment for rust-1.25 package.
>>
>>>
>>>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>>>
>>> Maybe add "[source]".
>> I repharase this comment and added source URL to patch file itself.
>>
>>>
>>>> -             (add-after 'configure 'enable-codegen-tests
>>>> -               (lambda _
>>>> -                 (substitute* "config.toml"
>>>> -                   (("codegen-tests = false") ""))
>>>> -                 #t))
>>>
>>> I think I had reproducibility problems when enabling codegen tests and
>>> parallel tests.  Is that not the case anymore?
>> Neither me nor Joe Hillenbrand in
>> https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
>> reproducibility issues with this tests.
>>>
>>>>               ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>>>               ;; See <https://github.com/rust-lang/rust/issues/54178>.
>>>
>>> Note to myself: I think the issue comments indicate that the newer gdb output
>>> is better - so we should create a patch similar to
>>> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.




Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Wed, 24 Oct 2018 19:44:02 GMT) Full text and rfc822 format available.

Message #32 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Wed, 24 Oct 2018 21:43:30 +0200
[Message part 1 (text/plain, inline)]
Hi Nikolai,

I have reviewed the patchset and am currently building the entire rust chain
with your patches with several rounds and plan on merging rust after a
successful run (probably Friday).
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Thu, 25 Oct 2018 05:45:02 GMT) Full text and rfc822 format available.

Message #35 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Thu, 25 Oct 2018 07:44:22 +0200
[Message part 1 (text/plain, inline)]
Hi Nikolai,

I've got a cargo reproducibility failure in rust 1.29.2.   Attached is diffoscope output :(

Seems a little similar to what we had before (https://github.com/rust-lang/rust/issues/50556 ).
[cargo-1.29.2.diff (text/x-patch, attachment)]
[Message part 3 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Fri, 26 Oct 2018 11:41:01 GMT) Full text and rfc822 format available.

Message #38 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Fri, 26 Oct 2018 16:40:34 +0500
Hi Danny,

I'm sad to hear it. All this issues looks for me as LLVM reproducibility
issue and there was some reproducibility-related changes after LLVM
6.0.1 release, e.g. https://reviews.llvm.org/D48369
https://reviews.llvm.org/D48392 and others. Note: I checked previously
mentioned changes individually and it did not solved reproducibility
issue for rust.

Unfortunately I'll be not able to return to this reproducibility
investigation for a while. So currently I can suggest to merge changes
without rust 1.29 to have reproducible chain from 1.23 to 1.28.

Regards,
Nikolai

Danny Milosavljevic <dannym <at> scratchpost.org> writes:

> Hi Nikolai,
>
> I've got a cargo reproducibility failure in rust 1.29.2.   Attached is diffoscope output :(
>
> Seems a little similar to what we had before (https://github.com/rust-lang/rust/issues/50556 ).




Information forwarded to guix-patches <at> gnu.org:
bug#33066; Package guix-patches. (Fri, 26 Oct 2018 16:13:02 GMT) Full text and rfc822 format available.

Message #41 received at 33066 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>
Cc: 33066 <at> debbugs.gnu.org
Subject: Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27
 reproducibility issues
Date: Fri, 26 Oct 2018 18:12:34 +0200
[Message part 1 (text/plain, inline)]
Hi Nikolai,

On Fri, 26 Oct 2018 16:40:34 +0500
Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:

> Unfortunately I'll be not able to return to this reproducibility
> investigation for a while. So currently I can suggest to merge changes
> without rust 1.29 to have reproducible chain from 1.23 to 1.28.

I did so now.

Thanks for the hard work!
[Message part 2 (application/pgp-signature, inline)]

Reply sent to Maxim Cournoyer <maxim.cournoyer <at> gmail.com>:
You have taken responsibility. (Fri, 05 Feb 2021 15:01:02 GMT) Full text and rfc822 format available.

Notification sent to Nikolai Merinov <nikolai.merinov <at> member.fsf.org>:
bug acknowledged by developer. (Fri, 05 Feb 2021 15:01:02 GMT) Full text and rfc822 format available.

Message #46 received at 33066-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: Nikolai Merinov <nikolai.merinov <at> member.fsf.org>,
 33066-done <at> debbugs.gnu.org
Subject: Re: bug#33066: [PATCH 0/2] Workaround rust reproducibility issues
Date: Fri, 05 Feb 2021 10:00:05 -0500
Hello,

Danny Milosavljevic <dannym <at> scratchpost.org> writes:

> Hi Nikolai,
>
> On Fri, 26 Oct 2018 16:40:34 +0500
> Nikolai Merinov <nikolai.merinov <at> member.fsf.org> wrote:
>
>> Unfortunately I'll be not able to return to this reproducibility
>> investigation for a while. So currently I can suggest to merge changes
>> without rust 1.29 to have reproducible chain from 1.23 to 1.28.
>
> I did so now.
>
> Thanks for the hard work!

I'm closing this; the work to bootstrap using mrustc 0.9 to start from
rust 1.29.2 has been merged to core-updates; it brought many changes.

Thank you,

Maxim




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sat, 06 Mar 2021 12:24:07 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 23 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.