GNU bug report logs - #46399
[PATCH] build-system/cargo: Propagate crates across builds.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 46399 in the body.
You can then email your comments to 46399 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: guix-patches <at> gnu.org
Cc: Efraim Flashner <efraim <at> flashner.co.il>
Subject: [PATCH] build-system/cargo: Propagate crates across builds.
Date: Tue,  9 Feb 2021 12:58:43 +0200

* guix/build-system/cargo.scm (cargo-build): Add cargo-package-flags,
install-source flags.
* guix/build/cargo-build-system.scm (unpack-rust-crates, package): New
procedures.
(install): Also install crate sources.
(%standard-phases): Add new phases.
* doc/guix.texi (Packaging-guidelines)[Rust Crates]: Adjust to changes
in the cargo-build-system.
---

With this patch, each current crate will also install its sources, both
in a .crate (tar.gz) and unpacked.

(ins)efraim <at> 3900XT ~$ tree -d /gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5
/gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5
└── share
    ├── cargo
    │   ├── registry
    │   └── src
    │       └── bencher-0.1.5
    │           └── benches
    └── doc
        └── rust-bencher-0.1.5

8 directories

A new phase after 'unpack will find the .crate files in
share/cargo/registry, copy them to target/registry and unpack them into
vendor-dir for use with the next build. At the end of that build all the
.crate tarballs are copied to that package's %out/share/cargo/registry,
and also unpacked into %out/share/cargo/src.

This means that with this patch and some modifications to existing
packages to use the new features it should be possible to run 'guix
environment rust-foo' and then in the .cargo/config set
$GUIX_ENVIRONMENT/share/cargo/src as a 'directory source' for cargo.

I have to look more into source-replacement¹ to see if I can organize
the .crate files in %out/share/cargo/registry into a local-registry
format for use as a 'cache' for crates.io.

¹ https://doc.rust-lang.org/cargo/reference/source-replacement.html


 doc/guix.texi                     | 15 ++++---
 guix/build-system/cargo.scm       |  4 ++
 guix/build/cargo-build-system.scm | 66 +++++++++++++++++++++++++++++--
 3 files changed, 77 insertions(+), 8 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 7d18703283..0d153a1470 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32,7 +32,7 @@ Copyright @copyright{} 2015, 2016, 2017, 2019, 2020 Leo Famulari@*
 Copyright @copyright{} 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus@*
 Copyright @copyright{} 2016 Ben Woodcroft@*
 Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
-Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@*
+Copyright @copyright{} 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner@*
 Copyright @copyright{} 2016 John Darrington@*
 Copyright @copyright{} 2016, 2017 Nikita Gillmann@*
 Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@*
@@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
 It adds @code{rustc} and @code{cargo} to the set of inputs.
 A different Rust package can be specified with the @code{#:rust} parameter.
 
-Regular cargo dependencies should be added to the package definition via the
-@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
+Regular cargo dependencies should be added to the package definition similarly
+to other packages; those needed only at build time to native-inputs, others to
+inputs.  If you need to add source-only crates then you should add them to via
+the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
 spec can be a package or a source definition.  Note that the spec must
 evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
 file at its root, or it will be ignored.  Similarly, cargo dev-dependencies
@@ -7449,8 +7451,11 @@ In its @code{configure} phase, this build system will make any source inputs
 specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs}
 parameters available to cargo.  It will also remove an included
 @code{Cargo.lock} file to be recreated by @code{cargo} during the
-@code{build} phase.  The @code{install} phase installs the binaries
-defined by the crate.
+@code{build} phase.  The @code{package} phase will run @code{cargo package}
+to create a source crate for future use.  The @code{install} phase installs
+the binaries defined by the crate.  Unless @code{install-source? #f} is
+defined it will also install a source crate repository of itself and unpacked
+sources, to ease in future hacking on rust packages.
 @end defvr
 
 @defvr {Scheme Variable} chicken-build-system
diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index 6c8edf6bac..01667ef045 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2016 David Craven <david <at> craven.ch>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov <at> gmail.com>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
+;;; Copyright © 2021 Efraim Flashner <efraim <at> flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -77,8 +78,10 @@ to NAME and VERSION."
                       (vendor-dir "guix-vendor")
                       (cargo-build-flags ''("--release"))
                       (cargo-test-flags ''("--release"))
+                      (cargo-package-flags ''("--no-metadata" "--no-verify"))
                       (features ''())
                       (skip-build? #f)
+                      (install-source? #t)
                       (phases '(@ (guix build cargo-build-system)
                                   %standard-phases))
                       (outputs '("out"))
@@ -106,6 +109,7 @@ to NAME and VERSION."
                     #:vendor-dir ,vendor-dir
                     #:cargo-build-flags ,cargo-build-flags
                     #:cargo-test-flags ,cargo-test-flags
+                    #:cargo-package-flags ,cargo-package-flags
                     #:features ,features
                     #:skip-build? ,skip-build?
                     #:tests? ,(and tests? (not skip-build?))
diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index 1d21b33895..0328ede4cb 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -2,7 +2,7 @@
 ;;; Copyright © 2016 David Craven <david <at> craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov <at> gmail.com>
-;;; Copyright © 2019, 2020 Efraim Flashner <efraim <at> flashner.co.il>
+;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba <at> kadziolka.net>
 ;;; Copyright © 2020 Marius Bakke <marius <at> gnu.org>
 ;;;
@@ -73,6 +73,38 @@ Cargo.toml file present at its root."
                                             " | cut -d/ -f2"
                                             " | grep -q '^Cargo.toml$'")))))
 
+(define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys)
+  (define (inputs->rust-inputs inputs)
+    "Filter using the label part from INPUTS."
+    (filter (lambda (input)
+              (match input
+                ((name . _) (rust-package? name))))
+            inputs))
+  (define (inputs->directories inputs)
+    "Extract the directory part from INPUTS."
+    (match inputs
+      (((names . directories) ...)
+       directories)))
+
+  (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs))))
+    (unless (null? rust-inputs)
+      (mkdir-p "target/package")
+      (mkdir-p vendor-dir)
+      ;; TODO: copy only regular inputs to target/package, not native-inputs.
+      (for-each (lambda (input-crate)
+                  (copy-recursively (string-append input-crate
+                                                   "/share/cargo/registry")
+                                    "target/package"))
+                (delete-duplicates rust-inputs))
+
+      (for-each (lambda (crate)
+                  (invoke "tar" "xzf" crate "-C" vendor-dir))
+                (find-files "target/package" "\\.crate$"))))
+  #t)
+
+(define (rust-package? name)
+  (string-prefix? "rust-" name))
+
 (define* (configure #:key inputs
                     (vendor-dir "guix-vendor")
                     #:allow-other-keys)
@@ -170,9 +202,23 @@ directory = '" port)
       (apply invoke "cargo" "test" cargo-test-flags)
       #t))
 
-(define* (install #:key inputs outputs skip-build? features #:allow-other-keys)
+(define* (package #:key
+                  (cargo-package-flags '("--no-metadata" "--no-verify"))
+                  #:allow-other-keys)
+  "Run 'cargo-package' for a given Cargo package."
+  (apply invoke `("cargo" "package" ,@cargo-package-flags)))
+
+(define* (install #:key
+                  inputs
+                  outputs
+                  skip-build?
+                  (install-source? #t)
+                  features
+                  #:allow-other-keys)
   "Install a given Cargo package."
-  (let* ((out (assoc-ref outputs "out")))
+  (let* ((out      (assoc-ref outputs "out"))
+         (registry (string-append out "/share/cargo/registry"))
+         (sources  (string-append out "/share/cargo/src")))
     (mkdir-p out)
 
     ;; Make cargo reuse all the artifacts we just built instead
@@ -186,6 +232,18 @@ directory = '" port)
         (invoke "cargo" "install" "--no-track" "--path" "." "--root" out
                 "--features" (string-join features)))
 
+    (when install-source?
+      ;; Install crate tarballs and unpacked sources for later use.
+      ;; TODO: Is there a better format/directory for these files?
+      (mkdir-p sources)
+      (for-each (lambda (crate)
+                  (install-file crate registry))
+                (find-files "target/package" "\\.crate$"))
+
+      (for-each (lambda (crate)
+                  (invoke "tar" "xzf" crate "-C" sources))
+                (find-files registry "\\.crate$")))
+
     #t))
 
 (define %standard-phases
@@ -195,6 +253,8 @@ directory = '" port)
     (replace 'build build)
     (replace 'check check)
     (replace 'install install)
+    (add-after 'build 'package package)
+    (add-after 'unpack 'unpack-rust-crates unpack-rust-crates)
     (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
-- 
2.30.0

Message #10 received at 46399-done <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: 46399-done <at> debbugs.gnu.org
Subject: Re: bug#46399: Acknowledgement ([PATCH] build-system/cargo:
 Propagate crates across builds.)
Date: Sun, 14 Mar 2021 19:59:39 +0200

[Message part 1 (text/plain, inline)]

Pushed with minor revisions. Also patched a few packages which needed
#:install-source #? in order to continue to build.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 46399 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 46399 <at> debbugs.gnu.org
Subject: Re: [bug#46399] [PATCH] build-system/cargo: Propagate crates across
 builds.
Date: Sun, 11 Apr 2021 17:11:21 +0200

Hello Efraim!

Thank you for improving Rust development with Guix.  I have a question
about this change to doc/guix.texi when translating.  Sorry for
noticing so late just before the string freeze.

On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 7d18703283..0d153a1470 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> […]
> @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
>  It adds @code{rustc} and @code{cargo} to the set of inputs.
>  A different Rust package can be specified with the @code{#:rust} parameter.
>  
> -Regular cargo dependencies should be added to the package definition via the
> -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> +Regular cargo dependencies should be added to the package definition similarly
> +to other packages; those needed only at build time to native-inputs, others to
> +inputs.  If you need to add source-only crates then you should add them to via
> +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
>  spec can be a package or a source definition.  Note that the spec must
>  evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
>  file at its root, or it will be ignored.  Similarly, cargo dev-dependencies

I have no experience in Rust.  Do I understand correctly that
source-only crates among the inputs should be added both to inputs and
to #:cargo-inputs?  Is the

> +inputs.  If you need to add source-only crates then you should add them to via

a typo which should read “add them too” instead of “add them to”?

I look at the subsequent change
f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons

> diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm
> index d67e9dc02f..7dde00ffdb 100644
> --- a/gnu/packages/crates-graphics.scm
> +++ b/gnu/packages/crates-graphics.scm
> @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")
>      (arguments
> -     `(#:skip-build? #t
> -       #:cargo-inputs
> +     `(#:cargo-inputs
>         (("rust-nix" ,rust-nix-0.18)
>          ("rust-once-cell" ,rust-once-cell-1)
> -        ("rust-smallvec" ,rust-smallvec-1)
> -        ("rust-wayland-sys" ,rust-wayland-sys-0.28))))
> +        ("rust-smallvec" ,rust-smallvec-1))))
> +    (inputs
> +     `(("rust-nix" ,rust-nix-0.18)
> +       ("rust-once-cell" ,rust-once-cell-1)
> +       ("rust-smallvec" ,rust-smallvec-1)
> +       ("rust-wayland-sys" ,rust-wayland-sys-0.28)))
>      (home-page "https://github.com/smithay/wayland-rs")

and do not understand why rust-wayland-sys is only part of inputs but
rust-once-cell is also among the cargo-inputs.  Also the inputs are
empty for older versions of rust-wayland-commons.

Could you clarify what is written in the doc/guix.texi?

Regards,
Florian

Message #16 received at 46399 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
Cc: 46399 <at> debbugs.gnu.org
Subject: Re: [bug#46399] [PATCH] build-system/cargo: Propagate crates across
 builds.
Date: Mon, 12 Apr 2021 09:40:17 +0300

[Message part 1 (text/plain, inline)]

On Sun, Apr 11, 2021 at 05:11:21PM +0200, pelzflorian (Florian Pelz) wrote:
> Hello Efraim!
> 
> Thank you for improving Rust development with Guix.  I have a question
> about this change to doc/guix.texi when translating.  Sorry for
> noticing so late just before the string freeze.
> 
> On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:
> > diff --git a/doc/guix.texi b/doc/guix.texi
> > index 7d18703283..0d153a1470 100644
> > --- a/doc/guix.texi
> > +++ b/doc/guix.texi
> > […]
> > @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
> >  It adds @code{rustc} and @code{cargo} to the set of inputs.
> >  A different Rust package can be specified with the @code{#:rust} parameter.
> >  
> > -Regular cargo dependencies should be added to the package definition via the
> > -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> > +Regular cargo dependencies should be added to the package definition similarly
> > +to other packages; those needed only at build time to native-inputs, others to
> > +inputs.  If you need to add source-only crates then you should add them to via
> > +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> >  spec can be a package or a source definition.  Note that the spec must
> >  evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
> >  file at its root, or it will be ignored.  Similarly, cargo dev-dependencies
> 
> I have no experience in Rust.  Do I understand correctly that
> source-only crates among the inputs should be added both to inputs and
> to #:cargo-inputs?  Is the
> 
> > +inputs.  If you need to add source-only crates then you should add them to via
> 
> a typo which should read “add them too” instead of “add them to”?

I went back and refreshed myself with what I wrote there. source-only
crates should be added to both regular inputs and to #:cargo-inputs.
Actually converting the ~1000 rust packages to actually make good use of
the changes will allow us to drop nearly all the #:cargo-inputs and just
use regular inputs, but until then we still need the propagated source
magic that comes from #:cargo-inputs.

Yes, definitely a typo, it should be "add them too".

> I look at the subsequent change
> f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons
> 
> > diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm
> > index d67e9dc02f..7dde00ffdb 100644
> > --- a/gnu/packages/crates-graphics.scm
> > +++ b/gnu/packages/crates-graphics.scm
> > @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")
> >      (arguments
> > -     `(#:skip-build? #t
> > -       #:cargo-inputs
> > +     `(#:cargo-inputs
> >         (("rust-nix" ,rust-nix-0.18)
> >          ("rust-once-cell" ,rust-once-cell-1)
> > -        ("rust-smallvec" ,rust-smallvec-1)
> > -        ("rust-wayland-sys" ,rust-wayland-sys-0.28))))
> > +        ("rust-smallvec" ,rust-smallvec-1))))
> > +    (inputs
> > +     `(("rust-nix" ,rust-nix-0.18)
> > +       ("rust-once-cell" ,rust-once-cell-1)
> > +       ("rust-smallvec" ,rust-smallvec-1)
> > +       ("rust-wayland-sys" ,rust-wayland-sys-0.28)))
> >      (home-page "https://github.com/smithay/wayland-rs")
> 
> and do not understand why rust-wayland-sys is only part of inputs but
> rust-once-cell is also among the cargo-inputs.  Also the inputs are
> empty for older versions of rust-wayland-commons.

#:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs
to future builds. The changes to the cargo-build-system mimic this so we
can use regular inputs and not #:cargo-inputs. rust-wayland-sys-0.28 has
enough of its dependencies converted that it can be used as just an
input but rust-once-cell-1 hasn't yet so it still needs to be in
#:cargo-inputs. As far as the other versions of rust-wayland-commons, I
started by focusing on alacritty so we could remove some of the build
phases and move them to the crates which actually needed patching.

> Could you clarify what is written in the doc/guix.texi?
> 
> Regards,
> Florian

I think for now the best option is to add the rust inputs of all types
to both inputs and #:cargo-inputs (or native-inputs and
#:cargo-development-inputs) and then remove them from #:cargo-inputs
when it's possible to do so. Ultimately Someone™ should revive the
wip-rust branch and convert all the crates over in one go and adjust the
importer and manual.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 46399 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 46399 <at> debbugs.gnu.org
Subject: Re: [bug#46399] [PATCH] build-system/cargo: Propagate crates across
 builds.
Date: Tue, 13 Apr 2021 14:31:39 +0200

On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:
> I went back and refreshed myself with what I wrote there. source-only
> crates should be added to both regular inputs and to #:cargo-inputs.

Thank you for clarifying.  I will translate the manual accordingly.

> #:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs
> to future builds.

I had not known about this difference.

> I think for now the best option is to add the rust inputs of all types
> to both inputs and #:cargo-inputs (or native-inputs and
> #:cargo-development-inputs) and then remove them from #:cargo-inputs
> when it's possible to do so. Ultimately Someone™ should revive the
> wip-rust branch and convert all the crates over in one go and adjust the
> importer and manual.

I suppose this paragraph of your reply should be added to the manual.

Regards,
Florian

Message #22 received at 46399 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
Cc: 46399 <at> debbugs.gnu.org, Efraim Flashner <efraim <at> flashner.co.il>
Subject: Re: [bug#46399] [PATCH] build-system/cargo: Propagate crates across
 builds.
Date: Tue, 13 Apr 2021 13:49:25 -0400

On Tue, Apr 13, 2021 at 02:31:39PM +0200, pelzflorian (Florian Pelz) wrote:
> On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:
> > I went back and refreshed myself with what I wrote there. source-only
> > crates should be added to both regular inputs and to #:cargo-inputs.
> 
> Thank you for clarifying.  I will translate the manual accordingly.

We are in "string freeze" for the upcoming release, so please delay your
changes to the manual until the release is completed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.