GNU bug report logs - #35155
[PATCH] build-system/cargo: refactor phases to successfully build

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35155 in the body.
You can then email your comments to 35155 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Chris Marusich <cmmarusich <at> gmail.com>
Subject: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Fri, 5 Apr 2019 00:07:27 -0700

[Message part 1 (text/plain, inline)]

This patch refactors the cargo-build-system builder code to correctly build
imported Rust crates. Specifically the changes:

* Do not rely on a Cargo.lock presence to determine installation. Cargo will
automatically create the file at its first invocation, so instead we check
the manifest for any executable targets before attempting an installation
* Do not attempt to modify the Cargo.toml file. There are many system specific
package in crates.io (e.g. for Windows, Redox, Fuschia, WASM, etc.) and
attempting to keep up with what crates must be patched out is futile.
* The build phases will honor a skip-build? flag which allows for
short-circuiting for optional packages which require nightly features or cannot
be built for the current platform.

Changes which still need to be done:
* Update the host-side code to expand transitive inputs: cargo requires that
all transitive crate dependencies are present in its (vendored) index, but
doing so by hand in the package definitions will become unwieldy.
* Update the host-side code to detect any "circular" dependencies which can
result from a naive import

Unfortunately there isn't a good way to test this patch at the moment.
Importing a non-trivial crate requires a lot of manual resolution, especially
with the points above remaining unimplemented.

If someone would really like to see the input package definitions I was using
to test, I'd be happy to share, though I'd advise that it's pretty hacked up
for my own convoluted testing at the moment.

—Ivan

[0001-build-system-cargo-refactor-phases-to-successfully-b.patch (application/octet-stream, attachment)]

Message #8 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: Chris Marusich <cmmarusich <at> gmail.com>, 35155 <at> debbugs.gnu.org
Subject: Re: [bug#35155] [PATCH] build-system/cargo: refactor phases to
 successfully build
Date: Sat, 6 Apr 2019 12:32:34 +0200

[Message part 1 (text/plain, inline)]

Hi Ivan,

good idea.

(Finally the hack with the Cargo.lock is gone :) )

>There are many system specific
> package in crates.io (e.g. for Windows, Redox, Fuschia, WASM, etc.) and
> attempting to keep up with what crates must be patched out is futile.

I agree.

> * The build phases will honor a skip-build? flag which allows for
> short-circuiting for optional packages which require nightly features or cannot
> be built for the current platform.

Ok, I guess.

> Changes which still need to be done:
> * Update the host-side code to expand transitive inputs: cargo requires that
> all transitive crate dependencies are present in its (vendored) index, but
> doing so by hand in the package definitions will become unwieldy.

Yeah.  Let's do that in an extra patch.

> * Update the host-side code to detect any "circular" dependencies which can
> result from a naive import

Yeah.

> Unfortunately there isn't a good way to test this patch at the moment.
> Importing a non-trivial crate requires a lot of manual resolution, especially
> with the points above remaining unimplemented.


> If someone would really like to see the input package definitions I was using
> to test, I'd be happy to share, though I'd advise that it's pretty hacked up
> for my own convoluted testing at the moment.

No, I can use my own set of hacked-together package definition to test it.

(The state of my hacked-together package definitions is not good enough to merge :) )

[Message part 2 (application/pgp-signature, inline)]

Message #11 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: Chris Marusich <cmmarusich <at> gmail.com>, 35155 <at> debbugs.gnu.org
Subject: Re: [bug#35155] [PATCH] build-system/cargo: refactor phases to
 successfully build
Date: Sat, 6 Apr 2019 12:41:25 +0200

[Message part 1 (text/plain, inline)]

>+    (generate-checksums rsrc "/dev/null")

Hmm, is the "package" entry in the final file ".cargo-checksums.json" useless?
Can it be left off completely?

[Message part 2 (application/pgp-signature, inline)]

Message #14 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: Chris Marusich <cmmarusich <at> gmail.com>, 35155 <at> debbugs.gnu.org
Subject: Re: [bug#35155] [PATCH] build-system/cargo: refactor phases to
 successfully build
Date: Sat, 6 Apr 2019 09:12:37 -0700

Hi Danny,

> On Apr 6, 2019, at 3:41 AM, Danny Milosavljevic <dannym <at> scratchpost.org> wrote:
> 
>> +    (generate-checksums rsrc "/dev/null")
> 
> Hmm, is the "package" entry in the final file ".cargo-checksums.json" useless?
> Can it be left off completely?

Yes, it appears that cargo expects there to be a “package” hash definition in
.cargo-checksum.json, but it doesn’t verify the contents (it only checks if they’ve
changed since the last time it updated its Cargo.lock file).

Since this procedure is directly used when building rust-1.20, I didn’t want to
change its definition and get blocked on bootstrapping all of rustc again.
Though I agree, we should update the procedure at some point.

—Ivan

Message #17 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 35155 <at> debbugs.gnu.org
Subject: Re: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Sat, 06 Apr 2019 16:27:14 -0700

[Message part 1 (text/plain, inline)]

Hi Ivan!

Ivan Petkov <ivanppetkov <at> gmail.com> writes:

> This patch refactors the cargo-build-system builder code to correctly build
> imported Rust crates.

This is great!  Thank you for taking the time to work on it.  I have
only some high level questions and minor comments.  It seems fine to me
in general.

Since no packages currently use the cargo-build-system, I think we can
basically just commit this on master and keep moving forward.  Danny, do
you agree?

> * Do not rely on a Cargo.lock presence to determine installation. Cargo will
> automatically create the file at its first invocation, so instead we check
> the manifest for any executable targets before attempting an installation

Sounds good.

> * Do not attempt to modify the Cargo.toml file. There are many system
> specific package in crates.io (e.g. for Windows, Redox, Fuschia, WASM,
> etc.) and attempting to keep up with what crates must be patched out
> is futile.

That seems reasonable, too.

> * The build phases will honor a skip-build? flag which allows for
> short-circuiting for optional packages which require nightly features
> or cannot be built for the current platform.

Can you elaborate on the motivation for this?  Are there situations in
which we need to build an optional package, but we aren't actually going
to use its build output anywhere?

If I'm understanding this correctly, it seems like this new flag would
allow us to write a package definition that doesn't actually build a
package.  If that's the case, I'm not sure why we would bother writing
the package definition in the first place.

> Changes which still need to be done:
> * Update the host-side code to expand transitive inputs: cargo requires that
> all transitive crate dependencies are present in its (vendored) index, but
> doing so by hand in the package definitions will become unwieldy.
> * Update the host-side code to detect any "circular" dependencies which can
> result from a naive import

I agree with that plan.  If I've been following along correctly, once we
figure out how to properly make all the transitive crate dependencies
(in source form) available in the build environment (and resolve the
circular dependencies), it should open the door to importing many Rust
packages.

> * guix/build-system/cargo.scm (%cargo-build-system-modules):
> Add (json parser)

Nitpick: You're missing a period here, and in a few more sentences in
the ChangeLog entry.

> Add #:cargo-tset-flags and pass it to cargo

Nitpick: The word "tset" should be "test".

> (install): Factor source logic to install-source.
> Define #:skip-build? flag and use it.
> Only install if executable targets are present.
> (install-source): Copy entire crate directory not just src.

I'm not as familiar with Rust packaging as you are, so the correctness
of this part is not as clear to me.

My understanding is that a Cargo package that is a library needs to
install its source (so that other Cargo libraries/applications can use
it) but not any executables.  On the other hand, a Cargo package that is
an application needs to install executables (so that a user can run it),
but not its source.  Is that right?  What about Cargo packages that are
both libraries and applications?  Do those even exist?

If you could help me understand (or point me to docs that will help me
understand) the model that Rust/Cargo follows here, it would be helpful.

> +(define* (configure #:key inputs
> +                    (vendor-dir "guix-vendor")
> +                    #:allow-other-keys)
>    "Replace Cargo.toml [dependencies] section with guix inputs."

Is this docstring still accurate after these changes?

> +  ;; Lift restriction on any lints: a crate author may have decided to opt
> +  ;; into stricter lints (e.g. #![deny(warnings)]) during their own builds
> +  ;; but we don't want any build failures that could be caused later by
> +  ;; upgrading the compiler for example.
> +  (setenv "RUSTFLAGS" "--cap-lints allow")

Is this necessary?  The docs seem to suggest that Cargo always sets it
to "allow" anyway:

https://doc.rust-lang.org/rustc/lints/levels.html

"This feature is used heavily by Cargo; it will pass --cap-lints allow
when compiling your dependencies, so that if they have any warnings,
they do not pollute the output of your build."

> @@ -122,22 +131,34 @@ directory = '" port)
>      ;; Until this changes we are working around this by
>      ;; distributing crates as source and replacing
>      ;; references in Cargo.toml with store paths.

Is this comment still accurate?

> -    (generate-checksums rsrc src)
> +    (generate-checksums rsrc "/dev/null")

This probably deserves a short comment to clarify the intent.

Really nice stuff - thank you for sharing it!

-- 
Chris

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 35155 <at> debbugs.gnu.org
Subject: Re: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Sat, 6 Apr 2019 19:02:35 -0700

Hi Chris!

Thank you for the detailed review! Happy to elaborate on any of your questions,
please let me know if you think any of my responses warrant additional code
comments.

> On Apr 6, 2019, at 4:27 PM, Chris Marusich <cmmarusich <at> gmail.com> wrote:
> 
>> * The build phases will honor a skip-build? flag which allows for
>> short-circuiting for optional packages which require nightly features
>> or cannot be built for the current platform.
> 
> Can you elaborate on the motivation for this?  Are there situations in
> which we need to build an optional package, but we aren't actually going
> to use its build output anywhere?

This is meant to be an escape hatch for to skip builds when necessary.
Nothing is setting this flag right now, but eventually the host-side code
may need to set this for certain situations.

> If I'm understanding this correctly, it seems like this new flag would
> allow us to write a package definition that doesn't actually build a
> package.  If that's the case, I'm not sure why we would bother writing
> the package definition in the first place.

That’s an accurate observation. Some context:

Cargo requires that all possible transitive dependencies are present in
its index/vendor directory. This is so it can deterministically build Cargo.lock
files independently of the current platform or what conditional features are
enabled.

There are several ways to package crates within guix with respect to dependent
crates:

a) Don’t pull in optional dependencies, or those for unsupported systems,
patch out the Cargo.toml file so they’re outright ignored by cargo
b) Create stubs for any unsupported/unpacked crates, basically a Cargo.toml
definition with the expected crate name/version and no code
c) Package all crates depended upon by a crate we’re interested in,
possibly annotating it in ways that it isn’t actually built by the CI (e.g. system
specific packages which there is no CI support for).

As mentioned in my earlier email, I believe option a to be a non-starter
since it will be a never-ending uphill battle. I personally believe option c
is the best long term approach (we may reserve option b for dire situations).
This way if guix is ever ported to other systems (e.g. Linux subsystem for Windows)
packages can still be used by consumers without them having to backfill
half the crates ecosystem.

Given this information, here’s how I anticipate we’ll want to skip doing actual
package builds:

* If the package is annotated as for a specific platform (e.g. Windows/Redox/Fuscia)
the host-side build code can populate (#:skip-build? #t) so it doesn’t fail during CI
builds but its still accessible for consuming crates.
* If we have a “circular” dependency as part of dev-dependencies (e.g. one crate
pulls in an upstream crate during testing to ensure it doesn’t break anything)
we’ll need to detangle the dependency graph by rewriting duplicate packages
to include the #:skip-build? flag. I can elaborate more on this in a separate email.

>> Changes which still need to be done:
>> * Update the host-side code to expand transitive inputs: cargo requires that
>> all transitive crate dependencies are present in its (vendored) index, but
>> doing so by hand in the package definitions will become unwieldy.
>> * Update the host-side code to detect any "circular" dependencies which can
>> result from a naive import
> 
> I agree with that plan.  If I've been following along correctly, once we
> figure out how to properly make all the transitive crate dependencies
> (in source form) available in the build environment (and resolve the
> circular dependencies), it should open the door to importing many Rust
> packages.

Yes, precisely, it will be much easier to spot an resolve any other bugs
or feature gaps once that’s in place.

> 
>> * guix/build-system/cargo.scm (%cargo-build-system-modules):
>> Add (json parser)
> 
> Nitpick: You're missing a period here, and in a few more sentences in
> the ChangeLog entry.

Happy to update. After skimming the contributing guidelines I was left
with the impression that commit messages are to only include the concrete
changes that were made, and any additional elaboration should be made
in code comments.

What’s the right process for fixing this? Just send an updated patch to this thread?

> 
>> Add #:cargo-tset-flags and pass it to cargo
> 
> Nitpick: The word "tset" should be "test”.

Whoops, that’s what I get for writing commit messages late at night :)

> 
>> (install): Factor source logic to install-source.
>> Define #:skip-build? flag and use it.
>> Only install if executable targets are present.
>> (install-source): Copy entire crate directory not just src.
> 
> I'm not as familiar with Rust packaging as you are, so the correctness
> of this part is not as clear to me.
> 
> My understanding is that a Cargo package that is a library needs to
> install its source (so that other Cargo libraries/applications can use
> it) but not any executables.  On the other hand, a Cargo package that is
> an application needs to install executables (so that a user can run it),
> but not its source.  Is that right?

You are correct. When building a crate, cargo needs access to the source
of all transitive dependencies, but it’s no longer needed after the build.

The reason we now copy the entire crate directory (rather than just the src
directory) is that some crates have build scripts which usually live outside
of `src` and are needed to successfully build. Although the Cargo.toml
has the path to the root build script, there are crates (like serde) which
have auxiliary build-script modules which aren’t shown in the manifest
contents. Rather than muck around and try to guess where the build script
code is, we can copy it all and let cargo sort it out.

> What about Cargo packages that are
> both libraries and applications?  Do those even exist?

Yes these are a bit rare but they do exist. I don’t have any examples on
hand, but you can have something akin to curl which can be used
as a binary, as well as imported as a library to other projects.

> 
>> +(define* (configure #:key inputs
>> +                    (vendor-dir "guix-vendor")
>> +                    #:allow-other-keys)
>>   "Replace Cargo.toml [dependencies] section with guix inputs."
> 
> Is this docstring still accurate after these changes?

I think the intent is still accurate, though I’ll tweak this to note vendoring
dependencies instead of updating the Cargo.toml.
> 
>> +  ;; Lift restriction on any lints: a crate author may have decided to opt
>> +  ;; into stricter lints (e.g. #![deny(warnings)]) during their own builds
>> +  ;; but we don't want any build failures that could be caused later by
>> +  ;; upgrading the compiler for example.
>> +  (setenv "RUSTFLAGS" "--cap-lints allow")
> 
> Is this necessary?  The docs seem to suggest that Cargo always sets it
> to "allow" anyway:
> 
> https://doc.rust-lang.org/rustc/lints/levels.html
> 
> "This feature is used heavily by Cargo; it will pass --cap-lints allow
> when compiling your dependencies, so that if they have any warnings,
> they do not pollute the output of your build.”

It’s true that cargo applies this to dependencies, but it doesn’t do this
for the top level package that’s currently being built (e.g. if the CI is
building some library crate in isolation). As Guix maintainers, we
wouldn’t want jobs to start failing because of a new lint cropping up
somewhere in between versions.

> 
>> @@ -122,22 +131,34 @@ directory = '" port)
>>     ;; Until this changes we are working around this by
>>     ;; distributing crates as source and replacing
>>     ;; references in Cargo.toml with store paths.
> 
> Is this comment still accurate?

I think this is still accurate (modulo Cargo.toml/vendoring word choice
which I can tweak).

> 
>> -    (generate-checksums rsrc src)
>> +    (generate-checksums rsrc "/dev/null")
> 
> This probably deserves a short comment to clarify the intent.

Do you mean commenting on the intent of `generate-checksums`
or the intent of the /dev/null parameter?

—Ivan

Message #23 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 35155 <at> debbugs.gnu.org
Subject: Re: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Sat, 6 Apr 2019 20:05:09 -0700

[Message part 1 (text/plain, inline)]

Hi Chris!

> On Apr 6, 2019, at 7:02 PM, Ivan Petkov <ivanppetkov <at> gmail.com> wrote:
> 
>> 
>>> * guix/build-system/cargo.scm (%cargo-build-system-modules):
>>> Add (json parser)
>> 
>> Nitpick: You're missing a period here, and in a few more sentences in
>> the ChangeLog entry.
> 
> Happy to update. After skimming the contributing guidelines I was left
> with the impression that commit messages are to only include the concrete
> changes that were made, and any additional elaboration should be made
> in code comments.

Please disregard this comment, I had misread your original email.

I’ve attached an updated version of my original patch.

—Ivan

[0001-build-system-cargo-refactor-phases-to-successfully-b.patch (application/octet-stream, attachment)]

Message #26 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ivan Petkov <ivanppetkov <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 35155 <at> debbugs.gnu.org
Subject: Re: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Sun, 07 Apr 2019 01:40:27 -0700

[Message part 1 (text/plain, inline)]

Hi Ivan,

Ivan Petkov <ivanppetkov <at> gmail.com> writes:

>> On Apr 6, 2019, at 4:27 PM, Chris Marusich <cmmarusich <at> gmail.com> wrote:
>> 
>>> * The build phases will honor a skip-build? flag which allows for
>>> short-circuiting for optional packages which require nightly features
>>> or cannot be built for the current platform.
>> 
>> Can you elaborate on the motivation for this?  Are there situations in
>> which we need to build an optional package, but we aren't actually going
>> to use its build output anywhere?
>
> This is meant to be an escape hatch for to skip builds when necessary.
> Nothing is setting this flag right now, but eventually the host-side code
> may need to set this for certain situations.

Understood.

>> If I'm understanding this correctly, it seems like this new flag would
>> allow us to write a package definition that doesn't actually build a
>> package.  If that's the case, I'm not sure why we would bother writing
>> the package definition in the first place.
>
> That’s an accurate observation. Some context:
>
> Cargo requires that all possible transitive dependencies are present in
> its index/vendor directory. This is so it can deterministically build Cargo.lock
> files independently of the current platform or what conditional features are
> enabled.

Do you mean that if a crate X has an optional feature that requires
crate Y, then Cargo requires Y to be present when building X even if X
is being built with that feature disabled?

> * If we have a “circular” dependency as part of dev-dependencies
> (e.g. one crate pulls in an upstream crate during testing to ensure it
> doesn’t break anything) we’ll need to detangle the dependency graph by
> rewriting duplicate packages to include the #:skip-build? flag. I can
> elaborate more on this in a separate email.

I think here, you're talking about the situation in which crate A
depends on crate B, which in turn depends on crate A's source, and to
break the cycle we will replace B's dependency on A with a dependency on
A', where A' is effectively just a source build of A (via #:skip-build?
#t).  Is that basically right?

I agree it would be good to discuss the resolution of circular
dependencies in another email thread, but I just wanted to double check
with you that my basic understanding of your intent is correct.

>>> Add #:cargo-tset-flags and pass it to cargo
>> 
>> Nitpick: The word "tset" should be "test”.
>
> Whoops, that’s what I get for writing commit messages late at night :)

It happens to the best of us!  :-)

> The reason we now copy the entire crate directory (rather than just the src
> directory) is that some crates have build scripts which usually live outside
> of `src` and are needed to successfully build. Although the Cargo.toml
> has the path to the root build script, there are crates (like serde) which
> have auxiliary build-script modules which aren’t shown in the manifest
> contents. Rather than muck around and try to guess where the build script
> code is, we can copy it all and let cargo sort it out.

OK, that makes sense.  I wasn't exactly sure why we changed this part,
but now it makes sense.  Thank you for explaining it!

>> What about Cargo packages that are
>> both libraries and applications?  Do those even exist?
>
> Yes these are a bit rare but they do exist. I don’t have any examples on
> hand, but you can have something akin to curl which can be used
> as a binary, as well as imported as a library to other projects.

In those rare cases, your changes are still good to go, right?  We don't
actually interact with the Cargo.lock file, and if there are any
executables, your code will install them.

Something else occurred to me.  In packages of C libraries, such as the
glibc package, we install libraries (e.g., ".so" files go into the "out"
output, and ".a" files go into "static" output).  However, here we are
not installing any libraries like that.  Do all Rust developers just use
Cargo.toml files to declare their dependencies, and then let Cargo
manage all the actual compiling and linking?  Do they ever manually
install a Rust library (without using Cargo) when hacking on a project?

I may be way off base here, since I'm not (yet!) an expert Rustacean.
If I'm confused, please help me to understand.

>>> +  ;; Lift restriction on any lints: a crate author may have decided to opt
>>> +  ;; into stricter lints (e.g. #![deny(warnings)]) during their own builds
>>> +  ;; but we don't want any build failures that could be caused later by
>>> +  ;; upgrading the compiler for example.
>>> +  (setenv "RUSTFLAGS" "--cap-lints allow")
>> 
>> Is this necessary?  The docs seem to suggest that Cargo always sets it
>> to "allow" anyway:
>> 
>> https://doc.rust-lang.org/rustc/lints/levels.html
>> 
>> "This feature is used heavily by Cargo; it will pass --cap-lints allow
>> when compiling your dependencies, so that if they have any warnings,
>> they do not pollute the output of your build.”
>
> It’s true that cargo applies this to dependencies, but it doesn’t do this
> for the top level package that’s currently being built (e.g. if the CI is
> building some library crate in isolation). As Guix maintainers, we
> wouldn’t want jobs to start failing because of a new lint cropping up
> somewhere in between versions.

Ah, I see.  Then yes, I agree: we should set it.

>>> -    (generate-checksums rsrc src)
>>> +    (generate-checksums rsrc "/dev/null")
>> 
>> This probably deserves a short comment to clarify the intent.
>
> Do you mean commenting on the intent of `generate-checksums`
> or the intent of the /dev/null parameter?

I mean the "/dev/null" argument, mainly.  As far as I can tell, it looks
like the generate-checksums procedure builds a file with checksums to
satisfy some requirement of the cargo tool.  That seems reasonable, but
I'm not sure why we use "/dev/null" here.

Finally, two more minor comments about code style which I don't think
you need to change but are good to know going forward:

- Instead of "system*", we prefer to use "invoke" (defined in (guix
  build utils)) whenever possible.  It throws an exception when an error
  occurs, so it's harder to accidentally ignore errors.

- Using "and" and "or" statements for control flow is OK, especially
  since you're using "system*".  In fact, we do this in other parts of
  Guix, too.  However, I personally feel that forms like "if", "when",
  and "unless" are clearer in some cases and should probably be
  preferred, especially when using "invoke" instead of "system*".

That said, I think it's fine as is.  Unless Danny (or someone else) has
some more comments, I'll merge your changes in the next few days.

-- 
Chris

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 35155 <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 35155 <at> debbugs.gnu.org
Subject: Re: [PATCH] build-system/cargo: refactor phases to successfully build
Date: Sun, 7 Apr 2019 08:49:09 -0700

[Message part 1 (text/plain, inline)]

Hi Chris,

> On Apr 7, 2019, at 1:40 AM, Chris Marusich <cmmarusich <at> gmail.com> wrote:
> 
> Do you mean that if a crate X has an optional feature that requires
> crate Y, then Cargo requires Y to be present when building X even if X
> is being built with that feature disabled?

Correct, the source to crate Y must be present when crate X is being built,
transitively, or otherwise.

>> * If we have a “circular” dependency as part of dev-dependencies
>> (e.g. one crate pulls in an upstream crate during testing to ensure it
>> doesn’t break anything) we’ll need to detangle the dependency graph by
>> rewriting duplicate packages to include the #:skip-build? flag. I can
>> elaborate more on this in a separate email.
> 
> I think here, you're talking about the situation in which crate A
> depends on crate B, which in turn depends on crate A's source, and to
> break the cycle we will replace B's dependency on A with a dependency on
> A', where A' is effectively just a source build of A (via #:skip-build?
> #t).  Is that basically right?
> 
> I agree it would be good to discuss the resolution of circular
> dependencies in another email thread, but I just wanted to double check
> with you that my basic understanding of your intent is correct.

Yep, your understanding of the situation and potential solution is correct.
I’ll send out a different email thread around this since we might need some
small tweaks to the overall build system to support this.

>>> What about Cargo packages that are
>>> both libraries and applications?  Do those even exist?
>> 
>> Yes these are a bit rare but they do exist. I don’t have any examples on
>> hand, but you can have something akin to curl which can be used
>> as a binary, as well as imported as a library to other projects.
> 
> In those rare cases, your changes are still good to go, right?  We don't
> actually interact with the Cargo.lock file, and if there are any
> executables, your code will install them.

Yep, there’s no harm in building the “src” output of an application crate
(outside of using up some store space) in case something else wants to
depend on it.

> Something else occurred to me.  In packages of C libraries, such as the
> glibc package, we install libraries (e.g., ".so" files go into the "out"
> output, and ".a" files go into "static" output).  However, here we are
> not installing any libraries like that.  Do all Rust developers just use
> Cargo.toml files to declare their dependencies, and then let Cargo
> manage all the actual compiling and linking?  Do they ever manually
> install a Rust library (without using Cargo) when hacking on a project?

In general, cargo is used to perform all building an linking of the final rust
outputs (binaries, shared libraries, and other rust artifacts). Cargo also supports
defining arbitrary build scripts such as building some other non-rust dependency
which is to be linked in the final outputs.

https://doc.rust-lang.org/cargo/reference/build-scripts.html <https://doc.rust-lang.org/cargo/reference/build-scripts.html>

However, cargo does not perform the actual distribution of these “external”
dependencies. Crates may vendor their own external sources, or they may
expect them to be present in the usual places, or they may support an environment
variable which points them in the right direction.

One example is the `jemalloc-sys` crate. If built with an environment variable
pointing to a compiled version of jemalloc, it will build rust bindings which link
to that binary. Otherwise it will build its own vendored version of the jemalloc source.

Handling these packages will need to be done on a case-by-case basis
with some additional setup glue in the package definitions, as necessary.

>>>> -    (generate-checksums rsrc src)
>>>> +    (generate-checksums rsrc "/dev/null")
>>> 
>>> This probably deserves a short comment to clarify the intent.
>> 
>> Do you mean commenting on the intent of `generate-checksums`
>> or the intent of the /dev/null parameter?
> 
> I mean the "/dev/null" argument, mainly.  As far as I can tell, it looks
> like the generate-checksums procedure builds a file with checksums to
> satisfy some requirement of the cargo tool.  That seems reasonable, but
> I'm not sure why we use "/dev/null" here.

Cargo expects the checksum package to include a checksum of each individual
file as well as a checksum for the entire directory. The `generate-checksums`
procedure doesn’t correctly handle the second parameter being a directory
and raises an error. Luckily, cargo doesn’t seem to care about the contents
of this checksum, as long as the declaration is there.

I didn’t want to change the `generate-checksums` procedure just yet since
it’s used during building of rust-1.20, and doing so will require a full bootstrap
of the compiler chain, which would have blocked me for a while.

> Finally, two more minor comments about code style which I don't think
> you need to change but are good to know going forward:
> 
> - Instead of "system*", we prefer to use "invoke" (defined in (guix
>  build utils)) whenever possible.  It throws an exception when an error
>  occurs, so it's harder to accidentally ignore errors.
> 
> - Using "and" and "or" statements for control flow is OK, especially
>  since you're using "system*".  In fact, we do this in other parts of
>  Guix, too.  However, I personally feel that forms like "if", "when",
>  and "unless" are clearer in some cases and should probably be
>  preferred, especially when using "invoke" instead of "system*”.

Thank you for the tips, I’ll keep these in mind going forward.
Still pretty new to guile, its APIs, and guix’s utilities on top of that! :)

—Ivan

[Message part 2 (text/html, inline)]

Message #34 received at 35155-done <at> debbugs.gnu.org (full text, mbox):

From: Ivan Petkov <ivanppetkov <at> gmail.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 35155-done <at> debbugs.gnu.org
Subject: Re: [bug#35155] [PATCH] build-system/cargo: refactor phases to
 successfully build
Date: Tue, 9 Apr 2019 09:01:20 -0700

Thank you Chris!

—Ivan

> On Apr 9, 2019, at 3:12 AM, Chris Marusich <cmmarusich <at> gmail.com> wrote:
> 
> Hi Ivan,
> 
> I've committed this as 1d3acde5087d50af6a4901fd7614f0940eb7b41d on
> master.  Thank you!
> 
> -- 
> Chris

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.