GNU bug report logs - #45104
pull: Add a "with-substitutes" option.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45104 in the body.
You can then email your comments to 45104 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: guix-patches <at> gnu.org
Subject: pull: Add a "with-substitutes" option.
Date: Mon, 07 Dec 2020 16:39:59 +0100

[Message part 1 (text/plain, inline)]

Hello,

When "guix pull" is run before the CI server is done baking substitutes,
the user may need to build some derivations, which can be long on
low-end hardware.

This patch adds an option to "guix pull" so that it updates to the
latest commit with available substitutes.

This work is still in progress, but I'd like to gather some impressions
on that before going further.

Thanks,

Mathieu

[0001-scripts-pull-Add-with-substitutes-option.patch (text/x-diff, inline)]

From d399f8dbb9e38a82241b9048b8b04758fae10005 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe <at> gnu.org>
Date: Mon, 7 Dec 2020 16:12:22 +0100
Subject: [PATCH] scripts: pull: Add "with-substitutes" option.

---
 guix/scripts/pull.scm | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 83cdc1d1eb..4609f8614e 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2013, 2014, 2015, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
 ;;; Copyright © 2020 Tobias Geerinckx-Rice <me <at> tobias.gr>
+;;; Copyright © 2020 Mathieu Othacehe <othacehe <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
 
 (define-module (guix scripts pull)
   #:use-module ((guix ui) #:hide (display-profile-content))
+  #:use-module (guix ci)
   #:use-module (guix colors)
   #:use-module (guix utils)
   #:use-module ((guix status) #:select (with-status-verbosity))
@@ -64,6 +66,7 @@
   #:re-export (display-profile-content
                channel-commit-hyperlink)
   #:export (channel-list
+            find-lastest-commit-with-substitutes
             guix-pull))
 
 
@@ -169,6 +172,9 @@ Download and deploy the latest version of Guix.\n"))
                  (lambda (opt name arg result)
                    (alist-cons 'validate-pull warn-about-backward-updates
                                result)))
+         (option '("with-substitutes") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'with-substitutes? #t result)))
          (option '("disable-authentication") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'authenticate-channels? #f result)))
@@ -526,6 +532,26 @@ true, display what would be built without actually building it."
           (leave (G_ "while creating symlink '~a': ~a~%")
                  link (strerror (system-error-errno args))))))))
 
+
+;;;
+;;; Substitutes.
+;;;
+
+(define (find-lastest-commit-with-substitutes)
+  (let ((urls %default-substitute-urls))
+    (any (lambda (url)
+           (let* ((build (match (latest-builds url 1
+                                               #:job "guix.x86_64-linux"
+                                               #:status 0) ;success
+                           ((build) build)))
+                  (evaluation (evaluation url
+                                          (build-evaluation build)))
+                  (commit (match (evaluation-checkouts evaluation)
+                            ((checkout)
+                             (checkout-commit checkout)))))
+             commit))
+         urls)))
+
 
 ;;;
 ;;; Queries.
@@ -731,8 +757,9 @@ Use '~/.config/guix/channels.scm' instead."))
 
   (let ((ref (assoc-ref opts 'ref))
         (url (or (assoc-ref opts 'repository-url)
-                 (environment-variable))))
-    (if (or ref url)
+                 (environment-variable)))
+        (with-substitutes? (assoc-ref opts 'with-substitutes?)))
+    (if (or ref url with-substitutes?)
         (match (find guix-channel? channels)
           ((? channel? guix)
            ;; Apply '--url', '--commit', and '--branch' to the 'guix' channel.
@@ -745,7 +772,12 @@ Use '~/.config/guix/channels.scm' instead."))
                       (channel (inherit guix)
                                (url url) (commit #f) (branch branch)))
                      (#f
-                      (channel (inherit guix) (url url))))
+                      (let ((commit
+                             (and with-substitutes?
+                                  (find-lastest-commit-with-substitutes))))
+                        (channel (inherit guix)
+                                 (url url)
+                                 (commit commit)))))
                    (remove guix-channel? channels))))
           (#f                           ;no 'guix' channel, failure will ensue
            channels))
-- 
2.29.2

Message #8 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Mathieu Othacehe <othacehe <at> gnu.org>, 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Mon, 07 Dec 2020 18:05:16 +0100

Hi Mathieu,

On Mon, 07 Dec 2020 at 16:39, Mathieu Othacehe <othacehe <at> gnu.org> wrote:

> When "guix pull" is run before the CI server is done baking substitutes,
> the user may need to build some derivations, which can be long on
> low-end hardware.

Yeah…


> This patch adds an option to "guix pull" so that it updates to the
> latest commit with available substitutes.

Cool!  It improves my ugly bash to pull 2 weeks behind. :-)


> This work is still in progress, but I'd like to gather some impressions
> on that before going further.

With your patch, it is possible to never pull something.  Because the
push are faster than the CI is able to build.

Therefore, an improvement could to be to check the latest commit on the
CI, if not available fallback to the previous, repeat until one is
available or repeat X time and raise a warning.


Cheers,
simon

Message #11 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Tue, 08 Dec 2020 19:17:47 +0000

[Message part 1 (text/plain, inline)]

Mathieu Othacehe <othacehe <at> gnu.org> writes:

> When "guix pull" is run before the CI server is done baking substitutes,
> the user may need to build some derivations, which can be long on
> low-end hardware.
>
> This patch adds an option to "guix pull" so that it updates to the
> latest commit with available substitutes.
>
> This work is still in progress, but I'd like to gather some impressions
> on that before going further.

Hey!

I think it's definitely a nice idea, although I think there are some
things in the naming.

My first thought is that the option doesn't really do what it says it
will do. The option is named --with-substitutes, but that actually means
something like "with successful build on ci.guix.gnu.org". This could
cause confusion if you're not fetching substitutes from ci.guix.gnu.org,
and it starts building things, or perhaps if you're fetching substitutes
from two servers, one of which is ci.guix.gnu.org which doesn't have a
substitute, but the other server does, this option would fetch an older
revision than what's truly available.

Maybe part of what I've said above is incorrect if a users substitute
URLs are used, but there also seems to be an assuption that something
which provides substitutes provides a Cuirass compatible API, which
isn't always the case.

One more thought on this is that it's probably important to convey that
this doesn't pick a revision where substitutes will be available. I
think this is something some users would be eager to assume, and then be
confused when it doesn't work.

Thanks,

Chris

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: Mathieu Othacehe <othacehe <at> gnu.org>, 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Mon, 14 Dec 2020 12:05:54 +0100

Hi!

Christopher Baines <mail <at> cbaines.net> skribis:

> My first thought is that the option doesn't really do what it says it
> will do. The option is named --with-substitutes, but that actually means
> something like "with successful build on ci.guix.gnu.org". This could
> cause confusion if you're not fetching substitutes from ci.guix.gnu.org,
> and it starts building things, or perhaps if you're fetching substitutes
> from two servers, one of which is ci.guix.gnu.org which doesn't have a
> substitute, but the other server does, this option would fetch an older
> revision than what's truly available.
>
> Maybe part of what I've said above is incorrect if a users substitute
> URLs are used, but there also seems to be an assuption that something
> which provides substitutes provides a Cuirass compatible API, which
> isn't always the case.
>
> One more thought on this is that it's probably important to convey that
> this doesn't pick a revision where substitutes will be available. I
> think this is something some users would be eager to assume, and then be
> confused when it doesn't work.

Agreed on these points.

Another option would be to leave it up to users to customize their
channel file to run pretty much the code you posted, like:

--8<---------------cut here---------------start------------->8---
(use-modules (guix ci)
             (srfi srfi-1)
             (ice-9 match))

(define (latest-commit-successfully-built)
  "Return the latest commit for which substitutes are (potentially)
available."
  (let* ((evaluations (filter (lambda (evaluation)
                                (and (evaluation-complete? evaluation)
                                     (string=? "guix-modular-master"
                                               (evaluation-spec
                                                evaluation))))
                              (latest-evaluations "https://ci.guix.gnu.org"))))
    (any (lambda (evaluation)
           (match (evaluation-checkouts evaluation)
             ((checkout)
              (checkout-commit checkout))
             (_ #f)))
         evaluations)))

;; Pull the latest commit fully built on berlin.guixsd.org.
(list (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (commit (pk 'commit (latest-commit-successfully-built)))))
--8<---------------cut here---------------end--------------->8---

We don’t want people to copy/paste all this, so we could instead provide
and document a procedure that takes a channel and substitute URL and
returns a channel, like:

  (channel-with-substitutes-available
    (channel (name 'guix) …)
    "https://ci.guix.gnu.org")

and optionally with a manifest or a list of packages that should be
available as substitutes:

  (channel-with-substitutes-available
    (channel (name 'guix) …)
    "https://ci.guix.gnu.org"
    (specifications->manifest '("emacs" "guile")))

WDYT?

It does mean that we’re asking users to do extra work.  Perhaps there
could still be a command-line option that would call
‘channel-with-substitutes-available’ for you, but at least it would take
an explicit URL and clarify what Chris mentioned?

BTW, doing all this is safer today because ‘guix pull’ will detect and
prevent downgrades.  Though an attacker who manages to break into
ci.guix.gnu.org could cause all the users of
‘channel-with-substitutes-available’ to no longer receive updates or to
receive them more slowly than they appear in Git simply by making CI
even slower than it currently is.

Thoughts?

Ludo’.

Message #17 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, Christopher Baines
 <mail <at> cbaines.net>
Cc: Mathieu Othacehe <othacehe <at> gnu.org>, 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Mon, 14 Dec 2020 12:39:21 +0100

Hi,

On Mon, 14 Dec 2020 at 12:05, Ludovic Courtès <ludo <at> gnu.org> wrote:

>   (channel-with-substitutes-available
>     (channel (name 'guix) …)
>     "https://ci.guix.gnu.org")
>
> and optionally with a manifest or a list of packages that should be
> available as substitutes:
>
>   (channel-with-substitutes-available
>     (channel (name 'guix) …)
>     "https://ci.guix.gnu.org"
>     (specifications->manifest '("emacs" "guile")))

Sounds good to me.


> BTW, doing all this is safer today because ‘guix pull’ will detect and
> prevent downgrades.  Though an attacker who manages to break into
> ci.guix.gnu.org could cause all the users of
> ‘channel-with-substitutes-available’ to no longer receive updates or to
> receive them more slowly than they appear in Git simply by making CI
> even slower than it currently is.

As mentioned earlier, if “guix pull” completes only when substitutes is
available, then depending on the CI, the user can never complete the
“pull” and so stay “blocked“.

For example, reusing your example, Emacs is not currently available and
it has been since say 1 month.  Therefore until Emacs becomes available
for the fast moving HEAD, the user would be “blocked”.  Or I miss
something.

<https://data.guix.gnu.org/repository/1/branch/master/package/emacs/output-history>

All the best,
simon

Message #20 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Tue, 15 Dec 2020 11:24:55 +0100

Hey Chris and Ludo,

> Agreed on these points.

Yes I think you are definitely right on that point.

>                                 (and (evaluation-complete? evaluation)
>                                      (string=? "guix-modular-master"
>                                                (evaluation-spec
>                                                 evaluation))))

On Berlin, evaluations can be completed for days, but the associated
builds never started. I think that searching directly for a completed
build provides a stronger guarantee of available substitutes.

> ;; Pull the latest commit fully built on berlin.guixsd.org.
> (list (channel
>        (name 'guix)
>        (url "https://git.savannah.gnu.org/git/guix.git")
>        (commit (pk 'commit (latest-commit-successfully-built)))))

Providing such a procedure definitely makes sense though.

>   (channel-with-substitutes-available
>     (channel (name 'guix) …)
>     "https://ci.guix.gnu.org"
>     (specifications->manifest '("emacs" "guile")))

Yes it would be the ultimate thing! However, while finding the latest
commit with an available substitute for a derivation is quite easy,
finding a commit with available derivations for N derivations seems way
more difficult.

> It does mean that we’re asking users to do extra work.  Perhaps there
> could still be a command-line option that would call
> ‘channel-with-substitutes-available’ for you, but at least it would take
> an explicit URL and clarify what Chris mentioned?

Yes, the user would then have to provide the channels that need
available substitutes, the URL to use for the substitution check and
maybe a manifest that also needs available substitutes.

The channels list could default to '("guix") and the URL to
"https://ci.guix.gnu.org" as it would be a sensible default for most
Guix users I think.

> BTW, doing all this is safer today because ‘guix pull’ will detect and
> prevent downgrades.  Though an attacker who manages to break into
> ci.guix.gnu.org could cause all the users of
> ‘channel-with-substitutes-available’ to no longer receive updates or to
> receive them more slowly than they appear in Git simply by making CI
> even slower than it currently is.

Yes, the downgrade check definitely helps here, as it's often what will
happen with our lagging CI. Regarding the security aspect, I think that
breaking into ci.guix.gnu.org can have other way more impacting
consequences.

Thanks,

Mathieu

Message #23 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 45104 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>,
 Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Tue, 15 Dec 2020 11:30:12 +0100

Hello zimoun,

> As mentioned earlier, if “guix pull” completes only when substitutes is
> available, then depending on the CI, the user can never complete the
> “pull” and so stay “blocked“.

There's nothing blocking in what I'm proposing. "guix pull" first asks
to the CI what's the latest commit with available substitutes for
"guix", and then tries to update to that commit. If it results in a
downgrade then, "guix pull" detects it.

The corner case where there are no commits with available substitutes is
not handled, but I guess we could decide to update to the latest commit
in that case.

Thanks,

Mathieu

Message #26 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>,
 Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Tue, 15 Dec 2020 13:51:17 +0100

Hi Mathieu,

On Tue, 15 Dec 2020 at 11:30, Mathieu Othacehe <othacehe <at> gnu.org> wrote:

> There's nothing blocking in what I'm proposing. "guix pull" first asks
> to the CI what's the latest commit with available substitutes for
> "guix", and then tries to update to that commit. If it results in a
> downgrade then, "guix pull" detects it.

Thanks for the explanation.

I have actually overlooked the code. Well, the proposal was what your
patch is doing somehow. :-)

Now, since I have played with “guix repl”, all is clear. ;-)

Aside, the option name ’--with-substitutes’ could be misleading.  Or
clearly state that it is substitutes for “guix”.


Cheers,
simon

Message #29 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Tue, 15 Dec 2020 23:03:45 +0100

Hi,

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

>>                                 (and (evaluation-complete? evaluation)
>>                                      (string=? "guix-modular-master"
>>                                                (evaluation-spec
>>                                                 evaluation))))
>
> On Berlin, evaluations can be completed for days, but the associated
> builds never started. I think that searching directly for a completed
> build provides a stronger guarantee of available substitutes.

Yes, something like you proposed probably makes more sense.

My point is just that we could make the procedure available as part of
the API and document it as something people can use in their channels
file.

>> ;; Pull the latest commit fully built on berlin.guixsd.org.
>> (list (channel
>>        (name 'guix)
>>        (url "https://git.savannah.gnu.org/git/guix.git")
>>        (commit (pk 'commit (latest-commit-successfully-built)))))
>
> Providing such a procedure definitely makes sense though.
>
>>   (channel-with-substitutes-available
>>     (channel (name 'guix) …)
>>     "https://ci.guix.gnu.org"
>>     (specifications->manifest '("emacs" "guile")))
>
> Yes it would be the ultimate thing! However, while finding the latest
> commit with an available substitute for a derivation is quite easy,
> finding a commit with available derivations for N derivations seems way
> more difficult.

Right!

>> It does mean that we’re asking users to do extra work.  Perhaps there
>> could still be a command-line option that would call
>> ‘channel-with-substitutes-available’ for you, but at least it would take
>> an explicit URL and clarify what Chris mentioned?
>
> Yes, the user would then have to provide the channels that need
> available substitutes, the URL to use for the substitution check and
> maybe a manifest that also needs available substitutes.
>
> The channels list could default to '("guix") and the URL to
> "https://ci.guix.gnu.org" as it would be a sensible default for most
> Guix users I think.

Yes, choosing good defaults can make it less intimidating.

>> BTW, doing all this is safer today because ‘guix pull’ will detect and
>> prevent downgrades.  Though an attacker who manages to break into
>> ci.guix.gnu.org could cause all the users of
>> ‘channel-with-substitutes-available’ to no longer receive updates or to
>> receive them more slowly than they appear in Git simply by making CI
>> even slower than it currently is.
>
> Yes, the downgrade check definitely helps here, as it's often what will
> happen with our lagging CI. Regarding the security aspect, I think that
> breaking into ci.guix.gnu.org can have other way more impacting
> consequences.

Yeah, though here we’re opening a new vulnerability channel, independent
of substitutes.  It changes the threat model.

Thanks,
Ludo’.

Message #32 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Fri, 29 Jan 2021 14:23:18 +0100

[Message part 1 (text/plain, inline)]

Hello,

Here's a patch implementing "channel-with-substitutes-available". The
following step will be to add the matching option for the "guix pull"
command, as discussed previously.

Thanks,

Mathieu

[0001-guix-channels-Introduce-channel-with-substitutes-ava.patch (text/x-diff, inline)]

From 8f93ced042f7f694c0c7a473686faa3c00497aa5 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe <at> gnu.org>
Date: Fri, 29 Jan 2021 13:48:44 +0100
Subject: [PATCH] guix: channels: Introduce
 "channel-with-substitutes-available".

* guix/channels.scm (find-latest-commit-with-substitutes,
channel-with-substitutes-available): New procedures.
* guix/scripts/pull.scm (guix-pull): Move "channel-list" call inside the
%current-system parameter scope.
* doc/guix.texi (Channels with substitutes): New section.
---
 doc/guix.texi         |  27 ++++++++++-
 guix/channels.scm     |  29 +++++++++++-
 guix/scripts/pull.scm | 103 +++++++++++++++++++++---------------------
 3 files changed, 106 insertions(+), 53 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index ff9e8da2e0..6587a49d0e 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -40,7 +40,7 @@ Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Julien Lepiller@*
 Copyright @copyright{} 2016 Alex ter Weele@*
 Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@*
 Copyright @copyright{} 2017, 2018, 2019 Clément Lassieur@*
-Copyright @copyright{} 2017, 2018, 2020 Mathieu Othacehe@*
+Copyright @copyright{} 2017, 2018, 2020, 2021 Mathieu Othacehe@*
 Copyright @copyright{} 2017 Federico Beffa@*
 Copyright @copyright{} 2017, 2018 Carlo Zancanaro@*
 Copyright @copyright{} 2017 Thomas Danckaert@*
@@ -245,6 +245,7 @@ Channels
 * Specifying Channel Authorizations::  Defining channel authors authorizations.
 * Primary URL::                 Distinguishing mirror to original.
 * Writing Channel News::        Communicating information to channel's users.
+* Channels with substitutes::   Using channels with available substitutes.
 
 Development
 
@@ -4919,6 +4920,7 @@ updates.
 * Specifying Channel Authorizations::  Defining channel authors authorizations.
 * Primary URL::                 Distinguishing mirror to original.
 * Writing Channel News::        Communicating information to channel's users.
+* Channels with substitutes::   Using channels with available substitutes.
 @end menu
 
 @node Specifying Additional Channels
@@ -5390,6 +5392,29 @@ xgettext -o news.po -l scheme -ken etc/news.txt
 To sum up, yes, you could use your channel as a blog.  But beware, this
 is @emph{not quite} what your users might expect.
 
+@node Channels with substitutes
+@section Channels with substitutes
+
+When running @command{guix pull}, Guix will first compile the
+definitions of every available package.  This is an expensive operation
+for which substitutes (@pxref{Substitutes}) may be available.  The
+following snippet in @file{channels.scm} will ensure that @command{guix
+pull} uses the latest commit with available substitutes for the package
+definitions.  This is done by querying the continuous integration
+server at @url{https://ci.guix.gnu.org}.
+
+Be careful, this does not mean that all the packages that you will
+install after running @command{guix pull} will have available
+substitutes.  It only ensures that @command{guix pull} will not try to
+compile package definitions.  This is particularly useful when using
+machines with limited resources.
+
+@lisp
+(list
+ (channel-with-substitutes-available
+  %default-guix-channel
+  "https://ci.guix.gnu.org"))
+@end lisp
 
 @c *********************************************************************
 @node Development
diff --git a/guix/channels.scm b/guix/channels.scm
index 0c84eed477..817e30d738 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -20,6 +20,7 @@
 
 (define-module (guix channels)
   #:use-module (git)
+  #:use-module (guix ci)
   #:use-module (guix git)
   #:use-module (guix git-authenticate)
   #:use-module ((guix openpgp)
@@ -98,7 +99,9 @@
             channel-news-entry-title
             channel-news-entry-body
 
-            channel-news-for-commit))
+            channel-news-for-commit
+
+            channel-with-substitutes-available))
 
 ;;; Commentary:
 ;;;
@@ -1044,6 +1047,30 @@ NEW.  When OLD is omitted or is #f, return all the news entries of CHANNEL."
           '()
           (apply throw key error rest)))))
 
+(define (find-latest-commit-with-substitutes url)
+  (let* ((job-name (string-append "guix." (%current-system)))
+         (build (match (latest-builds url 1
+                                      #:job job-name
+                                      #:status 0) ;success
+                  ((build) build)
+                  (_ #f)))
+         (evaluation (and build
+                          (evaluation url (build-evaluation build))))
+         (commit (and evaluation
+                      (match (evaluation-checkouts evaluation)
+                        ((checkout)
+                         (checkout-commit checkout))))))
+    commit))
+
+(define (channel-with-substitutes-available chan url)
+  (let ((commit (find-latest-commit-with-substitutes url)))
+    (unless commit
+      (warning (G_ "could not find available substitutes at ~a~%")
+               url))
+    (channel
+     (inherit chan)
+     (commit commit))))
+
 ;;; Local Variables:
 ;;; eval: (put 'with-guile 'scheme-indent-function 1)
 ;;; End:
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 83cdc1d1eb..4e0ab5d341 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -765,60 +765,61 @@ Use '~/.config/guix/channels.scm' instead."))
                                               #:argument-handler no-arguments))
             (substitutes? (assoc-ref opts 'substitutes?))
             (dry-run?     (assoc-ref opts 'dry-run?))
-            (channels     (channel-list opts))
             (profile      (or (assoc-ref opts 'profile) %current-profile))
             (current-channels (profile-channels profile))
             (validate-pull    (assoc-ref opts 'validate-pull))
             (authenticate?    (assoc-ref opts 'authenticate-channels?)))
-       (cond ((assoc-ref opts 'query)
-              (process-query opts profile))
-             ((assoc-ref opts 'generation)
-              (process-generation-change opts profile))
-             (else
-              (with-store store
-                (with-status-verbosity (assoc-ref opts 'verbosity)
-                  (parameterize ((%current-system (assoc-ref opts 'system))
-                                 (%graft? (assoc-ref opts 'graft?)))
-                    (with-build-handler (build-notifier #:use-substitutes?
-                                                        substitutes?
-                                                        #:verbosity
-                                                        (assoc-ref opts 'verbosity)
-                                                        #:dry-run? dry-run?)
-                      (set-build-options-from-command-line store opts)
-                      (ensure-default-profile)
-                      (honor-x509-certificates store)
-
-                      (let ((instances
-                             (latest-channel-instances store channels
-                                                       #:current-channels
-                                                       current-channels
-                                                       #:validate-pull
-                                                       validate-pull
-                                                       #:authenticate?
-                                                       authenticate?)))
-                        (format (current-error-port)
-                                (N_ "Building from this channel:~%"
-                                    "Building from these channels:~%"
-                                    (length instances)))
-                        (for-each (lambda (instance)
-                                    (let ((channel
-                                           (channel-instance-channel instance)))
-                                      (format (current-error-port)
-                                              "  ~10a~a\t~a~%"
-                                              (channel-name channel)
-                                              (channel-url channel)
-                                              (string-take
-                                               (channel-instance-commit instance)
-                                               7))))
-                                  instances)
-                        (parameterize ((%guile-for-build
-                                        (package-derivation
-                                         store
-                                         (if (assoc-ref opts 'bootstrap?)
-                                             %bootstrap-guile
-                                             (default-guile)))))
-                          (with-profile-lock profile
-                            (run-with-store store
-                              (build-and-install instances profile)))))))))))))))
+       (cond
+        ((assoc-ref opts 'query)
+         (process-query opts profile))
+        ((assoc-ref opts 'generation)
+         (process-generation-change opts profile))
+        (else
+         (with-store store
+           (with-status-verbosity (assoc-ref opts 'verbosity)
+             (parameterize ((%current-system (assoc-ref opts 'system))
+                            (%graft? (assoc-ref opts 'graft?)))
+               (with-build-handler (build-notifier #:use-substitutes?
+                                                   substitutes?
+                                                   #:verbosity
+                                                   (assoc-ref opts 'verbosity)
+                                                   #:dry-run? dry-run?)
+                 (set-build-options-from-command-line store opts)
+                 (ensure-default-profile)
+                 (honor-x509-certificates store)
+
+                 (let* ((channels (channel-list opts))
+                        (instances
+                         (latest-channel-instances store channels
+                                                   #:current-channels
+                                                   current-channels
+                                                   #:validate-pull
+                                                   validate-pull
+                                                   #:authenticate?
+                                                   authenticate?)))
+                   (format (current-error-port)
+                           (N_ "Building from this channel:~%"
+                               "Building from these channels:~%"
+                               (length instances)))
+                   (for-each (lambda (instance)
+                               (let ((channel
+                                      (channel-instance-channel instance)))
+                                 (format (current-error-port)
+                                         "  ~10a~a\t~a~%"
+                                         (channel-name channel)
+                                         (channel-url channel)
+                                         (string-take
+                                          (channel-instance-commit instance)
+                                          7))))
+                             instances)
+                   (parameterize ((%guile-for-build
+                                   (package-derivation
+                                    store
+                                    (if (assoc-ref opts 'bootstrap?)
+                                        %bootstrap-guile
+                                        (default-guile)))))
+                     (with-profile-lock profile
+                       (run-with-store store
+                         (build-and-install instances profile)))))))))))))))
 
 ;;; pull.scm ends here
-- 
2.29.2

Message #35 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Fri, 29 Jan 2021 14:36:30 +0100

[Message part 1 (text/plain, inline)]

Here's a v2 with the missing docstrings.

Thanks,

Mathieu

[0001-guix-channels-Introduce-channel-with-substitutes-ava.patch (text/x-diff, inline)]

From 31dad6456825a329ba0f07c95e3e99258d186a8f Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe <at> gnu.org>
Date: Fri, 29 Jan 2021 13:48:44 +0100
Subject: [PATCH v2] guix: channels: Introduce
 "channel-with-substitutes-available".

* guix/channels.scm (find-latest-commit-with-substitutes,
channel-with-substitutes-available): New procedures.
* guix/scripts/pull.scm (guix-pull): Move "channel-list" call inside the
%current-system parameter scope.
* doc/guix.texi (Channels with substitutes): New section.
---
 doc/guix.texi         |  27 ++++++++++-
 guix/channels.scm     |  37 ++++++++++++++-
 guix/scripts/pull.scm | 103 +++++++++++++++++++++---------------------
 3 files changed, 114 insertions(+), 53 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index ff9e8da2e0..6587a49d0e 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -40,7 +40,7 @@ Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Julien Lepiller@*
 Copyright @copyright{} 2016 Alex ter Weele@*
 Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@*
 Copyright @copyright{} 2017, 2018, 2019 Clément Lassieur@*
-Copyright @copyright{} 2017, 2018, 2020 Mathieu Othacehe@*
+Copyright @copyright{} 2017, 2018, 2020, 2021 Mathieu Othacehe@*
 Copyright @copyright{} 2017 Federico Beffa@*
 Copyright @copyright{} 2017, 2018 Carlo Zancanaro@*
 Copyright @copyright{} 2017 Thomas Danckaert@*
@@ -245,6 +245,7 @@ Channels
 * Specifying Channel Authorizations::  Defining channel authors authorizations.
 * Primary URL::                 Distinguishing mirror to original.
 * Writing Channel News::        Communicating information to channel's users.
+* Channels with substitutes::   Using channels with available substitutes.
 
 Development
 
@@ -4919,6 +4920,7 @@ updates.
 * Specifying Channel Authorizations::  Defining channel authors authorizations.
 * Primary URL::                 Distinguishing mirror to original.
 * Writing Channel News::        Communicating information to channel's users.
+* Channels with substitutes::   Using channels with available substitutes.
 @end menu
 
 @node Specifying Additional Channels
@@ -5390,6 +5392,29 @@ xgettext -o news.po -l scheme -ken etc/news.txt
 To sum up, yes, you could use your channel as a blog.  But beware, this
 is @emph{not quite} what your users might expect.
 
+@node Channels with substitutes
+@section Channels with substitutes
+
+When running @command{guix pull}, Guix will first compile the
+definitions of every available package.  This is an expensive operation
+for which substitutes (@pxref{Substitutes}) may be available.  The
+following snippet in @file{channels.scm} will ensure that @command{guix
+pull} uses the latest commit with available substitutes for the package
+definitions.  This is done by querying the continuous integration
+server at @url{https://ci.guix.gnu.org}.
+
+Be careful, this does not mean that all the packages that you will
+install after running @command{guix pull} will have available
+substitutes.  It only ensures that @command{guix pull} will not try to
+compile package definitions.  This is particularly useful when using
+machines with limited resources.
+
+@lisp
+(list
+ (channel-with-substitutes-available
+  %default-guix-channel
+  "https://ci.guix.gnu.org"))
+@end lisp
 
 @c *********************************************************************
 @node Development
diff --git a/guix/channels.scm b/guix/channels.scm
index 0c84eed477..a04eae1c10 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -20,6 +20,7 @@
 
 (define-module (guix channels)
   #:use-module (git)
+  #:use-module (guix ci)
   #:use-module (guix git)
   #:use-module (guix git-authenticate)
   #:use-module ((guix openpgp)
@@ -98,7 +99,9 @@
             channel-news-entry-title
             channel-news-entry-body
 
-            channel-news-for-commit))
+            channel-news-for-commit
+
+            channel-with-substitutes-available))
 
 ;;; Commentary:
 ;;;
@@ -1044,6 +1047,38 @@ NEW.  When OLD is omitted or is #f, return all the news entries of CHANNEL."
           '()
           (apply throw key error rest)))))
 
+(define (find-latest-commit-with-substitutes url)
+  "Return the latest commit with available substitutes for the Guix package
+definitions at URL.  Return false if no commit were found."
+  (let* ((job-name (string-append "guix." (%current-system)))
+         (build (match (latest-builds url 1
+                                      #:job job-name
+                                      #:status 0) ;success
+                  ((build) build)
+                  (_ #f)))
+         (evaluation (and build
+                          (evaluation url (build-evaluation build))))
+         (commit (and evaluation
+                      (match (evaluation-checkouts evaluation)
+                        ((checkout)
+                         (checkout-commit checkout))))))
+    commit))
+
+(define (channel-with-substitutes-available chan url)
+  "Return a channel inheriting from CHAN but which commit field is set to the
+latest commit with available substitutes for the Guix package definitions at
+URL.  The current system is taken into account.
+
+If no commit with available substitutes were found, the commit field is set to
+false and a warning message is printed."
+  (let ((commit (find-latest-commit-with-substitutes url)))
+    (unless commit
+      (warning (G_ "could not find available substitutes at ~a~%")
+               url))
+    (channel
+     (inherit chan)
+     (commit commit))))
+
 ;;; Local Variables:
 ;;; eval: (put 'with-guile 'scheme-indent-function 1)
 ;;; End:
diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 83cdc1d1eb..4e0ab5d341 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -765,60 +765,61 @@ Use '~/.config/guix/channels.scm' instead."))
                                               #:argument-handler no-arguments))
             (substitutes? (assoc-ref opts 'substitutes?))
             (dry-run?     (assoc-ref opts 'dry-run?))
-            (channels     (channel-list opts))
             (profile      (or (assoc-ref opts 'profile) %current-profile))
             (current-channels (profile-channels profile))
             (validate-pull    (assoc-ref opts 'validate-pull))
             (authenticate?    (assoc-ref opts 'authenticate-channels?)))
-       (cond ((assoc-ref opts 'query)
-              (process-query opts profile))
-             ((assoc-ref opts 'generation)
-              (process-generation-change opts profile))
-             (else
-              (with-store store
-                (with-status-verbosity (assoc-ref opts 'verbosity)
-                  (parameterize ((%current-system (assoc-ref opts 'system))
-                                 (%graft? (assoc-ref opts 'graft?)))
-                    (with-build-handler (build-notifier #:use-substitutes?
-                                                        substitutes?
-                                                        #:verbosity
-                                                        (assoc-ref opts 'verbosity)
-                                                        #:dry-run? dry-run?)
-                      (set-build-options-from-command-line store opts)
-                      (ensure-default-profile)
-                      (honor-x509-certificates store)
-
-                      (let ((instances
-                             (latest-channel-instances store channels
-                                                       #:current-channels
-                                                       current-channels
-                                                       #:validate-pull
-                                                       validate-pull
-                                                       #:authenticate?
-                                                       authenticate?)))
-                        (format (current-error-port)
-                                (N_ "Building from this channel:~%"
-                                    "Building from these channels:~%"
-                                    (length instances)))
-                        (for-each (lambda (instance)
-                                    (let ((channel
-                                           (channel-instance-channel instance)))
-                                      (format (current-error-port)
-                                              "  ~10a~a\t~a~%"
-                                              (channel-name channel)
-                                              (channel-url channel)
-                                              (string-take
-                                               (channel-instance-commit instance)
-                                               7))))
-                                  instances)
-                        (parameterize ((%guile-for-build
-                                        (package-derivation
-                                         store
-                                         (if (assoc-ref opts 'bootstrap?)
-                                             %bootstrap-guile
-                                             (default-guile)))))
-                          (with-profile-lock profile
-                            (run-with-store store
-                              (build-and-install instances profile)))))))))))))))
+       (cond
+        ((assoc-ref opts 'query)
+         (process-query opts profile))
+        ((assoc-ref opts 'generation)
+         (process-generation-change opts profile))
+        (else
+         (with-store store
+           (with-status-verbosity (assoc-ref opts 'verbosity)
+             (parameterize ((%current-system (assoc-ref opts 'system))
+                            (%graft? (assoc-ref opts 'graft?)))
+               (with-build-handler (build-notifier #:use-substitutes?
+                                                   substitutes?
+                                                   #:verbosity
+                                                   (assoc-ref opts 'verbosity)
+                                                   #:dry-run? dry-run?)
+                 (set-build-options-from-command-line store opts)
+                 (ensure-default-profile)
+                 (honor-x509-certificates store)
+
+                 (let* ((channels (channel-list opts))
+                        (instances
+                         (latest-channel-instances store channels
+                                                   #:current-channels
+                                                   current-channels
+                                                   #:validate-pull
+                                                   validate-pull
+                                                   #:authenticate?
+                                                   authenticate?)))
+                   (format (current-error-port)
+                           (N_ "Building from this channel:~%"
+                               "Building from these channels:~%"
+                               (length instances)))
+                   (for-each (lambda (instance)
+                               (let ((channel
+                                      (channel-instance-channel instance)))
+                                 (format (current-error-port)
+                                         "  ~10a~a\t~a~%"
+                                         (channel-name channel)
+                                         (channel-url channel)
+                                         (string-take
+                                          (channel-instance-commit instance)
+                                          7))))
+                             instances)
+                   (parameterize ((%guile-for-build
+                                   (package-derivation
+                                    store
+                                    (if (assoc-ref opts 'bootstrap?)
+                                        %bootstrap-guile
+                                        (default-guile)))))
+                     (with-profile-lock profile
+                       (run-with-store store
+                         (build-and-install instances profile)))))))))))))))
 
 ;;; pull.scm ends here
-- 
2.29.2

Message #38 received at 45104 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 45104 <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Sun, 31 Jan 2021 17:18:55 +0100

Hi!

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

> From 31dad6456825a329ba0f07c95e3e99258d186a8f Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <othacehe <at> gnu.org>
> Date: Fri, 29 Jan 2021 13:48:44 +0100
> Subject: [PATCH v2] guix: channels: Introduce
>  "channel-with-substitutes-available".
>
> * guix/channels.scm (find-latest-commit-with-substitutes,
> channel-with-substitutes-available): New procedures.
> * guix/scripts/pull.scm (guix-pull): Move "channel-list" call inside the
> %current-system parameter scope.
> * doc/guix.texi (Channels with substitutes): New section.

Yay!

> +@node Channels with substitutes
> +@section Channels with substitutes

Should be “Channels With Substitutes”

> +When running @command{guix pull}, Guix will first compile the
> +definitions of every available package.  This is an expensive operation
> +for which substitutes (@pxref{Substitutes}) may be available.  The
> +following snippet in @file{channels.scm} will ensure that @command{guix
> +pull} uses the latest commit with available substitutes for the package
> +definitions.  This is done by querying the continuous integration
               ^
I’d replace the period with a colon and insert the example right here.

> +server at @url{https://ci.guix.gnu.org}.
> +
> +Be careful, this does not mean that all the packages that you will

s/Be careful,/Note that/

> +install after running @command{guix pull} will have available
> +substitutes.  It only ensures that @command{guix pull} will not try to
> +compile package definitions.  This is particularly useful when using
> +machines with limited resources.
> +
> +@lisp
> +(list
> + (channel-with-substitutes-available
    ^
Nipick: maybe move on the previous line.

>  (define-module (guix channels)
>    #:use-module (git)
> +  #:use-module (guix ci)

Could we instead move these two procedures to (guix ci), with an
#:autoload for (guix channels), so that (guix channels) remains focused
on core functionality?

It does mean that in the example above we need to add an explicit
(use-modules (guix ci)).

Otherwise LGTM, thanks!

Ludo’.

Message #43 received at 45104-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 45104-done <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
Date: Sun, 31 Jan 2021 18:37:57 +0100

Hey Ludo,

> It does mean that in the example above we need to add an explicit
> (use-modules (guix ci)).
>
> Otherwise LGTM, thanks!

I took your remarks into account and pushed as
041a9466ea23d6ae811491bcf529bf9487317b48.

Thanks for reviewing!

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.