GNU bug report logs - #45104
pull: Add a "with-substitutes" option.

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Reported by: Mathieu Othacehe <othacehe@HIDDEN>; dated Mon, 7 Dec 2020 15:41:02 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 45104 <at> debbugs.gnu.org:


Received: (at 45104) by debbugs.gnu.org; 7 Dec 2020 17:12:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 07 12:12:00 2020
Received: from localhost ([127.0.0.1]:55349 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kmK3L-0002Zn-Ux
	for submit <at> debbugs.gnu.org; Mon, 07 Dec 2020 12:12:00 -0500
Received: from mail-wr1-f41.google.com ([209.85.221.41]:43260)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1kmK3K-0002ZU-Nj
 for 45104 <at> debbugs.gnu.org; Mon, 07 Dec 2020 12:11:59 -0500
Received: by mail-wr1-f41.google.com with SMTP id y17so4080868wrr.10
 for <45104 <at> debbugs.gnu.org>; Mon, 07 Dec 2020 09:11:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:in-reply-to:references:date:message-id:mime-version
 :content-transfer-encoding;
 bh=om+lleiirzyMuPAXOh0kk5eg3+YBg5eeG6AyF2Y5TAc=;
 b=gDDpVTcq0H8vOrXgg+HjnxP2+Ml0K+oD7ExTkP1/H8Yxa903K4ehstJbVelVI+Kgxx
 KfINLA7cAoxTv608svK7hjry/hdVZPI3QkhLpIEJLT9sNFgv8MFroqCUlIaN3sbI50nC
 lNnZn4cxj3TZFGmqD82BW9V6Gwyi0wBW0KubSWE1+SAcu39PVBMRNxog6lqsp+qPQZDR
 /1P94RR6ETvV47HzjFyRju++WMYAb9r/FTmx/N8V9pPIvQ6qzAM3Jr9h2OrLNxv4rcZt
 jFHDTN72TTyG3eN19rvkfX6IZiEk2Tfq3T26PRAuGhPAVQqmFsYfxG50eNz9HDQjDHB9
 ncqA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:in-reply-to:references:date
 :message-id:mime-version:content-transfer-encoding;
 bh=om+lleiirzyMuPAXOh0kk5eg3+YBg5eeG6AyF2Y5TAc=;
 b=t9Ks8MSB/sxKsYcStwdwO4VajxegYB3fyCfN0iOlwj11y4BdKNOQCjQtpFgItk6Jd3
 /FFyx8km8izL7MnBXugf3PkwIZZ7T6+PeuIrbRaiyeK4MkBu5vU6s2TQVxqBAwi8OQTt
 nr6S/jX+wmwWOj7TE9CZwg9SZGFXlCK5LnU423FSp8YwMf8TgvVUaRl3gFBLNxJQtsbU
 vESaf41AQVZnFA0x9SFx8Oeh9dFUqKKIM08ujajEzCGWW5fYkfMso+AqQlOi3C9MU9tK
 zM4g2lpOgAQW+U7XyzpFrSTOroLMV6VJuxj3yBiMOTwdeO6qJ6iLO0XCIRbFDCWgBRQx
 0hVg==
X-Gm-Message-State: AOAM532SvNKf6MTdPvYaatSPONA9nqyW35dAfqLsnST+zfPCGzlR1gs7
 2c7T8B+C4w4ZYT2Vz3+d1y87JFB3ie4=
X-Google-Smtp-Source: ABdhPJwsLZNH0KcZVj1WTEOOKjdpQB0ohSk6QrKkx1ga1JwgPqOuhu5mRU/7hocbiUd1zF/7bqwE0A==
X-Received: by 2002:adf:9b95:: with SMTP id d21mr21024846wrc.335.1607361112562; 
 Mon, 07 Dec 2020 09:11:52 -0800 (PST)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id r2sm14477813wrn.83.2020.12.07.09.11.51
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 07 Dec 2020 09:11:52 -0800 (PST)
From: zimoun <zimon.toutoune@HIDDEN>
To: Mathieu Othacehe <othacehe@HIDDEN>, 45104 <at> debbugs.gnu.org
Subject: Re: [bug#45104] pull: Add a "with-substitutes" option.
In-Reply-To: <87eek1vd4g.fsf@HIDDEN>
References: <87eek1vd4g.fsf@HIDDEN>
Date: Mon, 07 Dec 2020 18:05:16 +0100
Message-ID: <86k0tt5yyb.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 45104
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Mathieu,

On Mon, 07 Dec 2020 at 16:39, Mathieu Othacehe <othacehe@HIDDEN> wrote:

> When "guix pull" is run before the CI server is done baking substitutes,
> the user may need to build some derivations, which can be long on
> low-end hardware.

Yeah=E2=80=A6


> This patch adds an option to "guix pull" so that it updates to the
> latest commit with available substitutes.

Cool!  It improves my ugly bash to pull 2 weeks behind. :-)


> This work is still in progress, but I'd like to gather some impressions
> on that before going further.

With your patch, it is possible to never pull something.  Because the
push are faster than the CI is able to build.

Therefore, an improvement could to be to check the latest commit on the
CI, if not available fallback to the previous, repeat until one is
available or repeat X time and raise a warning.


Cheers,
simon





Information forwarded to guix-patches@HIDDEN:
bug#45104; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 7 Dec 2020 15:40:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 07 10:40:05 2020
Received: from localhost ([127.0.0.1]:55002 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1kmIcP-0004F9-23
	for submit <at> debbugs.gnu.org; Mon, 07 Dec 2020 10:40:05 -0500
Received: from lists.gnu.org ([209.51.188.17]:59678)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1kmIcM-0004Ez-Md
 for submit <at> debbugs.gnu.org; Mon, 07 Dec 2020 10:40:03 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:44856)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1kmIcM-0002Zp-Dc
 for guix-patches@HIDDEN; Mon, 07 Dec 2020 10:40:02 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:57408)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1kmIcM-0000wv-4V
 for guix-patches@HIDDEN; Mon, 07 Dec 2020 10:40:02 -0500
Received: from [2a01:e0a:19b:d9a0:34f0:bcad:cad1:16d6] (port=34604 helo=cervin)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <othacehe@HIDDEN>) id 1kmIcL-0006bB-7Q
 for guix-patches@HIDDEN; Mon, 07 Dec 2020 10:40:01 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
To: guix-patches@HIDDEN
Subject: pull: Add a "with-substitutes" option.
Date: Mon, 07 Dec 2020 16:39:59 +0100
Message-ID: <87eek1vd4g.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

--=-=-=
Content-Type: text/plain


Hello,

When "guix pull" is run before the CI server is done baking substitutes,
the user may need to build some derivations, which can be long on
low-end hardware.

This patch adds an option to "guix pull" so that it updates to the
latest commit with available substitutes.

This work is still in progress, but I'd like to gather some impressions
on that before going further.

Thanks,

Mathieu

--=-=-=
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: inline;
 filename=0001-scripts-pull-Add-with-substitutes-option.patch
Content-Transfer-Encoding: quoted-printable

From d399f8dbb9e38a82241b9048b8b04758fae10005 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@HIDDEN>
Date: Mon, 7 Dec 2020 16:12:22 +0100
Subject: [PATCH] scripts: pull: Add "with-substitutes" option.

---
 guix/scripts/pull.scm | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
index 83cdc1d1eb..4609f8614e 100644
--- a/guix/scripts/pull.scm
+++ b/guix/scripts/pull.scm
@@ -2,6 +2,7 @@
 ;;; Copyright =C2=A9 2013, 2014, 2015, 2017, 2018, 2019, 2020 Ludovic Cour=
t=C3=A8s <ludo@HIDDEN>
 ;;; Copyright =C2=A9 2017 Marius Bakke <mbakke@HIDDEN>
 ;;; Copyright =C2=A9 2020 Tobias Geerinckx-Rice <me@HIDDEN>
+;;; Copyright =C2=A9 2020 Mathieu Othacehe <othacehe@HIDDEN>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
=20
 (define-module (guix scripts pull)
   #:use-module ((guix ui) #:hide (display-profile-content))
+  #:use-module (guix ci)
   #:use-module (guix colors)
   #:use-module (guix utils)
   #:use-module ((guix status) #:select (with-status-verbosity))
@@ -64,6 +66,7 @@
   #:re-export (display-profile-content
                channel-commit-hyperlink)
   #:export (channel-list
+            find-lastest-commit-with-substitutes
             guix-pull))
=20
 
@@ -169,6 +172,9 @@ Download and deploy the latest version of Guix.\n"))
                  (lambda (opt name arg result)
                    (alist-cons 'validate-pull warn-about-backward-updates
                                result)))
+         (option '("with-substitutes") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'with-substitutes? #t result)))
          (option '("disable-authentication") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'authenticate-channels? #f result)))
@@ -526,6 +532,26 @@ true, display what would be built without actually bui=
lding it."
           (leave (G_ "while creating symlink '~a': ~a~%")
                  link (strerror (system-error-errno args))))))))
=20
+
+;;;
+;;; Substitutes.
+;;;
+
+(define (find-lastest-commit-with-substitutes)
+  (let ((urls %default-substitute-urls))
+    (any (lambda (url)
+           (let* ((build (match (latest-builds url 1
+                                               #:job "guix.x86_64-linux"
+                                               #:status 0) ;success
+                           ((build) build)))
+                  (evaluation (evaluation url
+                                          (build-evaluation build)))
+                  (commit (match (evaluation-checkouts evaluation)
+                            ((checkout)
+                             (checkout-commit checkout)))))
+             commit))
+         urls)))
+
 
 ;;;
 ;;; Queries.
@@ -731,8 +757,9 @@ Use '~/.config/guix/channels.scm' instead."))
=20
   (let ((ref (assoc-ref opts 'ref))
         (url (or (assoc-ref opts 'repository-url)
-                 (environment-variable))))
-    (if (or ref url)
+                 (environment-variable)))
+        (with-substitutes? (assoc-ref opts 'with-substitutes?)))
+    (if (or ref url with-substitutes?)
         (match (find guix-channel? channels)
           ((? channel? guix)
            ;; Apply '--url', '--commit', and '--branch' to the 'guix' chan=
nel.
@@ -745,7 +772,12 @@ Use '~/.config/guix/channels.scm' instead."))
                       (channel (inherit guix)
                                (url url) (commit #f) (branch branch)))
                      (#f
-                      (channel (inherit guix) (url url))))
+                      (let ((commit
+                             (and with-substitutes?
+                                  (find-lastest-commit-with-substitutes))))
+                        (channel (inherit guix)
+                                 (url url)
+                                 (commit commit)))))
                    (remove guix-channel? channels))))
           (#f                           ;no 'guix' channel, failure will e=
nsue
            channels))
--=20
2.29.2


--=-=-=--




Acknowledgement sent to Mathieu Othacehe <othacehe@HIDDEN>:
New bug report received and forwarded. Copy sent to guix-patches@HIDDEN. Full text available.
Report forwarded to guix-patches@HIDDEN:
bug#45104; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 7 Dec 2020 17:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.