Ludovic Courtès <ludo@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at 63646) by debbugs.gnu.org; 22 May 2023 15:45:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon May 22 11:45:40 2023
Received: from localhost ([127.0.0.1]:35994 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1q17j9-0005GX-TU
for submit <at> debbugs.gnu.org; Mon, 22 May 2023 11:45:40 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36838)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1q17j7-0005GJ-Q8
for 63646 <at> debbugs.gnu.org; Mon, 22 May 2023 11:45:38 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1q17j0-0004Ss-Lv; Mon, 22 May 2023 11:45:30 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
From; bh=jNGDR7jAhZT1/H2LdJMZCjKtyNMIxzu14LgJZEUFzmw=; b=QOtBR7v20b/OdOKJaOtL
eUQrtowVRm4UPSvX5j8bew3Lx1TdY8cbfJoXBYt01uirqBdDM79SFp9e7Rr2YJCqVljR7+QnHB6c9
kwokEspHARbPF8i3Jeyz06liWeZKhIbeU+Br0+7Zco3DcAWh3RuEidyCUaNfV6UNyyy6Lp3h+NzT2
HNAgDeSibRt8E3x+Dipx0W0nkXl3h6yYOSXN8lX8pIoX9YmmLDcsFkaErySrC+H6AYwetmBJ/1wyj
iYRc+R7fxUNBUvK8TBN/qeW8l5/hrGlnXtiaAQreNAcjgMR66Wq+regBv1rMyIfZwDdpHuNw+qf5w
7R8adxxXeu1bvA==;
Received: from [193.50.110.247] (helo=ribbon)
by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1q17j0-0002kE-7L; Mon, 22 May 2023 11:45:30 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: 63646 <at> debbugs.gnu.org
Subject: Re: bug#63646: [PATCH] substitute: If a server's nar URL is 404,
try the next one(s).
References: <d49d7ab95b1c904d9a7299f51b6d4d56487d270c.1684769418.git.ludo@HIDDEN>
Date: Mon, 22 May 2023 17:45:27 +0200
In-Reply-To: <d49d7ab95b1c904d9a7299f51b6d4d56487d270c.1684769418.git.ludo@HIDDEN>
("Ludovic =?utf-8?Q?Court=C3=A8s=22's?= message of "Mon, 22 May 2023
17:30:50 +0200")
Message-ID: <87v8gkv088.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 63646
Cc: Josselin Poiret <dev@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>,
Simon Tournier <zimon.toutoune@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>,
Christopher Baines <mail@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:
> + (define (try-fetch choices)
> + (match choices
> + (((uri compression file-size) rest ...)
> + (guard (c ((and (pair? rest) (network-error? c))
> + (warning (G_ "download from '~a' failed, trying next U=
RL~%")
> + (uri->string uri))
I realized we can change =E2=80=98network-error?=E2=80=99 to =E2=80=98http-=
get-error?=E2=80=99 above.
Otherwise, we could find ourselves trying several nar URLs on the same
server when the error is ETIMEDOUT or ECONNREFUSED, which would be a
waste of time.
Ludo=E2=80=99.
guix-patches@HIDDEN:bug#63646; Package guix-patches.
Full text available.
Received: (at submit) by debbugs.gnu.org; 22 May 2023 15:31:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon May 22 11:31:09 2023
Received: from localhost ([127.0.0.1]:35934 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1q17V6-0004sV-Kw
for submit <at> debbugs.gnu.org; Mon, 22 May 2023 11:31:09 -0400
Received: from lists.gnu.org ([209.51.188.17]:43202)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1q17V4-0004sN-F8
for submit <at> debbugs.gnu.org; Mon, 22 May 2023 11:31:07 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1q17V3-0007JC-CE
for guix-patches@HIDDEN; Mon, 22 May 2023 11:31:06 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1q17V1-0001E5-PC; Mon, 22 May 2023 11:31:04 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
references; bh=W8FgW+p0K19l9XTbOwJpeMnNIILn3UGIhOSd+7VQ6f4=; b=WMKZnyqJO8dQpq
ehNwetFFkQANvEyS2EOB5ibdDFU8hFVRlzuJECOyndywjI6tKiPFz38iMvXaYEM10bPSH9qk5MCAl
NHx81MVHbamkLOuZeS8H31hZ0Kxal5VI78bt/mm+Utn8ixgCaGqDbYt/eXHPHiNsy0ylIXR+DwpAA
cYV+ft79LqFlcjg8erkwN7psQG2tEDjE4vC/IOX2gre10KDf02wViXvIAPGIkw6VnZtjzI+7RNvgm
I+Cbe/ec6RzNB7gEO4CFck3EbH5biPeBLF7EzjYdd3CS3i2i/h8v8iXg876GK45InULqX4aDHxE07
QcSMdPHJsAWhkcceLWOw==;
Received: from [193.50.110.247] (helo=gnu.org)
by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1q17V0-0003C9-65; Mon, 22 May 2023 11:31:02 -0400
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN>
To: guix-patches@HIDDEN
Subject: [PATCH] substitute: If a server's nar URL is 404, try the next one(s).
Date: Mon, 22 May 2023 17:30:50 +0200
Message-Id: <d49d7ab95b1c904d9a7299f51b6d4d56487d270c.1684769418.git.ludo@HIDDEN>
X-Mailer: git-send-email 2.40.1
MIME-Version: 1.0
X-Debbugs-Cc: Christopher Baines <mail@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
If a substitute server advertises in its narinfo, for example, both a
/zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip
URL.
Fixes <https://issues.guix.gnu.org/63634>.
* guix/narinfo.scm (narinfo-preferred-uris): New procedure.
(narinfo-best-uri): Rebase on top of it.
* guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure.
Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of
NARINFO.
* tests/substitute.scm (request-substitution): Remove 'parameterize'.
Delete DESTINATION.
("substitute, preferred nar URL is 404, other is 200"): New test.
---
guix/narinfo.scm | 21 +++++++++++++++++----
guix/scripts/substitute.scm | 33 ++++++++++++++++++++++-----------
tests/substitute.scm | 36 +++++++++++++++++++++++++++++++-----
3 files changed, 70 insertions(+), 20 deletions(-)
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 741c7ad406..a149d9a901 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -54,6 +54,7 @@ (define-module (guix narinfo)
narinfo-hash-algorithm+value
narinfo-hash->sha256
+ narinfo-preferred-uris
narinfo-best-uri
valid-narinfo?
@@ -309,9 +310,11 @@ (define (decompresses-faster? compression1 compression2)
("gzip" (string=? compression2 "lzip"))
(_ #f)))
-(define* (narinfo-best-uri narinfo #:key fast-decompression?)
- "Select the \"best\" URI to download NARINFO's nar, and return three values:
-the URI, its compression method (a string), and the compressed file size.
+(define* (narinfo-preferred-uris narinfo #:key fast-decompression?)
+ "Return the sorted list of \"preferred\" nar URIs from NARINFO (preferred
+comes first) where each entry is a tuple containing: the URI, its compression
+method (a string), and the compressed file size.
+
When FAST-DECOMPRESSION? is true, prefer substitutes with faster
decompression (typically zstd) rather than substitutes with a higher
compression ratio (typically lzip)."
@@ -343,6 +346,16 @@ (define* (narinfo-best-uri narinfo #:key fast-decompression?)
((uri2 compression2 . _)
(decompresses-faster? compression2 compression1))))))
- (match (sort choices (if fast-decompression? (negate speed<?) file-size<?))
+ (sort choices (if fast-decompression? (negate speed<?) file-size<?)))
+
+(define* (narinfo-best-uri narinfo #:key fast-decompression?)
+ "Select the \"best\" URI to download NARINFO's nar, and return three values:
+the URI, its compression method (a string), and the compressed file size.
+
+When FAST-DECOMPRESSION? is true, prefer substitutes with faster
+decompression (typically zstd) rather than substitutes with a higher
+compression ratio (typically lzip)."
+ (match (narinfo-preferred-uris narinfo
+ #:fast-decompression? fast-decompression?)
(((uri compression file-size) _ ...)
(values uri compression file-size))))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 0b27ebb0fc..7a612c971c 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -481,18 +481,29 @@ (define* (download-nar narinfo destination
(leave (G_ "unsupported substitute URI scheme: ~a~%")
(uri->string uri)))))
- (let ((uri compression file-size
- (narinfo-best-uri narinfo
- #:fast-decompression?
- %prefer-fast-decompression?)))
- (unless print-build-trace?
- (format (current-error-port)
- (G_ "Downloading ~a...~%") (uri->string uri)))
+ (define (try-fetch choices)
+ (match choices
+ (((uri compression file-size) rest ...)
+ (guard (c ((and (pair? rest) (network-error? c))
+ (warning (G_ "download from '~a' failed, trying next URL~%")
+ (uri->string uri))
+ (try-fetch rest)))
+ (let ((port download-size (fetch uri)))
+ (unless print-build-trace?
+ (format (current-error-port)
+ (G_ "Downloading ~a...~%") (uri->string uri)))
+ (values port uri compression download-size))))
+ (()
+ (leave (G_ "no valid nar URLs for ~a at ~a~%")
+ (narinfo-path narinfo)
+ (narinfo-uri-base narinfo)))))
- (let* ((raw download-size
- ;; 'guix publish' without '--cache' doesn't specify a
- ;; Content-Length, so DOWNLOAD-SIZE is #f in this case.
- (fetch uri))
+ (let ((choices (narinfo-preferred-uris narinfo
+ #:fast-decompression?
+ %prefer-fast-decompression?)))
+ ;; 'guix publish' without '--cache' doesn't specify a Content-Length, so
+ ;; DOWNLOAD-SIZE is #f in this case.
+ (let* ((raw uri compression download-size (try-fetch choices))
(progress
(let* ((dl-size (or download-size
(and (equal? compression "none")
diff --git a/tests/substitute.scm b/tests/substitute.scm
index 9032a50268..8df3938b59 100644
--- a/tests/substitute.scm
+++ b/tests/substitute.scm
@@ -64,11 +64,11 @@ (define-syntax-rule (test-quit name error-rx exp)
(define (request-substitution item destination)
"Run 'guix substitute --substitute' to fetch ITEM to DESTINATION."
- (parameterize ((guix-warning-port (current-error-port)))
- (with-input-from-string (string-append "substitute " item " "
- destination "\n")
- (lambda ()
- (guix-substitute "--substitute")))))
+ (false-if-exception (delete-file destination))
+ (with-input-from-string (string-append "substitute " item " "
+ destination "\n")
+ (lambda ()
+ (guix-substitute "--substitute"))))
(define %public-key
;; This key is known to be in the ACL by default.
@@ -613,6 +613,32 @@ (define-syntax-rule (with-narinfo* narinfo directory body ...)
(lambda ()
(false-if-exception (delete-file "substitute-retrieved")))))))))))
+(test-equal "substitute, preferred nar URL is 404, other is 200"
+ "Substitutable data."
+ (with-narinfo* (string-append %narinfo "Signature: " (signature-field %narinfo))
+ %main-substitute-directory
+
+ (with-http-server `((200 ,(string-append %narinfo "Signature: "
+ (signature-field %narinfo)
+ "\n"
+ "URL: example.nar.lz\n"
+ "Compression: lzip\n"))
+ (404 "Sorry, nar.lz is missing!")
+ (200 ,(call-with-input-file
+ (string-append %main-substitute-directory
+ "/example.nar")
+ get-bytevector-all)))
+ (dynamic-wind
+ (const #t)
+ (lambda ()
+ (parameterize ((substitute-urls (list (%local-url))))
+ (request-substitution (string-append (%store-prefix)
+ "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo")
+ "substitute-retrieved"))
+ (call-with-input-file "substitute-retrieved" get-string-all))
+ (lambda ()
+ (false-if-exception (delete-file "substitute-retrieved")))))))
+
(test-quit "substitute, narinfo is available but nar is missing"
"failed to find alternative substitute"
(with-narinfo*
base-commit: dff1689bb37e5303868584d3f1d7a33cbcb7f51e
--
2.40.1
Ludovic Courtès <ludo@HIDDEN>:mail@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN.
Full text available.mail@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:bug#63646; Package guix-patches.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.