GNU logs - #47174, boring messages


Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH 0/2] substitute: Handle closing connections to substitute servers.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Mar 2021 19:22:01 +0000
Resent-Message-ID: <handler.47174.B.161583611726811 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 47174 <at> debbugs.gnu.org
X-Debbugs-Original-To: guix-patches@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.161583611726811
          (code B ref -1); Mon, 15 Mar 2021 19:22:01 +0000
Received: (at submit) by debbugs.gnu.org; 15 Mar 2021 19:21:57 +0000
Received: from localhost ([127.0.0.1]:37209 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lLsmq-0006yN-S4
	for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:21:57 -0400
Received: from lists.gnu.org ([209.51.188.17]:50388)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lLsmo-0006yF-FR
 for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:21:55 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:58256)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>) id 1lLsml-0000NJ-Jg
 for guix-patches@HIDDEN; Mon, 15 Mar 2021 15:21:54 -0400
Received: from mira.cbaines.net ([212.71.252.8]:53620)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <mail@HIDDEN>) id 1lLsme-0008W0-Lq
 for guix-patches@HIDDEN; Mon, 15 Mar 2021 15:21:51 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id 2E6D727BC52
 for <guix-patches@HIDDEN>; Mon, 15 Mar 2021 19:21:43 +0000 (GMT)
Received: from capella (localhost [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id 75122fc1
 for <guix-patches@HIDDEN>; Mon, 15 Mar 2021 19:21:42 +0000 (UTC)
User-agent: mu4e 1.4.15; emacs 27.1
From: Christopher Baines <mail@HIDDEN>
Date: Mon, 15 Mar 2021 19:21:41 +0000
Message-ID: <87y2eodxyy.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@HIDDEN;
 helo=mira.cbaines.net
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

--=-=-=
Content-Type: text/plain


Christopher Baines (2):
  guix: Alter http-fetch to return the response.
  substitute: Handle closing connections to substitute servers.

 guix/build/download-nar.scm |  5 +++--
 guix/build/download.scm     |  9 ++++++---
 guix/http-client.scm        | 12 ++++++------
 guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++-------
 4 files changed, 39 insertions(+), 18 deletions(-)

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBPs8VfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xef2A/+Lm+zmZgKgVlBPQz+dN6k5H9MnTXQNU3m
iPPZ0BeaY9d8ZTg1zSnwLeF9kRQsm3Z/VlU0wmwv56AO3RS2U6q8iXzk+V6kmui/
0a+c84k+TsIkFcyEJmQlGXXvcoGZpGi2BHMo4fdAsKdKkTW3nIk+fHfc0fJ31wuO
skHZF5Q6KRIGyURtym9w2NB4fBjoesjJqsZ7Q55OXgViGITfDqCGprdlF1cxdGvt
bNNCoNj26bOzlT7D10rW9IFZWJrG+6CGxfM0+d2LaW30viO+P8RPxLs5UfkcGsTE
41/1VCXUoCBAR2/GRoPGiZsYNIznLuV+/hqD+vNLh1nIctybX0cld74uttSROImp
4RiJy4YBdjnYu4e//rzui0jIU7QfX/iVSCYtgt6ymmsNVNfklzU0CEgrBHln8Dha
RRWVGY2R4SWN57K2p5CSjDQiQzH37IP+N1WRKeebCR8Brub7mcA3d5kFprvz/BS3
2AvrNUkVnxBw2D4Vk30QB2OMDofIfxLBJmNsZbPsG87BUjaspOB8kPvzSl00RnBr
WFICIPwEa5vAFv/N6fOQrkNCH2eNbK3R/2LEcLeNWTRq7UiXu1zL7kNtfjyZ4sXZ
C/zE+xMXGkCcLD9QeubMCuW3usRJAQgdNra6pl71FCE1VxyjC30/JBOTjBlXmHPG
ICJyXcsD3QQ=
=SskM
-----END PGP SIGNATURE-----
--=-=-=--




Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Christopher Baines <mail@HIDDEN>
Subject: bug#47174: Acknowledgement ([PATCH 0/2] substitute: Handle
 closing connections to substitute servers.)
Message-ID: <handler.47174.B.161583611726811.ack <at> debbugs.gnu.org>
References: <87y2eodxyy.fsf@HIDDEN>
X-Gnu-PR-Message: ack 47174
X-Gnu-PR-Package: guix-patches
X-Gnu-PR-Keywords: patch
Reply-To: 47174 <at> debbugs.gnu.org
Date: Mon, 15 Mar 2021 19:22:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 guix-patches@HIDDEN

If you wish to submit further information on this problem, please
send it to 47174 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
47174: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47174
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems


Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH 2/2] substitute: Handle closing connections to substitute servers.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Mar 2021 19:25:02 +0000
Resent-Message-ID: <handler.47174.B47174.161583629227114 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.161583629227114
          (code B ref 47174); Mon, 15 Mar 2021 19:25:02 +0000
Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 19:24:52 +0000
Received: from localhost ([127.0.0.1]:37215 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lLspg-00073F-Bn
	for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:24:52 -0400
Received: from mira.cbaines.net ([212.71.252.8]:35640)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lLspe-000731-CP
 for 47174 <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:24:50 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id 6C29B27BC54
 for <47174 <at> debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 1dfa5663
 for <47174 <at> debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
Date: Mon, 15 Mar 2021 19:24:49 +0000
Message-Id: <20210315192449.16248-2-mail@HIDDEN>
X-Mailer: git-send-email 2.30.1
In-Reply-To: <20210315192449.16248-1-mail@HIDDEN>
References: <20210315192449.16248-1-mail@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

When reusing a HTTP connection to fetch multiple nars, and the remote server
signals that the connection should be closed.

* guix/scripts/substitute.scm (process-substitution): Close connections to
substitute servers when a Connection: close header is specified in the
response.
---
 guix/scripts/substitute.scm | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index cb79ea6927..deb6fbdaa2 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -406,7 +406,9 @@ the current output port."
     (case (uri-scheme uri)
       ((file)
        (let ((port (open-file (uri-path uri) "r0b")))
-         (values port (stat:size (stat port)))))
+         (values port
+                 (stat:size (stat port))
+                 (const #t))))          ; no cleanup to do
       ((http https)
        (guard (c ((http-get-error? c)
                   (leave (G_ "download from '~a' failed: ~a, ~s~%")
@@ -434,7 +436,12 @@ the current output port."
                              #:buffered? #f
                              #:verify-certificate? #f)))
                 (values raw
-                        (response-content-length response))))))))
+                        (response-content-length response)
+                        (match (assq 'connection (response-headers response))
+                          (('connection 'close)
+                           (lambda ()
+                             (close-port (response-port response))))
+                          (_ (const #t))))))))))
       (else
        (leave (G_ "unsupported substitute URI scheme: ~a~%")
               (uri->string uri)))))
@@ -449,7 +456,7 @@ the current output port."
       (format (current-error-port)
               (G_ "Downloading ~a...~%") (uri->string uri)))
 
-    (let*-values (((raw download-size)
+    (let*-values (((raw download-size post-fetch-cleanup)
                    ;; 'guix publish' without '--cache' doesn't specify a
                    ;; Content-Length, so DOWNLOAD-SIZE is #f in this case.
                    (fetch uri))
@@ -493,6 +500,10 @@ the current output port."
       ;; Wait for the reporter to finish.
       (every (compose zero? cdr waitpid) pids)
 
+      ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTTP is
+      ;; being used, and the connection should be closed
+      (post-fetch-cleanup)
+
       ;; Skip a line after what 'progress-reporter/file' printed, and another
       ;; one to visually separate substitutions.
       (display "\n\n" (current-error-port))
-- 
2.30.1





Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH 1/2] guix: Alter http-fetch to return the response.
References: <87y2eodxyy.fsf@HIDDEN>
In-Reply-To: <87y2eodxyy.fsf@HIDDEN>
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Mar 2021 19:25:02 +0000
Resent-Message-ID: <handler.47174.B47174.161583629327121 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.161583629327121
          (code B ref 47174); Mon, 15 Mar 2021 19:25:02 +0000
Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 19:24:53 +0000
Received: from localhost ([127.0.0.1]:37217 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lLspg-00073H-Mh
	for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:24:53 -0400
Received: from mira.cbaines.net ([212.71.252.8]:35638)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lLspe-000730-CM
 for 47174 <at> debbugs.gnu.org; Mon, 15 Mar 2021 15:24:50 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id 5FFE527BC52
 for <47174 <at> debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (GMT)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 55fff41d
 for <47174 <at> debbugs.gnu.org>; Mon, 15 Mar 2021 19:24:49 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
Date: Mon, 15 Mar 2021 19:24:48 +0000
Message-Id: <20210315192449.16248-1-mail@HIDDEN>
X-Mailer: git-send-email 2.30.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Rather than just the port and response-content-length.  I'm looking at using
the response headers within the substitute script to work out when to close
the connection.

* guix/http-client.scm (http-fetch): Return the response as the second value,
rather than the response-content-length.
* guix/build/download-nar.scm (download-nar): Adapt accordingly.
* guix/build/download.scm (url-fetch): Adapt accordingly.
* guix/scripts/substitute.scm (process-substitution): Adapt accordingly.
---
 guix/build/download-nar.scm |  5 +++--
 guix/build/download.scm     |  9 ++++++---
 guix/http-client.scm        | 12 ++++++------
 guix/scripts/substitute.scm | 16 +++++++++++-----
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm
index 867f3c10bb..fbb5d37c0a 100644
--- a/guix/build/download-nar.scm
+++ b/guix/build/download-nar.scm
@@ -23,6 +23,7 @@
   #:autoload   (zlib) (call-with-gzip-input-port)
   #:use-module (guix progress)
   #:use-module (web uri)
+  #:use-module (web response)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 format)
@@ -101,7 +102,7 @@ success, #f otherwise."
       ((url rest ...)
        (format #t "Trying content-addressed mirror at ~a...~%"
                (uri-host (string->uri url)))
-       (let-values (((port size)
+       (let-values (((port resp)
                      (catch #t
                        (lambda ()
                          (http-fetch (string->uri url)))
@@ -109,7 +110,7 @@ success, #f otherwise."
                          (values #f #f)))))
          (if (not port)
              (loop rest)
-             (begin
+             (let ((size (response-content-length resp)))
                (if size
                    (format #t "Downloading from ~a (~,2h MiB)...~%" url
                            (/ size (expt 2 20.)))
diff --git a/guix/build/download.scm b/guix/build/download.scm
index f24a1e20df..437184b9cb 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -21,6 +21,7 @@
 (define-module (guix build download)
   #:use-module (web uri)
   #:use-module (web http)
+  #:use-module (web response)
   #:use-module ((web client) #:hide (open-socket-for-uri))
   #:use-module (web response)
   #:use-module (guix base64)
@@ -647,7 +648,7 @@ otherwise simply ignore them."
     (case (uri-scheme uri)
       ((http https)
        (false-if-exception*
-        (let-values (((port size)
+        (let-values (((port resp)
                       (http-fetch uri
                                   #:verify-certificate? verify-certificate?
                                   #:timeout timeout)))
@@ -657,9 +658,11 @@ otherwise simply ignore them."
                           #:buffer-size %http-receive-buffer-size
                           #:reporter (if print-build-trace?
                                          (progress-reporter/trace
-                                          file (uri->string uri) size)
+                                          file (uri->string uri)
+                                          (response-content-length resp))
                                          (progress-reporter/file
-                                          (uri-abbreviation uri) size)))
+                                          (uri-abbreviation uri)
+                                          (response-content-length resp))))
               (newline)))
           file)))
       ((ftp)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 2d7458a56e..47076d41f6 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -80,11 +80,11 @@
                      (verify-certificate? #t)
                      (headers '((user-agent . "GNU Guile")))
                      timeout)
-  "Return an input port containing the data at URI, and the expected number of
-bytes available or #f.  If TEXT? is true, the data at URI is considered to be
-textual.  Follow any HTTP redirection.  When BUFFERED? is #f, return an
-unbuffered port, suitable for use in `filtered-port'.  HEADERS is an alist of
-extra HTTP headers.
+  "Return an input port containing the data at URI, and the HTTP response from
+the server.  If TEXT? is true, the data at URI is considered to be textual.
+Follow any HTTP redirection.  When BUFFERED? is #f, return an unbuffered port,
+suitable for use in `filtered-port'.  HEADERS is an alist of extra HTTP
+headers.
 
 When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is
 not closed upon completion.
@@ -120,7 +120,7 @@ Raise an '&http-get-error' condition if downloading fails."
                      (response-code resp)))
         (case code
           ((200)
-           (values data (response-content-length resp)))
+           (values data resp))
           ((301                                   ; moved permanently
             302                                   ; found (redirection)
             303                                   ; see other
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 6892aa999b..cb79ea6927 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -60,6 +60,7 @@
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
   #:use-module (web uri)
+  #:use-module (web response)
   #:use-module (guix http-client)
   #:export (%allow-unauthenticated-substitutes?
             %error-to-file-descriptor-4?
@@ -424,11 +425,16 @@ the current output port."
            (call-with-connection-error-handling
             uri
             (lambda ()
-              (http-fetch uri #:text? #f
-                          #:open-connection open-connection-for-uri/cached
-                          #:keep-alive? #t
-                          #:buffered? #f
-                          #:verify-certificate? #f))))))
+              (let-values (((raw response)
+                            (http-fetch
+                             uri
+                             #:text? #f
+                             #:open-connection open-connection-for-uri/cached
+                             #:keep-alive? #t
+                             #:buffered? #f
+                             #:verify-certificate? #f)))
+                (values raw
+                        (response-content-length response))))))))
       (else
        (leave (G_ "unsupported substitute URI scheme: ~a~%")
               (uri->string uri)))))
-- 
2.30.1





Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH 0/2] substitute: Handle closing connections to substitute servers.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Mar 2021 20:37:02 +0000
Resent-Message-ID: <handler.47174.B47174.16158405711297 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.16158405711297
          (code B ref 47174); Mon, 15 Mar 2021 20:37:02 +0000
Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 20:36:11 +0000
Received: from localhost ([127.0.0.1]:37306 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lLtwh-0000Kr-BG
	for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 16:36:11 -0400
Received: from eggs.gnu.org ([209.51.188.92]:35428)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lLtwf-0000Ke-2s
 for 47174 <at> debbugs.gnu.org; Mon, 15 Mar 2021 16:36:09 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51905)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lLtwZ-0000FY-64; Mon, 15 Mar 2021 16:36:03 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=46674 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1lLtwY-0007JT-Bt; Mon, 15 Mar 2021 16:36:02 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20210315192449.16248-1-mail@HIDDEN>
 <20210315192449.16248-2-mail@HIDDEN>
Date: Mon, 15 Mar 2021 21:36:01 +0100
In-Reply-To: <20210315192449.16248-2-mail@HIDDEN> (Christopher Baines's
 message of "Mon, 15 Mar 2021 19:24:49 +0000")
Message-ID: <874khccfym.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Christopher Baines <mail@HIDDEN> skribis:

> When reusing a HTTP connection to fetch multiple nars, and the remote ser=
ver
> signals that the connection should be closed.
>
> * guix/scripts/substitute.scm (process-substitution): Close connections to
> substitute servers when a Connection: close header is specified in the
> response.

In the context of <https://issues.guix.gnu.org/47157>, honoring
=E2=80=9CConnection: close=E2=80=9D isn=E2=80=99t enough.  We need to handl=
e the case where the
server didn=E2=80=99t express the intent to close the connection but eventu=
ally
closed it after some time.

Does that make sense?

Ludo=E2=80=99.




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH 0/2] substitute: Handle closing connections to substitute servers.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Mar 2021 20:43:02 +0000
Resent-Message-ID: <handler.47174.B47174.16158409581930 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.16158409581930
          (code B ref 47174); Mon, 15 Mar 2021 20:43:02 +0000
Received: (at 47174) by debbugs.gnu.org; 15 Mar 2021 20:42:38 +0000
Received: from localhost ([127.0.0.1]:37322 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lLu2v-0000V4-VT
	for submit <at> debbugs.gnu.org; Mon, 15 Mar 2021 16:42:38 -0400
Received: from mira.cbaines.net ([212.71.252.8]:36892)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lLu2t-0000Uv-Nw
 for 47174 <at> debbugs.gnu.org; Mon, 15 Mar 2021 16:42:36 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id 2097327BC52;
 Mon, 15 Mar 2021 20:42:35 +0000 (GMT)
Received: from capella (localhost [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id 71a5997a;
 Mon, 15 Mar 2021 20:42:34 +0000 (UTC)
References: <20210315192449.16248-1-mail@HIDDEN>
 <20210315192449.16248-2-mail@HIDDEN> <874khccfym.fsf_-_@HIDDEN>
User-agent: mu4e 1.4.15; emacs 27.1
From: Christopher Baines <mail@HIDDEN>
In-reply-to: <874khccfym.fsf_-_@HIDDEN>
Date: Mon, 15 Mar 2021 20:42:34 +0000
Message-ID: <87pn00du85.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Christopher Baines <mail@HIDDEN> skribis:
>
>> When reusing a HTTP connection to fetch multiple nars, and the remote se=
rver
>> signals that the connection should be closed.
>>
>> * guix/scripts/substitute.scm (process-substitution): Close connections =
to
>> substitute servers when a Connection: close header is specified in the
>> response.
>
> In the context of <https://issues.guix.gnu.org/47157>, honoring
> =E2=80=9CConnection: close=E2=80=9D isn=E2=80=99t enough.  We need to han=
dle the case where the
> server didn=E2=80=99t express the intent to close the connection but even=
tually
> closed it after some time.
>
> Does that make sense?

Yeah, of course, this was something I was thinking about in addition to
the changes in [1].

1: https://issues.guix.gnu.org/47160

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBPxrpfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xd0Jg//cyAQJF65Ezu0jNXzIoiUOwNZ5xc2mfsO
Rnbn/G3E+LiZnjzEBXRLLXobWuVON8jIwKcrmroVB+BIJbbEzTLCSu4hahmT/0SE
osbOr0hC9e0a9XcVTRxj0w9raRfyAhtvOl7IANH/Nl+SfM9ZXU1BOve5VRGPnBwZ
5QCLfh/m0naWoSErs8KTtz4zfFPJPS8K/3F7euiH6wzbgv27LK7e9z1Le4MnmbAJ
p47/Qt9i/mBPMYXkCs1uN0ciylC59u12lUPPLAyUk/5J+VBRyGCdy1rxRTVfogRv
k7PtVOPvkjSwwYWIRF1YAU/EGlM3IurHj/EnbBLAKM8Nuaa9V27puwzs44TCWCz5
weArBWRuusiYQuCXHoJSb2x14MXWADZacTslEnbVqCqmnsq8osqq8COvySJHg+HU
RSKWCXU+7g/ePqEjIfr4Wt/slBrawvl4eP6IYf4Gv8xAV5XeIQNdGBe8ZeqcBK6O
yB/R4rd9bnikZwHWTWiximsNdXoSoHCGthi5D+1jFrHOjM/u3FAD/7KOLUjaRcL8
12PVpMFfz2XY0T2obZa/DtZxLKadjAxJZ03L6uyLbC+diNMlyj/hiy3iJ4SRonRp
8A0siXnnPWcHb4yYbufKFs8Fhj9kA3oQsAPnwvkdZGq6bQkD1gZ80msDS7r7jRBn
EOtl2pIqcQY=
=PQKS
-----END PGP SIGNATURE-----
--=-=-=--




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH v2 2/2] substitute: Handle closing connections to substitute servers.
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 16 May 2021 22:12:02 +0000
Resent-Message-ID: <handler.47174.B47174.16212030848677 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.16212030848677
          (code B ref 47174); Sun, 16 May 2021 22:12:02 +0000
Received: (at 47174) by debbugs.gnu.org; 16 May 2021 22:11:24 +0000
Received: from localhost ([127.0.0.1]:51628 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1liOyq-0002Fs-2w
	for submit <at> debbugs.gnu.org; Sun, 16 May 2021 18:11:24 -0400
Received: from mira.cbaines.net ([212.71.252.8]:38896)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1liOyo-0002Fe-Ih
 for 47174 <at> debbugs.gnu.org; Sun, 16 May 2021 18:11:22 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id C995427BC81
 for <47174 <at> debbugs.gnu.org>; Sun, 16 May 2021 23:11:21 +0100 (BST)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 29660e40
 for <47174 <at> debbugs.gnu.org>; Sun, 16 May 2021 22:11:21 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
Date: Sun, 16 May 2021 23:11:21 +0100
Message-Id: <20210516221121.16705-2-mail@HIDDEN>
X-Mailer: git-send-email 2.30.1
In-Reply-To: <20210516221121.16705-1-mail@HIDDEN>
References: <20210516221121.16705-1-mail@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

When reusing a HTTP connection to fetch multiple nars, and the remote server
signals that the connection should be closed.

* guix/scripts/substitute.scm (process-substitution): Close connections to
substitute servers when a Connection: close header is specified in the
response.
---
 guix/scripts/substitute.scm | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 96f425eaa0..208b8f1273 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -464,7 +464,9 @@ PORT."
     (case (uri-scheme uri)
       ((file)
        (let ((port (open-file (uri-path uri) "r0b")))
-         (values port (stat:size (stat port)))))
+         (values port
+                 (stat:size (stat port))
+                 (const #t))))          ; no cleanup to do
       ((http https)
        (guard (c ((http-get-error? c)
                   (leave (G_ "download from '~a' failed: ~a, ~s~%")
@@ -487,7 +489,12 @@ PORT."
                                        #:keep-alive? #t
                                        #:buffered? #f)))
                (values raw
-                       (response-content-length response)))))))
+                       (response-content-length response)
+                       (match (assq 'connection (response-headers response))
+                         (('connection 'close)
+                          (lambda ()
+                            (close-port port)))
+                         (_ (const #t)))))))))
       (else
        (leave (G_ "unsupported substitute URI scheme: ~a~%")
               (uri->string uri)))))
@@ -504,7 +511,7 @@ PORT."
       (format (current-error-port)
               (G_ "Downloading ~a...~%") (uri->string uri)))
 
-    (let*-values (((raw download-size)
+    (let*-values (((raw download-size post-fetch-cleanup)
                    ;; 'guix publish' without '--cache' doesn't specify a
                    ;; Content-Length, so DOWNLOAD-SIZE is #f in this case.
                    (fetch uri))
@@ -565,6 +572,10 @@ PORT."
       ;; Wait for the reporter to finish.
       (every (compose zero? cdr waitpid) pids)
 
+      ;; Do post-fetch cleanup, maybe closing the HTTP connection if HTTP is
+      ;; being used, and the connection should be closed
+      (post-fetch-cleanup)
+
       ;; Skip a line after what 'progress-reporter/file' printed, and another
       ;; one to visually separate substitutions.  When PRINT-BUILD-TRACE? is
       ;; true, leave it up to (guix status) to prettify things.
-- 
2.30.1





Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH v2 1/2] guix: Alter http-fetch to return the response.
References: <87y2eodxyy.fsf@HIDDEN>
In-Reply-To: <87y2eodxyy.fsf@HIDDEN>
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 16 May 2021 22:12:02 +0000
Resent-Message-ID: <handler.47174.B47174.16212030848685 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.16212030848685
          (code B ref 47174); Sun, 16 May 2021 22:12:02 +0000
Received: (at 47174) by debbugs.gnu.org; 16 May 2021 22:11:24 +0000
Received: from localhost ([127.0.0.1]:51630 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1liOyq-0002Fu-BD
	for submit <at> debbugs.gnu.org; Sun, 16 May 2021 18:11:24 -0400
Received: from mira.cbaines.net ([212.71.252.8]:38894)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1liOyo-0002Ff-Ih
 for 47174 <at> debbugs.gnu.org; Sun, 16 May 2021 18:11:23 -0400
Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa])
 by mira.cbaines.net (Postfix) with ESMTPSA id C0B0627BC78
 for <47174 <at> debbugs.gnu.org>; Sun, 16 May 2021 23:11:21 +0100 (BST)
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 309302e4
 for <47174 <at> debbugs.gnu.org>; Sun, 16 May 2021 22:11:21 +0000 (UTC)
From: Christopher Baines <mail@HIDDEN>
Date: Sun, 16 May 2021 23:11:20 +0100
Message-Id: <20210516221121.16705-1-mail@HIDDEN>
X-Mailer: git-send-email 2.30.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Rather than just the port and response-content-length.  I'm looking at using
the response headers within the substitute script to work out when to close
the connection.

* guix/http-client.scm (http-fetch): Return the response as the second value,
rather than the response-content-length.
* guix/build/download-nar.scm (download-nar): Adapt accordingly.
* guix/build/download.scm (url-fetch): Adapt accordingly.
* guix/scripts/substitute.scm (process-substitution): Adapt accordingly.
---
 guix/build/download-nar.scm |  5 +++--
 guix/build/download.scm     |  9 ++++++---
 guix/http-client.scm        | 12 ++++++------
 guix/scripts/substitute.scm | 12 ++++++++----
 4 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scm
index 867f3c10bb..fbb5d37c0a 100644
--- a/guix/build/download-nar.scm
+++ b/guix/build/download-nar.scm
@@ -23,6 +23,7 @@
   #:autoload   (zlib) (call-with-gzip-input-port)
   #:use-module (guix progress)
   #:use-module (web uri)
+  #:use-module (web response)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 format)
@@ -101,7 +102,7 @@ success, #f otherwise."
       ((url rest ...)
        (format #t "Trying content-addressed mirror at ~a...~%"
                (uri-host (string->uri url)))
-       (let-values (((port size)
+       (let-values (((port resp)
                      (catch #t
                        (lambda ()
                          (http-fetch (string->uri url)))
@@ -109,7 +110,7 @@ success, #f otherwise."
                          (values #f #f)))))
          (if (not port)
              (loop rest)
-             (begin
+             (let ((size (response-content-length resp)))
                (if size
                    (format #t "Downloading from ~a (~,2h MiB)...~%" url
                            (/ size (expt 2 20.)))
diff --git a/guix/build/download.scm b/guix/build/download.scm
index b14db42352..d2006cc1fd 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -22,6 +22,7 @@
 (define-module (guix build download)
   #:use-module (web uri)
   #:use-module (web http)
+  #:use-module (web response)
   #:use-module ((web client) #:hide (open-socket-for-uri))
   #:use-module (web response)
   #:use-module (guix base64)
@@ -706,7 +707,7 @@ otherwise simply ignore them."
     (case (uri-scheme uri)
       ((http https)
        (false-if-exception*
-        (let-values (((port size)
+        (let-values (((port resp)
                       (http-fetch uri
                                   #:verify-certificate? verify-certificate?
                                   #:timeout timeout)))
@@ -716,9 +717,11 @@ otherwise simply ignore them."
                           #:buffer-size %http-receive-buffer-size
                           #:reporter (if print-build-trace?
                                          (progress-reporter/trace
-                                          file (uri->string uri) size)
+                                          file (uri->string uri)
+                                          (response-content-length resp))
                                          (progress-reporter/file
-                                          (uri-abbreviation uri) size)))
+                                          (uri-abbreviation uri)
+                                          (response-content-length resp))))
               (newline)))
           file)))
       ((ftp)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 10bc278023..189535079b 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -81,11 +81,11 @@
                      (headers '((user-agent . "GNU Guile")))
                      (log-port (current-error-port))
                      timeout)
-  "Return an input port containing the data at URI, and the expected number of
-bytes available or #f.  If TEXT? is true, the data at URI is considered to be
-textual.  Follow any HTTP redirection.  When BUFFERED? is #f, return an
-unbuffered port, suitable for use in `filtered-port'.  HEADERS is an alist of
-extra HTTP headers.
+  "Return an input port containing the data at URI, and the HTTP response from
+the server.  If TEXT? is true, the data at URI is considered to be textual.
+Follow any HTTP redirection.  When BUFFERED? is #f, return an unbuffered port,
+suitable for use in `filtered-port'.  HEADERS is an alist of extra HTTP
+headers.
 
 When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is
 not closed upon completion.
@@ -123,7 +123,7 @@ Raise an '&http-get-error' condition if downloading fails."
                      (response-code resp)))
         (case code
           ((200)
-           (values data (response-content-length resp)))
+           (values data resp))
           ((301                                   ; moved permanently
             302                                   ; found (redirection)
             303                                   ; see other
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 8e4eae00b3..96f425eaa0 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -61,6 +61,7 @@
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
   #:use-module (web uri)
+  #:use-module (web response)
   #:use-module (guix http-client)
   #:export (%allow-unauthenticated-substitutes?
             %reply-file-descriptor
@@ -480,10 +481,13 @@ PORT."
                       (uri->string uri))
              (warning (G_ "try `--no-substitutes' if the problem persists~%")))
            (with-cached-connection uri port
-             (http-fetch uri #:text? #f
-                         #:port port
-                         #:keep-alive? #t
-                         #:buffered? #f)))))
+             (let-values (((raw response)
+                           (http-fetch uri #:text? #f
+                                       #:port port
+                                       #:keep-alive? #t
+                                       #:buffered? #f)))
+               (values raw
+                       (response-content-length response)))))))
       (else
        (leave (G_ "unsupported substitute URI scheme: ~a~%")
               (uri->string uri)))))
-- 
2.30.1





Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH v2 1/2] guix: Alter http-fetch to return the response.
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 17 May 2021 14:45:01 +0000
Resent-Message-ID: <handler.47174.B47174.162126266822120 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.162126266822120
          (code B ref 47174); Mon, 17 May 2021 14:45:01 +0000
Received: (at 47174) by debbugs.gnu.org; 17 May 2021 14:44:28 +0000
Received: from localhost ([127.0.0.1]:53382 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lieTr-0005kg-M4
	for submit <at> debbugs.gnu.org; Mon, 17 May 2021 10:44:27 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38988)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lieTo-0005kO-UB
 for 47174 <at> debbugs.gnu.org; Mon, 17 May 2021 10:44:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44900)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lieTj-0000ET-7e; Mon, 17 May 2021 10:44:19 -0400
Received: from [109.190.253.11] (port=55366 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1lieTi-0001Ww-Dn; Mon, 17 May 2021 10:44:19 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87y2eodxyy.fsf@HIDDEN>
 <20210516221121.16705-1-mail@HIDDEN>
Date: Mon, 17 May 2021 16:44:15 +0200
In-Reply-To: <20210516221121.16705-1-mail@HIDDEN> (Christopher Baines's
 message of "Sun, 16 May 2021 23:11:20 +0100")
Message-ID: <87o8d9a08w.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello Chis,

> * guix/http-client.scm (http-fetch): Return the response as the second value,
> rather than the response-content-length.

I think there is a missing adaptation in the call-with-nar procedure of
the (guix scripts challenge) module.

Otherwise, looks fine!

Thanks,

Mathieu




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47174] [PATCH v2 2/2] substitute: Handle closing connections to substitute servers.
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 17 May 2021 14:47:02 +0000
Resent-Message-ID: <handler.47174.B47174.162126279827377 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47174
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Christopher Baines <mail@HIDDEN>
Cc: 47174 <at> debbugs.gnu.org
Received: via spool by 47174-submit <at> debbugs.gnu.org id=B47174.162126279827377
          (code B ref 47174); Mon, 17 May 2021 14:47:02 +0000
Received: (at 47174) by debbugs.gnu.org; 17 May 2021 14:46:38 +0000
Received: from localhost ([127.0.0.1]:53386 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lieVy-00077G-3d
	for submit <at> debbugs.gnu.org; Mon, 17 May 2021 10:46:38 -0400
Received: from eggs.gnu.org ([209.51.188.92]:39606)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lieVv-00072R-UB
 for 47174 <at> debbugs.gnu.org; Mon, 17 May 2021 10:46:36 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45020)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lieVq-0001kn-Jb; Mon, 17 May 2021 10:46:30 -0400
Received: from [109.190.253.11] (port=55380 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1lieVp-0001sr-Qs; Mon, 17 May 2021 10:46:30 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <20210516221121.16705-1-mail@HIDDEN>
 <20210516221121.16705-2-mail@HIDDEN>
Date: Mon, 17 May 2021 16:46:27 +0200
In-Reply-To: <20210516221121.16705-2-mail@HIDDEN> (Christopher Baines's
 message of "Sun, 16 May 2021 23:11:21 +0100")
Message-ID: <87k0nxa058.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


> +                       (match (assq 'connection (response-headers response))
> +                         (('connection 'close)
> +                          (lambda ()
> +                            (close-port port)))

You could maybe factorize it in a close-connection? procedure. Out of
curiosity, when does the remote server asks for connection closing?

Thanks,

Mathieu





Last modified: Mon, 17 May 2021 14:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.