GNU logs - #39530, boring messages


Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#39530] [PATCH] guix: Support partial download
Resent-From: Julien Lepiller <julien@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 09 Feb 2020 19:25:02 +0000
Resent-Message-ID: <handler.39530.B.158127626310950 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 39530
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 39530 <at> debbugs.gnu.org
X-Debbugs-Original-To: guix-patches@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.158127626310950
          (code B ref -1); Sun, 09 Feb 2020 19:25:02 +0000
Received: (at submit) by debbugs.gnu.org; 9 Feb 2020 19:24:23 +0000
Received: from localhost ([127.0.0.1]:53183 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j0sBq-0002qX-KP
	for submit <at> debbugs.gnu.org; Sun, 09 Feb 2020 14:24:23 -0500
Received: from lists.gnu.org ([209.51.188.17]:43082)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <julien@HIDDEN>) id 1j0sBo-0002qQ-P0
 for submit <at> debbugs.gnu.org; Sun, 09 Feb 2020 14:24:21 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:55872)
 by lists.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <julien@HIDDEN>) id 1j0sBm-0006oh-Li
 for guix-patches@HIDDEN; Sun, 09 Feb 2020 14:24:20 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,URIBL_BLOCKED
 autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <julien@HIDDEN>) id 1j0sBl-0008BE-0I
 for guix-patches@HIDDEN; Sun, 09 Feb 2020 14:24:18 -0500
Received: from lepiller.eu ([2a00:5884:8208::1]:56492)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <julien@HIDDEN>) id 1j0sBk-00084m-AR
 for guix-patches@HIDDEN; Sun, 09 Feb 2020 14:24:16 -0500
Received: from lepiller.eu (localhost [127.0.0.1])
 by lepiller.eu (OpenSMTPD) with ESMTP id 24279709
 for <guix-patches@HIDDEN>; Sun, 9 Feb 2020 19:24:09 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=lepiller.eu; h=date:from
 :to:subject:message-id:mime-version:content-type; s=dkim; bh=IqI
 n9FtfuknaFYZT0ZnnnbSJi9k=; b=DRIGhk6XKQvV2T0J9gZleHcXwYn/ijoo8Gy
 F90vJmgfism9oIZ2f1u4AbsuhM4NRrNcYHT6sQpFnJ9b/apA8u5+DND0q2+EJkYm
 YAUlD14pU5mO2K4iZgeVlGwxZeV5AmyXl+5EYHu8RnxWhwfOA7O9Uua5vuYSvZtz
 aVMo2LubC0Psv+99DXlNSYMEpiHcPxvmskRVF83jtN7wTFnLt+/pF7ByzUovc2nU
 uXc0QmyHeGoQpNlwi9rYMeqPOSi5rPRNUeiLk2eNMquIoXWGcwYIKLe8pb0W8F4h
 1qR85qQLwlDv3cjnDd0QOI8Pb+lL+8cLxP8YD/dz4XQyYXAvscA==
Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 9a79115c
 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for <guix-patches@HIDDEN>;
 Sun, 9 Feb 2020 19:24:09 +0000 (UTC)
Date: Sun, 9 Feb 2020 20:23:58 +0100
From: Julien Lepiller <julien@HIDDEN>
Message-ID: <20200209202358.17bb4a39@HIDDEN>
X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-unknown-linux-gnu)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MP_/4vB8hBS2hkleG6Iiq6Nx7.Q"
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-Received-From: 2a00:5884:8208::1
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--MP_/4vB8hBS2hkleG6Iiq6Nx7.Q
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi Guix!

This patch adds support for partial download of fixed-output
derivations. I'm not very confident it can be pushed as-is, and it has
some shortcomings.

First, I make sure that the guix daemon will not remove previously
failed attempts when trying to build something again, when that is a
fixed-output derivation. Then, I add a Range HTTP header when
performing an HTTP fetch; this ensures that we only query for the part
we don't already have, and append it to the target file.

If a partial download fails, the same mirror/url is tried again, but
the partial file is removed first, ensuring we do a complete fetch this
time around. If that failed too, we try with the following url. If we
only perform a complete fetch, we proceed as usual. The next url will
be a partial fetch if there is already something locally.

The use-case is: I have a very unreliable wifi currently and when
downloading a big source (or substitute, but this patch doesn't address
that use-case), the connection is sometimes dropped in the middle and i
have to fetch everything from scratch. With this patch, the download
resumes.

Some issues that might need to be fixed: progress only shows for the
rest of the file, it would be nicer if it could start again where it
was before (say the connection dropped at 34%, then the progress bar
should start from 34%). When there are at least two urls it goes like
that: fetch a partial file, connection drops. Remove the file and try
again, connection drops. Go to the next mirror, fetch a partial file.
The first mirror restarted the download from the beginning, but we'd
like it not to, and skip to the following mirror instead. When there is
a hash mismatch, the file is fetched twice on a further attempt.

When testing locally with guix build -S ghostscript (and running the
daemon from ./pre-inst-env), the download went fine. Cancelling it in
the middle and restarting it did continue the download instead of
starting again, which is nice :).

However, with that daemon there was a lot of new builds required to run
guix environment guix as my user (and nothing was substituted, which
is weird), whereas with the system's daemon, there was nothing to
build. Maybe there's something fishy in that patch...

--MP_/4vB8hBS2hkleG6Iiq6Nx7.Q
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename=0001-guix-download-Add-partial-download-support.patch

From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@HIDDEN>
Date: Sun, 9 Feb 2020 19:47:27 +0100
Subject: [PATCH] guix: download: Add partial download support.

* nix/libstore/build.cc (tryToBuild): Do not remove invalid fixed-output
derivations.
* guix/build/download.scm (http-fetch): Add a range argument.
(url-fetch): Performa partial download if a file already exists.
---
 guix/build/download.scm | 36 +++++++++++++++++++++++++++---------
 nix/libstore/build.cc   |  1 +
 2 files changed, 28 insertions(+), 9 deletions(-)

diff --git a/guix/build/download.scm b/guix/build/download.scm
index 0f2d5f402a..c2b710becc 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -654,11 +654,13 @@ Return the resulting target URI."
                     #:query    (uri-query    ref)
                     #:fragment (uri-fragment ref)))))
 
-(define* (http-fetch uri #:key timeout (verify-certificate? #t))
+(define* (http-fetch uri #:key timeout (verify-certificate? #t) range)
   "Return an input port containing the data at URI, and the expected number of
 bytes available or #f.  When TIMEOUT is true, bail out if the connection could
 not be established in less than TIMEOUT seconds.  When VERIFY-CERTIFICATE? is
-true, verify HTTPS certificates; otherwise simply ignore them."
+true, verify HTTPS certificates; otherwise simply ignore them.  When RANGE is
+a number, it is the number of bytes we want to skip from the data at URI;
+otherwise the full document is requested."
 
   (define headers
     `(;; Some web sites, such as http://dist.schmorp.de, would block you if
@@ -670,6 +672,10 @@ true, verify HTTPS certificates; otherwise simply ignore them."
       ;; Acceptable" when not explicitly told that everything is accepted.
       (Accept . "*/*")
 
+      ,@(if range
+            `((Range . ,(string-append "bytes=" (number->string range) "-")))
+            '())
+
       ;; Basic authentication, if needed.
       ,@(match (uri-userinfo uri)
           ((? string? str)
@@ -690,7 +696,8 @@ true, verify HTTPS certificates; otherwise simply ignore them."
                 ((code)
                  (response-code resp)))
     (case code
-      ((200)                                      ; OK
+      ((200                                       ; OK
+        206)                                      ; Partial content
        (values port (response-content-length resp)))
       ((301                                       ; moved permanently
         302                                       ; found (redirection)
@@ -703,7 +710,8 @@ true, verify HTTPS certificates; otherwise simply ignore them."
          (close connection)
          (http-fetch uri
                      #:timeout timeout
-                     #:verify-certificate? verify-certificate?)))
+                     #:verify-certificate? verify-certificate?
+                     #:range range)))
       (else
        (error "download failed" (uri->string uri)
               code (response-reason-phrase resp))))))
@@ -775,10 +783,15 @@ otherwise simply ignore them."
       ((http https)
        (false-if-exception*
         (let-values (((port size)
-                      (http-fetch uri
-                                  #:verify-certificate? verify-certificate?
-                                  #:timeout timeout)))
-          (call-with-output-file file
+                      (if (file-exists? file)
+                        (http-fetch uri
+                                    #:verify-certificate? verify-certificate?
+                                    #:timeout timeout
+                                    #:range (stat:size (stat file)))
+                        (http-fetch uri
+                                    #:verify-certificate? verify-certificate?
+                                    #:timeout timeout))))
+          (call-with-port (open-file file (if (file-exists? file) "a" "w"))
             (lambda (output)
               (dump-port* port output
                           #:buffer-size %http-receive-buffer-size
@@ -817,7 +830,12 @@ otherwise simply ignore them."
   (let try ((uri (append uri content-addressed-uris)))
     (match uri
       ((uri tail ...)
-       (or (fetch uri file)
+       (or (if (file-exists? file)
+	       ;; If the file already exists, we do a partial fetch for the
+	       ;; remainder.  If something went wrong, we remove the file
+	       ;; and try to fetch the whole file instead.
+               (or (fetch uri file) (begin (delete-file file) (fetch uri file)))
+               (fetch uri file))
            (try tail)))
       (()
        (format (current-error-port) "failed to download ~s from ~s~%"
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 17e92c68a7..176ab40226 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild()
         Path path = i->second.path;
         if (worker.store.isValidPath(path)) continue;
         if (!pathExists(path)) continue;
+	if (fixedOutput) continue;
         debug(format("removing invalid path `%1%'") % path);
         deletePath(path);
     }
-- 
2.24.0


--MP_/4vB8hBS2hkleG6Iiq6Nx7.Q--




Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Julien Lepiller <julien@HIDDEN>
Subject: bug#39530: Acknowledgement ([PATCH] guix: Support partial download)
Message-ID: <handler.39530.B.158127626310950.ack <at> debbugs.gnu.org>
References: <20200209202358.17bb4a39@HIDDEN>
X-Gnu-PR-Message: ack 39530
X-Gnu-PR-Package: guix-patches
X-Gnu-PR-Keywords: patch
Reply-To: 39530 <at> debbugs.gnu.org
Date: Sun, 09 Feb 2020 19:25:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 guix-patches@HIDDEN

If you wish to submit further information on this problem, please
send it to 39530 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
39530: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D39530
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems


Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#39530] [PATCH] guix: Support partial download
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Wed, 19 Feb 2020 16:05:02 +0000
Resent-Message-ID: <handler.39530.B39530.158212828519645 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 39530
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Julien Lepiller <julien@HIDDEN>
Cc: 39530 <at> debbugs.gnu.org
Received: via spool by 39530-submit <at> debbugs.gnu.org id=B39530.158212828519645
          (code B ref 39530); Wed, 19 Feb 2020 16:05:02 +0000
Received: (at 39530) by debbugs.gnu.org; 19 Feb 2020 16:04:45 +0000
Received: from localhost ([127.0.0.1]:43304 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j4Rq9-00056m-A4
	for submit <at> debbugs.gnu.org; Wed, 19 Feb 2020 11:04:45 -0500
Received: from eggs.gnu.org ([209.51.188.92]:49312)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1j4Rq6-00056X-Uw
 for 39530 <at> debbugs.gnu.org; Wed, 19 Feb 2020 11:04:43 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:40228)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1j4Rq1-0006KE-Bj; Wed, 19 Feb 2020 11:04:37 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48422 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1j4Rpy-0007tb-9Z; Wed, 19 Feb 2020 11:04:35 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20200209202358.17bb4a39@HIDDEN>
Date: Wed, 19 Feb 2020 17:04:32 +0100
In-Reply-To: <20200209202358.17bb4a39@HIDDEN> (Julien
 Lepiller's message of "Sun, 9 Feb 2020 20:23:58 +0100")
Message-ID: <874kvmmulr.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi,

Julien Lepiller <julien@HIDDEN> skribis:

> First, I make sure that the guix daemon will not remove previously
> failed attempts when trying to build something again, when that is a
> fixed-output derivation. Then, I add a Range HTTP header when
> performing an HTTP fetch; this ensures that we only query for the part
> we don't already have, and append it to the target file.
>
> If a partial download fails, the same mirror/url is tried again, but
> the partial file is removed first, ensuring we do a complete fetch this
> time around. If that failed too, we try with the following url. If we
> only perform a complete fetch, we proceed as usual. The next url will
> be a partial fetch if there is already something locally.

Nice!

> However, with that daemon there was a lot of new builds required to run
> guix environment guix as my user (and nothing was substituted, which
> is weird), whereas with the system's daemon, there was nothing to
> build. Maybe there's something fishy in that patch...

Hmm, that sounds really weird.  Could you clarify what you did?

>>From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@HIDDEN>
> Date: Sun, 9 Feb 2020 19:47:27 +0100
> Subject: [PATCH] guix: download: Add partial download support.

Nitpick: you can remove =E2=80=9Cguix:=E2=80=9D from the subject.

> * nix/libstore/build.cc (tryToBuild): Do not remove invalid fixed-output
> derivations.
> * guix/build/download.scm (http-fetch): Add a range argument.
> (url-fetch): Performa partial download if a file already exists.

[...]

> -(define* (http-fetch uri #:key timeout (verify-certificate? #t))
> +(define* (http-fetch uri #:key timeout (verify-certificate? #t) range)
>    "Return an input port containing the data at URI, and the expected num=
ber of
>  bytes available or #f.  When TIMEOUT is true, bail out if the connection=
 could
>  not be established in less than TIMEOUT seconds.  When VERIFY-CERTIFICAT=
E? is
> -true, verify HTTPS certificates; otherwise simply ignore them."
> +true, verify HTTPS certificates; otherwise simply ignore them.  When RAN=
GE is
> +a number, it is the number of bytes we want to skip from the data at URI;
> +otherwise the full document is requested."

I=E2=80=99d suggest to rename #:range to #:offset because it denotes the st=
art
offset.

What response do we get if the server doesn=E2=80=99t support =E2=80=9CRang=
e=E2=80=9D?

Can servers silently ignore =E2=80=9CRange=E2=80=9D?

> +                      (if (file-exists? file)
> +                        (http-fetch uri
> +                                    #:verify-certificate? verify-certifi=
cate?
> +                                    #:timeout timeout
> +                                    #:range (stat:size (stat file)))
> +                        (http-fetch uri
> +                                    #:verify-certificate? verify-certifi=
cate?
> +                                    #:timeout timeout))))

I=E2=80=99d remove the =E2=80=98if=E2=80=99:

  (http-fetch =E2=80=A6
              #:offset (and=3D> (stat file #f) stat:size))

> --- a/nix/libstore/build.cc
> +++ b/nix/libstore/build.cc
> @@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild()
>          Path path =3D i->second.path;
>          if (worker.store.isValidPath(path)) continue;
>          if (!pathExists(path)) continue;
> +	if (fixedOutput) continue;

Please add a comment above explaining why fixed outputs are not deleted.

Also please: not tabs.  :-)

Thanks!

Ludo=E2=80=99.




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#39530] [PATCH] guix: Support partial download
Resent-From: Julien Lepiller <julien@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Wed, 19 Feb 2020 16:26:02 +0000
Resent-Message-ID: <handler.39530.B39530.158212952821550 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 39530
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 39530 <at> debbugs.gnu.org
Received: via spool by 39530-submit <at> debbugs.gnu.org id=B39530.158212952821550
          (code B ref 39530); Wed, 19 Feb 2020 16:26:02 +0000
Received: (at 39530) by debbugs.gnu.org; 19 Feb 2020 16:25:28 +0000
Received: from localhost ([127.0.0.1]:43318 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j4SAC-0005bV-CK
	for submit <at> debbugs.gnu.org; Wed, 19 Feb 2020 11:25:28 -0500
Received: from lepiller.eu ([89.234.186.109]:46764)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <julien@HIDDEN>) id 1j4SAA-0005bM-GK
 for 39530 <at> debbugs.gnu.org; Wed, 19 Feb 2020 11:25:27 -0500
Received: from lepiller.eu (localhost [127.0.0.1])
 by lepiller.eu (OpenSMTPD) with ESMTP id 53bd1e10;
 Wed, 19 Feb 2020 16:25:23 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=lepiller.eu; h=date
 :in-reply-to:references:mime-version:content-type
 :content-transfer-encoding:subject:to:cc:from:message-id; s=
 dkim; bh=y5E9Zh00kmFueuUpjoIv5PFBRQM=; b=JF4qH4xKOi9wAUcB/dp4Lo+
 /8VT3WkubuNtDFtDt72RD5UGWTY4IHrlGWx/e8f1rcMybiHt1VKO/MMqeQIuHrk0
 WIgc5v1h6KiwmZHwpdcYQXtpPRLMSYBMDCgLIhyqcsN73iJdane0o5v2Az55z9G3
 aeLmtM46C3Ruw+ULNC9/e2Wr2BOEOqGCmXJCAmnxs81yVApb6czxOhzYCg7jVMWs
 JEtX3Tp+jqV8oMWddRk5pLqSmgm/duXGSTMk0Vj/MxZBtJDiYkQuoKQkMZYINjP2
 OQDfFk0GJKhSfKMMQOwOAsYOfvMpUdqx733kARIjA24crMWbRu+xOaRnbx1nyzg=
 =
Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 9851b528
 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); 
 Wed, 19 Feb 2020 16:25:22 +0000 (UTC)
Date: Wed, 19 Feb 2020 11:25:05 -0500
User-Agent: K-9 Mail for Android
In-Reply-To: <874kvmmulr.fsf@HIDDEN>
References: <20200209202358.17bb4a39@HIDDEN>
 <874kvmmulr.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Julien Lepiller <julien@HIDDEN>
Message-ID: <8F45CBA3-9DCE-4BDD-A23E-543F035F87DC@HIDDEN>
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Le 19 f=C3=A9vrier 2020 11:04:32 GMT-05:00, "Ludovic Court=C3=A8s" <ludo@gn=
u=2Eorg> a =C3=A9crit :
>Hi,
>
>Julien Lepiller <julien@lepiller=2Eeu> skribis:
>
>> First, I make sure that the guix daemon will not remove previously
>> failed attempts when trying to build something again, when that is a
>> fixed-output derivation=2E Then, I add a Range HTTP header when
>> performing an HTTP fetch; this ensures that we only query for the
>part
>> we don't already have, and append it to the target file=2E
>>
>> If a partial download fails, the same mirror/url is tried again, but
>> the partial file is removed first, ensuring we do a complete fetch
>this
>> time around=2E If that failed too, we try with the following url=2E If =
we
>> only perform a complete fetch, we proceed as usual=2E The next url will
>> be a partial fetch if there is already something locally=2E
>
>Nice!
>
>> However, with that daemon there was a lot of new builds required to
>run
>> guix environment guix as my user (and nothing was substituted, which
>> is weird), whereas with the system's daemon, there was nothing to
>> build=2E Maybe there's something fishy in that patch=2E=2E=2E
>
>Hmm, that sounds really weird=2E  Could you clarify what you did?

Actually I'm not sure how to test the new daemon=2E I ran the guix daemon =
from a checkout and probably forgot to set sysconfdir to /etc, so it was mi=
ssing authorized keys, hence no substitutes=2E

>
>>>From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00
>2001
>> From: Julien Lepiller <julien@lepiller=2Eeu>
>> Date: Sun, 9 Feb 2020 19:47:27 +0100
>> Subject: [PATCH] guix: download: Add partial download support=2E
>
>Nitpick: you can remove =E2=80=9Cguix:=E2=80=9D from the subject=2E
>
>> * nix/libstore/build=2Ecc (tryToBuild): Do not remove invalid
>fixed-output
>> derivations=2E
>> * guix/build/download=2Escm (http-fetch): Add a range argument=2E
>> (url-fetch): Performa partial download if a file already exists=2E
>
>[=2E=2E=2E]
>
>> -(define* (http-fetch uri #:key timeout (verify-certificate? #t))
>> +(define* (http-fetch uri #:key timeout (verify-certificate? #t)
>range)
>>    "Return an input port containing the data at URI, and the expected
>number of
>>  bytes available or #f=2E  When TIMEOUT is true, bail out if the
>connection could
>>  not be established in less than TIMEOUT seconds=2E  When
>VERIFY-CERTIFICATE? is
>> -true, verify HTTPS certificates; otherwise simply ignore them=2E"
>> +true, verify HTTPS certificates; otherwise simply ignore them=2E  When
>RANGE is
>> +a number, it is the number of bytes we want to skip from the data at
>URI;
>> +otherwise the full document is requested=2E"
>
>I=E2=80=99d suggest to rename #:range to #:offset because it denotes the =
start
>offset=2E
>
>What response do we get if the server doesn=E2=80=99t support =E2=80=9CRa=
nge=E2=80=9D?
>
>Can servers silently ignore =E2=80=9CRange=E2=80=9D?
>
>> +                      (if (file-exists? file)
>> +                        (http-fetch uri
>> +                                    #:verify-certificate?
>verify-certificate?
>> +                                    #:timeout timeout
>> +                                    #:range (stat:size (stat file)))
>> +                        (http-fetch uri
>> +                                    #:verify-certificate?
>verify-certificate?
>> +                                    #:timeout timeout))))
>
>I=E2=80=99d remove the =E2=80=98if=E2=80=99:
>
>  (http-fetch =E2=80=A6
>              #:offset (and=3D> (stat file #f) stat:size))
>
>> --- a/nix/libstore/build=2Ecc
>> +++ b/nix/libstore/build=2Ecc
>> @@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild()
>>          Path path =3D i->second=2Epath;
>>          if (worker=2Estore=2EisValidPath(path)) continue;
>>          if (!pathExists(path)) continue;
>> +	if (fixedOutput) continue;
>
>Please add a comment above explaining why fixed outputs are not
>deleted=2E
>
>Also please: not tabs=2E  :-)
>
>Thanks!
>
>Ludo=E2=80=99=2E

Thanks!




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#39530] [PATCH] guix: Support partial download
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Wed, 19 Feb 2020 22:08:01 +0000
Resent-Message-ID: <handler.39530.B39530.158215006421310 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 39530
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Julien Lepiller <julien@HIDDEN>
Cc: 39530 <at> debbugs.gnu.org
Received: via spool by 39530-submit <at> debbugs.gnu.org id=B39530.158215006421310
          (code B ref 39530); Wed, 19 Feb 2020 22:08:01 +0000
Received: (at 39530) by debbugs.gnu.org; 19 Feb 2020 22:07:44 +0000
Received: from localhost ([127.0.0.1]:43501 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1j4XVP-0005Xe-Qd
	for submit <at> debbugs.gnu.org; Wed, 19 Feb 2020 17:07:44 -0500
Received: from eggs.gnu.org ([209.51.188.92]:48092)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1j4XVO-0005XQ-Ek
 for 39530 <at> debbugs.gnu.org; Wed, 19 Feb 2020 17:07:43 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48083)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1j4XVI-00031z-Hg; Wed, 19 Feb 2020 17:07:36 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49352 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1j4XVG-0007s2-GH; Wed, 19 Feb 2020 17:07:35 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20200209202358.17bb4a39@HIDDEN>
 <874kvmmulr.fsf@HIDDEN>
 <8F45CBA3-9DCE-4BDD-A23E-543F035F87DC@HIDDEN>
Date: Wed, 19 Feb 2020 23:07:31 +0100
In-Reply-To: <8F45CBA3-9DCE-4BDD-A23E-543F035F87DC@HIDDEN> (Julien
 Lepiller's message of "Wed, 19 Feb 2020 11:25:05 -0500")
Message-ID: <87v9o2kz8c.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Julien Lepiller <julien@HIDDEN> skribis:

> Actually I'm not sure how to test the new daemon. I ran the guix daemon f=
rom a checkout and probably forgot to set sysconfdir to /etc, so it was mis=
sing authorized keys, hence no substitutes.

Ah yes.  This should work:

  sudo -E ./pre-inst-env guix-daemon --build-users-group=3D=E2=80=A6 =E2=80=
=A6

Ludo=E2=80=99.





Last modified: Wed, 19 Feb 2020 22:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.