GNU logs - #47336, boring messages


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#47336: Disarchive as a fallback for downloads
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 23 Mar 2021 04:43:01 +0000
Resent-Message-ID: <handler.47336.B.16164745447069 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 47336
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 47336 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.16164745447069
          (code B ref -1); Tue, 23 Mar 2021 04:43:01 +0000
Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000
Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOYs3-0001pw-WB
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400
Received: from lists.gnu.org ([209.51.188.17]:51240)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOYs2-0001pp-0u
 for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59600)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYs1-0007wV-QR
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYry-0007rZ-7A
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id D60141373
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=content-transfer-encoding:content-type
 :date:from:message-id:mime-version:subject:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC
 aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6
 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0
 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5
 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/
 YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl
 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA
 ==
X-ME-Sender: <xms:pnFZYL4J4-5z0UZC1ft_G_1Nhsyy0ABL5PdIDX1VYRe_v_ZNezhl0Q>
 <xme:pnFZYA72ibiOrA_3NYhBmyqncC-HLuV5_9ESOPa2h89glclhmwXbINrXztgsUVuiS
 Nn5X2IxUhkya46jtA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd
 dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn
 ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf
 evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg
 rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh
 grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:pnFZYCewFnYM7r_tIF4Gu6oiXAXAN2KxThYXRJT-H0FxCRxRUwWJ1g>
 <xmx:pnFZYMJ2wySGLcQuD8EahxJhuyn31BqGxftQ0WUWVl2XhDswjbEdJA>
 <xmx:pnFZYPKgBzcoSJayN2aN5uB3u8JxwErxvQGYMGsUOtf1v3urevXYcw>
 <xmx:pnFZYEWuF8VgmOhBe8DcSbP1nEDNVy-Fk6x0yrhtHdZww14o8f3bug>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
Date: Tue, 23 Mar 2021 00:42:12 -0400
Message-ID: <87eeg6o50b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@HIDDEN;
 helo=wout3-smtp.messagingengine.com
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.6 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)

Hello,

This patch series adds Disarchive assembly (backed by SWH lookup) as a
fallback for downloads.

To try it, make sure you are running the daemon in an environment with
Disarchive available:

    $ ./pre-inst-env guix environment --ad-hoc guile disarchive
    # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild

Don=E2=80=99t forget to stop your existing Guix Daemon.  :)

You also need to make sure that regular downloads are unavailable.  I do
this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl=
-fetch=E2=80=9D in
=E2=80=9Cguix/build/download.scm=E2=80=9D.  I replace the usual list of URL=
s with =E2=80=98()=E2=80=99:

    (let try ((uri (append uri content-addressed-uris)))
      (match '() ; uri
        ...))

Now you can ask Guix for a recent .tar.gz source package:

    $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

You should see:

    Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v=
diwlv19y9-httpretty-1.0.5.tar.gz
    Assembling the directory httpretty-1.0.5
    Downloading from Software Heritage...
    7903d608efc89c14afb4d692a3721156e31a43e2/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
    [...]
    Checking httpretty-1.0.5 digest... ok
    Assembling the tarball httpretty-1.0.5.tar
    Checking httpretty-1.0.5.tar digest... ok
    Assembling the Gzip file httpretty-1.0.5.tar.gz
    Checking httpretty-1.0.5.tar.gz digest... ok
    Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty=
-1.0.5.tar.gz
    successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett=
y-1.0.5.tar.gz.drv

There=E2=80=99s lots to talk about though....

First, it looks up the metadata on my server.  This is fine for a demo,
but not what we want forever.  The patch series supports adding several
mirrors for looking up the metadata.  In the past, we talked about
putting everything on one or a few of the big Git hosting platforms like
GitHub or Gitlab.  That way, it would be easily picked up by SWH and
archived =E2=80=9Cforever=E2=80=9D.  Right now, I have Cuirass set up to bu=
ild the
metadata, and a little script that moves it from the build server to my
Web server.  It would be simple enough to adjust that script to push it
to a remote Git repo.  (Of course, the next step is to move this setup
to Guix infrastructure.)  Thoughts?

On the code level, there were two things I couldn=E2=80=99t figure out for
myself.

I made the mirror list just simple strings.  AIUI, the client and the
daemon have to agree about the format of the mirror list.  Given that
running old daemons is common, changing the format is difficult.  Is it
worth it to copy the more flexible interface used by the content
addressed mirrors?  If yes, do I have to do the same =E2=80=98module-autolo=
ad!=E2=80=99
dance to use =E2=80=98bytevector->base16-string=E2=80=99?  :)  (I probably =
would have
just copied it, but that part confused me a bit.)

I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel=
l, just
=E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D).  It feels weird to use=
 a bunch of host-side modules
from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module.  This =
is okay because
=E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. =
 It=E2=80=99s more
like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2=
=80=9D, which
is why importing host-side modules is OK... right?

Hopefully everything else is more-or-less fine.  :)


-- Tim




Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Timothy Sample <samplet@HIDDEN>
Subject: bug#47336: Acknowledgement (Disarchive as a fallback for downloads)
Message-ID: <handler.47336.B.16164745447069.ack <at> debbugs.gnu.org>
References: <87eeg6o50b.fsf@HIDDEN>
X-Gnu-PR-Message: ack 47336
X-Gnu-PR-Package: guix
Reply-To: 47336 <at> debbugs.gnu.org
Date: Tue, 23 Mar 2021 04:43:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 47336 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47336
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems


Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#47336: [PATCH 1/2] swh: Add a directory download procedure.
References: <87eeg6o50b.fsf@HIDDEN>
In-Reply-To: <87eeg6o50b.fsf@HIDDEN>
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 23 Mar 2021 04:53:02 +0000
Resent-Message-ID: <handler.47336.B47336.16164751648063 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 47336 <at> debbugs.gnu.org
Cc: Timothy Sample <samplet@HIDDEN>
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.16164751648063
          (code B ref 47336); Tue, 23 Mar 2021 04:53:02 +0000
Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:52:44 +0000
Received: from localhost ([127.0.0.1]:59255 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOZ24-00025z-Ds
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:52:44 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:42087)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOZ23-00025o-5S
 for 47336 <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:52:43 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id 5C2E14CE;
 Tue, 23 Mar 2021 00:52:37 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:52:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:date:from
 :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=9GhkL9aQDX+o8K1Ib
 LLy06+cd1v9DpMcsGDadXOnnPk=; b=mD0rvrRxnVCL73RGYUbbdN8818yUnGh58
 WE9F+dIqu3kz3MZxL1mHEkGX5ba3xkVlxU0Jk++lJ93qngoCNF5t9lxuMudGhDgG
 YaIBxdv+jeNcLz36AbuwJgtQCT4he/E9bBtXImaqdWFyLyJ8DNCZb6c+j9wX/c2g
 Ys/CmVIQkVKME2MiVs1SdTS3AL9gzTsGMAow2k9UkdKWb4RKX0AT7QziTP8bokdi
 a6wpQbxcNE2qz1LgmAZX195fBJt4JbWUNUgyxIpJtMMixqgBz7SPR+lhrAQxolMh
 oU+h6LKZRvuT6cOaXWFNYCE3s59m1uWyFi6I1CsN1qsafElYVts0g==
X-ME-Sender: <xms:FHRZYIFMBUxlRwRbEeCI7nZyySuOOgfdgiEZP4WXOXPFJ05TnbLJGw>
 <xme:FHRZYBWV381BoaWFJlT5UfNkmTIOAGEs7a9pZIEZ8jqxL_Ad_w7ep4eILFXjOw1qr
 62YqVxXOiY0VrKqQQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgggfestdekredtre
 dttdenucfhrhhomhepvfhimhhothhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhg
 hihrohdrtghomheqnecuggftrfgrthhtvghrnhepvdehudduieektedugeevgfeutdffff
 dujeeifffhudehvdelveduvedtfefhtdejnecukfhppeejgedrudduiedrudekiedrgeeg
 necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshgrmh
 hplhgvthesnhhghihrohdrtghomh
X-ME-Proxy: <xmx:FHRZYCKw3yT6Kp75_05dZfoF0i-EwgBdJU6paPS82UrRWm2I9r6_vw>
 <xmx:FHRZYKH2t8sc_OfKxgxjiohl5MC4af54HATL7buOdjRTm-x2iZZaPg>
 <xmx:FHRZYOVsmDskESdVvnTflZEB5Mq0xbfUXjhjIBnx4RWJmF0rNPTNdg>
 <xmx:FXRZYJB1N25-tQV0A4oZimmTrMM_V4SSLzfJMXLGFHV4SEphx6mNyA>
Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 738071080054;
 Tue, 23 Mar 2021 00:52:36 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
Date: Tue, 23 Mar 2021 00:52:12 -0400
Message-Id: <20210323045213.9419-1-samplet@HIDDEN>
X-Mailer: git-send-email 2.31.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

* guix/swh.scm (swh-directory-download): New procedure (with
implementation extracted from 'swh-download').
(swh-download): Use it to download the revision directory.
---
 guix/swh.scm | 65 +++++++++++++++++++++++++++++-----------------------
 1 file changed, 36 insertions(+), 29 deletions(-)

diff --git a/guix/swh.scm b/guix/swh.scm
index f11b7ea2d5..2402ec98e6 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -108,6 +108,7 @@
 
             commit-id?
 
+            swh-download-directory
             swh-download))
 
 ;;; Commentary:
@@ -558,12 +559,6 @@ requested bundle cooking, waiting for completion...~%"))
 ;;; High-level interface.
 ;;;
 
-(define (commit-id? reference)
-  "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if
-it is a tag name.  This is based on a simple heuristic so use with care!"
-  (and (= (string-length reference) 40)
-       (string-every char-set:hex-digit reference)))
-
 (define (call-with-temporary-directory proc)      ;FIXME: factorize
   "Call PROC with a name of a temporary directory; close the directory and
 delete it when leaving the dynamic extent of this call."
@@ -577,6 +572,39 @@ delete it when leaving the dynamic extent of this call."
       (lambda ()
         (false-if-exception (delete-file-recursively tmp-dir))))))
 
+(define* (swh-download-directory id output
+                                 #:key (log-port (current-error-port)))
+  "Download from Software Heritage the directory with the given ID, and
+unpack it to OUTPUT.  Return #t on success and #f on failure"
+  (call-with-temporary-directory
+   (lambda (directory)
+     (match (vault-fetch id 'directory #:log-port log-port)
+       (#f
+        (format log-port
+                "SWH: directory ~a could not be fetched from the vault~%"
+                id)
+        #f)
+       ((? port? input)
+        (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-")))
+          (dump-port input tar)
+          (close-port input)
+          (let ((status (close-pipe tar)))
+            (unless (zero? status)
+              (error "tar extraction failure" status)))
+
+          (match (scandir directory)
+            (("." ".." sub-directory)
+             (copy-recursively (string-append directory "/" sub-directory)
+                               output
+                               #:log (%make-void-port "w"))
+             #t))))))))
+
+(define (commit-id? reference)
+  "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if
+it is a tag name.  This is based on a simple heuristic so use with care!"
+  (and (= (string-length reference) 40)
+       (string-every char-set:hex-digit reference)))
+
 (define* (swh-download url reference output
                        #:key (log-port (current-error-port)))
   "Download from Software Heritage a checkout of the Git tag or commit
@@ -593,28 +621,7 @@ wait until it becomes available, which could take several minutes."
      (format log-port "SWH: found revision ~a with directory at '~a'~%"
              (revision-id revision)
              (swh-url (revision-directory-url revision)))
-     (call-with-temporary-directory
-      (lambda (directory)
-        (match (vault-fetch (revision-directory revision) 'directory
-                            #:log-port log-port)
-          (#f
-           (format log-port
-                   "SWH: directory ~a could not be fetched from the vault~%"
-                   (revision-directory revision))
-           #f)
-          ((? port? input)
-           (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-")))
-             (dump-port input tar)
-             (close-port input)
-             (let ((status (close-pipe tar)))
-               (unless (zero? status)
-                 (error "tar extraction failure" status)))
-
-             (match (scandir directory)
-               (("." ".." sub-directory)
-                (copy-recursively (string-append directory "/" sub-directory)
-                                  output
-                                  #:log (%make-void-port "w"))
-                #t))))))))
+     (swh-download-directory (revision-directory revision) output
+                             #:log-port log-port))
     (#f
      #f)))
-- 
2.31.0





Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#47336: [PATCH 2/2] download: Use Disarchive as a last resort.
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 23 Mar 2021 04:54:02 +0000
Resent-Message-ID: <handler.47336.B47336.16164752218171 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 47336 <at> debbugs.gnu.org
Cc: Timothy Sample <samplet@HIDDEN>
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.16164752218171
          (code B ref 47336); Tue, 23 Mar 2021 04:54:02 +0000
Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:53:41 +0000
Received: from localhost ([127.0.0.1]:59262 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOZ2y-00027i-R7
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:53:41 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:51937)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOZ2i-000276-Gd
 for 47336 <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:53:39 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id BCA45140F;
 Tue, 23 Mar 2021 00:53:18 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:53:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=Sh8vx4Va/ITkI6SHjW/BifjchdPveiaTGxtGoj7SZ
 yI=; b=rr9U1Tsdg37OXQSTnqsTREm3ajQAetH/38BeY8pNp0b45p9UqZgJKKqZ3
 Ee1wD0JjdU39XHa5sgY4qkYehCHXrzwGf0U6EYRD2vCH5B+fMllBeutBXksK68Xn
 dpxyKXZX7IdwcEYpPfV6MR2u4gvSLsHiVhxsCXBBltlpAEUoS/y2ywNrpV67Mcar
 rlUGCaM3+eABsrn9ZtG4SVYCtJRCdhpmsictRc8zp16rdDGtBts8acpmXGj05sRO
 3zGPzrB11LEl8n6MWVf5PXNksxAtyKsKRZH78r5h4id1jQC99R8tnZ+AL1BpGmkq
 M9yGmRv/gqrfkAo3U7euPQ2gnK/6g==
X-ME-Sender: <xms:PnRZYD1C6eA5uuUUumCmpMiZj-jRkkF7SMbuqfZP1Ewq3gqzQnRmuQ>
 <xme:PnRZYCEeBpMZwM6CJYoRhT4uezn9wUN_2bSWQZGPK2Olg1plmY8aOPqrRzvWAAMl1
 5i0kEOhraZwouAOTg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhggtgfgsehtke
 ertdertdejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvght
 sehnghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpeehtdeuteejkeevhfeftdeltd
 etjeejhfettefgleeigedtgfeguddtfffgtdevgeenucffohhmrghinhepnhhghihrohdr
 tghomhenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpe
 dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:PnRZYD7SeHR657BL3SDRRwWdpLnU0flAiW9rsvSyuYX4JVFCu0SDHA>
 <xmx:PnRZYI2kYP5lEf_s6Wy9K-Rd_u477TRncH_bPh8YPhB-wwOQyv0UkA>
 <xmx:PnRZYGGCPFfzCeQLjpAiXd1YrzjElFUjNpZK_2mtCcASpMQdC7A82A>
 <xmx:PnRZYByUCQpDUlpKSFBIkojd-qXdYnfNcN7PsCLNzFZx1JD_F6fjfA>
Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 257551080054;
 Tue, 23 Mar 2021 00:53:18 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
Date: Tue, 23 Mar 2021 00:52:13 -0400
Message-Id: <20210323045213.9419-2-samplet@HIDDEN>
X-Mailer: git-send-email 2.31.0
In-Reply-To: <20210323045213.9419-1-samplet@HIDDEN>
References: <20210323045213.9419-1-samplet@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

* guix/download.scm (%disarchive-mirrors): New variable.
(%disarchive-mirror-file): New variable.
(built-in-download): Add 'disarchive-mirrors' keyword argument and
pass its value along to the 'builtin:download' derivation.
(url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'.
* guix/scripts/perform-download.scm (perform-download): Read
Disarchive mirrors from the environment and pass them to
'url-fetch'.
* guix/build/download.scm (disarchive-fetch/any): New procedure.
(url-fetch): Add 'disarchive-mirrors' keyword argument, use it to
make a list of URIs, and use the new procedure to fetch the file if
all other methods fail.
---
 guix/build/download.scm           | 77 +++++++++++++++++++++++++++----
 guix/download.scm                 | 19 ++++++--
 guix/scripts/perform-download.scm |  7 ++-
 3 files changed, 89 insertions(+), 14 deletions(-)

diff --git a/guix/build/download.scm b/guix/build/download.scm
index a22d4064ca..f476d0f8ec 100644
--- a/guix/build/download.scm
+++ b/guix/build/download.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@HIDDEN>
 ;;; Copyright © 2015 Mark H Weaver <mhw@HIDDEN>
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@HIDDEN>
+;;; Copyright © 2021 Timothy Sample <samplet@HIDDEN>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -23,10 +24,12 @@
   #:use-module (web http)
   #:use-module ((web client) #:hide (open-socket-for-uri))
   #:use-module (web response)
+  #:use-module (guix base16)
   #:use-module (guix base64)
   #:use-module (guix ftp-client)
   #:use-module (guix build utils)
   #:use-module (guix progress)
+  #:use-module (guix swh)
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -626,10 +629,50 @@ Return a list of URIs."
     (else
      (list uri))))
 
+(define* (disarchive-fetch/any uris file
+                               #:key (timeout 10))
+  "Fetch a Disarchive specification from any of URIS, assemble it,
+and write the output to FILE."
+  (define (fetch-specification uris)
+    (any (lambda (uri)
+           (false-if-exception*
+            (let-values (((port size) (http-fetch uri
+                                                  #:verify-certificate? #t
+                                                  #:timeout timeout)))
+              (let ((specification (read port)))
+                (close-port port)
+                specification))))
+         uris))
+
+  (define (resolve addresses output)
+    (any (match-lambda
+           (('swhid swhid)
+            (match (string-split swhid #\:)
+              (("swh" "1" "dir" id)
+               (format #t "Downloading from Software Heritage...~%" file)
+               (false-if-exception*
+                (swh-download-directory id output)))
+              (_ #f)))
+           (_ #f))
+         addresses))
+
+  (match (and=> (resolve-module '(disarchive) #:ensure #f)
+                (lambda (disarchive)
+                  (cons (module-ref disarchive '%disarchive-log-port)
+                        (module-ref disarchive 'disarchive-assemble))))
+    (#f #f)
+    ((%disarchive-log-port . disarchive-assemble)
+     (format #t "Trying to use Disarchive to assemble ~a~%" file)
+     (match (fetch-specification uris)
+       (#f #f)
+       (spec (parameterize ((%disarchive-log-port (current-output-port)))
+               (disarchive-assemble spec file #:resolver resolve)))))))
+
 (define* (url-fetch url file
                     #:key
                     (timeout 10) (verify-certificate? #t)
                     (mirrors '()) (content-addressed-mirrors '())
+                    (disarchive-mirrors '())
                     (hashes '())
                     print-build-trace?)
   "Fetch FILE from URL; URL may be either a single string, or a list of
@@ -693,6 +736,17 @@ otherwise simply ignore them."
                               hashes))
                 content-addressed-mirrors))
 
+  (define disarchive-uris
+    (append-map (lambda (mirror)
+                  (map (match-lambda
+                         ((hash-algo . hash)
+                          (string->uri
+                           (string-append mirror
+                                          (symbol->string hash-algo) "/"
+                                          (bytevector->base16-string hash)))))
+                       hashes))
+                disarchive-mirrors))
+
   ;; Make this unbuffered so 'progress-report/file' works as expected.  'line
   ;; means '\n', not '\r', so it's not appropriate here.
   (setvbuf (current-output-port) 'none)
@@ -705,15 +759,18 @@ otherwise simply ignore them."
        (or (fetch uri file)
            (try tail)))
       (()
-       (format (current-error-port) "failed to download ~s from ~s~%"
-               file url)
-
-       ;; Remove FILE in case we made an incomplete download, for example due
-       ;; to ENOSPC.
-       (catch 'system-error
-         (lambda ()
-           (delete-file file))
-         (const #f))
-       #f))))
+       ;; If we are looking for a software archive, one last thing we
+       ;; can try is to use Disarchive to assemble it.
+       (or (disarchive-fetch/any disarchive-uris file #:timeout timeout)
+           (begin
+             (format (current-error-port) "failed to download ~s from ~s~%"
+                     file url)
+             ;; Remove FILE in case we made an incomplete download, for
+             ;; example due to ENOSPC.
+             (catch 'system-error
+               (lambda ()
+                 (delete-file file))
+               (const #f))
+             #f))))))
 
 ;;; download.scm ends here
diff --git a/guix/download.scm b/guix/download.scm
index 30f69c0325..72094e7318 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -406,12 +406,19 @@
   (plain-file "content-addressed-mirrors"
               (object->string %content-addressed-mirrors)))
 
+(define %disarchive-mirrors
+  '("https://disarchive.ngyro.com/"))
+
+(define %disarchive-mirror-file
+  (plain-file "disarchive-mirrors" (object->string %disarchive-mirrors)))
+
 (define built-in-builders*
   (store-lift built-in-builders))
 
 (define* (built-in-download file-name url
                             #:key system hash-algo hash
                             mirrors content-addressed-mirrors
+                            disarchive-mirrors
                             executable?
                             (guile 'unused))
   "Download FILE-NAME from URL using the built-in 'download' builder.  When
@@ -422,13 +429,16 @@ explicitly depend on Guile, GnuTLS, etc.  Instead, the daemon performs the
 download by itself using its own dependencies."
   (mlet %store-monad ((mirrors (lower-object mirrors))
                       (content-addressed-mirrors
-                       (lower-object content-addressed-mirrors)))
+                       (lower-object content-addressed-mirrors))
+                      (disarchive-mirrors (lower-object disarchive-mirrors)))
     (raw-derivation file-name "builtin:download" '()
                     #:system system
                     #:hash-algo hash-algo
                     #:hash hash
                     #:recursive? executable?
-                    #:sources (list mirrors content-addressed-mirrors)
+                    #:sources (list mirrors
+                                    content-addressed-mirrors
+                                    disarchive-mirrors)
 
                     ;; Honor the user's proxy and locale settings.
                     #:leaked-env-vars '("http_proxy" "https_proxy"
@@ -439,6 +449,7 @@ download by itself using its own dependencies."
                                  ("mirrors" . ,mirrors)
                                  ("content-addressed-mirrors"
                                   . ,content-addressed-mirrors)
+                                 ("disarchive-mirrors" . ,disarchive-mirrors)
                                  ,@(if executable?
                                        '(("executable" . "1"))
                                        '()))
@@ -492,7 +503,9 @@ name in the store."
                              #:executable? executable?
                              #:mirrors %mirror-file
                              #:content-addressed-mirrors
-                             %content-addressed-mirror-file)))))
+                             %content-addressed-mirror-file
+                             #:disarchive-mirrors
+                             %disarchive-mirror-file)))))
 
 (define* (url-fetch/executable url hash-algo hash
                                #:optional name
diff --git a/guix/scripts/perform-download.scm b/guix/scripts/perform-download.scm
index 8d409092ba..6889bcef79 100644
--- a/guix/scripts/perform-download.scm
+++ b/guix/scripts/perform-download.scm
@@ -54,7 +54,8 @@ actual output is different from that when we're doing a 'bmCheck' or
                        (output* "out")
                        (executable "executable")
                        (mirrors "mirrors")
-                       (content-addressed-mirrors "content-addressed-mirrors"))
+                       (content-addressed-mirrors "content-addressed-mirrors")
+                       (disarchive-mirrors "disarchive-mirrors"))
     (unless url
       (leave (G_ "~a: missing URL~%") (derivation-file-name drv)))
 
@@ -79,6 +80,10 @@ actual output is different from that when we're doing a 'bmCheck' or
                              (lambda (port)
                                (eval (read port) %user-module)))
                            '())
+                       #:disarchive-mirrors
+                       (if disarchive-mirrors
+                           (call-with-input-file disarchive-mirrors read)
+                           '())
                        #:hashes `((,algo . ,hash))
 
                        ;; Since DRV's output hash is known, X.509 certificate
-- 
2.31.0





Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#47336: Disarchive as a fallback for downloads
References: <87eeg6o50b.fsf@HIDDEN>
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 23 Mar 2021 05:12:01 +0000
Resent-Message-ID: <handler.47336.B47336.161647631510444 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 47336 <at> debbugs.gnu.org
Cc: control <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161647631510444
          (code B ref 47336); Tue, 23 Mar 2021 05:12:01 +0000
Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 05:11:55 +0000
Received: from localhost ([127.0.0.1]:59289 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOZKc-0002iN-TI
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 01:11:55 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:38467)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>)
 id 1lOZKa-0002i5-RD; Tue, 23 Mar 2021 01:11:53 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id E892813F4;
 Tue, 23 Mar 2021 01:11:46 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 01:11:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:from:in-reply-to
 :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=wybaihPhH17Op27Ve
 rjCZANcJ6v7ziL1qmf92ho7/Ps=; b=pmHzZxAHMWUl71ZqS+t3OMHmTjpXYYB/5
 OBvA0IC0vdBX7d7UovyCQ9k8lzWxH2hZplXrA0YljWIE/xYeCrWEZDWRj+697Q6Z
 JnjQLvtqDFTyfVxHDbb+95JU+rbiqW9zTp0tpx6Dv3AqmBF7ljILmRSEWVj+W34X
 Co2MTr72eglXrYSYE55r1R8H7qSQGUrqYtZd6uY7Ns+FIrwE1qM5wUUoh+fxt1Na
 0n+SX1vO1sCLM6G0JBA5NvTpw5qBD6ibI8DORF0c1I0WezqNPf4dMZ+jjUWnNQSL
 s57ULcbA55ObOtkqvIUvWnaPwT5cBydswLlf16E89TIHWdtdW98ew==
X-ME-Sender: <xms:knhZYG8aohIHYuEKd0HpVYjPVXktP1X29Shpynh5YsBqq42jFWoFwg>
 <xme:knhZYGsXD3ssi_klL5FACO-UqhnoDIspfizqNcqXZaS9cufPv3oPUE0owAT3hvNOc
 hA3Q-UafkuyKagEYQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgkedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffuffgjkfgfgggtsehttdertd
 dtredtnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn
 ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpedtheehudeltefgffdtgffhhedvvd
 fhffelfeetteffhfekueeglefhhfdvteeuveenucfkphepjeegrdduudeirddukeeirdeg
 geenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrg
 hmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:knhZYMBye0ym_2lxfaZocFAewiKK284EjnjDSJexnkyTXMA4gceeNg>
 <xmx:knhZYOfHiwdpxcRqBHbn73lYgyWM7DJfQY0e6BiRzKALiKebbOlw4g>
 <xmx:knhZYLNrVaMfVeC8b0u5MXYliegpVrlQSregMtpUN3A_80QG1qioJw>
 <xmx:knhZYKasuIPoRjl9U-LG9B64_ztchuCWXtz8ODwDIIAek0sUJSmhMQ>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1F221108005F;
 Tue, 23 Mar 2021 01:11:46 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
Date: Tue, 23 Mar 2021 01:11:45 -0400
In-Reply-To: bug's message of "Tue, 23 Mar 2021 04:53:56 +0000"
Message-ID: <8735wmo3n2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

reassign 47336 guix-patches
thanks

Oops!  I sent this to the wrong list.  My apologies.




Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 23 Mar 2021 05:11:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Mar 23 01:11:55 2021
Received: from localhost ([127.0.0.1]:59291 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOZKd-0002iP-3S
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 01:11:55 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:38467)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>)
 id 1lOZKa-0002i5-RD; Tue, 23 Mar 2021 01:11:53 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id E892813F4;
 Tue, 23 Mar 2021 01:11:46 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 01:11:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-type:date:from:in-reply-to
 :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=wybaihPhH17Op27Ve
 rjCZANcJ6v7ziL1qmf92ho7/Ps=; b=pmHzZxAHMWUl71ZqS+t3OMHmTjpXYYB/5
 OBvA0IC0vdBX7d7UovyCQ9k8lzWxH2hZplXrA0YljWIE/xYeCrWEZDWRj+697Q6Z
 JnjQLvtqDFTyfVxHDbb+95JU+rbiqW9zTp0tpx6Dv3AqmBF7ljILmRSEWVj+W34X
 Co2MTr72eglXrYSYE55r1R8H7qSQGUrqYtZd6uY7Ns+FIrwE1qM5wUUoh+fxt1Na
 0n+SX1vO1sCLM6G0JBA5NvTpw5qBD6ibI8DORF0c1I0WezqNPf4dMZ+jjUWnNQSL
 s57ULcbA55ObOtkqvIUvWnaPwT5cBydswLlf16E89TIHWdtdW98ew==
X-ME-Sender: <xms:knhZYG8aohIHYuEKd0HpVYjPVXktP1X29Shpynh5YsBqq42jFWoFwg>
 <xme:knhZYGsXD3ssi_klL5FACO-UqhnoDIspfizqNcqXZaS9cufPv3oPUE0owAT3hvNOc
 hA3Q-UafkuyKagEYQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgkedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffuffgjkfgfgggtsehttdertd
 dtredtnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn
 ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpedtheehudeltefgffdtgffhhedvvd
 fhffelfeetteffhfekueeglefhhfdvteeuveenucfkphepjeegrdduudeirddukeeirdeg
 geenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrg
 hmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:knhZYMBye0ym_2lxfaZocFAewiKK284EjnjDSJexnkyTXMA4gceeNg>
 <xmx:knhZYOfHiwdpxcRqBHbn73lYgyWM7DJfQY0e6BiRzKALiKebbOlw4g>
 <xmx:knhZYLNrVaMfVeC8b0u5MXYliegpVrlQSregMtpUN3A_80QG1qioJw>
 <xmx:knhZYKasuIPoRjl9U-LG9B64_ztchuCWXtz8ODwDIIAek0sUJSmhMQ>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1F221108005F;
 Tue, 23 Mar 2021 01:11:46 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
To: 47336 <at> debbugs.gnu.org
Subject: Re: bug#47336: Disarchive as a fallback for downloads
Date: Tue, 23 Mar 2021 01:11:45 -0400
In-Reply-To: bug's message of "Tue, 23 Mar 2021 04:53:56 +0000"
Message-ID: <8735wmo3n2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: control
Cc: control <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

reassign 47336 guix-patches
thanks

Oops!  I sent this to the wrong list.  My apologies.




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: zimoun <zimon.toutoune@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Tue, 23 Mar 2021 09:46:01 +0000
Resent-Message-ID: <handler.47336.B47336.161649270212351 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>, 47336 <at> debbugs.gnu.org, Mathieu Othacehe <othacehe@HIDDEN>
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161649270212351
          (code B ref 47336); Tue, 23 Mar 2021 09:46:01 +0000
Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 09:45:02 +0000
Received: from localhost ([127.0.0.1]:59537 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOdav-0003Cz-Cr
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 05:45:01 -0400
Received: from mail-wm1-f50.google.com ([209.85.128.50]:55998)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1lOdat-0003Ck-Fq
 for 47336 <at> debbugs.gnu.org; Tue, 23 Mar 2021 05:44:59 -0400
Received: by mail-wm1-f50.google.com with SMTP id 12so10656916wmf.5
 for <47336 <at> debbugs.gnu.org>; Tue, 23 Mar 2021 02:44:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:subject:in-reply-to:references:date:message-id:mime-version
 :content-transfer-encoding;
 bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=;
 b=a1kWhOty04rSkuZn+yOOy75QNSCxZU3tKaRoQzcBkwzUhLzQRGMidjhxwTCzmeQctS
 ujC55BPtDGxZVtFweOzRrWLjaea8ykqig+xmrybhktNCqYZZ9OwoS/Hx5fBtX7AKaHJs
 bSurhEv0RzAsjKnBE7drV8Hi2KxB7QCqXt0RfVdthLcy28j6WCVUSppEVi7ucmgMb2ph
 skLA9ocM+AYUU22B51RMDoyTuq4UwLQ7Ox9FMI2ApiZnNUunp+TRF+LWf6QxVKmXpgkB
 fUdTJFwHRzd6iuFDxCbUNBDAH2vITfyRpBbTFvGPJXcj1v3Vqm+xcmi2lixcnYGJmZsk
 nztw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:subject:in-reply-to:references:date
 :message-id:mime-version:content-transfer-encoding;
 bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=;
 b=j1s02Fyb9xGS3D9uIpCmDT7+9dPrkxnTE2BtP3W6bowPwzQN5MrTntBrrV4Xgzyo7h
 Hqv2mSJyi7t5AggIjot1kn4HxsvvFjfrMu8SMA+pp8gKHtWTCyYcZo3g66mud9s/7OYm
 KPmRsdWM+Sh21aEL/tghxI8x9Z9NHPH15tqpHGFDxQTfIFVCm3C5TwifjapaC8z62hIW
 Rr+atHIUYVtrEa/TRjvF7eBB1BYIoEmQfqcrLEhRWCV3y6LLpHrgSItWqVd68JJcw5aN
 Ck9Pn3VetH+QPk9n3tEgrbq2/xkqNGMInzDZUW6DPQ/9QqirvJAK9aYKVnFbda0jvHXP
 nu7w==
X-Gm-Message-State: AOAM531oRr/FTg6xo2XF6gFkoebVpwVtFhTSEAHbdnoCK9klky+yS3/9
 JcIQxjFzAdChFx4cNLG6q7YAG9RZo2I=
X-Google-Smtp-Source: ABdhPJzJrqraIVEvPjdWBzhyCgBvir/3yhgtfd13IxE/sysUfOFhuI40WjeXUCiXzi1VC5UM04lndw==
X-Received: by 2002:a1c:66c4:: with SMTP id a187mr2474403wmc.164.1616492693386; 
 Tue, 23 Mar 2021 02:44:53 -0700 (PDT)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id s83sm2074886wmf.26.2021.03.23.02.44.52
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 23 Mar 2021 02:44:53 -0700 (PDT)
From: zimoun <zimon.toutoune@HIDDEN>
In-Reply-To: <87eeg6o50b.fsf@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN>
Date: Tue, 23 Mar 2021 10:35:53 +0100
Message-ID: <86sg4mnreu.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Timothy,

(CC Mathieu to advice if it could be a feature of Cuirass.)


On Tue, 23 Mar 2021 at 00:42, Timothy Sample <samplet@HIDDEN> wrote:

> This patch series adds Disarchive assembly (backed by SWH lookup) as a
> fallback for downloads.

Awesome!


> You also need to make sure that regular downloads are unavailable.  I do
> this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Cu=
rl-fetch=E2=80=9D in
> =E2=80=9Cguix/build/download.scm=E2=80=9D.  I replace the usual list of U=
RLs with =E2=80=98()=E2=80=99:
>
>     (let try ((uri (append uri content-addressed-uris)))
>       (match '() ; uri
>         ...))
>
> Now you can ask Guix for a recent .tar.gz source package:
>
>     $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

Neat!  Now, there is a way to easily check the coverage, right?  Since
SWH is ingesting the tarball using <http://guix.gnu.org/sources.json>,
there is now a mean to report what Guix is able to rebuild.

>     Checking httpretty-1.0.5 digest... ok

What happens if it is not ok?

>     Assembling the tarball httpretty-1.0.5.tar
>     Checking httpretty-1.0.5.tar digest... ok
>     Assembling the Gzip file httpretty-1.0.5.tar.gz
>     Checking httpretty-1.0.5.tar.gz digest... ok
>     Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpret=
ty-1.0.5.tar.gz

Where is the assembly done?  In /tmp/, right?

>     successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpre=
tty-1.0.5.tar.gz.drv

Just to be sure, when does Guix check the integrity checksum?  I mean,
does Guix check the checksum after =E2=80=99disassemble=E2=80=99 re-assembl=
ed the source?


> First, it looks up the metadata on my server.  This is fine for a demo,
> but not what we want forever.  The patch series supports adding
> several

As we talked before, how does the database scale?  Do you have some
numbers for the current demo?  In order to try to extrapolate what does
it mean for a server to =C2=ABstore the metadata=C2=BB.

> mirrors for looking up the metadata.  In the past, we talked about
> putting everything on one or a few of the big Git hosting platforms like
> GitHub or Gitlab.  That way, it would be easily picked up by SWH and
> archived =E2=80=9Cforever=E2=80=9D.  Right now, I have Cuirass set up to =
build the
> metadata, and a little script that moves it from the build server to my
> Web server.  It would be simple enough to adjust that script to push it
> to a remote Git repo.  (Of course, the next step is to move this setup
> to Guix infrastructure.)  Thoughts?

Maybe this database could be a package, say =E2=80=9Cguix-tarball-db=E2=80=
=9D, updated
in agreement with the package =E2=80=9Cguix=E2=80=9D.  The source of this
=E2=80=9Cguix-tarball-db=E2=80=9D would be a remote big Git hosting platfor=
ms like
GitHub or whatever and not stored on Guix infrastructure, or maybe
stored on Guix infra.

Regularly, i.e., when the package =E2=80=9Cguix=E2=80=9D is updated, in the=
 same time,
the package =E2=80=9Cguix-tarball-db=E2=80=9D is updated too.  The =E2=80=
=9Cguix lint -c
archival=E2=80=9D sends the saving request to SWH.  Even if this saving req=
uest
should be automated soon. :-)

Then if Cuirass would have a feature to disassemble and update the Git
repo.

Last, a service should run as your demo.  But for long-term, this
service could disappear=E2=80=93=E2=80=93assuming SWH not :-).  Therefore, =
we could
imagine installing =E2=80=9Cguix-tarball-db=E2=80=9D then tweak some parame=
ters of the
guix-daemon and =E2=80=9Cguix build <foo>=E2=80=9D.  Both installing and bu=
ilding would
fetch from SWH if both upstream disappear.

Or this =E2=80=9Cguix-tarball-db=E2=80=9D should not be a plain package but=
 only an
input as origin for the package =E2=80=9Cguix=E2=80=9D.


> Hopefully everything else is more-or-less fine.  :)

Thanks!  That=E2=80=99s awesome!


Cheers,
simon




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Tue, 23 Mar 2021 14:32:02 +0000
Resent-Message-ID: <handler.47336.B47336.161650988425671 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: zimoun <zimon.toutoune@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161650988425671
          (code B ref 47336); Tue, 23 Mar 2021 14:32:02 +0000
Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 14:31:24 +0000
Received: from localhost ([127.0.0.1]:60913 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOi44-0006fy-BI
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 10:31:24 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:56299)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOi42-0006fk-Ml
 for 47336 <at> debbugs.gnu.org; Tue, 23 Mar 2021 10:31:23 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 842A05C00D7;
 Tue, 23 Mar 2021 10:31:17 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 10:31:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=MZOGcucNXJCcVOURYWb89EzsG/A48KOsBZjXvVtAo
 MM=; b=MQKGGVHnN1CXz9SGX+OTCpaG7PZ/x0DwKjIXqjWZcZ3Luo7adfatsuHAR
 j0mV3bvla//JvxvUlipL04UE5jFWH8DSnCOVqR9dBXJ9xCoL55XVT36h0lQwD8tT
 9a+8VDrAnjzIkIsYohhK257/H8AFRdPXF8ggD2tpfvNY1QySvlFGH9EnYb33Wpr9
 5g8Ner1rxahmOFLpTqbqUFQ61IxzXNloA/UyYJ8bUyXuoq9fTonPe1GzaIAh8jgY
 z8mOB8/v+dtQ11PURRpSCFWJOeBZee6nbuX/xCt8PRBF9dr19CfFaraYxG0y7iQo
 KrD7ZVIXJGMxyF5C4WgJJ+ymR0VNg==
X-ME-Sender: <xms:tPtZYMkA3gbb2t7Z62Hc8wNBYUUHW4phgWTYhr8hPcQZ9_UUr_MAKg>
 <xme:tPtZYL0HfbDWnpm24LedtnddxLGBisTAwJTfr8-R9WzCIMWHyMGjJo12HDF65ah6a
 eMa_7yhI7ncpA17Dw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudegiedgieeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho
 thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf
 grthhtvghrnhepledvlefhtdfhgfeuhfekiefhuefhfeevieeuheeuuedufefhudegffeg
 vedvjeefnecuffhomhgrihhnpehgnhhurdhorhhgpdhnghihrhhordgtohhmpdehnecukf
 hppeejgedrudduiedrudekiedrgeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr
 rghmpehmrghilhhfrhhomhepshgrmhhplhgvthesnhhghihrohdrtghomh
X-ME-Proxy: <xmx:tPtZYKpxzHnMPHfveiQ5WuoobX1j3ezM9NvajhhbXwy6G3yiaVV0QA>
 <xmx:tPtZYImNxgNk1ekcT4wk2eZDFwCgXHwin0cQ3gfL6HEZmUBSRfT_fg>
 <xmx:tPtZYK0dQ4Pdrxeb0jo6xK1iWZhUaJRkkEjiIee2NuTImmVXdpC95w>
 <xmx:tftZYP9Z0tm4H9x_CLcAUbtUfkmJiZPtpCqjkuff8LpJXn-u7jal3g>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 6D8F51080066;
 Tue, 23 Mar 2021 10:31:16 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN> <86sg4mnreu.fsf@HIDDEN>
Date: Tue, 23 Mar 2021 10:31:15 -0400
In-Reply-To: <86sg4mnreu.fsf@HIDDEN> (zimoun's message of "Tue, 23 Mar 2021
 10:35:53 +0100")
Message-ID: <87sg4mt00c.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi zimoun,

You make a lot of good points here.  Let me at least provide some quick
answers even if I=E2=80=99m not ready to comment on some of the bigger pict=
ure
stuff.

zimoun <zimon.toutoune@HIDDEN> writes:

> (CC Mathieu to advice if it could be a feature of Cuirass.)

So far I have been using Cuirass with only a tiny patch.  I=E2=80=99m not s=
ure
we need anything more than what Cuirass already provides.  (The tiny
patch is for allowing sorting the =E2=80=9Clatestbuilds=E2=80=9D results by=
 =E2=80=9Cstoptime=E2=80=9D
and =E2=80=9Cid=E2=80=9D.  This in turn allows paging through all the build=
s from the
API.)

> On Tue, 23 Mar 2021 at 00:42, Timothy Sample <samplet@HIDDEN> wrote:
>
>> Now you can ask Guix for a recent .tar.gz source package:
>>
>>     $ ./pre-inst-env guix build --no-substitutes -S python-httpretty
>
> Neat!  Now, there is a way to easily check the coverage, right?  Since
> SWH is ingesting the tarball using <http://guix.gnu.org/sources.json>,
> there is now a mean to report what Guix is able to rebuild.

I=E2=80=99m not sure I fully understand.  Disarchive covers about 4,300 Gzi=
p=E2=80=99ed
tarballs (no XZ yet).  There are about 100 for which compression
parameters cannot be found, and a handful (about 5) that have a
particularly funny idea about what a tarball is.  The metadata builds
for my database started one week ago and have been continuously updating
since then.

Are you asking if we could check what SWH has?  Yes!  Each metadata
file contains the SWHID of the input directory.  You could use
Disarchive to get this value or a simple =E2=80=9Cgrep swhid=E2=80=9D would=
 do it.  :)

    $ curl https://disarchive.ngyro.com/sha256/67989614004773db349791c37675=
efb914d084bdb221356a05e4369c35e7eb62 | grep swhid

It would be neat to have a big database of archive coverage from Guix
1.0 through to the present.  It=E2=80=99s quite a big project though.

Of course, you know all about the SWH rate limit....

>>     Checking httpretty-1.0.5 digest... ok
>
> What happens if it is not ok?

For that particular digest, it means the source directory is wrong.
Since we get the source from SWH, it means that the SWH archive is
wrong.  You will have to look elsewhere, I guess (this seems pretty
unlikely).  (There is a vanishing possibility that Disarchive
miscomputed the SWHID and managed to come up with a different, but still
valid SWHID....)

The other digest checks are more likely to fail.  They would indicate
that Disarchive no longer knows how to interpret the metadata.  Maybe
there will be a subtle bug in Disarchive 0.3.0 that causes this.  Either
use an old version of Disarchive or try to fix the current version.  :)
I worry about this, because it would be annoying, but the metadata does
have all the information needed to recover the original archive, so
nothing is really lost (except the user=E2=80=99s time).

>>     Assembling the tarball httpretty-1.0.5.tar
>>     Checking httpretty-1.0.5.tar digest... ok
>>     Assembling the Gzip file httpretty-1.0.5.tar.gz
>>     Checking httpretty-1.0.5.tar.gz digest... ok
>>     Copying result to
>> /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
>
> Where is the assembly done?  In /tmp/, right?

Yes.

>>     successfully built
>> /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv
>
> Just to be sure, when does Guix check the integrity checksum?  I mean,
> does Guix check the checksum after =E2=80=99disassemble=E2=80=99 re-assem=
bled the source?

Disarchive checks the result against the metadata to make sure it didn=E2=
=80=99t
make a mistake.  Guix also checks the final result to make sure the
fixed-output derivation is correct.  A fixed-output derivation is
basically just a checksum with a hint about how the data can be
obtained.  Guix really only cares about the checksum, the hint can do
whatever as long as it produces the result Guix wants.  With this patch
series, Disarchive is part of the hint.

>> First, it looks up the metadata on my server.  This is fine for a demo,
>> but not what we want forever.  The patch series supports adding
>> several
>
> As we talked before, how does the database scale?  Do you have some
> numbers for the current demo?  In order to try to extrapolate what does
> it mean for a server to =C2=ABstore the metadata=C2=BB.

With =E2=80=9Cgzip -9=E2=80=9D, the average metadata file is 6.8KiB.  It=E2=
=80=99s pretty
manageable.  There=E2=80=99s room for improvement on the Disarchive side, t=
oo.
It still stores some redundant information.  Uncompressed, it=E2=80=99s more
like 112KiB per file.  This is still pretty okay, really.  It means we
might hit tens of GiB over a couple years.  (It would take just over
100GiB to store a million uncompressed metadata files.)  The compression
ratio is what drove me to skip Git for now.

>> mirrors for looking up the metadata.  In the past, we talked about
>> putting everything on one or a few of the big Git hosting platforms like
>> GitHub or Gitlab.  That way, it would be easily picked up by SWH and
>> archived =E2=80=9Cforever=E2=80=9D.  Right now, I have Cuirass set up to=
 build the
>> metadata, and a little script that moves it from the build server to my
>> Web server.  It would be simple enough to adjust that script to push it
>> to a remote Git repo.  (Of course, the next step is to move this setup
>> to Guix infrastructure.)  Thoughts?
>
> Maybe this database could be a package, say =E2=80=9Cguix-tarball-db=E2=
=80=9D, updated
> in agreement with the package =E2=80=9Cguix=E2=80=9D.  The source of this
> =E2=80=9Cguix-tarball-db=E2=80=9D would be a remote big Git hosting platf=
orms like
> GitHub or whatever and not stored on Guix infrastructure, or maybe
> stored on Guix infra.
>
> Regularly, i.e., when the package =E2=80=9Cguix=E2=80=9D is updated, in t=
he same time,
> the package =E2=80=9Cguix-tarball-db=E2=80=9D is updated too.  The =E2=80=
=9Cguix lint -c
> archival=E2=80=9D sends the saving request to SWH.  Even if this saving r=
equest
> should be automated soon. :-)
>
> Then if Cuirass would have a feature to disassemble and update the Git
> repo.
>
> Last, a service should run as your demo.  But for long-term, this
> service could disappear=E2=80=93=E2=80=93assuming SWH not :-).  Therefore=
, we could
> imagine installing =E2=80=9Cguix-tarball-db=E2=80=9D then tweak some para=
meters of the
> guix-daemon and =E2=80=9Cguix build <foo>=E2=80=9D.  Both installing and =
building would
> fetch from SWH if both upstream disappear.
>
> Or this =E2=80=9Cguix-tarball-db=E2=80=9D should not be a plain package b=
ut only an
> input as origin for the package =E2=80=9Cguix=E2=80=9D.

This is an interesting idea, but one that I would have to think about
more.  :)


-- Tim




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sat, 27 Mar 2021 10:40:01 +0000
Resent-Message-ID: <handler.47336.B47336.16168415729325 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 47336 <at> debbugs.gnu.org, zimoun <zimon.toutoune@HIDDEN>
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.16168415729325
          (code B ref 47336); Sat, 27 Mar 2021 10:40:01 +0000
Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:39:32 +0000
Received: from localhost ([127.0.0.1]:43457 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lQ6Ls-0002QL-Ay
	for submit <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:39:32 -0400
Received: from eggs.gnu.org ([209.51.188.92]:33608)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lQ6Lp-0002Q7-Hf
 for 47336 <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:39:30 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34494)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lQ6Lj-00046H-Lv; Sat, 27 Mar 2021 06:39:23 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41636 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1lQ6Lh-0003Y1-1l; Sat, 27 Mar 2021 06:39:22 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN> <86sg4mnreu.fsf@HIDDEN>
 <87sg4mt00c.fsf@HIDDEN>
Date: Sat, 27 Mar 2021 11:39:18 +0100
In-Reply-To: <87sg4mt00c.fsf@HIDDEN> (Timothy Sample's message of "Tue, 23
 Mar 2021 10:31:15 -0400")
Message-ID: <87eeg0284p.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi!

Timothy Sample <samplet@HIDDEN> skribis:

> With =E2=80=9Cgzip -9=E2=80=9D, the average metadata file is 6.8KiB.  It=
=E2=80=99s pretty
> manageable.  There=E2=80=99s room for improvement on the Disarchive side,=
 too.
> It still stores some redundant information.  Uncompressed, it=E2=80=99s m=
ore
> like 112KiB per file.  This is still pretty okay, really.  It means we
> might hit tens of GiB over a couple years.  (It would take just over
> 100GiB to store a million uncompressed metadata files.)  The compression
> ratio is what drove me to skip Git for now.

If needed, the sexp serialization could still be made more compact:
using =E2=80=98write=E2=80=99 instead of =E2=80=98pretty-print=E2=80=99, sh=
ortening field names (but
that=E2=80=99d be incompatible).

We could also use CBOR or canonical sexp serialization, though maybe
gzipped sexps are more compact than what we could achieve?

Anyway, these are surface syntax optimizations that can always be made
at a later point in time when we feel a need for them.

Ludo=E2=80=99.




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sat, 27 Mar 2021 10:41:01 +0000
Resent-Message-ID: <handler.47336.B47336.16168416259432 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>
Cc: 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.16168416259432
          (code B ref 47336); Sat, 27 Mar 2021 10:41:01 +0000
Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:40:25 +0000
Received: from localhost ([127.0.0.1]:43461 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lQ6Mj-0002S3-K7
	for submit <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:40:25 -0400
Received: from eggs.gnu.org ([209.51.188.92]:33754)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lQ6Mh-0002Rp-A1
 for 47336 <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:40:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34499)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lQ6Mc-0004bG-37; Sat, 27 Mar 2021 06:40:18 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41638 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1lQ6Ma-0006GK-4Q; Sat, 27 Mar 2021 06:40:17 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN>
Date: Sat, 27 Mar 2021 11:40:14 +0100
In-Reply-To: <20210323045213.9419-1-samplet@HIDDEN> (Timothy Sample's
 message of "Tue, 23 Mar 2021 00:52:12 -0400")
Message-ID: <87a6qo2835.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Timothy Sample <samplet@HIDDEN> skribis:

> * guix/swh.scm (swh-directory-download): New procedure (with
> implementation extracted from 'swh-download').
> (swh-download): Use it to download the revision directory.

LGTM!




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sat, 27 Mar 2021 10:58:02 +0000
Resent-Message-ID: <handler.47336.B47336.161684264210939 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>
Cc: 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161684264210939
          (code B ref 47336); Sat, 27 Mar 2021 10:58:02 +0000
Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:57:22 +0000
Received: from localhost ([127.0.0.1]:43471 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lQ6d8-0002qN-Ey
	for submit <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:57:22 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36022)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lQ6d6-0002q9-Om
 for 47336 <at> debbugs.gnu.org; Sat, 27 Mar 2021 06:57:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34573)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lQ6d1-0006gL-8A; Sat, 27 Mar 2021 06:57:15 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41646 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1lQ6d0-00007a-Mw; Sat, 27 Mar 2021 06:57:15 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20210323045213.9419-1-samplet@HIDDEN>
 <20210323045213.9419-2-samplet@HIDDEN>
Date: Sat, 27 Mar 2021 11:57:12 +0100
In-Reply-To: <20210323045213.9419-2-samplet@HIDDEN> (Timothy Sample's
 message of "Tue, 23 Mar 2021 00:52:13 -0400")
Message-ID: <87y2e8zwxj.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi!

Timothy Sample <samplet@HIDDEN> skribis:

> * guix/download.scm (%disarchive-mirrors): New variable.
> (%disarchive-mirror-file): New variable.
> (built-in-download): Add 'disarchive-mirrors' keyword argument and
> pass its value along to the 'builtin:download' derivation.
> (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'.
> * guix/scripts/perform-download.scm (perform-download): Read
> Disarchive mirrors from the environment and pass them to
> 'url-fetch'.
> * guix/build/download.scm (disarchive-fetch/any): New procedure.
> (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to
> make a list of URIs, and use the new procedure to fetch the file if
> all other methods fail.

[...]

> +  #:use-module (guix base16)
>    #:use-module (guix base64)
>    #:use-module (guix ftp-client)
>    #:use-module (guix build utils)
>    #:use-module (guix progress)
> +  #:use-module (guix swh)

Maybe #:autoload them.

> +(define* (disarchive-fetch/any uris file
> +                               #:key (timeout 10))
> +  "Fetch a Disarchive specification from any of URIS, assemble it,
> +and write the output to FILE."
> +  (define (fetch-specification uris)
> +    (any (lambda (uri)
> +           (false-if-exception*
> +            (let-values (((port size) (http-fetch uri
> +                                                  #:verify-certificate? =
#t
> +                                                  #:timeout timeout)))

Perhaps add #:key (verify-certificate? #t) and have the caller pass it?
Currently (guix scripts perform-download) sets it to #f, which is a good
idea IMO.

> +  (match (and=3D> (resolve-module '(disarchive) #:ensure #f)
> +                (lambda (disarchive)
> +                  (cons (module-ref disarchive '%disarchive-log-port)
> +                        (module-ref disarchive 'disarchive-assemble))))
> +    (#f #f)
> +    ((%disarchive-log-port . disarchive-assemble)
> +     (format #t "Trying to use Disarchive to assemble ~a~%" file)
> +     (match (fetch-specification uris)
> +       (#f #f)
> +       (spec (parameterize ((%disarchive-log-port (current-output-port)))
> +               (disarchive-assemble spec file #:resolver resolve)))))))

So we would normally arrange so that the =E2=80=98guix=E2=80=99 package dep=
ends on
Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call works=
 when done
via =E2=80=98guix perform-download=E2=80=99, right?

In the #f case, perhaps we should print something like =E2=80=9CDisarchive =
not
found, bailing out=E2=80=9D?

That=E2=80=99s all I have to say; it looks great to me!

That=E2=80=99s quite a milestone, it=E2=80=99d be great to have that in the=
 upcoming
release.  Next we can discuss how to populate the Disarchive database
and where to do that (or your hosting fees could easily skyrocket :-)).
I suppose we could run that in Berlin and/or we could make an argument
about using SWH or Inria resources for that.

Thanks,
Ludo=E2=80=99.




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sat, 10 Apr 2021 20:53:02 +0000
Resent-Message-ID: <handler.47336.B47336.161808796210630 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>
Cc: 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161808796210630
          (code B ref 47336); Sat, 10 Apr 2021 20:53:02 +0000
Received: (at 47336) by debbugs.gnu.org; 10 Apr 2021 20:52:42 +0000
Received: from localhost ([127.0.0.1]:53418 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lVKaw-0002lO-1b
	for submit <at> debbugs.gnu.org; Sat, 10 Apr 2021 16:52:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47430)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1lVKau-0002l9-OE
 for 47336 <at> debbugs.gnu.org; Sat, 10 Apr 2021 16:52:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:50810)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1lVKap-00022A-DR; Sat, 10 Apr 2021 16:52:35 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=42156 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1lVKao-000323-Sf; Sat, 10 Apr 2021 16:52:35 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN> <87a6qo2835.fsf_-_@HIDDEN>
Date: Sat, 10 Apr 2021 22:52:33 +0200
In-Reply-To: <87a6qo2835.fsf_-_@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 27 Mar 2021 11:40:14 +0100")
Message-ID: <87blal2772.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ping!  :-)

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> Timothy Sample <samplet@HIDDEN> skribis:
>
>> * guix/swh.scm (swh-directory-download): New procedure (with
>> implementation extracted from 'swh-download').
>> (swh-download): Use it to download the revision directory.
>
> LGTM!




Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 26 Apr 2021 09:51:02 +0000
Resent-Message-ID: <handler.47336.B47336.161943061512602 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Timothy Sample <samplet@HIDDEN>
Cc: 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161943061512602
          (code B ref 47336); Mon, 26 Apr 2021 09:51:02 +0000
Received: (at 47336) by debbugs.gnu.org; 26 Apr 2021 09:50:15 +0000
Received: from localhost ([127.0.0.1]:44619 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1laxsd-0003HC-7V
	for submit <at> debbugs.gnu.org; Mon, 26 Apr 2021 05:50:15 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36402)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1laxsa-0003Gx-TV
 for 47336 <at> debbugs.gnu.org; Mon, 26 Apr 2021 05:50:14 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:52159)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1laxsV-0002Az-Gf; Mon, 26 Apr 2021 05:50:07 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55298 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1laxsL-0005MF-CU; Mon, 26 Apr 2021 05:50:06 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN> <87a6qo2835.fsf_-_@HIDDEN>
Date: Mon, 26 Apr 2021 11:49:55 +0200
In-Reply-To: <87a6qo2835.fsf_-_@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 27 Mar 2021 11:40:14 +0100")
Message-ID: <87v989e5oc.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Timothy,

Ping=C2=B2!

Let me know if you=E2=80=99re like me to apply the patches on your behalf.

Ludo=E2=80=99.

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> Timothy Sample <samplet@HIDDEN> skribis:
>
>> * guix/swh.scm (swh-directory-download): New procedure (with
>> implementation extracted from 'swh-download').
>> (swh-download): Use it to download the revision directory.
>
> LGTM!




Message sent:


MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Timothy Sample <samplet@HIDDEN>
Subject: bug#47336: closed (Disarchive as a fallback for downloads)
CC: tracker <at> debbugs.gnu.org
Message-ID: <handler.47336.D47336.161957705128544.ackdone <at> debbugs.gnu.org>
References: <87o8dzkunk.fsf_-_@HIDDEN> <87eeg6o50b.fsf@HIDDEN>
X-Gnu-PR-Message: closed 47336
X-Gnu-PR-Package: guix-patches
Date: Wed, 28 Apr 2021 02:31:02 +0000
Content-Type: multipart/mixed; boundary="----------=_1619577062-28565-0"

This is a multi-part message in MIME format...

------------=_1619577062-28565-0
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=utf-8

Your message dated Tue, 27 Apr 2021 22:30:39 -0400
with message-id <87o8dzkunk.fsf_-_@HIDDEN>
and subject line Re: bug#47336: Disarchive as a fallback for downloads
has caused the debbugs.gnu.org bug report #47336,
regarding Disarchive as a fallback for downloads
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs@HIDDEN)


--=20
47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47336
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

------------=_1619577062-28565-0
Content-Type: message/rfc822
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000
Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOYs3-0001pw-WB
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400
Received: from lists.gnu.org ([209.51.188.17]:51240)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOYs2-0001pp-0u
 for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59600)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYs1-0007wV-QR
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYry-0007rZ-7A
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id D60141373
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=content-transfer-encoding:content-type
 :date:from:message-id:mime-version:subject:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC
 aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6
 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0
 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5
 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/
 YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl
 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA
 ==
X-ME-Sender: <xms:pnFZYL4J4-5z0UZC1ft_G_1Nhsyy0ABL5PdIDX1VYRe_v_ZNezhl0Q>
 <xme:pnFZYA72ibiOrA_3NYhBmyqncC-HLuV5_9ESOPa2h89glclhmwXbINrXztgsUVuiS
 Nn5X2IxUhkya46jtA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd
 dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn
 ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf
 evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg
 rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh
 grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:pnFZYCewFnYM7r_tIF4Gu6oiXAXAN2KxThYXRJT-H0FxCRxRUwWJ1g>
 <xmx:pnFZYMJ2wySGLcQuD8EahxJhuyn31BqGxftQ0WUWVl2XhDswjbEdJA>
 <xmx:pnFZYPKgBzcoSJayN2aN5uB3u8JxwErxvQGYMGsUOtf1v3urevXYcw>
 <xmx:pnFZYEWuF8VgmOhBe8DcSbP1nEDNVy-Fk6x0yrhtHdZww14o8f3bug>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
To: bug-guix@HIDDEN
Subject: Disarchive as a fallback for downloads
Date: Tue, 23 Mar 2021 00:42:12 -0400
Message-ID: <87eeg6o50b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@HIDDEN;
 helo=wout3-smtp.messagingengine.com
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.6 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)

Hello,

This patch series adds Disarchive assembly (backed by SWH lookup) as a
fallback for downloads.

To try it, make sure you are running the daemon in an environment with
Disarchive available:

    $ ./pre-inst-env guix environment --ad-hoc guile disarchive
    # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild

Don=E2=80=99t forget to stop your existing Guix Daemon.  :)

You also need to make sure that regular downloads are unavailable.  I do
this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl=
-fetch=E2=80=9D in
=E2=80=9Cguix/build/download.scm=E2=80=9D.  I replace the usual list of URL=
s with =E2=80=98()=E2=80=99:

    (let try ((uri (append uri content-addressed-uris)))
      (match '() ; uri
        ...))

Now you can ask Guix for a recent .tar.gz source package:

    $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

You should see:

    Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v=
diwlv19y9-httpretty-1.0.5.tar.gz
    Assembling the directory httpretty-1.0.5
    Downloading from Software Heritage...
    7903d608efc89c14afb4d692a3721156e31a43e2/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
    [...]
    Checking httpretty-1.0.5 digest... ok
    Assembling the tarball httpretty-1.0.5.tar
    Checking httpretty-1.0.5.tar digest... ok
    Assembling the Gzip file httpretty-1.0.5.tar.gz
    Checking httpretty-1.0.5.tar.gz digest... ok
    Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty=
-1.0.5.tar.gz
    successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett=
y-1.0.5.tar.gz.drv

There=E2=80=99s lots to talk about though....

First, it looks up the metadata on my server.  This is fine for a demo,
but not what we want forever.  The patch series supports adding several
mirrors for looking up the metadata.  In the past, we talked about
putting everything on one or a few of the big Git hosting platforms like
GitHub or Gitlab.  That way, it would be easily picked up by SWH and
archived =E2=80=9Cforever=E2=80=9D.  Right now, I have Cuirass set up to bu=
ild the
metadata, and a little script that moves it from the build server to my
Web server.  It would be simple enough to adjust that script to push it
to a remote Git repo.  (Of course, the next step is to move this setup
to Guix infrastructure.)  Thoughts?

On the code level, there were two things I couldn=E2=80=99t figure out for
myself.

I made the mirror list just simple strings.  AIUI, the client and the
daemon have to agree about the format of the mirror list.  Given that
running old daemons is common, changing the format is difficult.  Is it
worth it to copy the more flexible interface used by the content
addressed mirrors?  If yes, do I have to do the same =E2=80=98module-autolo=
ad!=E2=80=99
dance to use =E2=80=98bytevector->base16-string=E2=80=99?  :)  (I probably =
would have
just copied it, but that part confused me a bit.)

I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel=
l, just
=E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D).  It feels weird to use=
 a bunch of host-side modules
from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module.  This =
is okay because
=E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. =
 It=E2=80=99s more
like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2=
=80=9D, which
is why importing host-side modules is OK... right?

Hopefully everything else is more-or-less fine.  :)


-- Tim



------------=_1619577062-28565-0
Content-Type: message/rfc822
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Received: (at 47336-done) by debbugs.gnu.org; 28 Apr 2021 02:30:51 +0000
Received: from localhost ([127.0.0.1]:51379 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lbZyU-0007QK-Ou
	for submit <at> debbugs.gnu.org; Tue, 27 Apr 2021 22:30:50 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:46999)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lbZyR-0007Pz-0T
 for 47336-done <at> debbugs.gnu.org; Tue, 27 Apr 2021 22:30:48 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id 6D226EA3;
 Tue, 27 Apr 2021 22:30:41 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 27 Apr 2021 22:30:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=4XqfWTcO+597/Uu3CUK5wZtE/MGAF4i4wRDNBVMPa
 PE=; b=dnUGU82lUM+ZvPwwEaP024vphaFUxxnh2jZ0RakQDnp5zCitB4Ze3qUBm
 6FnYQbD6G3F0WiTBY5HdhCfTZTP59mJKEaKkMnHrowEAFUO7ZnFC171Y+Zxypmxw
 pY7gCYMA2h8y3lPh/oQNqwYjJnw/DHS6/g/uUbc+rvKld6F0fstpyD+iqAD4A6Dh
 g0KKAstsgtiACMuKqCh+iylNoRabaPhW+nPtXH47IovAArQuirOcL0IVVBz371S+
 E+f1SplzgwrurRxuo8BeMLHDvLgb48RvIyCt442JytOAjR74SwzdY7JxG0h7a4ZF
 XjBzmPCr3YxZn3X3YGXgbAoRxE73g==
X-ME-Sender: <xms:0MiIYEaBVTwlNOpF8POEMXnb2bSUC_hWJU9IzCG904jY6WPuvF3GKg>
 <xme:0MiIYPYKC6s34__K3MIEyngHUIr3CfOVqQM035khW_-gdMxe7CkhD7L0K-myzoMx2
 uLZE3zT6ddpbUEUaQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvuddgheelucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho
 thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf
 grthhtvghrnhepvdelvefhheegudevjedvfedtveeileffveehveejgefgteetteejteel
 tdefueevnecuffhomhgrihhnpehgnhhurdhorhhgnecukfhppeejgedrudduiedrudekie
 drgeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep
 shgrmhhplhgvthesnhhghihrohdrtghomh
X-ME-Proxy: <xmx:0MiIYO8GtSUzpVhOEXfLd8igopqwJL431w_h2q8PGeDKL-YDEgVEgQ>
 <xmx:0MiIYOoZhp3vtkfg5O8nm0DdXRp5M9-WrP7axZGVi0qrun3SCJXGog>
 <xmx:0MiIYPoIXZnQfnhSF5iUuYwNUp8BZJIxnci1q8ZyiRlGZyM5i7UnUQ>
 <xmx:0ciIYJFmjbcPLm7EVDSteLyPPbatadNMSiK88c7CLC__uLhFdgYp5A>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA;
 Tue, 27 Apr 2021 22:30:40 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#47336: Disarchive as a fallback for downloads
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN> <87a6qo2835.fsf_-_@HIDDEN>
 <87v989e5oc.fsf_-_@HIDDEN>
Date: Tue, 27 Apr 2021 22:30:39 -0400
In-Reply-To: <87v989e5oc.fsf_-_@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s?=
 =?utf-8?Q?=22's?= message of "Mon, 26 Apr 2021 11:49:55 +0200")
Message-ID: <87o8dzkunk.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 47336-done
Cc: 47336-done <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Ping=C2=B2!
>
> Let me know if you=E2=80=99re like me to apply the patches on your behalf.

No, no.  I=E2=80=99m just a little distracted over here.  I just pushed this
series with the updates you suggested (using #:autoload, passing
#:verify-certificates?, and being a bit more chatty).  Sorry for the
delay and thanks for the reminder.

Next, I=E2=80=99ll convert my Cuirass 0.x setup to a Cuirass 1.x setup, and=
 then
I can start a discussion about moving the metadata builds to
ci.guix.gnu.org.

Also, to answer your other question:

> So we would normally arrange so that the =E2=80=98guix=E2=80=99 package d=
epends on
> Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call wor=
ks when done
> via =E2=80=98guix perform-download=E2=80=99, right?

That=E2=80=99s the idea.  I=E2=80=99m not confident about updating the =E2=
=80=98guix=E2=80=99 package
myself, though....


-- Tim


------------=_1619577062-28565-0--


Message sent:


MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Timothy Sample <samplet@HIDDEN>
Subject: bug#47336: closed (Re: bug#47336: Disarchive as a fallback for
 downloads)
Message-ID: <handler.47336.D47336.161957705128544.notifdone <at> debbugs.gnu.org>
References: <87o8dzkunk.fsf_-_@HIDDEN> <87eeg6o50b.fsf@HIDDEN>
X-Gnu-PR-Message: they-closed 47336
X-Gnu-PR-Package: guix-patches
Reply-To: 47336 <at> debbugs.gnu.org
Date: Wed, 28 Apr 2021 02:31:02 +0000
Content-Type: multipart/mixed; boundary="----------=_1619577062-28565-1"

This is a multi-part message in MIME format...

------------=_1619577062-28565-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"

Your bug report

#47336: Disarchive as a fallback for downloads

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 47336 <at> debbugs.gnu.org.

--=20
47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47336
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

------------=_1619577062-28565-1
Content-Type: message/rfc822
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Received: (at 47336-done) by debbugs.gnu.org; 28 Apr 2021 02:30:51 +0000
Received: from localhost ([127.0.0.1]:51379 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lbZyU-0007QK-Ou
	for submit <at> debbugs.gnu.org; Tue, 27 Apr 2021 22:30:50 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:46999)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lbZyR-0007Pz-0T
 for 47336-done <at> debbugs.gnu.org; Tue, 27 Apr 2021 22:30:48 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id 6D226EA3;
 Tue, 27 Apr 2021 22:30:41 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 27 Apr 2021 22:30:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=4XqfWTcO+597/Uu3CUK5wZtE/MGAF4i4wRDNBVMPa
 PE=; b=dnUGU82lUM+ZvPwwEaP024vphaFUxxnh2jZ0RakQDnp5zCitB4Ze3qUBm
 6FnYQbD6G3F0WiTBY5HdhCfTZTP59mJKEaKkMnHrowEAFUO7ZnFC171Y+Zxypmxw
 pY7gCYMA2h8y3lPh/oQNqwYjJnw/DHS6/g/uUbc+rvKld6F0fstpyD+iqAD4A6Dh
 g0KKAstsgtiACMuKqCh+iylNoRabaPhW+nPtXH47IovAArQuirOcL0IVVBz371S+
 E+f1SplzgwrurRxuo8BeMLHDvLgb48RvIyCt442JytOAjR74SwzdY7JxG0h7a4ZF
 XjBzmPCr3YxZn3X3YGXgbAoRxE73g==
X-ME-Sender: <xms:0MiIYEaBVTwlNOpF8POEMXnb2bSUC_hWJU9IzCG904jY6WPuvF3GKg>
 <xme:0MiIYPYKC6s34__K3MIEyngHUIr3CfOVqQM035khW_-gdMxe7CkhD7L0K-myzoMx2
 uLZE3zT6ddpbUEUaQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvuddgheelucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho
 thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf
 grthhtvghrnhepvdelvefhheegudevjedvfedtveeileffveehveejgefgteetteejteel
 tdefueevnecuffhomhgrihhnpehgnhhurdhorhhgnecukfhppeejgedrudduiedrudekie
 drgeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep
 shgrmhhplhgvthesnhhghihrohdrtghomh
X-ME-Proxy: <xmx:0MiIYO8GtSUzpVhOEXfLd8igopqwJL431w_h2q8PGeDKL-YDEgVEgQ>
 <xmx:0MiIYOoZhp3vtkfg5O8nm0DdXRp5M9-WrP7axZGVi0qrun3SCJXGog>
 <xmx:0MiIYPoIXZnQfnhSF5iUuYwNUp8BZJIxnci1q8ZyiRlGZyM5i7UnUQ>
 <xmx:0ciIYJFmjbcPLm7EVDSteLyPPbatadNMSiK88c7CLC__uLhFdgYp5A>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA;
 Tue, 27 Apr 2021 22:30:40 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#47336: Disarchive as a fallback for downloads
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN> <87a6qo2835.fsf_-_@HIDDEN>
 <87v989e5oc.fsf_-_@HIDDEN>
Date: Tue, 27 Apr 2021 22:30:39 -0400
In-Reply-To: <87v989e5oc.fsf_-_@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s?=
 =?utf-8?Q?=22's?= message of "Mon, 26 Apr 2021 11:49:55 +0200")
Message-ID: <87o8dzkunk.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 47336-done
Cc: 47336-done <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hi,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Ping=C2=B2!
>
> Let me know if you=E2=80=99re like me to apply the patches on your behalf.

No, no.  I=E2=80=99m just a little distracted over here.  I just pushed this
series with the updates you suggested (using #:autoload, passing
#:verify-certificates?, and being a bit more chatty).  Sorry for the
delay and thanks for the reminder.

Next, I=E2=80=99ll convert my Cuirass 0.x setup to a Cuirass 1.x setup, and=
 then
I can start a discussion about moving the metadata builds to
ci.guix.gnu.org.

Also, to answer your other question:

> So we would normally arrange so that the =E2=80=98guix=E2=80=99 package d=
epends on
> Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call wor=
ks when done
> via =E2=80=98guix perform-download=E2=80=99, right?

That=E2=80=99s the idea.  I=E2=80=99m not confident about updating the =E2=
=80=98guix=E2=80=99 package
myself, though....


-- Tim


------------=_1619577062-28565-1
Content-Type: message/rfc822
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000
Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lOYs3-0001pw-WB
	for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400
Received: from lists.gnu.org ([209.51.188.17]:51240)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>) id 1lOYs2-0001pp-0u
 for submit <at> debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59600)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYs1-0007wV-QR
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <samplet@HIDDEN>) id 1lOYry-0007rZ-7A
 for bug-guix@HIDDEN; Tue, 23 Mar 2021 00:42:21 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id D60141373
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=content-transfer-encoding:content-type
 :date:from:message-id:mime-version:subject:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC
 aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6
 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0
 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5
 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/
 YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl
 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA
 ==
X-ME-Sender: <xms:pnFZYL4J4-5z0UZC1ft_G_1Nhsyy0ABL5PdIDX1VYRe_v_ZNezhl0Q>
 <xme:pnFZYA72ibiOrA_3NYhBmyqncC-HLuV5_9ESOPa2h89glclhmwXbINrXztgsUVuiS
 Nn5X2IxUhkya46jtA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd
 dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn
 ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf
 evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg
 rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh
 grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm
X-ME-Proxy: <xmx:pnFZYCewFnYM7r_tIF4Gu6oiXAXAN2KxThYXRJT-H0FxCRxRUwWJ1g>
 <xmx:pnFZYMJ2wySGLcQuD8EahxJhuyn31BqGxftQ0WUWVl2XhDswjbEdJA>
 <xmx:pnFZYPKgBzcoSJayN2aN5uB3u8JxwErxvQGYMGsUOtf1v3urevXYcw>
 <xmx:pnFZYEWuF8VgmOhBe8DcSbP1nEDNVy-Fk6x0yrhtHdZww14o8f3bug>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356
 for <bug-guix@HIDDEN>; Tue, 23 Mar 2021 00:42:14 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
To: bug-guix@HIDDEN
Subject: Disarchive as a fallback for downloads
Date: Tue, 23 Mar 2021 00:42:12 -0400
Message-ID: <87eeg6o50b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@HIDDEN;
 helo=wout3-smtp.messagingengine.com
X-Spam_score_int: -25
X-Spam_score: -2.6
X-Spam_bar: --
X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.6 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)

Hello,

This patch series adds Disarchive assembly (backed by SWH lookup) as a
fallback for downloads.

To try it, make sure you are running the daemon in an environment with
Disarchive available:

    $ ./pre-inst-env guix environment --ad-hoc guile disarchive
    # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild

Don=E2=80=99t forget to stop your existing Guix Daemon.  :)

You also need to make sure that regular downloads are unavailable.  I do
this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl=
-fetch=E2=80=9D in
=E2=80=9Cguix/build/download.scm=E2=80=9D.  I replace the usual list of URL=
s with =E2=80=98()=E2=80=99:

    (let try ((uri (append uri content-addressed-uris)))
      (match '() ; uri
        ...))

Now you can ask Guix for a recent .tar.gz source package:

    $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

You should see:

    Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v=
diwlv19y9-httpretty-1.0.5.tar.gz
    Assembling the directory httpretty-1.0.5
    Downloading from Software Heritage...
    7903d608efc89c14afb4d692a3721156e31a43e2/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
    7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
    [...]
    Checking httpretty-1.0.5 digest... ok
    Assembling the tarball httpretty-1.0.5.tar
    Checking httpretty-1.0.5.tar digest... ok
    Assembling the Gzip file httpretty-1.0.5.tar.gz
    Checking httpretty-1.0.5.tar.gz digest... ok
    Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty=
-1.0.5.tar.gz
    successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett=
y-1.0.5.tar.gz.drv

There=E2=80=99s lots to talk about though....

First, it looks up the metadata on my server.  This is fine for a demo,
but not what we want forever.  The patch series supports adding several
mirrors for looking up the metadata.  In the past, we talked about
putting everything on one or a few of the big Git hosting platforms like
GitHub or Gitlab.  That way, it would be easily picked up by SWH and
archived =E2=80=9Cforever=E2=80=9D.  Right now, I have Cuirass set up to bu=
ild the
metadata, and a little script that moves it from the build server to my
Web server.  It would be simple enough to adjust that script to push it
to a remote Git repo.  (Of course, the next step is to move this setup
to Guix infrastructure.)  Thoughts?

On the code level, there were two things I couldn=E2=80=99t figure out for
myself.

I made the mirror list just simple strings.  AIUI, the client and the
daemon have to agree about the format of the mirror list.  Given that
running old daemons is common, changing the format is difficult.  Is it
worth it to copy the more flexible interface used by the content
addressed mirrors?  If yes, do I have to do the same =E2=80=98module-autolo=
ad!=E2=80=99
dance to use =E2=80=98bytevector->base16-string=E2=80=99?  :)  (I probably =
would have
just copied it, but that part confused me a bit.)

I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel=
l, just
=E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D).  It feels weird to use=
 a bunch of host-side modules
from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module.  This =
is okay because
=E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. =
 It=E2=80=99s more
like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2=
=80=9D, which
is why importing host-side modules is OK... right?

Hopefully everything else is more-or-less fine.  :)


-- Tim



------------=_1619577062-28565-1--


Message sent to guix-patches@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: [bug#47336] Disarchive as a fallback for downloads
Resent-From: Timothy Sample <samplet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Wed, 28 Apr 2021 07:03:02 +0000
Resent-Message-ID: <handler.47336.B47336.161959332523960 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 47336
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: control <at> debbugs.gnu.org, 47336 <at> debbugs.gnu.org
Received: via spool by 47336-submit <at> debbugs.gnu.org id=B47336.161959332523960
          (code B ref 47336); Wed, 28 Apr 2021 07:03:02 +0000
Received: (at 47336) by debbugs.gnu.org; 28 Apr 2021 07:02:05 +0000
Received: from localhost ([127.0.0.1]:51583 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lbeCz-0006EO-4R
	for submit <at> debbugs.gnu.org; Wed, 28 Apr 2021 03:02:05 -0400
Received: from out5-smtp.messagingengine.com ([66.111.4.29]:55273)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <samplet@HIDDEN>)
 id 1lbeCw-0006Dp-3A; Wed, 28 Apr 2021 03:02:03 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id B99DC5C00BD;
 Wed, 28 Apr 2021 03:01:56 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Wed, 28 Apr 2021 03:01:56 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=73wGy0hp+Ms9wxJ0OS/8kZyUHxq2hBRI86EwfbO2J
 r0=; b=HElWpid1QUvLXjr5widnlf9WrYsJxxlpXpeyv320R5gGrr+hO2jwvJzQH
 YJCUsKt1tmiwdnjHRyr3c6TuW2p2Zd8fiQLeu35Dvz+nHfF3sztWCrY/Rp8tIM1W
 0rH46o8C0+da6KXnID89d2pV3ptgSam4Yrvmk2dZ92xX7Dd+5d7UOL2Aq9Y8X+Y8
 31xIqXrPR/IH6qUCC84azZuzUYQowvveYzB8f5TqlmeiX2RoSNiQTBhnXF9dkA2W
 wlaMyu24IJk5St/pcyY2XXpnvLCP6FoIqz8KLAOuuLw8y4vIeZoTEpn9HVRZw8OX
 2d9E1YhUbj9u/TSaq6IzDs7dK0g8Q==
X-ME-Sender: <xms:ZAiJYIGRtFqbjp1FmhwIWy4ySpTMuQ53Aq02kaPLmH4s-nInYtxUzg>
 <xme:ZAiJYBX2fSKnVGxwhqFXBg4qM5bD3pgOBbI3tbTh7UihQpopqeOHcFPzApci47ldG
 tUoVTMzTvsmHPB16w>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvuddguddufecutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefhvffufhffjgfkfgggtgfgsehtqhertddtreejnecuhfhrohhmpefvihhm
 ohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehnghihrhhordgtohhmqeenucggtf
 frrghtthgvrhhnpeeiudeuhfeggeelleevheegudfguefhieekffdtveeilefglefhvddt
 gfeiheetgfenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiii
 gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgt
 ohhm
X-ME-Proxy: <xmx:ZAiJYCL9dOMnYXONcBcxN4JpUKIaShRsGxWoeVEzfiCfaYST8hcDfQ>
 <xmx:ZAiJYKG_dxAdiqCx2BZJGwnd4OEABdlyjrldHT58mZBNAzDwwMJ6GA>
 <xmx:ZAiJYOUB1xswKxcw9y8o9MFBzBC-uE5HVjj5SiZaOzD-Si6ESPDLyA>
 <xmx:ZAiJYHfcg9c_K5F7V_KDmljw2y86AQFnFo50FnuvR3srO51PzzSklA>
Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44])
 by mail.messagingengine.com (Postfix) with ESMTPA;
 Wed, 28 Apr 2021 03:01:56 -0400 (EDT)
From: Timothy Sample <samplet@HIDDEN>
References: <87eeg6o50b.fsf@HIDDEN>
 <20210323045213.9419-1-samplet@HIDDEN> <87a6qo2835.fsf_-_@HIDDEN>
 <87v989e5oc.fsf_-_@HIDDEN> <87o8dzkunk.fsf_-_@HIDDEN>
Date: Wed, 28 Apr 2021 03:01:55 -0400
In-Reply-To: <87o8dzkunk.fsf_-_@HIDDEN> (Timothy Sample's message of "Tue, 
 27 Apr 2021 22:30:39 -0400")
Message-ID: <87h7jqlwnw.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

reopen 47336
thanks

Hi again,

Timothy Sample <samplet@HIDDEN> writes:

> I just pushed this series [...]

And broke =E2=80=9Cguix pull=E2=80=9D!!  (I somehow fooled myself into thin=
king that I
had already tested with =E2=80=9Cguix pull --url=3D...=E2=80=9D locally.)  =
I reverted the
offending commit.

It turns out that adding a reference from =E2=80=9C(guix build download)=E2=
=80=9D to
=E2=80=9C(guix swh)=E2=80=9D breaks =E2=80=9Ccompute-guix-derivation=E2=80=
=9D in
=E2=80=9Cbuild-aux/build-self.scm=E2=80=9D.  This is because =E2=80=9C(guix=
 swh)=E2=80=9D references
=E2=80=9C(json)=E2=80=9D, which is not available in the =E2=80=9Ccompute-gu=
ix-derivation=E2=80=9D
environment.  I tried mimicking the =E2=80=9Cfake-git=E2=80=9D trick, but i=
t didn=E2=80=99t work
(I guess it needs the =E2=80=9Cdefine-json-mapping=E2=80=9D macro at compil=
e time).

Everything works if I remove the #:autoload for =E2=80=9C(guix swh)=E2=80=
=9D and put

  ;; If we import (guix swh) directly, we introduce a compile-time
  ;; dependency on Guile-JSON.  This breaks the "build-self" code, which
  ;; needs to build this module without Guile-JSON.  Hence, we track
  ;; down the following procedure at runtime.
  (define swh-download-directory
    (module-ref (resolve-module '(guix swh)) 'swh-download-directory))

inside of =E2=80=9Cdisarchive-fetch/any=E2=80=9D (just before it=E2=80=99s =
needed).  Does this
approach look okay?


-- Tim




Message received at fakecontrol@fakecontrolmessage:


Received: (at fakecontrol) by fakecontrolmessage;
To: internal_control <at> debbugs.gnu.org
From: Debbugs Internal Request <help-debbugs@HIDDEN>
Subject: Internal Control
Message-Id: Did not alter fixed versions and reopened.
Date: Wed, 28 Apr 2021 07:03:02 +0000
User-Agent: Fakemail v42.6.9

# This is a fake control message.
#
# The action:
# Did not alter fixed versions and reopened.
thanks
# This fakemail brought to you by your local debbugs
# administrator



Last modified: Wed, 28 Apr 2021 07:15:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.