GNU bug report logs - #52555
[RFC PATCH 0/3] Decentralized substitute distribution with ERIS

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Severity: important; Reported by: pukkamustard <pukkamustard@HIDDEN>; Keywords: patch; dated Thu, 16 Dec 2021 16:18:02 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 52555 <at> debbugs.gnu.org:


Received: (at 52555) by debbugs.gnu.org; 23 Dec 2021 12:18:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 23 07:18:29 2021
Received: from localhost ([127.0.0.1]:60570 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1n0N3A-0003EI-Ic
	for submit <at> debbugs.gnu.org; Thu, 23 Dec 2021 07:18:29 -0500
Received: from mout02.posteo.de ([185.67.36.66]:52835)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@HIDDEN>) id 1n0N38-0003E4-FE
 for 52555 <at> debbugs.gnu.org; Thu, 23 Dec 2021 07:18:23 -0500
Received: from submission (posteo.de [89.146.220.130]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 2F987240105
 for <52555 <at> debbugs.gnu.org>; Thu, 23 Dec 2021 13:18:15 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1640261896; bh=JgqGKyEHLIdnmE6TeYENeP0wVNmtmDc4GKpcUJhHrYU=;
 h=From:To:Cc:Subject:Date:From;
 b=iW3bMzBy/BjgyPuHsO5+9LyOGNWnUQFwMfA/pKQ2N7sT2vDJlUho5P5oGdQg9U7cR
 2Ezr5kuKqzBVCIoJlu30LWxEMyyQ1OfjizzCEz3NNk5WlTjq7HkqxajPzcaeQJwH+w
 9XY7aXwKtfPqkEhYPq776u+6mnysrSdWphxAhC/ghdRNQgz9kT6a0a0bkFaRI1zLDi
 GiFDbSjV94XgkH54gVWFvomQOn+vccx+fur2ZEnOxTChnW6YYI19Y1k4svuCYlbOLW
 LPD1Is0csnx0JJkiZ8WexZlR2IFiMxvGLMwVVbs6T73h4WHmM72I8NHM8jXg4aWpOk
 +lQbMIAcjbkcA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4JKTjy5gg2z6tn0;
 Thu, 23 Dec 2021 13:18:14 +0100 (CET)
References: <20211216161724.547-1-pukkamustard@HIDDEN>
 <87h7b3gs64.fsf@HIDDEN>
From: pukkamustard <pukkamustard@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute
 distribution with ERIS
Date: Thu, 23 Dec 2021 11:42:46 +0000
In-reply-to: <87h7b3gs64.fsf@HIDDEN>
Message-ID: <86bl17ms56.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: ~pukkamustard/eris@HIDDEN, 52555 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Hi Ludo,

Thanks for your comments!

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: gzip
>> FileSize: 67363
>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDD=
VDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: zstd
>> FileSize: 64917
>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ=
6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>
> Do we really need one URN per compression method?  Couldn=E2=80=99t we le=
ave
> compression (of individual chunks, possibly) as a =E2=80=9Cdetail=E2=80=
=9D handled by
> the encoding or the transport layer?
>

I agree that it would be nice to leave this to the encoding layer as
that would allow certain optimizations (e.g. de-duplication).

Unfortunately, we haven't figured out yet what the most suitable
compression/format would be. Something like EROSFS seems good (as it
aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
just an archive format and there do not seem to be any libraries that we
could use to neatly integrate. It seems possible to block-align a Tar
archive, but that seems a bit hackey [2]. Other things to look into
might be Tarlz [3] and ZPAQ [4].

To get started I suggest just using one of the compressions/formats
already in Guix. zstd seems to be a reasonable choice (for the same
reasons why it makes sense to use zstd with `--discover` [5]).

Does that sound like a plan?

[1] https://inqlab.net/git/guile-eris.git/tree/examples/dedup-fs/Readme.org
[2] https://unix.stackexchange.com/questions/276908/make-tar-or-other-archi=
ve-with-data-block-aligned-like-in-original-files-for/279384#279384
[3] http://lzip.nongnu.org/tarlz.html
[4] http://mattmahoney.net/dc/zpaq.html
[5] https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

>> If the `--ipfs` is used for `guix publish` then the encoded blocks are a=
lso
>> uploaded to the IPFS daemon. The nar could then be retrieved from anywhe=
re like
>> this:
>>
>> (use-modules (eris)
>> 	     (eris blocks ipfs))
>>
>> (eris-decode->bytevector
>>  "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAG=
JUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>>  eris-blocks-ipfs-ref)
>>
>> These patches do not yet retrieve content from IPFS (TODO). But in princ=
iple,
>> anybody connected to IPFS can get the nar with the ERIS URN. This could =
be used
>> to reduce load on substitute server as they would only need to publish t=
he ERIS
>> URN directly - substitutes could be delivered much more peer-to-peer.
>
> Nice.  So adjusting =E2=80=98guix substitute=E2=80=99 should be relativel=
y easy?

Yes, relatively! :)

I meant to send in a V2 that does this before going on holidays, but I'm
afraid I won't make it. V2 will come in early January!

>> Other transports that I have been looking in to and am pretty sure will =
work
>> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
>> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
>> identifiers (URN) are abstracted away from specific transports (and also
>> applications). ERIS is almost exactly the same encoding as used in GNUNet
>> (ECRS).
>
> As a first step, =E2=80=98guix publish=E2=80=99 could implement RFC 2169,=
 too.
>
> I gather implementing the HTTP and IPFS backends in =E2=80=98guix substit=
ute=E2=80=99
> should be relatively easy, right?

Yes, those seem to be the two easiest backends to implement.

>> A tricky things is figuring out how to multiplex all these different
>> transports and storages...
>
> Yes.  We don=E2=80=99t know yet what performance and data availability wi=
ll be
> like on IPFS, for instance, so it=E2=80=99s important for users to be abl=
e to
> set priorities.  It=E2=80=99s also important to gracefully fall back to d=
irect
> HTTP downloads when fancier p2p methods fail, regardless of how they
> fail.

Agree.

Thanks,
-pukkamustard





Information forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.

Message received at 52555 <at> debbugs.gnu.org:


Received: (at 52555) by debbugs.gnu.org; 20 Dec 2021 16:25:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Dec 20 11:25:19 2021
Received: from localhost ([127.0.0.1]:51230 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mzLTT-00057j-1o
	for submit <at> debbugs.gnu.org; Mon, 20 Dec 2021 11:25:19 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:43422)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mzLTR-00057Q-De
 for 52555 <at> debbugs.gnu.org; Mon, 20 Dec 2021 11:25:18 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 927AC361;
 Mon, 20 Dec 2021 17:25:10 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 1Wfl5kQPKPVV; Mon, 20 Dec 2021 17:25:09 +0100 (CET)
Received: from ribbon (unknown [193.50.110.211])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id D191B63;
 Mon, 20 Dec 2021 17:25:08 +0100 (CET)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: pukkamustard <pukkamustard@HIDDEN>
Subject: Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute
 distribution with ERIS
References: <20211216161724.547-1-pukkamustard@HIDDEN>
Date: Mon, 20 Dec 2021 17:25:07 +0100
In-Reply-To: <20211216161724.547-1-pukkamustard@HIDDEN>
 (pukkamustard@HIDDEN's message of "Thu, 16 Dec 2021 16:17:24
 +0000")
Message-ID: <87h7b3gs64.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: /
Authentication-Results: hera.aquilenet.fr;
	none
X-Rspamd-Server: hera
X-Rspamd-Queue-Id: 927AC361
X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[];
 RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[];
 RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain];
 FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
 RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[];
 MID_RHS_MATCH_FROM(0.00)[]
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 52555
Cc: ~pukkamustard/eris@HIDDEN, 52555 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi pukkamustard,

pukkamustard <pukkamustard@HIDDEN> skribis:

> This is an initial patch and proposal towards decentralizing substitute
> distribution with ERIS.

Woohoo, sounds exciting!

> ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of conten=
t into
> uniformly sized, encryped and content-addressed blocks. The original cont=
ent
> can be reconstructed only with access to a read capability, which can be
> encoded as an URN.
>
> One key advantage of ERIS is that the encoding is protocol agnostic. Any
> protocol that can transfer small (32KiB) sized blocks referenced by the h=
ash of
> their content will do. This can be done with things such as GNUNet, IPFS,
> OpenDHT, HTTP or a USB stick on a bicycle.

Yes, that=E2=80=99s nice.

> The following patch allows substitutes to be published over IPFS using ER=
IS.
> This is inspired and very similar to previous work on distributing substi=
tutes
> over IPFS [2].
>
> The narinfos served by `guix publish` look like this:
>
> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: gzip
> FileSize: 67363
> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDV=
DAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: zstd
> FileSize: 64917
> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6=
FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM

Do we really need one URN per compression method?  Couldn=E2=80=99t we leave
compression (of individual chunks, possibly) as a =E2=80=9Cdetail=E2=80=9D =
handled by
the encoding or the transport layer?

> If the `--ipfs` is used for `guix publish` then the encoded blocks are al=
so
> uploaded to the IPFS daemon. The nar could then be retrieved from anywher=
e like
> this:
>
> (use-modules (eris)
> 	     (eris blocks ipfs))
>
> (eris-decode->bytevector
>  "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJ=
UEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>  eris-blocks-ipfs-ref)
>
> These patches do not yet retrieve content from IPFS (TODO). But in princi=
ple,
> anybody connected to IPFS can get the nar with the ERIS URN. This could b=
e used
> to reduce load on substitute server as they would only need to publish th=
e ERIS
> URN directly - substitutes could be delivered much more peer-to-peer.

Nice.  So adjusting =E2=80=98guix substitute=E2=80=99 should be relatively =
easy?

> Other transports that I have been looking in to and am pretty sure will w=
ork
> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
> identifiers (URN) are abstracted away from specific transports (and also
> applications). ERIS is almost exactly the same encoding as used in GNUNet
> (ECRS).

As a first step, =E2=80=98guix publish=E2=80=99 could implement RFC 2169, t=
oo.

I gather implementing the HTTP and IPFS backends in =E2=80=98guix substitut=
e=E2=80=99
should be relatively easy, right?

> Blocks can be stored in any kind of databases (see for example the GDBM
> bindings [4]).
>
> A tricky things is figuring out how to multiplex all these different
> transports and storages...

Yes.  We don=E2=80=99t know yet what performance and data availability will=
 be
like on IPFS, for instance, so it=E2=80=99s important for users to be able =
to
set priorities.  It=E2=80=99s also important to gracefully fall back to dir=
ect
HTTP downloads when fancier p2p methods fail, regardless of how they
fail.

> The ERIS specification is still considered "experimental". However we feel
> confident to stabilize it and intend to do so around February/March 2022 =
with a
> release 1.0.0 of the specification. This will ensure that the identifiers
> remain stable for the forseeable future (until the crypto breaks). Before=
 that
> there is also a small external security audit of the specification planned
> (thanks to NGI0/NLnet!).

Neat.

This is all very exciting.  I look forward to playing around with it!

Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.
Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 52555 <at> debbugs.gnu.org:


Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:21:01 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 16 11:21:00 2021
Received: from localhost ([127.0.0.1]:36508 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mxtV6-0003xO-Dp
	for submit <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:21:00 -0500
Received: from mout01.posteo.de ([185.67.36.65]:34165)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@HIDDEN>) id 1mxtV2-0003wj-VK
 for 52555 <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:20:57 -0500
Received: from submission (posteo.de [89.146.220.130]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 66D91240026
 for <52555 <at> debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:51 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1639671651; bh=8FUfzH+H19v7PInjtcmZ9loRZ+gv+GSpvf6oOoA0eR4=;
 h=From:To:Cc:Subject:Date:From;
 b=q3677HhT4ire8v1On3rXXk2hHSHVeTfXOJJMYNfGarL65V+/ZMjqP0fAYBPzROczE
 eas83QDMVCfP6Cru45FGGM4PktcKU+vVuCpRh3MmCqINvg7z46nVn8nJVxhE3r3HrW
 2KZZ2vrzt8gZYeVgqymH+gIqvJnG31N8yTe2L8JhgEe4TBxqd2gM4hjE9Re42GZpxx
 TTVrU15muau8PE+/Z3KNb3VoOrRor3jJvJ0j8aJRHpycDpvEaoOkmAy9S9E9Ho1LDL
 BWQY3CgIy8F7qnsaDG7ODIY1al8FAVlboSijotsprngtlipcaQeOfTo4blayBSJMgu
 1nMOmikrWOjTA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4JFHR64dFHz9rxS;
 Thu, 16 Dec 2021 17:20:50 +0100 (CET)
From: pukkamustard <pukkamustard@HIDDEN>
To: 52555 <at> debbugs.gnu.org
Subject: [RFC PATCH 3/3] publish: Add IPFS support.
Date: Thu, 16 Dec 2021 16:20:36 +0000
Message-Id: <20211216162036.694-3-pukkamustard@HIDDEN>
In-Reply-To: <20211216162036.694-1-pukkamustard@HIDDEN>
References: <20211216161724.547-1-pukkamustard@HIDDEN>
 <20211216162036.694-1-pukkamustard@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: pukkamustard <pukkamustard@HIDDEN>, ~pukkamustard/eris@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

* guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'.
  (render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it.
  (guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url.
---
 guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++---------
 1 file changed, 41 insertions(+), 11 deletions(-)

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 8e4b90789b..8e7fb47b9e 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,8 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (srfi srfi-171)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -52,6 +54,7 @@ (define-module (guix scripts publish)
   #:use-module (guix base64)
   #:use-module (guix config)
   #:use-module (guix derivations)
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:use-module (gcrypt hash)
   #:use-module (guix pki)
   #:use-module (gcrypt pk-crypto)
@@ -59,6 +62,8 @@ (define-module (guix scripts publish)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
   #:use-module (eris)
+  #:use-module (eris read-capability)
+  #:use-module (eris blocks ipfs)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -83,6 +88,7 @@ (define-module (guix scripts publish)
             run-publish-server
             guix-publish))
 
+
 (define (show-help)
   (format #t (G_ "Usage: guix publish [OPTION]...
 Publish ~a over HTTP.\n") %store-directory)
@@ -102,6 +108,8 @@ (define (show-help)
   (display (G_ "
       --cache-bypass-threshold=SIZE
                          serve store items below SIZE even when not cached"))
+  (display (G_ "
+      --ipfs[=GATEWAY]   publish items over IPFS via GATEWAY"))
   (display (G_ "
       --workers=N        use N workers to bake items"))
   (display (G_ "
@@ -220,6 +228,10 @@ (define %options
                 (lambda (opt name arg result)
                   (alist-cons 'cache-bypass-threshold (size->number arg)
                               result)))
+        (option '("ipfs") #f #t
+                (lambda (opt name arg result)
+                  (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url))
+                              result)))
         (option '("workers") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'workers (string->number* arg)
@@ -526,7 +538,7 @@ (define (bypass-cache? store item)
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compressions (list %no-compression))
                                 (nar-path "nar") negative-ttl
-                                cache pool)
+                                cache pool ipfs?)
   "Respond to the narinfo request for REQUEST.  If the narinfo is available in
 CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
 requested using POOL."
@@ -571,7 +583,8 @@ (define (delete-entry narinfo)
                  (bake-narinfo+nar cache item
                                    #:ttl ttl
                                    #:compressions compressions
-                                   #:nar-path nar-path)))
+                                   #:nar-path nar-path
+                                   #:ipfs? ipfs?)))
 
              (when ttl
                (single-baker 'cache-cleanup
@@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port)
 
 (define* (bake-narinfo+nar cache item
                            #:key ttl (compressions (list %no-compression))
-                           (nar-path "/nar"))
+                           (nar-path "/nar") ipfs?)
   "Write the narinfo and nar for ITEM to CACHE."
   (define (compressed-nar-size compression)
     (let* ((nar  (nar-cache-file cache item #:compression compression))
@@ -641,9 +654,19 @@ (define (compressed-nar-size compression)
 
   (define (compressed-eris-urn compression)
     (let* ((nar (nar-cache-file cache item #:compression compression))
-           (stat (stat nar #f)))
+           (stat (stat nar #f))
+           (block-reducer (if ipfs?
+                              (eris-blocks-ipfs-reducer
+                               #:ipfs-base-url (ipfs:%ipfs-base-url))
+                              rcount)))
       (and stat
-           (cons compression (call-with-input-file nar eris-encode->urn)))))
+           (cons compression
+                 (call-with-input-file nar
+                   (lambda (port)
+                     (let ((read-cap _
+                                     (eris-encode port #:block-reducer
+                                                  block-reducer)))
+                       (read-capability->string read-cap))))))))
 
   (let ((compression (actual-compressions item compressions)))
 
@@ -1115,7 +1138,8 @@ (define* (make-request-handler store
                                cache pool
                                narinfo-ttl narinfo-negative-ttl
                                (nar-path "nar")
-                               (compressions (list %no-compression)))
+                               (compressions (list %no-compression))
+                               ipfs?)
   (define compression-type?
     string->compression-type)
 
@@ -1147,7 +1171,8 @@ (define (handle request body)
                                       #:ttl narinfo-ttl
                                       #:negative-ttl narinfo-negative-ttl
                                       #:nar-path nar-path
-                                      #:compressions compressions)
+                                      #:compressions compressions
+                                      #:ipfs? ipfs?)
                (render-narinfo store request hash
                                #:ttl narinfo-ttl
                                #:negative-ttl narinfo-negative-ttl
@@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store
                              advertise? port
                              (compressions (list %no-compression))
                              (nar-path "nar") narinfo-ttl narinfo-negative-ttl
-                             cache pool)
+                             cache pool ipfs?)
   (when advertise?
     (let ((name (service-name)))
       ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a
@@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store
                                     #:nar-path nar-path
                                     #:narinfo-ttl narinfo-ttl
                                     #:narinfo-negative-ttl narinfo-negative-ttl
-                                    #:compressions compressions)
+                                    #:compressions compressions
+                                    #:ipfs? ipfs?)
               concurrent-http-server
               `(#:socket ,socket)))
 
@@ -1296,6 +1322,8 @@ (define-command (guix-publish . args)
            (repl-port (assoc-ref opts 'repl))
            (cache     (assoc-ref opts 'cache))
            (workers   (assoc-ref opts 'workers))
+           (ipfs      (assoc-ref opts 'ipfs))
+           (ipfs?     (if ipfs #t #f))
 
            ;; Read the key right away so that (1) we fail early on if we can't
            ;; access them, and (2) we can then drop privileges.
@@ -1315,7 +1343,8 @@ (define-command (guix-publish . args)
                      (%private-key private-key)
                      (cache-bypass-threshold
                       (or (assoc-ref opts 'cache-bypass-threshold)
-                          (cache-bypass-threshold))))
+                          (cache-bypass-threshold)))
+                     (ipfs:%ipfs-base-url ipfs))
         (info (G_ "publishing ~a on ~a, port ~d~%")
               %store-directory
               (inet-ntop (sockaddr:fam address) (sockaddr:addr address))
@@ -1344,7 +1373,8 @@ (define-command (guix-publish . args)
                               #:nar-path nar-path
                               #:compressions compressions
                               #:narinfo-negative-ttl negative-ttl
-                              #:narinfo-ttl ttl))))))
+                              #:narinfo-ttl ttl
+                              #:ipfs? ipfs?))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'single-baker 'scheme-indent-function 1)
-- 
2.34.0





Information forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.

Message received at 52555 <at> debbugs.gnu.org:


Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:20:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 16 11:20:57 2021
Received: from localhost ([127.0.0.1]:36504 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mxtV3-0003x3-0g
	for submit <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:20:57 -0500
Received: from mout01.posteo.de ([185.67.36.65]:44809)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@HIDDEN>) id 1mxtV0-0003wg-C9
 for 52555 <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:20:55 -0500
Received: from submission (posteo.de [89.146.220.130]) 
 by mout01.posteo.de (Postfix) with ESMTPS id C8A4A240027
 for <52555 <at> debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:48 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1639671648; bh=mzF6Dcz6gBsxGgNgXV2yX2To4wOq88FwLdvP0HAzkYo=;
 h=From:To:Cc:Subject:Date:From;
 b=n7IfKTN8HDGAo6xEuyFQh2PJ1Kld/Sj/lviOXqQSe/U0XMXzbw0JqTAzPeHhyH2W+
 frlB6+zNnf2ug2LBhzBLm+wPFER2hxpmv2TO2DN+nSS6SoCHOTNFVWNu9VhFs5DY2J
 AMR650+puRcRgAGAQk59mNxHhwGW5FMrTpQoq549I2fZFmf7kIS+/fgapxwxKLIGu9
 er1LA+mbm1F1JmBR/oz6HPjkZBFi8lMew85MZRatBy5ZMzqIa9N+W1Q0dnpbRiO8Qw
 KXAmAprHuQxwUlkG+3dJcIO9IZNW7mDpGjwhesz9C9uWq7x8K41KJwjdMN3vxddy/k
 6YE9/vRp4Ep9Q==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4JFHR400WSz9rxX;
 Thu, 16 Dec 2021 17:20:47 +0100 (CET)
From: pukkamustard <pukkamustard@HIDDEN>
To: 52555 <at> debbugs.gnu.org
Subject: [RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git
 version.
Date: Thu, 16 Dec 2021 16:20:35 +0000
Message-Id: <20211216162036.694-2-pukkamustard@HIDDEN>
In-Reply-To: <20211216162036.694-1-pukkamustard@HIDDEN>
References: <20211216161724.547-1-pukkamustard@HIDDEN>
 <20211216162036.694-1-pukkamustard@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: pukkamustard <pukkamustard@HIDDEN>, ~pukkamustard/eris@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
 gnu/packages/guile-xyz.scm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 8346d99996..5e6f31a6e2 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4325,15 +4325,15 @@ (define-public guile-sodium
 (define-public guile-eris
   (package
     (name "guile-eris")
-    (version "0.2.0")
+    (version "f1e4dd87988f9a80b05a8051d7f5ba3daf79dcc1")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://inqlab.net/git/eris.git")
-             (commit (string-append "v" version))))
+             (url "https://inqlab.net/git/guile-eris.git")
+             (commit version)))
        (file-name (git-file-name name version))
-       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+       (sha256 (base32 "0kgm4b4qn2s74wjvxy273gdi1l1m81i2k4kkk1zc6vlcg3np7p06"))))
     (build-system gnu-build-system)
     (arguments '())
     (native-inputs
@@ -4345,7 +4345,7 @@ (define-public guile-eris
            guile-srfi-180))
     (inputs (list guile-3.0))
     (propagated-inputs
-     (list guile-sodium))
+     (list guile-sodium guile-json-4 guile-gdbm-ffi))
     (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
     (description
      "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0





Information forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.

Message received at 52555 <at> debbugs.gnu.org:


Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:20:57 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 16 11:20:57 2021
Received: from localhost ([127.0.0.1]:36502 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mxtV2-0003x1-HG
	for submit <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:20:56 -0500
Received: from mout01.posteo.de ([185.67.36.65]:37321)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@HIDDEN>) id 1mxtUx-0003wb-RN
 for 52555 <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:20:54 -0500
Received: from submission (posteo.de [89.146.220.130]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 43FC9240026
 for <52555 <at> debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:45 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1639671645; bh=c506nOniPbxfWxa5Datyw8oJED0OKeKWWVS7Qozcw10=;
 h=From:To:Cc:Subject:Date:From;
 b=GgY5+Cdof2C9QGoRmOaV/LbGZ9OSD/kJwQwxbMx1W50K/MnV3OyMltuBEOI60iAWg
 nR0Yn7SK+x0Z2b4Epyp3+0gYiVGKkR06lmq2UnI+KYz/gSSr3wwtUD7/f5MFdGxBt8
 0BuI1Led8qyeypkADSsiR29ra02FPhd3oFu/TXE5/DiHS6iDjYUzmGpjINGoifaYmV
 bRtVXXWvp/4GUSzYGqVko3EgfwQnTPt4+tNpjlrhNCmgeoofH1OYeufxysrdBwsOOE
 zvNVgEOpGA2V9kjDQ+Lm9wSlZV5MdEgaBI0Mv92FojJy83vOrTqJsWZ9iWR+NUVIoB
 JpsinZnAntYtA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4JFHR01ll4z9rxT;
 Thu, 16 Dec 2021 17:20:43 +0100 (CET)
From: pukkamustard <pukkamustard@HIDDEN>
To: 52555 <at> debbugs.gnu.org
Subject: [RFC PATCH 1/3] publish: Add ERIS URN to narinfo
Date: Thu, 16 Dec 2021 16:20:34 +0000
Message-Id: <20211216162036.694-1-pukkamustard@HIDDEN>
In-Reply-To: <20211216161724.547-1-pukkamustard@HIDDEN>
References: <20211216161724.547-1-pukkamustard@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: pukkamustard <pukkamustard@HIDDEN>, ~pukkamustard/eris@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urns parameter and honor it.
(store-item->recutils): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-urns]: New field.
(narinfo-maker): Handle ERIS URN.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
 configure.ac                        |  5 +++++
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 10 ++++++----
 guix/scripts/publish.scm            | 31 +++++++++++++++++++++--------
 4 files changed, 35 insertions(+), 12 deletions(-)

diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9496499850..5c49167782 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -394,6 +394,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..0972ede3c1 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -67,13 +67,14 @@ (define-module (guix narinfo)
             equivalent-narinfo?))
 
 (define-record-type <narinfo>
-  (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
+  (%make-narinfo path uri-base uris eris-urns compressions file-sizes
+                 file-hashes nar-hash nar-size references deriver system
                  signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
   (uris         narinfo-uris)            ;list of strings
+  (eris-urns    narinfo-eris-urns)       ;list of (strings | #f)
   (compressions narinfo-compressions)    ;list of strings
   (file-sizes   narinfo-file-sizes)      ;list of (integers | #f)
   (file-hashes  narinfo-file-hashes)
@@ -134,7 +135,7 @@ (define (narinfo-signature->canonical-sexp str)
 (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
-  (lambda (path urls compressions file-hashes file-sizes
+  (lambda (path urls eris-urns compressions file-hashes file-sizes
                 nar-hash nar-size references deriver system
                 signature)
     "Return a new <narinfo> object."
@@ -148,6 +149,7 @@ (define len (length urls))
                                    (string-append cache-url url)
                                    (string-append cache-url "/" url)))))
                         urls)
+                   eris-urns
                    compressions
                    (match file-sizes
                      (()        (make-list len #f))
@@ -186,7 +188,7 @@ (define* (read-narinfo port #:optional url
                      "FileHash" "FileSize" "NarHash" "NarSize"
                      "References" "Deriver" "System"
                      "Signature")
-                   '("URL" "Compression" "FileSize" "FileHash"))))
+                   '("URL" "ERIS" "Compression" "FileSize" "FileHash"))))
 
 (define (narinfo-sha256 narinfo)
   "Return the sha256 hash of NARINFO as a bytevector, or #f if NARINFO lacks a
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..8e4b90789b 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -308,7 +309,7 @@ (define* (store-item->recutils store-item
                                #:key
                                (nar-path "nar")
                                (compression %no-compression)
-                               file-size)
+                               file-size eris-urn)
   "Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM,
 with COMPRESSION, starting at NAR-PATH."
   (let ((url (encode-and-join-uri-path
@@ -319,19 +320,22 @@ (define* (store-item->recutils store-item
                     (($ <compression> type)
                      (list (symbol->string type))))
                 ,(basename store-item)))))
-    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]"
-            url (compression-type compression) file-size)))
+    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]~@[ERIS: ~a~%~]"
+            url (compression-type compression) file-size eris-urn)))
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '()) (eris-urns '()))
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
 
 Optionally, FILE-SIZES is a list of compression/integer pairs, where the
 integer is size in bytes of the compressed NAR; it informs the client of how
-much needs to be downloaded."
+much needs to be downloaded.
+
+Optionally, ERIS-URNS is a list of compression/string pairs, where the
+string is the ERIS URN of the compressed NAR."
   (let* ((path-info  (query-path-info store store-path))
          (compressions (actual-compressions store-path compressions))
          (hash       (bytevector->nix-base32-string
@@ -352,9 +356,12 @@ (define* (narinfo-string store store-path
                              store-path
                              (map (lambda (compression)
                                     (let ((size (assoc-ref file-sizes
-                                                           compression)))
+                                                           compression))
+                                          (eris-urn (assoc-ref eris-urns
+                                                               compression)))
                                       (store-item->recutils store-path
                                                             #:file-size size
+                                                            #:eris-urn eris-urn
                                                             #:nar-path nar-path
                                                             #:compression
                                                             compression)))
@@ -632,6 +639,12 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (compressed-eris-urn compression)
+    (let* ((nar (nar-cache-file cache item #:compression compression))
+           (stat (stat nar #f)))
+      (and stat
+           (cons compression (call-with-input-file nar eris-encode->urn)))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -646,11 +659,13 @@ (define (compressed-nar-size compression)
              ;; thread's connection to the store since we would end up sending
              ;; stuff concurrently on the same channel.
              (with-store store
-               (let ((sizes (filter-map compressed-nar-size compression)))
+               (let ((sizes (filter-map compressed-nar-size compression))
+                     (eris-urns (filter-map compressed-eris-urn compression)))
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urns eris-urns)
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0





Information forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 16 Dec 2021 16:17:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Dec 16 11:17:56 2021
Received: from localhost ([127.0.0.1]:36493 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mxtS1-0003rY-Se
	for submit <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:17:56 -0500
Received: from lists.gnu.org ([209.51.188.17]:37690)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@HIDDEN>) id 1mxtRx-0003rN-Ab
 for submit <at> debbugs.gnu.org; Thu, 16 Dec 2021 11:17:48 -0500
Received: from eggs.gnu.org ([209.51.188.92]:55568)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <pukkamustard@HIDDEN>)
 id 1mxtRu-0007zZ-Gl
 for guix-patches@HIDDEN; Thu, 16 Dec 2021 11:17:43 -0500
Received: from mout01.posteo.de ([185.67.36.65]:54721)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <pukkamustard@HIDDEN>)
 id 1mxtRr-0002xU-RE
 for guix-patches@HIDDEN; Thu, 16 Dec 2021 11:17:42 -0500
Received: from submission (posteo.de [89.146.220.130]) 
 by mout01.posteo.de (Postfix) with ESMTPS id A7664240027
 for <guix-patches@HIDDEN>; Thu, 16 Dec 2021 17:17:36 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1639671456; bh=LCOS0Ruzr1yXnOxpCv/ZshQ+w7IBoj9ooeByvyiV5d8=;
 h=From:To:Cc:Subject:Date:From;
 b=owd1T1Cl32kJHXMZf3WHFiPOsjnFVOytH4ghqM0GkhJppwP2dm/+FhcWST8y0jQaY
 xaz4l+BbbwPuyq2VuvI6a5T3AR5gSTVIcDp78szjoVq5/1egww/hG5Xn+xt538NMSO
 bkLL6RW2bEKcnQ40FniNb4CeLpO5dJnjkHF7clurTmifj4HuQ1YIRf6FRtud1oivee
 3WSf/fz6ccTB+TzqjCLdVnYfCtK4iHo/Nscb5VOx/cB4qL2/zZUT+th9JHJ7O1SDyo
 oMWAft+8/jnyeQvogoCndtkL+Te/69ZSuMIB90XFQdk/MKs2NoXv7VXTSYp8MDL4so
 UMHM/LeItkubw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4JFHML4ndkz9rxQ;
 Thu, 16 Dec 2021 17:17:34 +0100 (CET)
From: pukkamustard <pukkamustard@HIDDEN>
To: guix-patches@HIDDEN
Subject: [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
Date: Thu, 16 Dec 2021 16:17:24 +0000
Message-Id: <20211216161724.547-1-pukkamustard@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Received-SPF: pass client-ip=185.67.36.65;
 envelope-from=pukkamustard@HIDDEN; helo=mout01.posteo.de
X-Spam_score_int: -43
X-Spam_score: -4.4
X-Spam_bar: ----
X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
Cc: pukkamustard <pukkamustard@HIDDEN>, ~pukkamustard/eris@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hello Guix,

This is an initial patch and proposal towards decentralizing substitute
distribution with ERIS.

ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
uniformly sized, encryped and content-addressed blocks. The original content
can be reconstructed only with access to a read capability, which can be
encoded as an URN.

One key advantage of ERIS is that the encoding is protocol agnostic. Any
protocol that can transfer small (32KiB) sized blocks referenced by the hash of
their content will do. This can be done with things such as GNUNet, IPFS,
OpenDHT, HTTP or a USB stick on a bicycle.

The following patch allows substitutes to be published over IPFS using ERIS.
This is inspired and very similar to previous work on distributing substitutes
over IPFS [2].

The narinfos served by `guix publish` look like this:

--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: gzip
FileSize: 67363
ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: zstd
FileSize: 64917
ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w
NarSize: 220704
References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib
Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

For every compressed nar the ERIS URN is computed and added.

If the `--ipfs` is used for `guix publish` then the encoded blocks are also
uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
this:

--8<---------------cut here---------------start------------->8---
(use-modules (eris)
	     (eris blocks ipfs))

(eris-decode->bytevector
 "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
 eris-blocks-ipfs-ref)
--8<---------------cut here---------------end--------------->8---

These patches do not yet retrieve content from IPFS (TODO). But in principle,
anybody connected to IPFS can get the nar with the ERIS URN. This could be used
to reduce load on substitute server as they would only need to publish the ERIS
URN directly - substitutes could be delivered much more peer-to-peer.

Other transports that I have been looking in to and am pretty sure will work
include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
identifiers (URN) are abstracted away from specific transports (and also
applications). ERIS is almost exactly the same encoding as used in GNUNet
(ECRS).

Blocks can be stored in any kind of databases (see for example the GDBM
bindings [4]).

A tricky things is figuring out how to multiplex all these different
transports and storages...

The ERIS specification is still considered "experimental". However we feel
confident to stabilize it and intend to do so around February/March 2022 with a
release 1.0.0 of the specification. This will ensure that the identifiers
remain stable for the forseeable future (until the crypto breaks). Before that
there is also a small external security audit of the specification planned
(thanks to NGI0/NLnet!).

This is just a little demo of the idea and some food for thought and
discussion. Give it a try and let me know what you think!

I've also pushed the patches to my personal Guix mirror if you want to check it
out from there:

https://inqlab.net/git/guix.git/log/?h=wip-eris

Also CCing ~pukkamustard/eris@HIDDEN where there is some general ERIS
related discussion.

Thanks,
-pukkamustard

[1] http://purl.org/eris
[2] https://issues.guix.gnu.org/33899
[3] https://www.ietf.org/rfc/rfc2169.txt
[4] https://inqlab.net/git/guile-eris.git/tree/eris/blocks/gdbm.scm


pukkamustard (3):
  publish: Add ERIS URN to narinfo
  WIP: gnu: guile-eris: Update to unreleased git version.
  publish: Add IPFS support.

 configure.ac                        |  5 ++
 gnu/packages/guile-xyz.scm          | 10 ++--
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 10 ++--
 guix/scripts/publish.scm            | 79 ++++++++++++++++++++++-------
 5 files changed, 79 insertions(+), 26 deletions(-)

-- 
2.34.0





Acknowledgement sent to pukkamustard <pukkamustard@HIDDEN>:
New bug report received and forwarded. Copy sent to guix-patches@HIDDEN. Full text available.
Report forwarded to guix-patches@HIDDEN:
bug#52555; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Thu, 23 Dec 2021 12:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.