GNU bug report logs - #76003
[PATCH] distribute NARs through IPFS.

Previous Next

To reply to this bug, email your comments to 76003 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Justin Veilleux <terramorpha <at> cock.li>
To: guix-patches <at> gnu.org, ludo <at> gnu.org
Subject: [PATCH] distribute NARs through IPFS.
Date: Sat, 01 Feb 2025 23:36:18 -0500

[Message part 1 (text/plain, inline)]

Hello everyone.

I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899 (which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.

However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.

The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.

on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.

These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.

WDYT?

[0001-gnu-substitute-Add-support-for-downloading-nars-over.patch (text/x-patch, attachment)]

[0002-gnu-publish-Add-support-for-distributing-nars-throug.patch (text/x-patch, attachment)]

[0003-doc-publish-Add-documentation-for-the-ipfs-api-argum.patch (text/x-patch, attachment)]

[0004-services-guix-publish-Expose-and-document-the-ipfs-a.patch (text/x-patch, attachment)]

Message #8 received at 76003 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Justin Veilleux <terramorpha <at> cock.li>
Cc: ludo <at> gnu.org, 76003 <at> debbugs.gnu.org
Subject: Re: bug#76003: [PATCH] distribute NARs through IPFS.
Date: Wed, 05 Feb 2025 15:22:05 +0900

Hi,

Justin Veilleux <terramorpha <at> cock.li> writes:

[...]

> These changes don't exploit every advantage of IPFS delivery
> (deduplication, "true decentralization", etc), but enable a non-trivial
> use-case: many machines on a local network can now distribute amongst
> themselves substitutes (thus reducing the total needed bandwidth)
> transparently.

Note that this is already possible via mdns discovery of substitute
servers advertising themselves via mDNS/DNS-SD (that's the
'--advertise?' option of guix-publish, coupled with the '--discover'
option of guix-daemon).

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.