GNU logs - #33899, boring messages

Message sent to guix-patches@HIDDEN:

Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:14:01 +0000
Resent-Message-ID: <handler.33899.B.15460387815093 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 33899 <at> debbugs.gnu.org
Cc: Hector Sanjuan <code@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:12:05 +0100
Message-Id: <20181228231205.8068-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello Guix!

Here is a first draft adding support to distribute and retrieve substitutes
over IPFS.  This builds on discussions at the R-B Summit with Héctor Sanjuan
of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian
Paul Schmidt posted on guix-devel last month.

The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides
bindings to a subset of that API.  This module also implements a custom
“directory” format to store directory trees in IPFS (IPFS already provides
“UnixFS” and “tar” but they store too many or too few file attributes.)

‘guix publish’ and ‘guix substitute’ use (guix ipfs) to
store and retrieve store items.  Complete directory trees are stored in
IPFS “as is”, rather than as compressed archives (nars).  This allows for
deduplication in IPFS.  ‘guix publish’ adds a new “IPFS” field in
narinfos and ‘guix substitute’ can then query those objects over IPFS.
So the idea is that you still get narinfos over HTTP(S), and then you
have the option of downloading substitutes over IPFS.

I’ve pushed these patches in ‘wip-ipfs-substitutes’.  This is rough on the
edges and probably buggy, but the adventurous among us might want to give
it a spin.  :-)

Thanks,
Ludo’.

Ludovic Courtès (5):
  Add (guix json).
  tests: 'file=?' now recurses on directories.
  Add (guix ipfs).
  publish: Add IPFS support.
  DRAFT substitute: Add IPFS support.

 Makefile.am                 |   3 +
 doc/guix.texi               |  33 +++++
 guix/ipfs.scm               | 250 ++++++++++++++++++++++++++++++++++++
 guix/json.scm               |  63 +++++++++
 guix/scripts/publish.scm    |  67 +++++++---
 guix/scripts/substitute.scm | 106 ++++++++-------
 guix/swh.scm                |  35 +----
 guix/tests.scm              |  26 +++-
 tests/ipfs.scm              |  55 ++++++++
 9 files changed, 535 insertions(+), 103 deletions(-)
 create mode 100644 guix/ipfs.scm
 create mode 100644 guix/json.scm
 create mode 100644 tests/ipfs.scm

-- 
2.20.1

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: bug#33899: Acknowledgement ([PATCH 0/5] Distributing substitutes
 over IPFS)
Message-ID: <handler.33899.B.15460387815093.ack <at> debbugs.gnu.org>
References: <20181228231205.8068-1-ludo@HIDDEN>
X-Gnu-PR-Message: ack 33899
X-Gnu-PR-Package: guix-patches
X-Gnu-PR-Keywords: patch
Reply-To: 33899 <at> debbugs.gnu.org
Date: Fri, 28 Dec 2018 23:14:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 guix-patches@HIDDEN

If you wish to submit further information on this problem, please
send it to 33899 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
33899: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33899
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to guix-patches@HIDDEN:

Subject: [bug#33899] [PATCH 2/5] tests: 'file=?' now recurses on directories.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:17:02 +0000
Resent-Message-ID: <handler.33899.B33899.15460389765450 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 33899 <at> debbugs.gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:15:51 +0100
Message-Id: <20181228231554.8220-2-ludo@HIDDEN>
In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN>
References: <20181228231554.8220-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

* guix/tests.scm (not-dot?): New procedure.
(file=?)[executable?]: New procedure.
In 'regular case, check whether the executable bit is preserved.
Add 'directory case.
---
 guix/tests.scm | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/guix/tests.scm b/guix/tests.scm
index f4948148c4..c9ae2718e4 100644
--- a/guix/tests.scm
+++ b/guix/tests.scm
@@ -26,9 +26,12 @@
   #:use-module (gcrypt hash)
   #:use-module (guix build-system gnu)
   #:use-module (gnu packages bootstrap)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-64)
   #:use-module (rnrs bytevectors)
+  #:use-module (ice-9 ftw)
   #:use-module (ice-9 binary-ports)
   #:use-module (web uri)
   #:export (open-connection-for-tests
@@ -138,16 +141,31 @@ too expensive to build entirely in the test store."
             (loop (1+ i)))
           bv))))
 
+(define (not-dot? entry)
+  (not (member entry '("." ".."))))
+
 (define (file=? a b)
-  "Return true if files A and B have the same type and same content."
+  "Return true if files A and B have the same type and same content,
+recursively."
+  (define (executable? file)
+    (->bool (logand (stat:mode (lstat file)) #o100)))
+
   (and (eq? (stat:type (lstat a)) (stat:type (lstat b)))
        (case (stat:type (lstat a))
          ((regular)
-          (equal?
-           (call-with-input-file a get-bytevector-all)
-           (call-with-input-file b get-bytevector-all)))
+          (and (eqv? (executable? a) (executable? b))
+               (equal?
+                (call-with-input-file a get-bytevector-all)
+                (call-with-input-file b get-bytevector-all))))
          ((symlink)
           (string=? (readlink a) (readlink b)))
+         ((directory)
+          (let ((lst1 (scandir a not-dot?))
+                (lst2 (scandir b not-dot?)))
+            (and (equal? lst1 lst2)
+                 (every file=?
+                        (map (cut string-append a "/" <>) lst1)
+                        (map (cut string-append b "/" <>) lst2)))))
          (else
           (error "what?" (lstat a))))))
 
-- 
2.20.1

Message sent to guix-patches@HIDDEN:

Subject: [bug#33899] [PATCH 1/5] Add (guix json).
References: <20181228231205.8068-1-ludo@HIDDEN>
In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN>
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:17:02 +0000
Resent-Message-ID: <handler.33899.B33899.15460389775457 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 33899 <at> debbugs.gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:15:50 +0100
Message-Id: <20181228231554.8220-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

* guix/swh.scm: Use (guix json).
(define-json-reader, define-json-mapping): Move to...
* guix/json.scm: ... here.  New file.
* Makefile.am (MODULES): Add it.
---
 Makefile.am   |  1 +
 guix/json.scm | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++
 guix/swh.scm  | 35 +---------------------------
 3 files changed, 65 insertions(+), 34 deletions(-)
 create mode 100644 guix/json.scm

diff --git a/Makefile.am b/Makefile.am
index 0e5ca02ed3..da3720e3a6 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -77,6 +77,7 @@ MODULES =					\
   guix/discovery.scm				\
   guix/git-download.scm				\
   guix/hg-download.scm				\
+  guix/json.scm					\
   guix/swh.scm					\
   guix/monads.scm				\
   guix/monad-repl.scm				\
diff --git a/guix/json.scm b/guix/json.scm
new file mode 100644
index 0000000000..d446f6894e
--- /dev/null
+++ b/guix/json.scm
@@ -0,0 +1,63 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix json)
+  #:use-module (json)
+  #:use-module (srfi srfi-9)
+  #:export (define-json-mapping))
+
+;;; Commentary:
+;;;
+;;; This module provides tools to define mappings from JSON objects to SRFI-9
+;;; records.  This is useful when writing bindings to HTTP APIs.
+;;;
+;;; Code:
+
+(define-syntax-rule (define-json-reader json->record ctor spec ...)
+  "Define JSON->RECORD as a procedure that converts a JSON representation,
+read from a port, string, or hash table, into a record created by CTOR and
+following SPEC, a series of field specifications."
+  (define (json->record input)
+    (let ((table (cond ((port? input)
+                        (json->scm input))
+                       ((string? input)
+                        (json-string->scm input))
+                       ((hash-table? input)
+                        input))))
+      (let-syntax ((extract-field (syntax-rules ()
+                                    ((_ table (field key json->value))
+                                     (json->value (hash-ref table key)))
+                                    ((_ table (field key))
+                                     (hash-ref table key))
+                                    ((_ table (field))
+                                     (hash-ref table
+                                               (symbol->string 'field))))))
+        (ctor (extract-field table spec) ...)))))
+
+(define-syntax-rule (define-json-mapping rtd ctor pred json->record
+                      (field getter spec ...) ...)
+  "Define RTD as a record type with the given FIELDs and GETTERs, à la SRFI-9,
+and define JSON->RECORD as a conversion from JSON to a record of this type."
+  (begin
+    (define-record-type rtd
+      (ctor field ...)
+      pred
+      (field getter) ...)
+
+    (define-json-reader json->record ctor
+      (field spec ...) ...)))
diff --git a/guix/swh.scm b/guix/swh.scm
index 89cddb2bdd..c5f2153a22 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -23,6 +23,7 @@
   #:use-module (web client)
   #:use-module (web response)
   #:use-module (json)
+  #:use-module (guix json)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-11)
@@ -127,40 +128,6 @@
       url
       (string-append url "/")))
 
-(define-syntax-rule (define-json-reader json->record ctor spec ...)
-  "Define JSON->RECORD as a procedure that converts a JSON representation,
-read from a port, string, or hash table, into a record created by CTOR and
-following SPEC, a series of field specifications."
-  (define (json->record input)
-    (let ((table (cond ((port? input)
-                        (json->scm input))
-                       ((string? input)
-                        (json-string->scm input))
-                       ((hash-table? input)
-                        input))))
-      (let-syntax ((extract-field (syntax-rules ()
-                                    ((_ table (field key json->value))
-                                     (json->value (hash-ref table key)))
-                                    ((_ table (field key))
-                                     (hash-ref table key))
-                                    ((_ table (field))
-                                     (hash-ref table
-                                               (symbol->string 'field))))))
-        (ctor (extract-field table spec) ...)))))
-
-(define-syntax-rule (define-json-mapping rtd ctor pred json->record
-                      (field getter spec ...) ...)
-  "Define RTD as a record type with the given FIELDs and GETTERs, à la SRFI-9,
-and define JSON->RECORD as a conversion from JSON to a record of this type."
-  (begin
-    (define-record-type rtd
-      (ctor field ...)
-      pred
-      (field getter) ...)
-
-    (define-json-reader json->record ctor
-      (field spec ...) ...)))
-
 (define %date-regexp
   ;; Match strings like "2014-11-17T22:09:38+01:00" or
   ;; "2018-09-30T23:20:07.815449+00:00"".
-- 
2.20.1

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 5/5] DRAFT substitute: Add IPFS support.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:17:03 +0000
Resent-Message-ID: <handler.33899.B33899.15460389825481 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 33899 <at> debbugs.gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15460389825481
          (code B ref 33899); Fri, 28 Dec 2018 23:17:03 +0000
Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:22 +0000
Received: from localhost ([127.0.0.1]:40713 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gd1Mb-0001QJ-W9
	for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:22 -0500
Received: from eggs.gnu.org ([208.118.235.92]:35129)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gd1Ma-0001Pn-2Y
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:20 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gd1MP-0002Hy-1D
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:14 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51207)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1gd1MJ-00028O-3F; Fri, 28 Dec 2018 18:16:03 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82)
 (envelope-from <ludo@HIDDEN>)
 id 1gd1MI-00077n-Pb; Fri, 28 Dec 2018 18:16:02 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:15:54 +0100
Message-Id: <20181228231554.8220-5-ludo@HIDDEN>
X-Mailer: git-send-email 2.20.1
In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN>
References: <20181228231554.8220-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

Missing:

  - documentation
  - command-line options
  - progress report when downloading over IPFS
  - fallback when we fail to fetch from IPFS

* guix/scripts/substitute.scm (<narinfo>)[ipfs]: New field.
(read-narinfo): Read "IPFS".
(process-substitution/http): New procedure, with code formerly in
'process-substitution'.
(process-substitution): Check for IPFS and call 'ipfs:restore-file-tree'
when IPFS is true.
---
 guix/scripts/substitute.scm | 106 +++++++++++++++++++++---------------
 1 file changed, 61 insertions(+), 45 deletions(-)

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 53b1777241..8be15e4f13 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -42,6 +42,7 @@
   #:use-module (guix progress)
   #:use-module ((guix build syscalls)
                 #:select (set-thread-name))
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:use-module (ice-9 rdelim)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 match)
@@ -281,7 +282,7 @@ failure, return #f and #f."
 
 (define-record-type <narinfo>
   (%make-narinfo path uri uri-base compression file-hash file-size nar-hash nar-size
-                 references deriver system signature contents)
+                 references deriver system ipfs signature contents)
   narinfo?
   (path         narinfo-path)
   (uri          narinfo-uri)
@@ -294,6 +295,7 @@ failure, return #f and #f."
   (references   narinfo-references)
   (deriver      narinfo-deriver)
   (system       narinfo-system)
+  (ipfs         narinfo-ipfs)
   (signature    narinfo-signature)      ; canonical sexp
   ;; The original contents of a narinfo file.  This field is needed because we
   ;; want to preserve the exact textual representation for verification purposes.
@@ -335,7 +337,7 @@ s-expression: ~s~%")
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
   (lambda (path url compression file-hash file-size nar-hash nar-size
-                references deriver system signature)
+                references deriver system ipfs signature)
     "Return a new <narinfo> object."
     (%make-narinfo path
                    ;; Handle the case where URL is a relative URL.
@@ -352,6 +354,7 @@ must contain the original contents of a narinfo file."
                      ((or #f "") #f)
                      (_ deriver))
                    system
+                   ipfs
                    (false-if-exception
                     (and=> signature narinfo-signature->canonical-sexp))
                    str)))
@@ -386,7 +389,7 @@ No authentication and authorization checks are performed here!"
                    (narinfo-maker str url)
                    '("StorePath" "URL" "Compression"
                      "FileHash" "FileSize" "NarHash" "NarSize"
-                     "References" "Deriver" "System"
+                     "References" "Deriver" "System" "IPFS"
                      "Signature"))))
 
 (define (narinfo-sha256 narinfo)
@@ -947,13 +950,58 @@ authorized substitutes."
     (wtf
      (error "unknown `--query' command" wtf))))
 
+(define* (process-substitution/http narinfo destination uri
+                                    #:key print-build-trace?)
+  (unless print-build-trace?
+    (format (current-error-port)
+            (G_ "Downloading ~a...~%") (uri->string uri)))
+
+  (let*-values (((raw download-size)
+                 ;; Note that Hydra currently generates Nars on the fly
+                 ;; and doesn't specify a Content-Length, so
+                 ;; DOWNLOAD-SIZE is #f in practice.
+                 (fetch uri #:buffered? #f #:timeout? #f))
+                ((progress)
+                 (let* ((comp     (narinfo-compression narinfo))
+                        (dl-size  (or download-size
+                                      (and (equal? comp "none")
+                                           (narinfo-size narinfo))))
+                        (reporter (if print-build-trace?
+                                      (progress-reporter/trace
+                                       destination
+                                       (uri->string uri) dl-size
+                                       (current-error-port))
+                                      (progress-reporter/file
+                                       (uri->string uri) dl-size
+                                       (current-error-port)
+                                       #:abbreviation nar-uri-abbreviation))))
+                   (progress-report-port reporter raw)))
+                ((input pids)
+                 ;; NOTE: This 'progress' port of current process will be
+                 ;; closed here, while the child process doing the
+                 ;; reporting will close it upon exit.
+                 (decompressed-port (and=> (narinfo-compression narinfo)
+                                           string->symbol)
+                                    progress)))
+    ;; Unpack the Nar at INPUT into DESTINATION.
+    (restore-file input destination)
+    (close-port input)
+
+    ;; Wait for the reporter to finish.
+    (every (compose zero? cdr waitpid) pids)
+
+    ;; Skip a line after what 'progress-reporter/file' printed, and another
+    ;; one to visually separate substitutions.
+    (display "\n\n" (current-error-port))))
+
 (define* (process-substitution store-item destination
                                #:key cache-urls acl print-build-trace?)
   "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
 DESTINATION as a nar file.  Verify the substitute against ACL."
   (let* ((narinfo (lookup-narinfo cache-urls store-item
                                   (cut valid-narinfo? <> acl)))
-         (uri     (and=> narinfo narinfo-uri)))
+         (uri     (and=> narinfo narinfo-uri))
+         (ipfs    (and=> narinfo narinfo-ipfs)))
     (unless uri
       (leave (G_ "no valid substitute for '~a'~%")
              store-item))
@@ -961,47 +1009,15 @@ DESTINATION as a nar file.  Verify the substitute against ACL."
     ;; Tell the daemon what the expected hash of the Nar itself is.
     (format #t "~a~%" (narinfo-hash narinfo))
 
-    (unless print-build-trace?
-      (format (current-error-port)
-              (G_ "Downloading ~a...~%") (uri->string uri)))
-
-    (let*-values (((raw download-size)
-                   ;; Note that Hydra currently generates Nars on the fly
-                   ;; and doesn't specify a Content-Length, so
-                   ;; DOWNLOAD-SIZE is #f in practice.
-                   (fetch uri #:buffered? #f #:timeout? #f))
-                  ((progress)
-                   (let* ((comp     (narinfo-compression narinfo))
-                          (dl-size  (or download-size
-                                        (and (equal? comp "none")
-                                             (narinfo-size narinfo))))
-                          (reporter (if print-build-trace?
-                                        (progress-reporter/trace
-                                         destination
-                                         (uri->string uri) dl-size
-                                         (current-error-port))
-                                        (progress-reporter/file
-                                         (uri->string uri) dl-size
-                                         (current-error-port)
-                                         #:abbreviation nar-uri-abbreviation))))
-                     (progress-report-port reporter raw)))
-                  ((input pids)
-                   ;; NOTE: This 'progress' port of current process will be
-                   ;; closed here, while the child process doing the
-                   ;; reporting will close it upon exit.
-                   (decompressed-port (and=> (narinfo-compression narinfo)
-                                             string->symbol)
-                                      progress)))
-      ;; Unpack the Nar at INPUT into DESTINATION.
-      (restore-file input destination)
-      (close-port input)
-
-      ;; Wait for the reporter to finish.
-      (every (compose zero? cdr waitpid) pids)
-
-      ;; Skip a line after what 'progress-reporter/file' printed, and another
-      ;; one to visually separate substitutions.
-      (display "\n\n" (current-error-port)))))
+    (if ipfs
+        (begin
+          (unless print-build-trace?
+            (format (current-error-port)
+                    (G_ "Downloading from IPFS ~s...~%") ipfs))
+          (ipfs:restore-file-tree ipfs destination))
+        (process-substitution/http narinfo destination uri
+                                   #:print-build-trace?
+                                   print-build-trace?))))
 
 
 ;;;
-- 
2.20.1

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 4/5] publish: Add IPFS support.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:17:03 +0000
Resent-Message-ID: <handler.33899.B33899.15460389835493 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 33899 <at> debbugs.gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15460389835493
          (code B ref 33899); Fri, 28 Dec 2018 23:17:03 +0000
Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:23 +0000
Received: from localhost ([127.0.0.1]:40716 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gd1Mc-0001QM-HW
	for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:23 -0500
Received: from eggs.gnu.org ([208.118.235.92]:32852)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gd1Ma-0001Pm-23
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:20 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gd1MP-0002Hs-0F
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:14 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51206)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1gd1MI-00028B-GX; Fri, 28 Dec 2018 18:16:03 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82)
 (envelope-from <ludo@HIDDEN>)
 id 1gd1MI-00077n-33; Fri, 28 Dec 2018 18:16:02 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:15:53 +0100
Message-Id: <20181228231554.8220-4-ludo@HIDDEN>
X-Mailer: git-send-email 2.20.1
In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN>
References: <20181228231554.8220-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

* guix/scripts/publish.scm (show-help, %options): Add '--ipfs'.
(narinfo-string): Add IPFS parameter and honor it.
(render-narinfo/cached): Add #:ipfs? and honor it.
(bake-narinfo+nar, make-request-handler, run-publish-server): Likewise.
(guix-publish): Honor '--ipfs' and parameterize %IPFS-BASE-URL.
---
 doc/guix.texi            | 33 ++++++++++++++++++++
 guix/scripts/publish.scm | 67 ++++++++++++++++++++++++++++------------
 2 files changed, 80 insertions(+), 20 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index fcb5b8c088..f2af5a1558 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -8470,6 +8470,15 @@ caching of the archives before they are sent to clients---see below for
 details.  The @command{guix weather} command provides a handy way to
 check what a server provides (@pxref{Invoking guix weather}).
 
+@cindex peer-to-peer, substitute distribution
+@cindex distributed storage, of substitutes
+@cindex IPFS, for substitutes
+It is also possible to publish substitutes over @uref{https://ipfs.io, IFPS},
+a distributed, peer-to-peer storage mechanism.  To enable it, pass the
+@option{--ipfs} option alongside @option{--cache}, and make sure you're
+running @command{ipfs daemon}.  Capable clients will then be able to choose
+whether to fetch substitutes over HTTP or over IPFS.
+
 As a bonus, @command{guix publish} also serves as a content-addressed
 mirror for source files referenced in @code{origin} records
 (@pxref{origin Reference}).  For instance, assuming @command{guix
@@ -8560,6 +8569,30 @@ thread per CPU core is created, but this can be customized.  See
 When @option{--ttl} is used, cached entries are automatically deleted
 when they have expired.
 
+@item --ifps[=@var{gateway}]
+When used in conjunction with @option{--cache}, instruct @command{guix
+publish} to publish substitutes over the @uref{https://ipfs.io, IPFS
+distributed data store} in addition to HTTP.
+
+@quotation Note
+As of version @value{VERSION}, IPFS support is experimental.  You're welcome
+to share your experience with the developers by emailing
+@email{guix-devel@@gnu.org}!
+@end quotation
+
+The IPFS HTTP interface must be reachable at @var{gateway}, by default
+@code{localhost:5001}.  To get it up and running, it is usually enough to
+install IPFS and start the IPFS daemon:
+
+@example
+$ guix package -i go-ipfs
+$ ipfs init
+$ ipfs daemon
+@end example
+
+For more information on how to get started with IPFS, please refer to the
+@uref{https://docs.ipfs.io/introduction/usage/, IPFS documentation}.
+
 @item --workers=@var{N}
 When @option{--cache} is used, request the allocation of @var{N} worker
 threads to ``bake'' archives.
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index a236f3e45c..2accd632ab 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -59,6 +59,7 @@
   #:use-module ((guix build utils)
                 #:select (dump-port mkdir-p find-files))
   #:use-module ((guix build syscalls) #:select (set-thread-name))
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:export (%public-key
             %private-key
 
@@ -78,6 +79,8 @@ Publish ~a over HTTP.\n") %store-directory)
                          compress archives at LEVEL"))
   (display (G_ "
   -c, --cache=DIRECTORY  cache published items to DIRECTORY"))
+  (display (G_ "
+      --ipfs[=GATEWAY]   publish items over IPFS via GATEWAY"))
   (display (G_ "
       --workers=N        use N workers to bake items"))
   (display (G_ "
@@ -168,6 +171,10 @@ compression disabled~%"))
         (option '(#\c "cache") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'cache arg result)))
+        (option '("ipfs") #f #t
+                (lambda (opt name arg result)
+                  (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url))
+                              result)))
         (option '("workers") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'workers (string->number* arg)
@@ -237,12 +244,15 @@ compression disabled~%"))
 
 (define* (narinfo-string store store-path key
                          #:key (compression %no-compression)
-                         (nar-path "nar") file-size)
+                         (nar-path "nar") file-size ipfs)
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
+
 Optionally, FILE-SIZE can specify the size in bytes of the compressed NAR; it
-informs the client of how much needs to be downloaded."
+informs the client of how much needs to be downloaded.
+
+When IPFS is true, it is the IPFS object identifier for STORE-PATH."
   (let* ((path-info  (query-path-info store store-path))
          (compression (actual-compression store-path compression))
          (url        (encode-and-join-uri-path
@@ -295,7 +305,12 @@ References: ~a~%~a"
                                  (apply throw args))))))
          (signature  (base64-encode-string
                       (canonical-sexp->string (signed-string info)))))
-    (format #f "~aSignature: 1;~a;~a~%" info (gethostname) signature)))
+    (format #f "~aSignature: 1;~a;~a~%~a" info (gethostname) signature
+
+            ;; Append IPFS info below the signed part.
+            (if ipfs
+                (string-append "IPFS: " ipfs "\n")
+                ""))))
 
 (define* (not-found request
                     #:key (phrase "Resource not found")
@@ -406,10 +421,12 @@ items.  Failing that, we could eventually have to recompute them and return
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compression %no-compression)
                                 (nar-path "nar")
-                                cache pool)
+                                cache pool ipfs?)
   "Respond to the narinfo request for REQUEST.  If the narinfo is available in
 CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
-requested using POOL."
+requested using POOL.
+
+When IPFS? is true, additionally publish binaries over IPFS."
   (define (delete-entry narinfo)
     ;; Delete NARINFO and the corresponding nar from CACHE.
     (let ((nar (string-append (string-drop-right narinfo
@@ -447,7 +464,8 @@ requested using POOL."
                  (bake-narinfo+nar cache item
                                    #:ttl ttl
                                    #:compression compression
-                                   #:nar-path nar-path)))
+                                   #:nar-path nar-path
+                                   #:ipfs? ipfs?)))
 
              (when ttl
                (single-baker 'cache-cleanup
@@ -465,7 +483,7 @@ requested using POOL."
 
 (define* (bake-narinfo+nar cache item
                            #:key ttl (compression %no-compression)
-                           (nar-path "/nar"))
+                           (nar-path "/nar") ipfs?)
   "Write the narinfo and nar for ITEM to CACHE."
   (let* ((compression (actual-compression item compression))
          (nar         (nar-cache-file cache item
@@ -502,7 +520,11 @@ requested using POOL."
                                    #:nar-path nar-path
                                    #:compression compression
                                    #:file-size (and=> (stat nar #f)
-                                                      stat:size))
+                                                      stat:size)
+                                   #:ipfs
+                                   (and ipfs?
+                                        (ipfs:content-name
+                                         (ipfs:add-file-tree item))))
                    port))))))
 
 ;; XXX: Declare the 'X-Nar-Compression' HTTP header, which is in fact for
@@ -766,7 +788,8 @@ blocking."
                                cache pool
                                narinfo-ttl
                                (nar-path "nar")
-                               (compression %no-compression))
+                               (compression %no-compression)
+                               ipfs?)
   (define nar-path?
     (let ((expected (split-and-decode-uri-path nar-path)))
       (cut equal? expected <>)))
@@ -793,7 +816,8 @@ blocking."
                                       #:pool pool
                                       #:ttl narinfo-ttl
                                       #:nar-path nar-path
-                                      #:compression compression)
+                                      #:compression compression
+                                      #:ipfs? ipfs?)
                (render-narinfo store request hash
                                #:ttl narinfo-ttl
                                #:nar-path nar-path
@@ -847,13 +871,14 @@ blocking."
 (define* (run-publish-server socket store
                              #:key (compression %no-compression)
                              (nar-path "nar") narinfo-ttl
-                             cache pool)
+                             cache pool ipfs?)
   (run-server (make-request-handler store
                                     #:cache cache
                                     #:pool pool
                                     #:nar-path nar-path
                                     #:narinfo-ttl narinfo-ttl
-                                    #:compression compression)
+                                    #:compression compression
+                                    #:ipfs? ipfs?)
               concurrent-http-server
               `(#:socket ,socket)))
 
@@ -902,6 +927,7 @@ blocking."
            (repl-port (assoc-ref opts 'repl))
            (cache     (assoc-ref opts 'cache))
            (workers   (assoc-ref opts 'workers))
+           (ipfs      (assoc-ref opts 'ipfs))
 
            ;; Read the key right away so that (1) we fail early on if we can't
            ;; access them, and (2) we can then drop privileges.
@@ -930,14 +956,15 @@ consider using the '--user' option!~%")))
         (set-thread-name "guix publish")
 
         (with-store store
-          (run-publish-server socket store
-                              #:cache cache
-                              #:pool (and cache (make-pool workers
-                                                           #:thread-name
-                                                           "publish worker"))
-                              #:nar-path nar-path
-                              #:compression compression
-                              #:narinfo-ttl ttl))))))
+          (parameterize ((ipfs:%ipfs-base-url ipfs))
+            (run-publish-server socket store
+                                #:cache cache
+                                #:pool (and cache (make-pool workers
+                                                             #:thread-name
+                                                             "publish worker"))
+                                #:nar-path nar-path
+                                #:compression compression
+                                #:narinfo-ttl ttl)))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'single-baker 'scheme-indent-function 1)
-- 
2.20.1

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 3/5] Add (guix ipfs).
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 28 Dec 2018 23:17:04 +0000
Resent-Message-ID: <handler.33899.B33899.15460389895506 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 33899 <at> debbugs.gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15460389895506
          (code B ref 33899); Fri, 28 Dec 2018 23:17:04 +0000
Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:29 +0000
Received: from localhost ([127.0.0.1]:40718 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gd1Mi-0001Qj-AP
	for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:28 -0500
Received: from eggs.gnu.org ([208.118.235.92]:56171)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gd1Mc-0001Q4-54
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:22 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <ludo@HIDDEN>) id 1gd1MT-0002LC-0x
 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:17 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51204)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1gd1MH-00027t-Pg; Fri, 28 Dec 2018 18:16:03 -0500
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82)
 (envelope-from <ludo@HIDDEN>)
 id 1gd1MH-00077n-Cc; Fri, 28 Dec 2018 18:16:01 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Date: Sat, 29 Dec 2018 00:15:52 +0100
Message-Id: <20181228231554.8220-3-ludo@HIDDEN>
X-Mailer: git-send-email 2.20.1
In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN>
References: <20181228231554.8220-1-ludo@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

* guix/ipfs.scm, tests/ipfs.scm: New files.
* Makefile.am (MODULES, SCM_TESTS): Add them.
---
 Makefile.am    |   2 +
 guix/ipfs.scm  | 250 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ipfs.scm |  55 +++++++++++
 3 files changed, 307 insertions(+)
 create mode 100644 guix/ipfs.scm
 create mode 100644 tests/ipfs.scm

diff --git a/Makefile.am b/Makefile.am
index da3720e3a6..975d83db6c 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -101,6 +101,7 @@ MODULES =					\
   guix/cve.scm					\
   guix/workers.scm				\
   guix/zlib.scm					\
+  guix/ipfs.scm					\
   guix/build-system.scm				\
   guix/build-system/android-ndk.scm		\
   guix/build-system/ant.scm			\
@@ -384,6 +385,7 @@ SCM_TESTS =					\
   tests/cve.scm					\
   tests/workers.scm				\
   tests/zlib.scm				\
+  tests/ipfs.scm				\
   tests/file-systems.scm			\
   tests/uuid.scm				\
   tests/system.scm				\
diff --git a/guix/ipfs.scm b/guix/ipfs.scm
new file mode 100644
index 0000000000..e941feda6f
--- /dev/null
+++ b/guix/ipfs.scm
@@ -0,0 +1,250 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix ipfs)
+  #:use-module (guix json)
+  #:use-module (guix base64)
+  #:use-module ((guix build utils) #:select (dump-port))
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-26)
+  #:use-module (rnrs io ports)
+  #:use-module (rnrs bytevectors)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 ftw)
+  #:use-module (web uri)
+  #:use-module (web client)
+  #:use-module (web response)
+  #:export (%ipfs-base-url
+            add-file
+            add-file-tree
+            restore-file-tree
+
+            content?
+            content-name
+            content-hash
+            content-size
+
+            add-empty-directory
+            add-to-directory
+            read-contents
+            publish-name))
+
+;;; Commentary:
+;;;
+;;; This module implements bindings for the HTTP interface of the IPFS
+;;; gateway, documented here: <https://docs.ipfs.io/reference/api/http/>.  It
+;;; allows you to add and retrieve files over IPFS, and a few other things.
+;;;
+;;; Code:
+
+(define %ipfs-base-url
+  ;; URL of the IPFS gateway.
+  (make-parameter "http://localhost:5001"))
+
+(define* (call url decode #:optional (method http-post)
+               #:key body (false-if-404? #t) (headers '()))
+  "Invoke the endpoint at URL using METHOD.  Decode the resulting JSON body
+using DECODE, a one-argument procedure that takes an input port; when DECODE
+is false, return the input port.  When FALSE-IF-404? is true, return #f upon
+404 responses."
+  (let*-values (((response port)
+                 (method url #:streaming? #t
+                         #:body body
+
+                         ;; Always pass "Connection: close".
+                         #:keep-alive? #f
+                         #:headers `((connection close)
+                                     ,@headers))))
+    (cond ((= 200 (response-code response))
+           (if decode
+               (let ((result (decode port)))
+                 (close-port port)
+                 result)
+               port))
+          ((and false-if-404?
+                (= 404 (response-code response)))
+           (close-port port)
+           #f)
+          (else
+           (close-port port)
+           (throw 'ipfs-error url response)))))
+
+;; Result of a file addition.
+(define-json-mapping <content> make-content content?
+  json->content
+  (name   content-name "Name")
+  (hash   content-hash "Hash")
+  (bytes  content-bytes "Bytes")
+  (size   content-size "Size" string->number))
+
+;; Result of a 'patch/add-link' operation.
+(define-json-mapping <directory> make-directory directory?
+  json->directory
+  (hash   directory-hash "Hash")
+  (links  directory-links "Links" json->links))
+
+;; A "link".
+(define-json-mapping <link> make-link link?
+  json->link
+  (name   link-name "Name")
+  (hash   link-hash "Hash")
+  (size   link-size "Size" string->number))
+
+;; A "binding", also known as a "name".
+(define-json-mapping <binding> make-binding binding?
+  json->binding
+  (name   binding-name "Name")
+  (value  binding-value "Value"))
+
+(define (json->links json)
+  (match json
+    (#f    '())
+    (links (map json->link links))))
+
+(define %multipart-boundary
+  ;; XXX: We might want to find a more reliable boundary.
+  (string-append (make-string 24 #\-) "2698127afd7425a6"))
+
+(define (bytevector->form-data bv port)
+  "Write to PORT a 'multipart/form-data' representation of BV."
+  (display (string-append "--" %multipart-boundary "\r\n"
+                          "Content-Disposition: form-data\r\n"
+                          "Content-Type: application/octet-stream\r\n\r\n")
+           port)
+  (put-bytevector port bv)
+  (display (string-append "\r\n--" %multipart-boundary "--\r\n")
+           port))
+
+(define* (add-data data #:key (name "file.txt") recursive?)
+  "Add DATA, a bytevector, to IPFS.  Return a content object representing it."
+  (call (string-append (%ipfs-base-url)
+                       "/api/v0/add?arg=" (uri-encode name)
+                       "&recursive="
+                       (if recursive? "true" "false"))
+        json->content
+        #:headers
+        `((content-type
+           . (multipart/form-data
+              (boundary . ,%multipart-boundary))))
+        #:body
+        (call-with-bytevector-output-port
+         (lambda (port)
+           (bytevector->form-data data port)))))
+
+(define (not-dot? entry)
+  (not (member entry '("." ".."))))
+
+(define (file-tree->sexp file)
+  "Add FILE, recursively, to the IPFS, and return an sexp representing the
+directory's tree structure.
+
+Unlike IPFS's own \"UnixFS\" structure, this format preserves exactly what we
+need: like the nar format, it preserves the executable bit, but does not save
+the mtime or other Unixy attributes irrelevant in the store."
+  ;; The natural approach would be to insert each directory listing as an
+  ;; object of its own in IPFS.  However, this does not buy us much in terms
+  ;; of deduplication, but it does cause a lot of extra round trips when
+  ;; fetching it.  Thus, this sexp is \"flat\" in that only the leaves are
+  ;; inserted into the IPFS.
+  (let ((st (lstat file)))
+    (match (stat:type st)
+      ('directory
+       (let* ((parent  file)
+              (entries (map (lambda (file)
+                              `(entry ,file
+                                      ,(file-tree->sexp
+                                        (string-append parent "/" file))))
+                            (scandir file not-dot?)))
+              (size    (fold (lambda (entry total)
+                               (match entry
+                                 (('entry name (kind value size))
+                                  (+ total size))))
+                             0
+                             entries)))
+         `(directory ,entries ,size)))
+      ('symlink
+       `(symlink ,(readlink file) 0))
+      ('regular
+       (let ((size (stat:size st)))
+         (if (zero? (logand (stat:mode st) #o100))
+             `(file ,(content-name (add-file file)) ,size)
+             `(executable ,(content-name (add-file file)) ,size)))))))
+
+(define (add-file-tree file)
+  "Add FILE to the IPFS, recursively, using our own canonical directory
+format.  Return the resulting content object."
+  (add-data (string->utf8 (object->string
+                           `(file-tree (version 0)
+                                       ,(file-tree->sexp file))))))
+
+(define (restore-file-tree object file)
+  "Restore to FILE the tree pointed to by OBJECT."
+  (let restore ((tree (match (read (read-contents object))
+                        (('file-tree ('version 0) tree)
+                         tree)))
+                (file file))
+    (match tree
+      (('file object size)
+       (call-with-output-file file
+         (lambda (output)
+           (dump-port (read-contents object) output))))
+      (('executable object size)
+       (call-with-output-file file
+         (lambda (output)
+           (dump-port (read-contents object) output)))
+       (chmod file #o555))
+      (('symlink target size)
+       (symlink target file))
+      (('directory (('entry names entries) ...) size)
+       (mkdir file)
+       (for-each restore entries
+                 (map (cut string-append file "/" <>) names))))))
+
+(define* (add-file file #:key (name (basename file)))
+  "Add FILE under NAME to the IPFS and return a content object for it."
+  (add-data (match (call-with-input-file file get-bytevector-all)
+              ((? eof-object?) #vu8())
+              (bv bv))
+            #:name name))
+
+(define* (add-empty-directory #:key (name "directory"))
+  "Return a content object for an empty directory."
+  (add-data #vu8() #:recursive? #t #:name name))
+
+(define* (add-to-directory directory file name)
+  "Add FILE to DIRECTORY under NAME, and return the resulting directory.
+DIRECTORY and FILE must be hashes identifying objects in the IPFS store."
+  (call (string-append (%ipfs-base-url)
+                       "/api/v0/object/patch/add-link?arg="
+                       (uri-encode directory)
+                       "&arg=" (uri-encode name) "&arg=" (uri-encode file)
+                       "&create=true")
+        json->directory))
+
+(define* (read-contents object #:key offset length)
+  "Return an input port to read the content of OBJECT from."
+  (call (string-append (%ipfs-base-url)
+                       "/api/v0/cat?arg=" object)
+        #f))
+
+(define* (publish-name object)
+  "Publish OBJECT under the current peer ID."
+  (call (string-append (%ipfs-base-url)
+                       "/api/v0/name/publish?arg=" object)
+        json->binding))
diff --git a/tests/ipfs.scm b/tests/ipfs.scm
new file mode 100644
index 0000000000..3b662b22bd
--- /dev/null
+++ b/tests/ipfs.scm
@@ -0,0 +1,55 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-ipfs)
+  #:use-module (guix ipfs)
+  #:use-module ((guix utils) #:select (call-with-temporary-directory))
+  #:use-module (guix tests)
+  #:use-module (web uri)
+  #:use-module (srfi srfi-64))
+
+;; Test the (guix ipfs) module.
+
+(define (ipfs-gateway-running?)
+  "Return true if the IPFS gateway is running at %IPFS-BASE-URL."
+  (let* ((uri    (string->uri (%ipfs-base-url)))
+         (socket (socket AF_INET SOCK_STREAM 0)))
+    (define connected?
+      (catch 'system-error
+        (lambda ()
+          (format (current-error-port)
+                  "probing IPFS gateway at localhost:~a...~%"
+                  (uri-port uri))
+          (connect socket AF_INET INADDR_LOOPBACK (uri-port uri))
+          #t)
+        (const #f)))
+
+    (close-port socket)
+    connected?))
+
+(unless (ipfs-gateway-running?)
+  (test-skip 1))
+
+(test-assert "add-file-tree + restore-file-tree"
+  (call-with-temporary-directory
+   (lambda (directory)
+     (let* ((source  (dirname (search-path %load-path "guix/base32.scm")))
+            (target  (string-append directory "/r"))
+            (content (pk 'content (add-file-tree source))))
+       (restore-file-tree (content-name content) target)
+       (file=? source target)))))
-- 
2.20.1

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 07 Jan 2019 16:42:02 +0000
Resent-Message-ID: <handler.33899.B.15468793112034 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Cc: ludo@HIDDEN, mail@HIDDEN, go-ipfs-wg@HIDDEN
X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "guix-patches@HIDDEN" <guix-patches@HIDDEN>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by submit <at> debbugs.gnu.org id=B.15468793112034
          (code B ref -1); Mon, 07 Jan 2019 16:42:02 +0000
Received: (at submit) by debbugs.gnu.org; 7 Jan 2019 16:41:51 +0000
Received: from localhost ([127.0.0.1]:48899 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ggXyJ-0000Wc-0W
	for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 11:41:51 -0500
Received: from eggs.gnu.org ([209.51.188.92]:55150)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1ggW8S-0003cI-GO
 for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:12 -0500
Received: from lists.gnu.org ([209.51.188.17]:46720)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8M-0005SD-0u
 for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:07 -0500
Received: from eggs.gnu.org ([209.51.188.92]:37113)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <code@HIDDEN>) id 1ggW8K-0007s6-TI
 for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:05 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.1 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_LOW
 autolearn=disabled version=3.3.2
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <code@HIDDEN>) id 1ggW8J-0005RN-PX
 for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:04 -0500
Received: from mail1.protonmail.ch ([185.70.40.18]:31631)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8J-0005QN-8L
 for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:03 -0500
Date: Mon, 07 Jan 2019 14:43:56 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1546872237;
 bh=lp764yzH6aORAEQQrqc5Xp6aLNoagBpe0MxADG4AtLk=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=OhgQyfiRy8hSgOA1WJRzOajYeo4JMif6M3sfNh7u1XF3qWu787NWniXdAr9/Q8rUf
 KtqWQUi3yccrDPcldPCyKMugLzgiPdLjPahb6apDMuzmgr5AvNcpC1XPa0vNyXtKfI
 6VfxS5PiC82FCqEFmD8aT1/47qH4PFyVRac1Rr5Q=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 185.70.40.18
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Spam-Score: 1.0 (+)
X-Mailman-Approved-At: Mon, 07 Jan 2019 11:41:48 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Saturday, December 29, 2018 12:12 AM, Ludovic Court=C3=A8s <ludo@HIDDEN=
> wrote:

> Hello Guix!
>
> Here is a first draft adding support to distribute and retrieve substitut=
es
> over IPFS. This builds on discussions at the R-B Summit with H=C3=A9ctor =
Sanjuan
> of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian
> Paul Schmidt posted on guix-devel last month.
>
> The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides
> bindings to a subset of that API. This module also implements a custom
> =E2=80=9Cdirectory=E2=80=9D format to store directory trees in IPFS (IPFS=
 already provides
> =E2=80=9CUnixFS=E2=80=9D and =E2=80=9Ctar=E2=80=9D but they store too man=
y or too few file attributes.)
>
> =E2=80=98guix publish=E2=80=99 and =E2=80=98guix substitute=E2=80=99 use =
(guix ipfs) to
> store and retrieve store items. Complete directory trees are stored in
> IPFS =E2=80=9Cas is=E2=80=9D, rather than as compressed archives (nars). =
This allows for
> deduplication in IPFS. =E2=80=98guix publish=E2=80=99 adds a new =
=E2=80=9CIPFS=E2=80=9D field in
> narinfos and =E2=80=98guix substitute=E2=80=99 can then query those objec=
ts over IPFS.
> So the idea is that you still get narinfos over HTTP(S), and then you
> have the option of downloading substitutes over IPFS.
>
> I=E2=80=99ve pushed these patches in =E2=80=98wip-ipfs-substitutes=
=E2=80=99. This is rough on the
> edges and probably buggy, but the adventurous among us might want to give
> it a spin. :-)
>
> Thanks,
> Ludo=E2=80=99.


Hey! Happy new year! This is great news. I'm very glad to see this.
I haven't tried this yet but looking at the code there are a couple
of things to point out.

1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
has a read/write API (on :5001) and a read-only API that we call "gateway"
and which runs on :8080. The gateway, apart from handling most of the
read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
directory-type CIDs. The gateway does not allow to "publish". Therefore I t=
hink
the doc strings should say "IPFS daemon API" rather than "GATEWAY".

2) I'm not proficient enough in schema to grasp the details of the
"directory" format. If I understand it right, you keep a separate manifest
object listing the directory structure, the contents and the executable bit
for each. Thus, when adding a store item you add all the files separately a=
nd
this manifest. And when retrieving a store item you fetch the manifest and
reconstruct the tree by fetching the contents in it (and applying the
executable flag). Is this correct? This works, but it can be improved:

You can add all the files/folders in a single request. If I'm
reading it right, now each files is added separately (and gets pinned
separately). It would probably make sense to add it all in a single request=
,
letting IPFS to store the directory structure as "unixfs". You can
additionally add the sexp file with the dir-structure and executable flags
as an extra file to the root folder. This would allow to fetch the whole th=
ing
with a single request too /api/v0/get?arg=3D<hash>. And to pin a single has=
h
recursively (and not each separately). After getting the whole thing, you
will need to chmod +x things accordingly.

It will probably need some trial an error to get the multi-part right
to upload all in a single request. The Go code HTTP Clients doing
this can be found at:

https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96

As you see, a directory part in the multipart will have the content-type He=
ader
set to "application/x-directory". The best way to see how "abspath" etc is =
set
is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:500=
1).

Once UnixFSv2 lands, you will be in a position to just drop the sexp file
altogether.

Let me know if you have any doubts, I'll make my best to answer them. In th=
e
meantime I'll try to get more familiar with Guix.

Cheers,

Hector

PS. There is a place where it says "ifps" instead of "ipfs". A very common =
typo.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 07 Jan 2019 16:42:02 +0000
Resent-Message-ID: <handler.33899.B33899.15468793112027 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Cc: ludo@HIDDEN, mail@HIDDEN, go-ipfs-wg@HIDDEN
X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "guix-patches@HIDDEN" <guix-patches@HIDDEN>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15468793112027
          (code B ref 33899); Mon, 07 Jan 2019 16:42:02 +0000
Received: (at 33899) by debbugs.gnu.org; 7 Jan 2019 16:41:51 +0000
Received: from localhost ([127.0.0.1]:48897 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ggXyH-0000WZ-Rj
	for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 11:41:50 -0500
Received: from mail2.protonmail.ch ([185.70.40.22]:26380)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1ggW8J-0003bW-9s
 for 33899 <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:04 -0500
Date: Mon, 07 Jan 2019 14:43:56 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1546872237;
 bh=lp764yzH6aORAEQQrqc5Xp6aLNoagBpe0MxADG4AtLk=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=OhgQyfiRy8hSgOA1WJRzOajYeo4JMif6M3sfNh7u1XF3qWu787NWniXdAr9/Q8rUf
 KtqWQUi3yccrDPcldPCyKMugLzgiPdLjPahb6apDMuzmgr5AvNcpC1XPa0vNyXtKfI
 6VfxS5PiC82FCqEFmD8aT1/47qH4PFyVRac1Rr5Q=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham
 autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch
X-Spam-Score: -0.7 (/)
X-Mailman-Approved-At: Mon, 07 Jan 2019 11:41:47 -0500
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Saturday, December 29, 2018 12:12 AM, Ludovic Court=C3=A8s <ludo@HIDDEN=
> wrote:

> Hello Guix!
>
> Here is a first draft adding support to distribute and retrieve substitut=
es
> over IPFS. This builds on discussions at the R-B Summit with H=C3=A9ctor =
Sanjuan
> of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian
> Paul Schmidt posted on guix-devel last month.
>
> The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides
> bindings to a subset of that API. This module also implements a custom
> =E2=80=9Cdirectory=E2=80=9D format to store directory trees in IPFS (IPFS=
 already provides
> =E2=80=9CUnixFS=E2=80=9D and =E2=80=9Ctar=E2=80=9D but they store too man=
y or too few file attributes.)
>
> =E2=80=98guix publish=E2=80=99 and =E2=80=98guix substitute=E2=80=99 use =
(guix ipfs) to
> store and retrieve store items. Complete directory trees are stored in
> IPFS =E2=80=9Cas is=E2=80=9D, rather than as compressed archives (nars). =
This allows for
> deduplication in IPFS. =E2=80=98guix publish=E2=80=99 adds a new =
=E2=80=9CIPFS=E2=80=9D field in
> narinfos and =E2=80=98guix substitute=E2=80=99 can then query those objec=
ts over IPFS.
> So the idea is that you still get narinfos over HTTP(S), and then you
> have the option of downloading substitutes over IPFS.
>
> I=E2=80=99ve pushed these patches in =E2=80=98wip-ipfs-substitutes=
=E2=80=99. This is rough on the
> edges and probably buggy, but the adventurous among us might want to give
> it a spin. :-)
>
> Thanks,
> Ludo=E2=80=99.

Hey! Happy new year! This is great news. I'm very glad to see this.
I haven't tried this yet but looking at the code there are a couple
of things to point out.

1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
has a read/write API (on :5001) and a read-only API that we call "gateway"
and which runs on :8080. The gateway, apart from handling most of the
read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
directory-type CIDs. The gateway does not allow to "publish". Therefore I t=
hink
the doc strings should say "IPFS daemon API" rather than "GATEWAY".

2) I'm not proficient enough in schema to grasp the details of the
"directory" format. If I understand it right, you keep a separate manifest
object listing the directory structure, the contents and the executable bit
for each. Thus, when adding a store item you add all the files separately a=
nd
this manifest. And when retrieving a store item you fetch the manifest and
reconstruct the tree by fetching the contents in it (and applying the
executable flag). Is this correct? This works, but it can be improved:

You can add all the files/folders in a single request. If I'm
reading it right, now each files is added separately (and gets pinned
separately). It would probably make sense to add it all in a single request=
,
letting IPFS to store the directory structure as "unixfs". You can
additionally add the sexp file with the dir-structure and executable flags
as an extra file to the root folder. This would allow to fetch the whole th=
ing
with a single request too /api/v0/get?arg=3D<hash>. And to pin a single has=
h
recursively (and not each separately). After getting the whole thing, you
will need to chmod +x things accordingly.

It will probably need some trial an error to get the multi-part right
to upload all in a single request. The Go code HTTP Clients doing
this can be found at:

https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96

As you see, a directory part in the multipart will have the content-type He=
ader
set to "application/x-directory". The best way to see how "abspath" etc is =
set
is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:500=
1).

Once UnixFSv2 lands, you will be in a position to just drop the sexp file
altogether.

Let me know if you have any doubts, I'll make my best to answer them. In th=
e
meantime I'll try to get more familiar with Guix.

Cheers,

Hector

PS. There is a place where it says "ifps" instead of "ipfs". A very common =
typo.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 14 Jan 2019 13:18:02 +0000
Resent-Message-ID: <handler.33899.B33899.154747186414203 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Hector Sanjuan <code@HIDDEN>
Cc: "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.154747186414203
          (code B ref 33899); Mon, 14 Jan 2019 13:18:02 +0000
Received: (at 33899) by debbugs.gnu.org; 14 Jan 2019 13:17:44 +0000
Received: from localhost ([127.0.0.1]:58369 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gj27a-0003gz-RH
	for submit <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:17:44 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:58392)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gj27Y-0003gp-BJ
 for 33899 <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:17:41 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id A4FB7D1F;
 Mon, 14 Jan 2019 14:17:38 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id fiz-E9TzlM6x; Mon, 14 Jan 2019 14:17:37 +0100 (CET)
Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 9A1E2AFB;
 Mon, 14 Jan 2019 14:17:37 +0100 (CET)
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 25 =?UTF-8?Q?Niv=C3=B4se?= an 227 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 14 Jan 2019 14:17:36 +0100
In-Reply-To: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 (Hector Sanjuan's message of "Mon, 07 Jan 2019 14:43:56 +0000")
Message-ID: <87r2dfv0nj.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi Hector,

Happy new year to you too!  :-)

Hector Sanjuan <code@HIDDEN> skribis:

> 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
> has a read/write API (on :5001) and a read-only API that we call "gateway"
> and which runs on :8080. The gateway, apart from handling most of the
> read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
> or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
> directory-type CIDs. The gateway does not allow to "publish". Therefore I=
 think
> the doc strings should say "IPFS daemon API" rather than "GATEWAY".

Indeed, I=E2=80=99ll change that.

> 2) I'm not proficient enough in schema to grasp the details of the
> "directory" format. If I understand it right, you keep a separate manifest
> object listing the directory structure, the contents and the executable b=
it
> for each. Thus, when adding a store item you add all the files separately=
 and
> this manifest. And when retrieving a store item you fetch the manifest and
> reconstruct the tree by fetching the contents in it (and applying the
> executable flag). Is this correct? This works, but it can be improved:

That=E2=80=99s correct.

> You can add all the files/folders in a single request. If I'm
> reading it right, now each files is added separately (and gets pinned
> separately). It would probably make sense to add it all in a single reque=
st,
> letting IPFS to store the directory structure as "unixfs". You can
> additionally add the sexp file with the dir-structure and executable flags
> as an extra file to the root folder. This would allow to fetch the whole =
thing
> with a single request too /api/v0/get?arg=3D<hash>. And to pin a single h=
ash
> recursively (and not each separately). After getting the whole thing, you
> will need to chmod +x things accordingly.

Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D.  The problems, as =
I see it, is that it
stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to store =
the mtimes or
permissions; we could ignore them upon reconstruction though), and =E2=80=
=9Cnot
enough=E2=80=9D in another way (the executable bit is lost, IIUC.)

> It will probably need some trial an error to get the multi-part right
> to upload all in a single request. The Go code HTTP Clients doing
> this can be found at:
>
> https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96
>
> As you see, a directory part in the multipart will have the content-type =
Header
> set to "application/x-directory". The best way to see how "abspath" etc i=
s set
> is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5=
001).
>
> Once UnixFSv2 lands, you will be in a position to just drop the sexp file
> altogether.

Yes, that makes sense.  In the meantime, I guess we have to keep using
our own format.

What are the performance implications of adding and retrieving files one
by one like I did?  I understand we=E2=80=99re doing N HTTP requests to the
local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single reques=
t, but this
alone can=E2=80=99t be much of a problem since communication is happening
locally.  Does pinning each file separately somehow incur additional
overhead?

Thanks for your feedback!

Ludo=E2=80=99.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 18 Jan 2019 09:09:01 +0000
Resent-Message-ID: <handler.33899.B33899.154780249827048 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.154780249827048
          (code B ref 33899); Fri, 18 Jan 2019 09:09:01 +0000
Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 09:08:18 +0000
Received: from localhost ([127.0.0.1]:35661 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gkQ8P-00072B-OR
	for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:08:18 -0500
Received: from mail2.protonmail.ch ([185.70.40.22]:18229)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1gkQ8L-00071w-TA
 for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:08:16 -0500
Date: Fri, 18 Jan 2019 09:08:02 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1547802487;
 bh=T2CaH9z+ADJ9S9e9RCY4+xyhaJgW2pJel/fjzCSPEXQ=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=jlvynOjlw+RGreZhyTBmiX39eeMW+1XKeOueyx5lFCjmOkWL0f5p3Jy3o2UuI9xu4
 rPGBKAk9/NOjOXmJQxfOHLTNMECFChPDt32W4vDbSqYjWlmNeS1XTI24XIgjUipEUB
 xc2ooFpTQg+foKk4ceF2/Up06S4fnEhP0kZAoxTs=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
In-Reply-To: <87r2dfv0nj.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham
 autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wr=
ote:

> Hi Hector,
>
> Happy new year to you too! :-)
>
> Hector Sanjuan code@HIDDEN skribis:
>
> > 1.  The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-i=
pfs
> >     has a read/write API (on :5001) and a read-only API that we call "g=
ateway"
> >     and which runs on :8080. The gateway, apart from handling most of t=
he
> >     read-only methods from the HTTP API, also handles paths like "/ipfs=
/<cid>"
> >     or "/ipns/<name>" gracefully, and returns an autogenerated webpage =
for
> >     directory-type CIDs. The gateway does not allow to "publish". There=
fore I think
> >     the doc strings should say "IPFS daemon API" rather than "GATEWAY".
> >
>
> Indeed, I=E2=80=99ll change that.
>
> > 2.  I'm not proficient enough in schema to grasp the details of the
> >     "directory" format. If I understand it right, you keep a separate m=
anifest
> >     object listing the directory structure, the contents and the execut=
able bit
> >     for each. Thus, when adding a store item you add all the files sepa=
rately and
> >     this manifest. And when retrieving a store item you fetch the manif=
est and
> >     reconstruct the tree by fetching the contents in it (and applying t=
he
> >     executable flag). Is this correct? This works, but it can be improv=
ed:
> >
>
> That=E2=80=99s correct.
>
> > You can add all the files/folders in a single request. If I'm
> > reading it right, now each files is added separately (and gets pinned
> > separately). It would probably make sense to add it all in a single req=
uest,
> > letting IPFS to store the directory structure as "unixfs". You can
> > additionally add the sexp file with the dir-structure and executable fl=
ags
> > as an extra file to the root folder. This would allow to fetch the whol=
e thing
> > with a single request too /api/v0/get?arg=3D<hash>. And to pin a single=
 hash
> > recursively (and not each separately). After getting the whole thing, y=
ou
> > will need to chmod +x things accordingly.
>
> Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, as=
 I see it, is that it
> stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to stor=
e the mtimes or
> permissions; we could ignore them upon reconstruction though), and =
=E2=80=9Cnot
> enough=E2=80=9D in another way (the executable bit is lost, IIUC.)

Actually the only metadata that Unixfs stores is size:
https://github.com/ipfs/go-unixfs/blob/master/pb/unixfs.proto and by all
means the amount of metadata is negligible for the actual data stored
and serves to give you a progress bar when you are downloading.

Having IPFS understand what files are part of a single item is important
because you can pin/unpin,diff,patch all of them as a whole. Unixfs
also takes care of handling the case where the directories need to
be sharded because there are too many entries. When the user
puts the single root hash in ipfs.io/ipfs/<hash>, it will display
correctly the underlying files and the people will be
able to navigate the actual tree with both web and cli. Note that
every file added to IPFS is getting wrapped as a Unixfs block
anyways. You are just saving some "directory" nodes by adding
them separately.

There is an alternative way which is using IPLD to implement a custom
block format that carries the executable bit information and nothing
else. But I don't see significant advantages at this point for the extra
work it requires.

>
> > It will probably need some trial an error to get the multi-part right
> > to upload all in a single request. The Go code HTTP Clients doing
> > this can be found at:
> > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L9=
6
> > As you see, a directory part in the multipart will have the content-typ=
e Header
> > set to "application/x-directory". The best way to see how "abspath" etc=
 is set
> > is probably to sniff an `ipfs add -r <testfolder>` operation (localhost=
:5001).
> > Once UnixFSv2 lands, you will be in a position to just drop the sexp fi=
le
> > altogether.
>
> Yes, that makes sense. In the meantime, I guess we have to keep using
> our own format.
>
> What are the performance implications of adding and retrieving files one
> by one like I did? I understand we=E2=80=99re doing N HTTP requests to th=
e
> local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single requ=
est, but this
> alone can=E2=80=99t be much of a problem since communication is happening
> locally. Does pinning each file separately somehow incur additional
> overhead?
>

Yes, pinning separately is slow and incurs in overhead. Pins are stored
in a merkle tree themselves so it involves reading, patching and saving. Th=
is
gets quite slow when you have very large pinsets because your pins block si=
ze
grow. Your pinset will grow very large if you do this. Additionally the
pinning operation itself requires global lock making it more slow.

But, even if it was fast, you will not have a way to easily unpin
anything that becomes obsolete or have an overview of to where things
belong. It is also unlikely that a single IPFS daemon will be able to
store everything you build, so you might find yourself using IPFS Cluster
soon to distribute the storage across multiple nodes and then you will
be effectively adding remotely.


> Thanks for your feedback!
>
> Ludo=E2=80=99.

Thanks for working on this!

Hector

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 18 Jan 2019 09:53:02 +0000
Resent-Message-ID: <handler.33899.B33899.15478051766205 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Hector Sanjuan <code@HIDDEN>
Cc: "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15478051766205
          (code B ref 33899); Fri, 18 Jan 2019 09:53:02 +0000
Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 09:52:56 +0000
Received: from localhost ([127.0.0.1]:35688 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gkQpb-0001c1-OB
	for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:52:56 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:37936)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1gkQpZ-0001br-NQ
 for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:52:54 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id E070B2341;
 Fri, 18 Jan 2019 10:52:51 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id ZQPwYQv1qiwm; Fri, 18 Jan 2019 10:52:51 +0100 (CET)
Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id A1334C39;
 Fri, 18 Jan 2019 10:52:50 +0100 (CET)
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 29 =?UTF-8?Q?Niv=C3=B4se?= an 227 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 18 Jan 2019 10:52:49 +0100
In-Reply-To: <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 (Hector Sanjuan's message of "Fri, 18 Jan 2019 09:08:02 +0000")
Message-ID: <8736pqthqm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hello,

Hector Sanjuan <code@HIDDEN> skribis:

> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original =
Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
> On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> =
wrote:

[...]

>> Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, a=
s I see it, is that it
>> stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to sto=
re the mtimes or
>> permissions; we could ignore them upon reconstruction though), and =E2=
=80=9Cnot
>> enough=E2=80=9D in another way (the executable bit is lost, IIUC.)
>
> Actually the only metadata that Unixfs stores is size:
> https://github.com/ipfs/go-unixfs/blob/master/pb/unixfs.proto and by all
> means the amount of metadata is negligible for the actual data stored
> and serves to give you a progress bar when you are downloading.

Yes, the format I came up with also store the size so we can eventually
display a progress bar.

> Having IPFS understand what files are part of a single item is important
> because you can pin/unpin,diff,patch all of them as a whole. Unixfs
> also takes care of handling the case where the directories need to
> be sharded because there are too many entries.

Isn=E2=80=99t there a way, then, to achieve the same behavior with the cust=
om
format?  The /api/v0/add entry point has a =E2=80=98pin=E2=80=99 argument; =
I suppose we
could leave it to false except when we add the top-level =E2=80=9Cdirectory=
=E2=80=9D
node?  Wouldn=E2=80=99t that give us behavior similar to that of Unixfs?

> When the user puts the single root hash in ipfs.io/ipfs/<hash>, it
> will display correctly the underlying files and the people will be
> able to navigate the actual tree with both web and cli.

Right, though that=E2=80=99s less important in my view.

> Note that every file added to IPFS is getting wrapped as a Unixfs
> block anyways. You are just saving some "directory" nodes by adding
> them separately.

Hmm weird.  When I do /api/v0/add, I=E2=80=99m really just passing a byte
vector; there=E2=80=99s no notion of a =E2=80=9Cfile=E2=80=9D here, AFAICS.=
  Or am I missing
something?

>> > It will probably need some trial an error to get the multi-part right
>> > to upload all in a single request. The Go code HTTP Clients doing
>> > this can be found at:
>> > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L=
96
>> > As you see, a directory part in the multipart will have the content-ty=
pe Header
>> > set to "application/x-directory". The best way to see how "abspath" et=
c is set
>> > is probably to sniff an `ipfs add -r <testfolder>` operation (localhos=
t:5001).
>> > Once UnixFSv2 lands, you will be in a position to just drop the sexp f=
ile
>> > altogether.
>>
>> Yes, that makes sense. In the meantime, I guess we have to keep using
>> our own format.
>>
>> What are the performance implications of adding and retrieving files one
>> by one like I did? I understand we=E2=80=99re doing N HTTP requests to t=
he
>> local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single req=
uest, but this
>> alone can=E2=80=99t be much of a problem since communication is happening
>> locally. Does pinning each file separately somehow incur additional
>> overhead?
>>
>
> Yes, pinning separately is slow and incurs in overhead. Pins are stored
> in a merkle tree themselves so it involves reading, patching and saving. =
This
> gets quite slow when you have very large pinsets because your pins block =
size
> grow. Your pinset will grow very large if you do this. Additionally the
> pinning operation itself requires global lock making it more slow.

OK, I see.

> But, even if it was fast, you will not have a way to easily unpin
> anything that becomes obsolete or have an overview of to where things
> belong. It is also unlikely that a single IPFS daemon will be able to
> store everything you build, so you might find yourself using IPFS Cluster
> soon to distribute the storage across multiple nodes and then you will
> be effectively adding remotely.

Currently, =E2=80=98guix publish=E2=80=99 stores things as long as they are=
 requested,
and then for the duration specified with =E2=80=98--ttl=E2=80=99.  I suppos=
e we could
have similar behavior with IPFS: if an item hasn=E2=80=99t been requested f=
or
the specified duration, then we unpin it.

Does that make sense?

Thanks for your help!

Ludo=E2=80=99.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 18 Jan 2019 11:27:02 +0000
Resent-Message-ID: <handler.33899.B33899.154781079130812 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.154781079130812
          (code B ref 33899); Fri, 18 Jan 2019 11:27:02 +0000
Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 11:26:31 +0000
Received: from localhost ([127.0.0.1]:35757 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gkSIA-00080t-RZ
	for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 06:26:31 -0500
Received: from mail-40132.protonmail.ch ([185.70.40.132]:58973)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1gkSI7-00080e-SG
 for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 06:26:29 -0500
Date: Fri, 18 Jan 2019 11:26:18 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1547810781;
 bh=4pQGBXxb3EphmMnCRcuTgN8S67MGMR15cim0fy1tBRA=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=JUHq75Hu99aZzuqyOEOZscwrocjODfW5i1KLzllAUJSK7+zeJrYH3dOcttQwc5yoH
 O9cITpViXuSiO+t1z1oIOGRit1EdT/mcqVNnOktCHkyrTxYGWoMqQIGsrAHr+8IPVX
 /eshhWg/uvqWT19jMa1sk+09ClZnODSZ3u/x9ps0=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
In-Reply-To: <8736pqthqm.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham
 autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Friday, January 18, 2019 10:52 AM, Ludovic Court=C3=A8s <ludo@HIDDEN> w=
rote:

> Hello,
>
> Hector Sanjuan code@HIDDEN skribis:
>
> > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Origina=
l Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=
=90
> > On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s ludo@HIDDEN =
wrote:
>
> [...]
>

>
> Isn=E2=80=99t there a way, then, to achieve the same behavior with the cu=
stom
> format? The /api/v0/add entry point has a =E2=80=98pin=E2=80=99 argument;=
 I suppose we
> could leave it to false except when we add the top-level =E2=80=9Cdirecto=
ry=E2=80=9D
> node? Wouldn=E2=80=99t that give us behavior similar to that of Unixfs?
>

Yes. What you could do is to add every file flatly/separately (with pin=3Df=
alse)
and at the end add an IPLD object with references to all the
files that you added and including the exec bit information (and size?).
This is just a JSON file:

{
   "name": "package name",
   "contents": [
       {
           "path": "/file/path", # so you know where to extract it later
           "exec": true,
           "ipfs": { "/": "Qmhash..." }
       },
       ...
}

This needs to be added to IPFS with the /api/v0/dag/put endpoint (this
converts it to CBOR - IPLD-Cbor is the actual block format used here).
When this is pinned (?pin=3Dtrue), this will pin all the things referenced
from it recursively in the way we want.

So this will be quite similar to unixfs. But note that if this blob
ever grows over the 2M block-size limit because you have a package with
many files, you will need to start solving problems that unixfs solves
automatically now (directory sharding).

Because IPLD-cbor is supported, ipfs, the gateway etc will know how to
display these manifests, the info in it and their links.


> > When the user puts the single root hash in ipfs.io/ipfs/<hash>, it
> > will display correctly the underlying files and the people will be
> > able to navigate the actual tree with both web and cli.
>
> Right, though that=E2=80=99s less important in my view.
>
> > Note that every file added to IPFS is getting wrapped as a Unixfs
> > block anyways. You are just saving some "directory" nodes by adding
> > them separately.
>
> Hmm weird. When I do /api/v0/add, I=E2=80=99m really just passing a byte
> vector; there=E2=80=99s no notion of a =E2=80=9Cfile=E2=80=9D here, AFAIC=
S. Or am I missing
> something?

They are wrapped in Unixfs blocks anyway by default. From the moment
the file is >256K it will get chunked into several  pieces and
a Unixfs block (or multiple, if a really big file) is necessary to
reference them. In this case the root hash will be a Unixfs node
with links to the parts.

There is a "raw-leaves" option which does not wrap the individual
blocks with unixfs, so if the file is small to not be chunked,
you can avoid the default unixfs-wrapping this way.


>
> > > > It will probably need some trial an error to get the multi-part rig=
ht
> > > > to upload all in a single request. The Go code HTTP Clients doing
> > > > this can be found at:
> > > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.g=
o#L96
> > > > As you see, a directory part in the multipart will have the content=
-type Header
> > > > set to "application/x-directory". The best way to see how "abspath"=
 etc is set
> > > > is probably to sniff an `ipfs add -r <testfolder>` operation (local=
host:5001).
> > > > Once UnixFSv2 lands, you will be in a position to just drop the sex=
p file
> > > > altogether.
> > >
> > > Yes, that makes sense. In the meantime, I guess we have to keep using
> > > our own format.
> > > What are the performance implications of adding and retrieving files =
one
> > > by one like I did? I understand we=E2=80=99re doing N HTTP requests t=
o the
> > > local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single =
request, but this
> > > alone can=E2=80=99t be much of a problem since communication is happe=
ning
> > > locally. Does pinning each file separately somehow incur additional
> > > overhead?
> >
> > Yes, pinning separately is slow and incurs in overhead. Pins are stored
> > in a merkle tree themselves so it involves reading, patching and saving=
. This
> > gets quite slow when you have very large pinsets because your pins bloc=
k size
> > grow. Your pinset will grow very large if you do this. Additionally the
> > pinning operation itself requires global lock making it more slow.
>
> OK, I see.

I should add that even if you want to /add all files separately (and then
put the IPLD manifest I described above), you can still add them all in the=
 same
request (it becomes easier as you just need to put more parts in the multip=
art
and don't have to worry about names/folders/paths).

The /add endpoint will forcefully close the HTTP connection for every
/add (long story) and small delays might add up to a big one. Specially rel=
evant
if using IPFS Cluster, where /add might send the blocks somewhere else and =
does
needs to do some other things.


>
> > But, even if it was fast, you will not have a way to easily unpin
> > anything that becomes obsolete or have an overview of to where things
> > belong. It is also unlikely that a single IPFS daemon will be able to
> > store everything you build, so you might find yourself using IPFS Clust=
er
> > soon to distribute the storage across multiple nodes and then you will
> > be effectively adding remotely.
>
> Currently, =E2=80=98guix publish=E2=80=99 stores things as long as they a=
re requested,
> and then for the duration specified with =E2=80=98--ttl=E2=80=99. I suppo=
se we could
> have similar behavior with IPFS: if an item hasn=E2=80=99t been requested=
 for
> the specified duration, then we unpin it.
>
> Does that make sense?

Yes, in fact I wanted IPFS Cluster to support a TTL so that things are
automatically unpinned when it expires too.

>
> Thanks for your help!
>
> Ludo=E2=80=99.

Thanks!

Hector

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
References: <20181228231205.8068-1-ludo@HIDDEN>
In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN>
Resent-From: "Alex Griffin" <a@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 13 May 2019 18:53:02 +0000
Resent-Message-ID: <handler.33899.B33899.155777354031499 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: 33899 <at> debbugs.gnu.org
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.155777354031499
          (code B ref 33899); Mon, 13 May 2019 18:53:02 +0000
Received: (at 33899) by debbugs.gnu.org; 13 May 2019 18:52:20 +0000
Received: from localhost ([127.0.0.1]:46178 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hQG3f-0008Bw-Uz
	for submit <at> debbugs.gnu.org; Mon, 13 May 2019 14:52:20 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:59557)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <a@HIDDEN>) id 1hQG3d-0008Bj-B7
 for 33899 <at> debbugs.gnu.org; Mon, 13 May 2019 14:52:18 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46])
 by mailout.nyi.internal (Postfix) with ESMTP id 0E46524622
 for <33899 <at> debbugs.gnu.org>; Mon, 13 May 2019 14:52:12 -0400 (EDT)
Received: from imap1 ([10.202.2.51])
 by compute6.internal (MEProxy); Mon, 13 May 2019 14:52:12 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ajgrf.com; h=
 mime-version:message-id:date:from:to:subject:content-type; s=
 fm2; bh=YwpmwH8PqqsqeXz7J0/EzZXIVEHJ4n28KJSWyGB79CQ=; b=nNgG37U/
 HyW30pWzPn24l6E3oDjnKCD0ffFEwyHtrAOHn22Z2ooNnCnAyXaqmsNLipqc1l4c
 P7SR6d6KaZBvLTbKLjK/v+t7VhQUCxnThaknZgXRFu9B2PB9dg5oTLfJ8Zws7Msm
 /Dv0bcIpaVbPDgCEFKiNiKCLpI2NU49xdUII7GJ9jRe2qSG8xAPQjyi6p8zvZzEA
 arWHzmPZekE9zyZNVXhJzIeF7vjapuWEg7LkO9Wwlx5hPSxff0xQjne2IrNtNEyW
 rpTOChWFN/eAx7Jo9w2e4y4LOEf0FqRpVSOv9QERj+uq2TGxI+w7+Rd5mSx6t1Ac
 gglKMbFuT2hJDQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=content-type:date:from:message-id
 :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender
 :x-me-sender:x-sasl-enc; s=fm2; bh=YwpmwH8PqqsqeXz7J0/EzZXIVEHJ4
 n28KJSWyGB79CQ=; b=zPnUXWIVcNL92dNDtKF8OWp1c8WwujrH2kMsdxuGVrEds
 a2AGlb/eKfMKGReyKY7Hxhyvil4BbD9drPjFjsekDZM26XmGhdnMbQM1pbWoxEPK
 S+V5XWO/VGLk9P90LLcwzZeQsQtgRxiFnsk4OGlQvpzI5Om/G9//mHVEoSEP/lAt
 pBd+fYQhhaawAWg5P5f+rvf0icJMUjrvrxgv+hb0jHNIEFf9ZjUsm599UeJkUbPR
 8MheuTEa3SQsjxui92Xu/+PFsyyAK9g4DAPPL7DLOYuZ8tHA8NG/JpYMFqqjVIpy
 NYGnAE8vje8Pk7XKUyOfUF33S43dPXle6KobcPWMQ==
X-ME-Sender: <xms:27zZXMBsMIdzxhKsG1wyOmdCU60vwuGhK5IfFdsp_Mpgl2_818290A>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrleeggddufedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsehttdertd
 erredtnecuhfhrohhmpedftehlvgigucfirhhifhhfihhnfdcuoegrsegrjhhgrhhfrdgt
 ohhmqeenucffohhmrghinhepghhithhhuhgsrdgtohhmnecurfgrrhgrmhepmhgrihhlfh
 hrohhmpegrsegrjhhgrhhfrdgtohhmnecuvehluhhsthgvrhfuihiivgeptd
X-ME-Proxy: <xmx:27zZXMbTJ0uhw2qZqWHoraR2mDDvpMPvZJzKqipkG-uQf6bbyqHSlA>
 <xmx:27zZXIqDlmIpMGXZ8dBbrCzerKlScC9JjMn7xicbQjAtjnehDbfnog>
 <xmx:27zZXEBbjr5dyLR_bu3UTFoVSNBlOWV4cEb-JehsNGpiT2tktBJYRA>
 <xmx:27zZXH6HLy0BecJilHdKeYxHGNyESfaxn2343S-yFBBvK5reSht0lg>
Received: by mailuser.nyi.internal (Postfix, from userid 501)
 id 7B8C4D4948; Mon, 13 May 2019 14:52:11 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.1.6-453-g9eaa09e-fmstable-20190430v2bis
Mime-Version: 1.0
Message-Id: <5e3fb8d9-1a83-4031-ab02-c4e10e2be1ea@HIDDEN>
Date: Mon, 13 May 2019 14:51:42 -0400
From: "Alex Griffin" <a@HIDDEN>
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Do I understand correctly that the only reason you don't just store nar files is for deduplication? Reading [this page][1] suggests to me that you might be overthinking it. IPFS already uses a content-driven chunking algorithm that might provide good enough deduplication on its own. It also looks like you can use your own chunker, so a future improvement could be implementing a custom chunker that makes sure to split nar files at the file boundaries within them.

[1]: https://github.com/ipfs/archives

-- 
Alex Griffin

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Pierre Neidhardt <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 01 Jul 2019 21:37:02 +0000
Resent-Message-ID: <handler.33899.B33899.15620170073602 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Hector Sanjuan <code@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Antoine Eiche <lewo@HIDDEN> 
Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.15620170073602
          (code B ref 33899); Mon, 01 Jul 2019 21:37:02 +0000
Received: (at 33899) by debbugs.gnu.org; 1 Jul 2019 21:36:47 +0000
Received: from localhost ([127.0.0.1]:46194 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hi3yg-0000w2-Us
	for submit <at> debbugs.gnu.org; Mon, 01 Jul 2019 17:36:47 -0400
Received: from relay10.mail.gandi.net ([217.70.178.230]:39679)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1hi3ye-0000vs-10
 for 33899 <at> debbugs.gnu.org; Mon, 01 Jul 2019 17:36:45 -0400
Received: from mimimi (unknown [212.81.186.37])
 (Authenticated sender: pierre@HIDDEN)
 by relay10.mail.gandi.net (Postfix) with ESMTPSA id AB329240007;
 Mon,  1 Jul 2019 21:36:35 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
In-Reply-To: <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
Date: Mon, 01 Jul 2019 23:36:34 +0200
Message-ID: <87zhlxe8t9.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.2 (/)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi!

(Re-sending to debbugs, sorry for the double email :p)

A little update/recap after many months! :)

I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo=
's
patch so now I have a little better understanding of the current state
of affair.

=2D We could store the substitutes as tarballs on IPFS, but this has
  some possible downsides:

  - We would need to use IPFS' tar chunker to deduplicate the content of
    the tarball.  But the tar chunker is not well maintained currently,
    and it's not clear whether it's reproducible at the moment, so it
    would need some more work.

  - Tarballs might induce some performance cost.  Nix had attempted
    something similar in the past and this may have incurred a significant
    performance penalty, although this remains to be confirmed.
    Lewo?

=2D Ludo's patch stores all files on IPFS individually.  This way we don't
  need to touch the tar chunker, so it's less work :)
  This raises some other issues however:

  - Extra metadata:  IPFS stores files on UnixFSv1 which does not
    include the executable bit.

    - Right now we store a s-exp manifest with a list of files and a
      list of executable bits.  But maybe we don't have to roll out our own.

    - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not
      recommend using it (not sure why though).

    - We could use UnixFSv2 but it's not released yet and it's unclear when
      it's going to be released.  So we can't really count on it right now.

    - IPLD: As H=C3=A9ctor suggested in the previous email, we could levera=
ge
      IPLD and generate a JSON object that references the files with
      their paths together with an "executable?" property.
      A problem would arise if this IPLD object grows over the 2M
      block-size limit because then we would have to shard it (something
      that UnixFS would do automatically for us).

  - Flat storage vs. tree storage: Right now we are storing the files
    separately, but this has some shortcomings, namely we need multiple
    "get" requests instead of just one, and that IPFS does
    not "know" that those files are related.  (We lose the web view of
    the tree, etc.)  Storing them as tree could be better.
    I don't understand if that would work with the "IPLD manifest"
    suggested above.  H=C3=A9ctor?

  - Pinning: Pinning all files separately incurs an overhead.  It's
    enough to just pin the IPLD object since it propagates recursively.
    When adding a tree, then it's no problem since pinning is only done onc=
e.

  - IPFS endpoint calls: instead of adding each file individually, it's
    possible to add them all in one go.  Can we add all files at once
    while using a flat storage? (I.e. not adding them all under a common
    root folder.)

To sum up, here is what remains to be done on the current patch:

=2D Add all files in one go without pinning them.
=2D Store as the file tree?  Can we still us the IPLD object to reference
  the files in the tree?  Else use the "raw-leaves" option to avoid
  wrapping small files in UnixFS blocks.
=2D Remove the Scheme manifest if IPLD can do.
=2D Generate the IPLD object and pin it.

Any corrections?
Thoughts?

Cheers!

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0afOIACgkQm9z0l6S7
zH+oEgf/U76CtGhULJqKEVxBD8+LYT2ggfqVlaC2NnMckdg+JUHCC4dpqMdBLks3
Wye4tFYLfXinaN8SYmLc2ubA0ZiD24xHC5qAO10PPsyiea8VUChJf3jyTkV1TNn/
Knp2Aj8ALk84nEPoTIkG9vBkKgXQj2szzD1lEOTCdyOcLaDFvBzV31uyDMnqbW18
bsvPDmWfoUsDVk8VB/sgrPBK9qA7UF2Hv5qvuAcf1Y5xJ1PTRbWhf2T3cNZe/Wt3
+/hYEwMdEx9UJvW9P71rhA0iATlOjqhpju0HG3MYN95VGYcKL9R9xDJJH1jqUde0
tZyA3prRss1O7/0JMP9+s+4XIuU7uw==
=WJum
-----END PGP SIGNATURE-----
--=-=-=--

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Pierre Neidhardt <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sat, 06 Jul 2019 08:45:02 +0000
Resent-Message-ID: <handler.33899.B33899.156240265615352 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Hector Sanjuan <code@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Antoine Eiche <lewo@HIDDEN>
Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156240265615352
          (code B ref 33899); Sat, 06 Jul 2019 08:45:02 +0000
Received: (at 33899) by debbugs.gnu.org; 6 Jul 2019 08:44:16 +0000
Received: from localhost ([127.0.0.1]:54191 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hjgIq-0003zX-Lg
	for submit <at> debbugs.gnu.org; Sat, 06 Jul 2019 04:44:16 -0400
Received: from relay8-d.mail.gandi.net ([217.70.183.201]:38019)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1hjgIn-0003zL-If
 for 33899 <at> debbugs.gnu.org; Sat, 06 Jul 2019 04:44:14 -0400
X-Originating-IP: 92.169.116.127
Received: from bababa (lfbn-1-4117-127.w92-169.abo.wanadoo.fr [92.169.116.127])
 (Authenticated sender: pierre@HIDDEN)
 by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 266AB1BF206;
 Sat,  6 Jul 2019 08:44:05 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
In-Reply-To: <87zhlxe8t9.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN>
Date: Sat, 06 Jul 2019 10:44:03 +0200
Message-ID: <87imsfd030.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.2 (/)

--=-=-=
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

Link to the Nix integration discussion:
https://github.com/NixOS/nix/issues/859.

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0gX1MACgkQm9z0l6S7
zH/1mAgAlaFU5i8qcYJsTuIa0XvGq8h0XCSzJmg9iImrYj+zqfxrZrHkl0PwdZGr
Z7vGKlxmIyCNRJuVTCi1wdyWcqzKVScJ1YI5FazCTR788pV8Qiu3ytmtcI5Vc6+a
2aYX3dcbYBhSQIYRH0Mr5CB9YfCPy5XhnTYuU8Pz2rkxm0h+fHKjVifzfmfnRHZz
NZ9M3+8ZN13mzI0j2nvltsAsVE9zMyD1BkREZ8itvsx9OuYLa9SBQ7sZawyNJKiI
LsBBr3AJHl3xCCVYNetY8tMDS98TPEtRiOKp9xro8crjhr/WzEWfCCyYmLwKK7a5
es7eWdkaB7ZY90pWy8gxQE2voTkkAw==
=pYr5
-----END PGP SIGNATURE-----
--=-=-=--

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 12 Jul 2019 20:16:02 +0000
Resent-Message-ID: <handler.33899.B33899.156296252415473 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Pierre Neidhardt <mail@HIDDEN>
Cc: Hector Sanjuan <code@HIDDEN>, Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156296252415473
          (code B ref 33899); Fri, 12 Jul 2019 20:16:02 +0000
Received: (at 33899) by debbugs.gnu.org; 12 Jul 2019 20:15:24 +0000
Received: from localhost ([127.0.0.1]:40775 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hm1wy-00041U-4Y
	for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:15:24 -0400
Received: from eggs.gnu.org ([209.51.188.92]:45851)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hm1ww-00041F-BW
 for 33899 <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:15:22 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47533)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hm1wp-0005Rr-8V; Fri, 12 Jul 2019 16:15:15 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52938 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hm1we-0004RU-PL; Fri, 12 Jul 2019 16:15:06 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 24 Messidor an 227 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 12 Jul 2019 22:15:02 +0200
In-Reply-To: <87zhlxe8t9.fsf@HIDDEN> (Pierre Neidhardt's message of
 "Mon, 01 Jul 2019 23:36:34 +0200")
Message-ID: <87ftnbf1rt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello!

Pierre Neidhardt <mail@HIDDEN> skribis:

> A little update/recap after many months! :)

Thank you, and apologies for the delay!

>   - Extra metadata:  IPFS stores files on UnixFSv1 which does not
>     include the executable bit.
>
>     - Right now we store a s-exp manifest with a list of files and a
>       list of executable bits.  But maybe we don't have to roll out our o=
wn.
>
>     - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not
>       recommend using it (not sure why though).
>
>     - We could use UnixFSv2 but it's not released yet and it's unclear wh=
en
>       it's going to be released.  So we can't really count on it right no=
w.

UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2
would be appropriate, though it stores timestamps that we don=E2=80=99t need
(not necessarily a problem).

>   - Flat storage vs. tree storage: Right now we are storing the files
>     separately, but this has some shortcomings, namely we need multiple
>     "get" requests instead of just one, and that IPFS does
>     not "know" that those files are related.  (We lose the web view of
>     the tree, etc.)  Storing them as tree could be better.
>     I don't understand if that would work with the "IPLD manifest"
>     suggested above.  H=C3=A9ctor?

I don=E2=80=99t consider the web view a strong argument :-) since we could =
write
tools to deal with whatever format we use.

Regarding multiple GET requests: we could pipeline them, and it seems
more like an implementation detail to me.  The real question is whether
making separate GET requests prevents some optimization in IPFS.

>   - Pinning: Pinning all files separately incurs an overhead.  It's
>     enough to just pin the IPLD object since it propagates recursively.
>     When adding a tree, then it's no problem since pinning is only done o=
nce.

Where=E2=80=99s the overhead exactly?

>   - IPFS endpoint calls: instead of adding each file individually, it's
>     possible to add them all in one go.  Can we add all files at once
>     while using a flat storage? (I.e. not adding them all under a common
>     root folder.)

Again, is the concern that we=E2=80=99re making one GET and thus one round =
trip
per file, or is there some additional cost under the hood?

Thanks for the summary and explanations!

Ludo=E2=80=99.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Molly Mackinlay <molly@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 12 Jul 2019 21:10:02 +0000
Resent-Message-ID: <handler.33899.B33899.156296576620687 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Pierre Neidhardt <mail@HIDDEN>
Cc: Alex Potsides <alex.potsides@HIDDEN>, Hector Sanjuan <code@HIDDEN>, Andrew Nesbitt <andrew@HIDDEN>, Eric Myhre <eric.myhre@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>, Antoine Eiche <lewo@HIDDEN>, Jessica Schilling <jessica@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156296576620687
          (code B ref 33899); Fri, 12 Jul 2019 21:10:02 +0000
Received: (at 33899) by debbugs.gnu.org; 12 Jul 2019 21:09:26 +0000
Received: from localhost ([127.0.0.1]:40806 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hm2nF-0005Na-WA
	for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 17:09:26 -0400
Received: from mail-io1-f53.google.com ([209.85.166.53]:46436)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <molly@HIDDEN>) id 1hm1lB-0003iJ-6K
 for 33899 <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:03:13 -0400
Received: by mail-io1-f53.google.com with SMTP id i10so22938897iol.13
 for <33899 <at> debbugs.gnu.org>; Fri, 12 Jul 2019 13:03:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protocol.ai; s=google;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=8czCWGdi1/j1frklq4/BID0J5QGNs9v0xuL0GjsLP/A=;
 b=PdiQOh+LudNgCjV/pNMfeUvqIHMu81fbF/srzxpcJw7JHtbXZH7vImVu0G5HS3FUUH
 TDn+nQCtLvEDAE94CCSDqKH3m1/pKWVpihKr14gPHeMmeT0Ig5IC3MJXkZ3Q+W1FCzAA
 uCmUDhsTGWoo7b0LtQFMaiW1bdAu2LNJEQy+QdHmlsXZwkcU5X3G3KthYiuBhgmbZH6l
 v7X2MsE28uuNpICshbE8ak6E7o9Lbl/17irrYt+xwr3XkkByyI6/K/Ljj4zct0BTfHaX
 AVi13EowUyyYemTuDxDk9bX8HUMw5m/tL62cNO6sfk3iLGVknVk4mzYWKILuScK2pf8t
 9aAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=8czCWGdi1/j1frklq4/BID0J5QGNs9v0xuL0GjsLP/A=;
 b=FJlTIAJORqrC4U78JCLCIzqd+DrZ/54Ac6sHVnKfRNoSFPiQkRPNPygcXKAXbFeIPH
 nYxjIHvvO0H6ehhp6EEJ1Ncyl3s4NlDz/vtvFwo5pAC6kMMsqaLh/A3DGLm7uc30HEiz
 D5Y13roExyWduY7M9/7UufPtYM98yf7VRiY6f10n4k0mu3aalXuVI+Kgd5xacpCd4l90
 WZ2Gp4vG5PK9LuA0VO/jqLggf9SmwKF81AddLYFTnXC2P4rwdGqkxyaQdVyjoCRS/d8l
 kWbXILiumHQyyXcqEVJXEjqoBtHG/SaZoaRBEyPOqEyMO50P7l09joAbD94S8VWTSFyZ
 ymqw==
X-Gm-Message-State: APjAAAUBQAbpc5Cf55pRN4z3EsQEfKP6dgPC6/nvdcNvoDt709/0oKUW
 OZ+JJMUl0YMzSf6HVNXhQjGXLOBoyHWZiesGjCDMkg==
X-Google-Smtp-Source: APXvYqwu2ITNZ/vYX12QdVFcDTU8nGxi3gLK4djlQXHTGzDBHiNSlLVmSOAmXzyElsB6CAZ+QSZCsA20J+Ik5jm9RjU=
X-Received: by 2002:a6b:dc17:: with SMTP id s23mr9250638ioc.56.1562961786040; 
 Fri, 12 Jul 2019 13:03:06 -0700 (PDT)
MIME-Version: 1.0
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN>
In-Reply-To: <87zhlxe8t9.fsf@HIDDEN>
From: Molly Mackinlay <molly@HIDDEN>
Date: Fri, 12 Jul 2019 13:02:55 -0700
Message-ID: <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN>
Content-Type: multipart/alternative; boundary="0000000000000e29ad058d8166f7"
X-Spam-Score: 0.0 (/)
X-Mailman-Approved-At: Fri, 12 Jul 2019 17:09:24 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--0000000000000e29ad058d8166f7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for the update Pierre! Also adding Alex, Jessica, Eric and Andrew
from the package managers discussions at IPFS Camp as FYI.

Generating the ipld manifest with the metadata and the tree of files should
also be fine AFAIK - I=E2=80=99m sure Hector and Eric can expand more on ho=
w to
compose them, but data storage format shouldn=E2=80=99t make a big differen=
ce for
the ipld manifest.

On Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt <mail@HIDDEN> wrote:

> Hi!
>
> (Re-sending to debbugs, sorry for the double email :p)
>
> A little update/recap after many months! :)
>
> I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Lu=
do's
> patch so now I have a little better understanding of the current state
> of affair.
>
> - We could store the substitutes as tarballs on IPFS, but this has
>   some possible downsides:
>
>   - We would need to use IPFS' tar chunker to deduplicate the content of
>     the tarball.  But the tar chunker is not well maintained currently,
>     and it's not clear whether it's reproducible at the moment, so it
>     would need some more work.
>
>   - Tarballs might induce some performance cost.  Nix had attempted
>     something similar in the past and this may have incurred a significan=
t
>     performance penalty, although this remains to be confirmed.
>     Lewo?
>
> - Ludo's patch stores all files on IPFS individually.  This way we don't
>   need to touch the tar chunker, so it's less work :)
>   This raises some other issues however:
>
>   - Extra metadata:  IPFS stores files on UnixFSv1 which does not
>     include the executable bit.
>
>     - Right now we store a s-exp manifest with a list of files and a
>       list of executable bits.  But maybe we don't have to roll out our
> own.
>
>     - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not
>       recommend using it (not sure why though).
>
>     - We could use UnixFSv2 but it's not released yet and it's unclear wh=
en
>       it's going to be released.  So we can't really count on it right no=
w.
>
>     - IPLD: As H=C3=A9ctor suggested in the previous email, we could leve=
rage
>       IPLD and generate a JSON object that references the files with
>       their paths together with an "executable?" property.
>       A problem would arise if this IPLD object grows over the 2M
>       block-size limit because then we would have to shard it (something
>       that UnixFS would do automatically for us).
>
>   - Flat storage vs. tree storage: Right now we are storing the files
>     separately, but this has some shortcomings, namely we need multiple
>     "get" requests instead of just one, and that IPFS does
>     not "know" that those files are related.  (We lose the web view of
>     the tree, etc.)  Storing them as tree could be better.
>     I don't understand if that would work with the "IPLD manifest"
>     suggested above.  H=C3=A9ctor?
>
>   - Pinning: Pinning all files separately incurs an overhead.  It's
>     enough to just pin the IPLD object since it propagates recursively.
>     When adding a tree, then it's no problem since pinning is only done
> once.
>
>   - IPFS endpoint calls: instead of adding each file individually, it's
>     possible to add them all in one go.  Can we add all files at once
>     while using a flat storage? (I.e. not adding them all under a common
>     root folder.)
>
> To sum up, here is what remains to be done on the current patch:
>
> - Add all files in one go without pinning them.
> - Store as the file tree?  Can we still us the IPLD object to reference
>   the files in the tree?  Else use the "raw-leaves" option to avoid
>   wrapping small files in UnixFS blocks.
> - Remove the Scheme manifest if IPLD can do.
> - Generate the IPLD object and pin it.
>
> Any corrections?
> Thoughts?
>
> Cheers!
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Go IPFS Working Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-ipfs-wg+unsubscribe@HIDDEN
> To view this discussion on the web visit
> https://groups.google.com/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40a=
mbrevar.xyz
> .
>

--0000000000000e29ad058d8166f7
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div><div dir=3D"auto">Thanks for the update Pierre! Also adding Alex, Jess=
ica, Eric and Andrew from the package managers discussions at IPFS Camp as =
FYI.</div><br></div><div dir=3D"auto">Generating the ipld manifest with the=
 metadata and the tree of files should also be fine AFAIK - I=E2=80=99m sur=
e Hector and Eric can expand more on how to compose them, but data storage =
format shouldn=E2=80=99t make a big difference for the ipld manifest.</div>=
<div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">O=
n Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt &lt;<a href=3D"mailto:mail@a=
mbrevar.xyz">mail@HIDDEN</a>&gt; wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex">Hi!<br>
<br>
(Re-sending to debbugs, sorry for the double email :p)<br>
<br>
A little update/recap after many months! :)<br>
<br>
I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo=
&#39;s<br>
patch so now I have a little better understanding of the current state<br>
of affair.<br>
<br>
- We could store the substitutes as tarballs on IPFS, but this has<br>
=C2=A0 some possible downsides:<br>
<br>
=C2=A0 - We would need to use IPFS&#39; tar chunker to deduplicate the cont=
ent of<br>
=C2=A0 =C2=A0 the tarball.=C2=A0 But the tar chunker is not well maintained=
 currently,<br>
=C2=A0 =C2=A0 and it&#39;s not clear whether it&#39;s reproducible at the m=
oment, so it<br>
=C2=A0 =C2=A0 would need some more work.<br>
<br>
=C2=A0 - Tarballs might induce some performance cost.=C2=A0 Nix had attempt=
ed<br>
=C2=A0 =C2=A0 something similar in the past and this may have incurred a si=
gnificant<br>
=C2=A0 =C2=A0 performance penalty, although this remains to be confirmed.<b=
r>
=C2=A0 =C2=A0 Lewo?<br>
<br>
- Ludo&#39;s patch stores all files on IPFS individually.=C2=A0 This way we=
 don&#39;t<br>
=C2=A0 need to touch the tar chunker, so it&#39;s less work :)<br>
=C2=A0 This raises some other issues however:<br>
<br>
=C2=A0 - Extra metadata:=C2=A0 IPFS stores files on UnixFSv1 which does not=
<br>
=C2=A0 =C2=A0 include the executable bit.<br>
<br>
=C2=A0 =C2=A0 - Right now we store a s-exp manifest with a list of files an=
d a<br>
=C2=A0 =C2=A0 =C2=A0 list of executable bits.=C2=A0 But maybe we don&#39;t =
have to roll out our own.<br>
<br>
=C2=A0 =C2=A0 - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex =
did not<br>
=C2=A0 =C2=A0 =C2=A0 recommend using it (not sure why though).<br>
<br>
=C2=A0 =C2=A0 - We could use UnixFSv2 but it&#39;s not released yet and it&=
#39;s unclear when<br>
=C2=A0 =C2=A0 =C2=A0 it&#39;s going to be released.=C2=A0 So we can&#39;t r=
eally count on it right now.<br>
<br>
=C2=A0 =C2=A0 - IPLD: As H=C3=A9ctor suggested in the previous email, we co=
uld leverage<br>
=C2=A0 =C2=A0 =C2=A0 IPLD and generate a JSON object that references the fi=
les with<br>
=C2=A0 =C2=A0 =C2=A0 their paths together with an &quot;executable?&quot; p=
roperty.<br>
=C2=A0 =C2=A0 =C2=A0 A problem would arise if this IPLD object grows over t=
he 2M<br>
=C2=A0 =C2=A0 =C2=A0 block-size limit because then we would have to shard i=
t (something<br>
=C2=A0 =C2=A0 =C2=A0 that UnixFS would do automatically for us).<br>
<br>
=C2=A0 - Flat storage vs. tree storage: Right now we are storing the files<=
br>
=C2=A0 =C2=A0 separately, but this has some shortcomings, namely we need mu=
ltiple<br>
=C2=A0 =C2=A0 &quot;get&quot; requests instead of just one, and that IPFS d=
oes<br>
=C2=A0 =C2=A0 not &quot;know&quot; that those files are related.=C2=A0 (We =
lose the web view of<br>
=C2=A0 =C2=A0 the tree, etc.)=C2=A0 Storing them as tree could be better.<b=
r>
=C2=A0 =C2=A0 I don&#39;t understand if that would work with the &quot;IPLD=
 manifest&quot;<br>
=C2=A0 =C2=A0 suggested above.=C2=A0 H=C3=A9ctor?<br>
<br>
=C2=A0 - Pinning: Pinning all files separately incurs an overhead.=C2=A0 It=
&#39;s<br>
=C2=A0 =C2=A0 enough to just pin the IPLD object since it propagates recurs=
ively.<br>
=C2=A0 =C2=A0 When adding a tree, then it&#39;s no problem since pinning is=
 only done once.<br>
<br>
=C2=A0 - IPFS endpoint calls: instead of adding each file individually, it&=
#39;s<br>
=C2=A0 =C2=A0 possible to add them all in one go.=C2=A0 Can we add all file=
s at once<br>
=C2=A0 =C2=A0 while using a flat storage? (I.e. not adding them all under a=
 common<br>
=C2=A0 =C2=A0 root folder.)<br>
<br>
To sum up, here is what remains to be done on the current patch:<br>
<br>
- Add all files in one go without pinning them.<br>
- Store as the file tree?=C2=A0 Can we still us the IPLD object to referenc=
e<br>
=C2=A0 the files in the tree?=C2=A0 Else use the &quot;raw-leaves&quot; opt=
ion to avoid<br>
=C2=A0 wrapping small files in UnixFS blocks.<br>
- Remove the Scheme manifest if IPLD can do.<br>
- Generate the IPLD object and pin it.<br>
<br>
Any corrections?<br>
Thoughts?<br>
<br>
Cheers!<br>
<br>
-- <br>
Pierre Neidhardt<br>
<a href=3D"https://ambrevar.xyz/" rel=3D"noreferrer" target=3D"_blank">http=
s://ambrevar.xyz/</a><br>
<br>
-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;Go IPFS Working Group&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:go-ipfs-wg%2Bunsubscribe@HIDDEN" target=3D"_blan=
k">go-ipfs-wg+unsubscribe@HIDDEN</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40ambrevar.xyz" rel=3D"noref=
errer" target=3D"_blank">https://groups.google.com/a/ipfs.io/d/msgid/go-ipf=
s-wg/87zhlxe8t9.fsf%40ambrevar.xyz</a>.<br>
</blockquote></div></div>

--0000000000000e29ad058d8166f7--

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 14 Jul 2019 22:33:02 +0000
Resent-Message-ID: <handler.33899.B33899.156314352330214 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156314352330214
          (code B ref 33899); Sun, 14 Jul 2019 22:33:02 +0000
Received: (at 33899) by debbugs.gnu.org; 14 Jul 2019 22:32:03 +0000
Received: from localhost ([127.0.0.1]:46174 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hmn2I-0007qp-GE
	for submit <at> debbugs.gnu.org; Sun, 14 Jul 2019 18:32:03 -0400
Received: from mail1.protonmail.ch ([185.70.40.18]:47974)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1hmn2E-0007jh-W4
 for 33899 <at> debbugs.gnu.org; Sun, 14 Jul 2019 18:32:00 -0400
Date: Sun, 14 Jul 2019 22:31:43 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1563143510;
 bh=A1ca9BbPvEBnUQhGhk34WkOtYemz98SjsNpEYJUXbZk=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=m29RboLIL4fAf+SslxgfY4xuNdk/Zig/YTJ4VGq3Z5KIZOEBBYS/iSYo5/w5WO1E3
 Z9ZUk57jK2C/+D4fBCR+/inlRzz+Dw13Kv3ZRRxQg/giSQrWAhTynXnP3gTZg+bEXP
 tTJ8iZ+YMz1DiKKomfydSB35KRVLTEjaKvohRxbA=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link>
In-Reply-To: <87ftnbf1rt.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham
 autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Hey! Thanks for reviving this discussion!

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wrot=
e:

> Hello!
>
> Pierre Neidhardt mail@HIDDEN skribis:
>
> > A little update/recap after many months! :)
>
> Thank you, and apologies for the delay!
>
> > -   Extra metadata: IPFS stores files on UnixFSv1 which does not
> >     include the executable bit.
> >     -   Right now we store a s-exp manifest with a list of files and a
> >         list of executable bits. But maybe we don't have to roll out ou=
r own.
> >
> >     -   UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did =
not
> >         recommend using it (not sure why though).
> >
> >     -   We could use UnixFSv2 but it's not released yet and it's unclea=
r when
> >         it's going to be released. So we can't really count on it right=
 now.
> >
>
> UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2
> would be appropriate, though it stores timestamps that we don=E2=80=99t n=
eed
> (not necessarily a problem).
>
> > -   Flat storage vs. tree storage: Right now we are storing the files
> >     separately, but this has some shortcomings, namely we need multiple
> >     "get" requests instead of just one, and that IPFS does
> >     not "know" that those files are related. (We lose the web view of
> >     the tree, etc.) Storing them as tree could be better.
> >     I don't understand if that would work with the "IPLD manifest"
> >     suggested above. H=C3=A9ctor?
> >
>
> I don=E2=80=99t consider the web view a strong argument :-) since we coul=
d write
> tools to deal with whatever format we use.
>
> Regarding multiple GET requests: we could pipeline them, and it seems
> more like an implementation detail to me. The real question is whether
> making separate GET requests prevents some optimization in IPFS.
>
> > -   Pinning: Pinning all files separately incurs an overhead. It's
> >     enough to just pin the IPLD object since it propagates recursively.
> >     When adding a tree, then it's no problem since pinning is only done=
 once.
> >
>
> Where=E2=80=99s the overhead exactly?

There are reasons why we are proposing to create a single DAG with an
IPLD object at the root. Pinning has a big overhead because it
involves locking, reading, parsing, and writing an internal pin-DAG. This
is specially relevant when the pinset is very large.

Doing multiple GET requests also has overhead, like being unable to use
a single bitswap session (which, when downloading something new means a
big overhead since every request will have to find providers).

And it's not just the web view, it's the ability to walk/traverse all
the object related to a given root natively, which allows also to compare
multiple trees and to be more efficient for some things ("pin update"
for example). Your original idea is to create a manifest with
references to different parts. I'm just asking you to
create that manifest in a format where those references are understood
not only by you, the file creator, but by IPFS and any tool that can
read IPLD, by making this a IPLD object (which is just a json).

The process of adding "something" to ipfs is as follows.

----
1. Add to IPFS: multipart upload equivalent to "ipfs add -r":

~/ipfstest $ ipfs add -r -q .
QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP

2. Add manifest as IPLD object. dag/put a json file like:

cat <<EOF | ipfs dag put
{
  "executables": ["ipfstest/1.txt"],
  "root": {
    "/": "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP"
  }
}
EOF
---

That's it "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP" is the root
of your package files.

"bafyreievcw5qoowhepwskcxybochrui65bbtsliuy7r6kyail4w5lyqnjm"
is the root of your manifest file with the list of executables
and a pointer to the other root.


Hector

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Jul 2019 09:26:01 +0000
Resent-Message-ID: <handler.33899.B33899.156318270724090 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Hector Sanjuan <code@HIDDEN>
Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156318270724090
          (code B ref 33899); Mon, 15 Jul 2019 09:26:01 +0000
Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 09:25:07 +0000
Received: from localhost ([127.0.0.1]:46393 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hmxEJ-0006GU-6I
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:25:07 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38340)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1hmxEI-0006Fo-26
 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:25:06 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43843)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>)
 id 1hmxEB-0008OF-2c; Mon, 15 Jul 2019 05:24:59 -0400
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=38884 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@HIDDEN>)
 id 1hmxEA-0000G8-JN; Mon, 15 Jul 2019 05:24:58 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN>
 <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 27 Messidor an 227 de la =?UTF-8?Q?R=C3=A9volution?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 15 Jul 2019 11:24:57 +0200
In-Reply-To: <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link>
 (Hector Sanjuan's message of "Sun, 14 Jul 2019 22:31:43 +0000")
Message-ID: <87ef2rr6om.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello H=C3=A9ctor!  :-)

Hector Sanjuan <code@HIDDEN> skribis:

> On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wr=
ote:

[...]

>> > -   Pinning: Pinning all files separately incurs an overhead. It's
>> >     enough to just pin the IPLD object since it propagates recursively.
>> >     When adding a tree, then it's no problem since pinning is only don=
e once.
>> >
>>
>> Where=E2=80=99s the overhead exactly?
>
> There are reasons why we are proposing to create a single DAG with an
> IPLD object at the root. Pinning has a big overhead because it
> involves locking, reading, parsing, and writing an internal pin-DAG. This
> is specially relevant when the pinset is very large.
>
> Doing multiple GET requests also has overhead, like being unable to use
> a single bitswap session (which, when downloading something new means a
> big overhead since every request will have to find providers).
>
> And it's not just the web view, it's the ability to walk/traverse all
> the object related to a given root natively, which allows also to compare
> multiple trees and to be more efficient for some things ("pin update"
> for example). Your original idea is to create a manifest with
> references to different parts. I'm just asking you to
> create that manifest in a format where those references are understood
> not only by you, the file creator, but by IPFS and any tool that can
> read IPLD, by making this a IPLD object (which is just a json).

OK, I see.  Put this way, it seems like creating a DAG with an IPLD
object as its root is pretty compelling.

Thanks for clarifying!

Ludo=E2=80=99.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Pierre Neidhardt <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Jul 2019 10:12:02 +0000
Resent-Message-ID: <handler.33899.B33899.156318549221173 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Hector Sanjuan <code@HIDDEN>
Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156318549221173
          (code B ref 33899); Mon, 15 Jul 2019 10:12:02 +0000
Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 10:11:32 +0000
Received: from localhost ([127.0.0.1]:46454 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hmxxD-0005VR-Nl
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:11:32 -0400
Received: from relay8-d.mail.gandi.net ([217.70.183.201]:58277)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1hmxxA-0005VD-E0
 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:11:30 -0400
X-Originating-IP: 92.169.116.127
Received: from bababa (lfbn-1-4117-127.w92-169.abo.wanadoo.fr [92.169.116.127])
 (Authenticated sender: pierre@HIDDEN)
 by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 29A9E1BF21A;
 Mon, 15 Jul 2019 10:10:51 +0000 (UTC)
From: Pierre Neidhardt <mail@HIDDEN>
In-Reply-To: <87ef2rr6om.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN>
 <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link>
 <87ef2rr6om.fsf@HIDDEN>
Date: Mon, 15 Jul 2019 12:10:51 +0200
Message-ID: <87zhlf7glw.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha256; protocol="application/pgp-signature"
X-Spam-Score: -0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.2 (/)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

H=C3=A9ctor mentioned a possible issue with the IPLD manifest growing too b=
ig
(in case of too many files in a package), that is, above 2MB.
Then we would need to implement some form of sharding.

H=C3=A9ctor, do you confirm?  Any idea on how to tackle this elegantly?

=2D-=20
Pierre Neidhardt
https://ambrevar.xyz/

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0sUSsACgkQm9z0l6S7
zH/mJQf+NTkbrr1KF6tXPlj0AoVhvX8pNkzoqt+3elFFZyXYmmAsegpCmM0u1p9T
wXijMS+RsKYQkDlq+45gyyXjUFnZFYBlRjY2CVJoYbfpK5BDGTgdxfdch7vTL9A0
4hL7drMpQkjrSpE9mVAwTF7ngGYNzZNyWOE+z3gp/u+bFb69a1SPh6WzCavtdCTs
jOnLCJJTFzKl9EaA3VvYFtZ8iS1eunC1qIk5dLjIkbcc0oWK4QQqyC3EZNtDOcyA
3eMwYkVzC8+56XCXT+x8qVgJWo4n73TErviCQEjG6P3ni+JqEPCqkK3tEzncrLw6
I3lUbvMHXWUGWxmMIf38/OKuD00Whg==
=bQTd
-----END PGP SIGNATURE-----
--=-=-=--

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Hector Sanjuan <code@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Jul 2019 10:22:02 +0000
Resent-Message-ID: <handler.33899.B33899.156318610530541 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Pierre Neidhardt <mail@HIDDEN>
Cc: Antoine Eiche <lewo@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Reply-To: Hector Sanjuan <code@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156318610530541
          (code B ref 33899); Mon, 15 Jul 2019 10:22:02 +0000
Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 10:21:45 +0000
Received: from localhost ([127.0.0.1]:46475 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hmy77-0007wX-73
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:21:45 -0400
Received: from mail1.protonmail.ch ([185.70.40.18]:12764)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1hmy75-0007wK-5z
 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:21:43 -0400
Date: Mon, 15 Jul 2019 10:21:29 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link;
 s=protonmail; t=1563186095;
 bh=TIpP7wYPxWbYeJf5L4gugk8l4GYnpT0V2wx9rOANvpI=;
 h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:
 Feedback-ID:From;
 b=VEvbU3Qm58WF1lDTfG0xttvMUI9OablWV5XHdKmjJHdM60QI3QvhddQnFsRpJ4NjS
 p8YrCBQ1lQyzk2+ZLpTcTOYTjhtTeiJsrpz3rbfEaHtEqhDTnDj4fNG19o8IEQ00Tk
 FU/PCbgrxWj05F/ACgUsy5YEpep0LH9Oh+Pjr6Ig=
From: Hector Sanjuan <code@HIDDEN>
Message-ID: <TbeM3tgSHfcbwcjeZ6QdzDTvWhrkzeTEnWCbfJxJfx4sPZRGupGThXLKTPafQnN2u-MrQ_zcST3TawHVICLzaQAq9gG11BXkNNqgzBkXg6U=@hector.link>
In-Reply-To: <87zhlf7glw.fsf@HIDDEN>
References: <20181228231205.8068-1-ludo@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN>
 <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link>
 <87ef2rr6om.fsf@HIDDEN> <87zhlf7glw.fsf@HIDDEN>
Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham
 autolearn_force=no version=3.4.2
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me=
ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90
On Monday, July 15, 2019 12:10 PM, Pierre Neidhardt <mail@HIDDEN> wro=
te:

> H=C3=A9ctor mentioned a possible issue with the IPLD manifest growing too=
 big
> (in case of too many files in a package), that is, above 2MB.
> Then we would need to implement some form of sharding.
>
> H=C3=A9ctor, do you confirm? Any idea on how to tackle this elegantly?
>

Doing the DAG node the way I proposed it (referencing a single root) should=
 be
ok... Unless you put too many executable files in that list, it should larg=
ely
stay within the 2MB limit.


--
Hector

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#33899] [PATCH 0/5] Distributing substitutes over IPFS
Resent-From: Alex Potsides <alex.potsides@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Mon, 15 Jul 2019 13:52:02 +0000
Resent-Message-ID: <handler.33899.B33899.156319871521166 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33899
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Molly Mackinlay <molly@HIDDEN>
Cc: Hector Sanjuan <code@HIDDEN>, Eric Myhre <eric.myhre@HIDDEN>, Andrew Nesbitt <andrew@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>, Antoine Eiche <lewo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, Jessica Schilling <jessica@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>
Received: via spool by 33899-submit <at> debbugs.gnu.org id=B33899.156319871521166
          (code B ref 33899); Mon, 15 Jul 2019 13:52:02 +0000
Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 13:51:55 +0000
Received: from localhost ([127.0.0.1]:46924 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1hn1OU-0005VJ-OZ
	for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 09:51:55 -0400
Received: from mail-io1-f44.google.com ([209.85.166.44]:36830)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <alex.potsides@HIDDEN>) id 1hmxA2-00067M-OB
 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:20:43 -0400
Received: by mail-io1-f44.google.com with SMTP id o9so33010855iom.3
 for <33899 <at> debbugs.gnu.org>; Mon, 15 Jul 2019 02:20:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protocol.ai; s=google;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=JVKsX9yXOVrQhKCE8ctMtZohF0HOELFZEQ9+EI7Wel8=;
 b=D+GEjm0Bq18c01eS0jOVsSWRLrt1ZKSFhNDSvElSj1KhkxCe/OkkjYkHkPGjmyunAb
 khhJJuw0x5NbVsVLCnbrJGsjkxYvy4Cd3eS6oVC8G8z4mguMmfhwab9SaipZ1y+9a2we
 y7S/mIsAMCIEnKeORdwUR6MlmhW5YGnSrzJ7zDTxedgncI5D9KS0DKpP2cdqZim8l9L8
 AKpeYycptlP/6ys9lh7nf970xLkPa6EpTmsl07sX0RkRm+9pImbNFrFN3drGaRRrxcNU
 PEhYxmmKjP47MmDjXP+yYSK7kejAhPmNFsjquZ5LEXFdHIdOBmZrIONkOGc+GjwOlg/K
 q99Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=JVKsX9yXOVrQhKCE8ctMtZohF0HOELFZEQ9+EI7Wel8=;
 b=qo6TiGYHKbx1ey+qmsnbcvkFErJKB33FMwFGNzc442L7WiJBYROxGc/NU3zofdN8po
 SOZ0lMlqFrpXi0rLhovDAO5XQU3v3S8Lbkc9UF126msGzCpMMl20nVg19BInzJ3bQpFb
 +qnI+1YKDavlTx3y2WjPTNwo3K0vNcYGJI6OaO+vGmpL+WaINSHsbjpfFNOANpUhpNVt
 zb+QQGTNRp94eeYAD27loCuYYphxP+xjJgkXosNZgzWMip16bcoadOqfO0vW2QqtWRe4
 UKD3f+GejUPE/gPkhPV1OuOuqPtkAESHdWj9bC6TfNVDIxX/LnhMBwObZ5zOXRsZuod6
 BAhA==
X-Gm-Message-State: APjAAAV7ICHsXns33Njm+ealNmlPaorTPS55W+HLSo0x5bXD7otik5Fq
 xgJlgSxdVC8diSxJnEf7yGxr3DhVt5Rn3mV/9U6npw==
X-Google-Smtp-Source: APXvYqxziFxpokSLRNCxyhUHQhm71Slye/PB2M3I3qEEEnni1w0sJJmhF7Ct5DPoKjQ4E3SljYMfV9e48F7LGWCb224=
X-Received: by 2002:a6b:ed01:: with SMTP id n1mr20901155iog.255.1563182436953; 
 Mon, 15 Jul 2019 02:20:36 -0700 (PDT)
MIME-Version: 1.0
References: <20181228231205.8068-1-ludo@HIDDEN>
 <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link>
 <87r2dfv0nj.fsf@HIDDEN>
 <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link>
 <8736pqthqm.fsf@HIDDEN>
 <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link>
 <87zhlxe8t9.fsf@HIDDEN>
 <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN>
In-Reply-To: <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN>
From: Alex Potsides <alex.potsides@HIDDEN>
Date: Mon, 15 Jul 2019 10:20:26 +0100
Message-ID: <CAL7-xizWV2EEtyCf=3HbEdDaTdz3qjskVqZfh6LrF-QBCfKNGg@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000dfcafe058db4c59e"
X-Spam-Score: 0.0 (/)
X-Mailman-Approved-At: Mon, 15 Jul 2019 09:51:53 -0400
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000dfcafe058db4c59e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

The reason not to use the UnixFSv1 metadata field was that it's in the spec
<https://github.com/ipfs/specs/tree/master/unixfs#data-format> but it's not
really been implemented.  As it stands in v1, you'd have to add explicit
metadata types to the spec (executable, owner?, group?, etc) because
protobufs need to know about everything ahead of time and each
implementation would have update to implement those.  This is all possible
& not a technical blocker, but since most effort is centred around UnixFSv2
the timescales might not fit with people's requirements.

The more pragmatic approach Hector suggested was to wrap a CID that
resolves to the UnixFSv1 file in a JSON object that you could use to store
application-specific metadata - something similar to the UnixFSv1.5 section
<https://github.com/ipfs/camp/blob/master/DEEP_DIVES/package-managers/READM=
E.md#unixfs-v15>
in our notes from the Package Managers deep dive we did at camp.

a.






On Fri, Jul 12, 2019 at 9:03 PM Molly Mackinlay <molly@HIDDEN> wrote:

> Thanks for the update Pierre! Also adding Alex, Jessica, Eric and Andrew
> from the package managers discussions at IPFS Camp as FYI.
>
> Generating the ipld manifest with the metadata and the tree of files
> should also be fine AFAIK - I=E2=80=99m sure Hector and Eric can expand m=
ore on how
> to compose them, but data storage format shouldn=E2=80=99t make a big dif=
ference
> for the ipld manifest.
>
> On Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt <mail@HIDDEN> wrote=
:
>
>> Hi!
>>
>> (Re-sending to debbugs, sorry for the double email :p)
>>
>> A little update/recap after many months! :)
>>
>> I talked with H=C3=A9ctor and some other people from IPFS + I reviewed L=
udo's
>> patch so now I have a little better understanding of the current state
>> of affair.
>>
>> - We could store the substitutes as tarballs on IPFS, but this has
>>   some possible downsides:
>>
>>   - We would need to use IPFS' tar chunker to deduplicate the content of
>>     the tarball.  But the tar chunker is not well maintained currently,
>>     and it's not clear whether it's reproducible at the moment, so it
>>     would need some more work.
>>
>>   - Tarballs might induce some performance cost.  Nix had attempted
>>     something similar in the past and this may have incurred a significa=
nt
>>     performance penalty, although this remains to be confirmed.
>>     Lewo?
>>
>> - Ludo's patch stores all files on IPFS individually.  This way we don't
>>   need to touch the tar chunker, so it's less work :)
>>   This raises some other issues however:
>>
>>   - Extra metadata:  IPFS stores files on UnixFSv1 which does not
>>     include the executable bit.
>>
>>     - Right now we store a s-exp manifest with a list of files and a
>>       list of executable bits.  But maybe we don't have to roll out our
>> own.
>>
>>     - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not
>>       recommend using it (not sure why though).
>>
>>     - We could use UnixFSv2 but it's not released yet and it's unclear
>> when
>>       it's going to be released.  So we can't really count on it right
>> now.
>>
>>     - IPLD: As H=C3=A9ctor suggested in the previous email, we could lev=
erage
>>       IPLD and generate a JSON object that references the files with
>>       their paths together with an "executable?" property.
>>       A problem would arise if this IPLD object grows over the 2M
>>       block-size limit because then we would have to shard it (something
>>       that UnixFS would do automatically for us).
>>
>>   - Flat storage vs. tree storage: Right now we are storing the files
>>     separately, but this has some shortcomings, namely we need multiple
>>     "get" requests instead of just one, and that IPFS does
>>     not "know" that those files are related.  (We lose the web view of
>>     the tree, etc.)  Storing them as tree could be better.
>>     I don't understand if that would work with the "IPLD manifest"
>>     suggested above.  H=C3=A9ctor?
>>
>>   - Pinning: Pinning all files separately incurs an overhead.  It's
>>     enough to just pin the IPLD object since it propagates recursively.
>>     When adding a tree, then it's no problem since pinning is only done
>> once.
>>
>>   - IPFS endpoint calls: instead of adding each file individually, it's
>>     possible to add them all in one go.  Can we add all files at once
>>     while using a flat storage? (I.e. not adding them all under a common
>>     root folder.)
>>
>> To sum up, here is what remains to be done on the current patch:
>>
>> - Add all files in one go without pinning them.
>> - Store as the file tree?  Can we still us the IPLD object to reference
>>   the files in the tree?  Else use the "raw-leaves" option to avoid
>>   wrapping small files in UnixFS blocks.
>> - Remove the Scheme manifest if IPLD can do.
>> - Generate the IPLD object and pin it.
>>
>> Any corrections?
>> Thoughts?
>>
>> Cheers!
>>
>> --
>> Pierre Neidhardt
>> https://ambrevar.xyz/
>>
>> --
>> You received this message because you are subscribed to the Google Group=
s
>> "Go IPFS Working Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send a=
n
>> email to go-ipfs-wg+unsubscribe@HIDDEN
>> To view this discussion on the web visit
>> https://groups.google.com/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40=
ambrevar.xyz
>> .
>>
>

--000000000000dfcafe058db4c59e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div>The reason not to use the UnixFSv1 metadata=
 field was that it&#39;s <a href=3D"https://github.com/ipfs/specs/tree/mast=
er/unixfs#data-format">in the spec</a> but it&#39;s not really been impleme=
nted.=C2=A0 As it stands in v1, you&#39;d have to add explicit metadata typ=
es to the spec (executable, owner?, group?, etc) because protobufs need to =
know about everything ahead of time and each implementation would have upda=
te to implement those.=C2=A0 This is all possible &amp; not a technical blo=
cker, but since most effort is centred around UnixFSv2 the timescales might=
 not fit with people&#39;s requirements.<div><br></div><div>The more pragma=
tic approach Hector suggested was to wrap a CID that resolves to the UnixFS=
v1 file in a JSON object that you could use to store application-specific m=
etadata - something similar to the <a href=3D"https://github.com/ipfs/camp/=
blob/master/DEEP_DIVES/package-managers/README.md#unixfs-v15">UnixFSv1.5 se=
ction</a> in our notes from the Package Managers deep dive we did at camp.<=
/div><div><br></div><div>a.</div><div><br></div><div><br></div><div><br></d=
iv><div><br></div><div><br></div></div><br><div class=3D"gmail_quote"><div =
dir=3D"ltr" class=3D"gmail_attr">On Fri, Jul 12, 2019 at 9:03 PM Molly Mack=
inlay &lt;<a href=3D"mailto:molly@HIDDEN">molly@HIDDEN</a>&gt; wr=
ote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div d=
ir=3D"auto">Thanks for the update Pierre! Also adding Alex, Jessica, Eric a=
nd Andrew from the package managers discussions at IPFS Camp as FYI.</div><=
br></div><div dir=3D"auto">Generating the ipld manifest with the metadata a=
nd the tree of files should also be fine AFAIK - I=E2=80=99m sure Hector an=
d Eric can expand more on how to compose them, but data storage format shou=
ldn=E2=80=99t make a big difference for the ipld manifest.</div><div><br><d=
iv class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jul =
1, 2019 at 2:36 PM Pierre Neidhardt &lt;<a href=3D"mailto:mail@HIDDEN=
" target=3D"_blank">mail@HIDDEN</a>&gt; wrote:<br></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex">Hi!<br>
<br>
(Re-sending to debbugs, sorry for the double email :p)<br>
<br>
A little update/recap after many months! :)<br>
<br>
I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo=
&#39;s<br>
patch so now I have a little better understanding of the current state<br>
of affair.<br>
<br>
- We could store the substitutes as tarballs on IPFS, but this has<br>
=C2=A0 some possible downsides:<br>
<br>
=C2=A0 - We would need to use IPFS&#39; tar chunker to deduplicate the cont=
ent of<br>
=C2=A0 =C2=A0 the tarball.=C2=A0 But the tar chunker is not well maintained=
 currently,<br>
=C2=A0 =C2=A0 and it&#39;s not clear whether it&#39;s reproducible at the m=
oment, so it<br>
=C2=A0 =C2=A0 would need some more work.<br>
<br>
=C2=A0 - Tarballs might induce some performance cost.=C2=A0 Nix had attempt=
ed<br>
=C2=A0 =C2=A0 something similar in the past and this may have incurred a si=
gnificant<br>
=C2=A0 =C2=A0 performance penalty, although this remains to be confirmed.<b=
r>
=C2=A0 =C2=A0 Lewo?<br>
<br>
- Ludo&#39;s patch stores all files on IPFS individually.=C2=A0 This way we=
 don&#39;t<br>
=C2=A0 need to touch the tar chunker, so it&#39;s less work :)<br>
=C2=A0 This raises some other issues however:<br>
<br>
=C2=A0 - Extra metadata:=C2=A0 IPFS stores files on UnixFSv1 which does not=
<br>
=C2=A0 =C2=A0 include the executable bit.<br>
<br>
=C2=A0 =C2=A0 - Right now we store a s-exp manifest with a list of files an=
d a<br>
=C2=A0 =C2=A0 =C2=A0 list of executable bits.=C2=A0 But maybe we don&#39;t =
have to roll out our own.<br>
<br>
=C2=A0 =C2=A0 - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex =
did not<br>
=C2=A0 =C2=A0 =C2=A0 recommend using it (not sure why though).<br>
<br>
=C2=A0 =C2=A0 - We could use UnixFSv2 but it&#39;s not released yet and it&=
#39;s unclear when<br>
=C2=A0 =C2=A0 =C2=A0 it&#39;s going to be released.=C2=A0 So we can&#39;t r=
eally count on it right now.<br>
<br>
=C2=A0 =C2=A0 - IPLD: As H=C3=A9ctor suggested in the previous email, we co=
uld leverage<br>
=C2=A0 =C2=A0 =C2=A0 IPLD and generate a JSON object that references the fi=
les with<br>
=C2=A0 =C2=A0 =C2=A0 their paths together with an &quot;executable?&quot; p=
roperty.<br>
=C2=A0 =C2=A0 =C2=A0 A problem would arise if this IPLD object grows over t=
he 2M<br>
=C2=A0 =C2=A0 =C2=A0 block-size limit because then we would have to shard i=
t (something<br>
=C2=A0 =C2=A0 =C2=A0 that UnixFS would do automatically for us).<br>
<br>
=C2=A0 - Flat storage vs. tree storage: Right now we are storing the files<=
br>
=C2=A0 =C2=A0 separately, but this has some shortcomings, namely we need mu=
ltiple<br>
=C2=A0 =C2=A0 &quot;get&quot; requests instead of just one, and that IPFS d=
oes<br>
=C2=A0 =C2=A0 not &quot;know&quot; that those files are related.=C2=A0 (We =
lose the web view of<br>
=C2=A0 =C2=A0 the tree, etc.)=C2=A0 Storing them as tree could be better.<b=
r>
=C2=A0 =C2=A0 I don&#39;t understand if that would work with the &quot;IPLD=
 manifest&quot;<br>
=C2=A0 =C2=A0 suggested above.=C2=A0 H=C3=A9ctor?<br>
<br>
=C2=A0 - Pinning: Pinning all files separately incurs an overhead.=C2=A0 It=
&#39;s<br>
=C2=A0 =C2=A0 enough to just pin the IPLD object since it propagates recurs=
ively.<br>
=C2=A0 =C2=A0 When adding a tree, then it&#39;s no problem since pinning is=
 only done once.<br>
<br>
=C2=A0 - IPFS endpoint calls: instead of adding each file individually, it&=
#39;s<br>
=C2=A0 =C2=A0 possible to add them all in one go.=C2=A0 Can we add all file=
s at once<br>
=C2=A0 =C2=A0 while using a flat storage? (I.e. not adding them all under a=
 common<br>
=C2=A0 =C2=A0 root folder.)<br>
<br>
To sum up, here is what remains to be done on the current patch:<br>
<br>
- Add all files in one go without pinning them.<br>
- Store as the file tree?=C2=A0 Can we still us the IPLD object to referenc=
e<br>
=C2=A0 the files in the tree?=C2=A0 Else use the &quot;raw-leaves&quot; opt=
ion to avoid<br>
=C2=A0 wrapping small files in UnixFS blocks.<br>
- Remove the Scheme manifest if IPLD can do.<br>
- Generate the IPLD object and pin it.<br>
<br>
Any corrections?<br>
Thoughts?<br>
<br>
Cheers!<br>
<br>
-- <br>
Pierre Neidhardt<br>
<a href=3D"https://ambrevar.xyz/" rel=3D"noreferrer" target=3D"_blank">http=
s://ambrevar.xyz/</a><br>
<br>
-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;Go IPFS Working Group&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:go-ipfs-wg%2Bunsubscribe@HIDDEN" target=3D"_blan=
k">go-ipfs-wg+unsubscribe@HIDDEN</a>.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40ambrevar.xyz" rel=3D"noref=
errer" target=3D"_blank">https://groups.google.com/a/ipfs.io/d/msgid/go-ipf=
s-wg/87zhlxe8t9.fsf%40ambrevar.xyz</a>.<br>
</blockquote></div></div>
</blockquote></div>

--000000000000dfcafe058db4c59e--

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.