Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 13:51:55 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 09:51:55 2019 Received: from localhost ([127.0.0.1]:46924 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hn1OU-0005VJ-OZ for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 09:51:55 -0400 Received: from mail-io1-f44.google.com ([209.85.166.44]:36830) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <alex.potsides@HIDDEN>) id 1hmxA2-00067M-OB for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:20:43 -0400 Received: by mail-io1-f44.google.com with SMTP id o9so33010855iom.3 for <33899 <at> debbugs.gnu.org>; Mon, 15 Jul 2019 02:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protocol.ai; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JVKsX9yXOVrQhKCE8ctMtZohF0HOELFZEQ9+EI7Wel8=; b=D+GEjm0Bq18c01eS0jOVsSWRLrt1ZKSFhNDSvElSj1KhkxCe/OkkjYkHkPGjmyunAb khhJJuw0x5NbVsVLCnbrJGsjkxYvy4Cd3eS6oVC8G8z4mguMmfhwab9SaipZ1y+9a2we y7S/mIsAMCIEnKeORdwUR6MlmhW5YGnSrzJ7zDTxedgncI5D9KS0DKpP2cdqZim8l9L8 AKpeYycptlP/6ys9lh7nf970xLkPa6EpTmsl07sX0RkRm+9pImbNFrFN3drGaRRrxcNU PEhYxmmKjP47MmDjXP+yYSK7kejAhPmNFsjquZ5LEXFdHIdOBmZrIONkOGc+GjwOlg/K q99Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JVKsX9yXOVrQhKCE8ctMtZohF0HOELFZEQ9+EI7Wel8=; b=qo6TiGYHKbx1ey+qmsnbcvkFErJKB33FMwFGNzc442L7WiJBYROxGc/NU3zofdN8po SOZ0lMlqFrpXi0rLhovDAO5XQU3v3S8Lbkc9UF126msGzCpMMl20nVg19BInzJ3bQpFb +qnI+1YKDavlTx3y2WjPTNwo3K0vNcYGJI6OaO+vGmpL+WaINSHsbjpfFNOANpUhpNVt zb+QQGTNRp94eeYAD27loCuYYphxP+xjJgkXosNZgzWMip16bcoadOqfO0vW2QqtWRe4 UKD3f+GejUPE/gPkhPV1OuOuqPtkAESHdWj9bC6TfNVDIxX/LnhMBwObZ5zOXRsZuod6 BAhA== X-Gm-Message-State: APjAAAV7ICHsXns33Njm+ealNmlPaorTPS55W+HLSo0x5bXD7otik5Fq xgJlgSxdVC8diSxJnEf7yGxr3DhVt5Rn3mV/9U6npw== X-Google-Smtp-Source: APXvYqxziFxpokSLRNCxyhUHQhm71Slye/PB2M3I3qEEEnni1w0sJJmhF7Ct5DPoKjQ4E3SljYMfV9e48F7LGWCb224= X-Received: by 2002:a6b:ed01:: with SMTP id n1mr20901155iog.255.1563182436953; Mon, 15 Jul 2019 02:20:36 -0700 (PDT) MIME-Version: 1.0 References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN> In-Reply-To: <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN> From: Alex Potsides <alex.potsides@HIDDEN> Date: Mon, 15 Jul 2019 10:20:26 +0100 Message-ID: <CAL7-xizWV2EEtyCf=3HbEdDaTdz3qjskVqZfh6LrF-QBCfKNGg@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS To: Molly Mackinlay <molly@HIDDEN> Content-Type: multipart/alternative; boundary="000000000000dfcafe058db4c59e" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 33899 X-Mailman-Approved-At: Mon, 15 Jul 2019 09:51:53 -0400 Cc: Hector Sanjuan <code@HIDDEN>, Eric Myhre <eric.myhre@HIDDEN>, Andrew Nesbitt <andrew@HIDDEN>, =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>, Antoine Eiche <lewo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, Jessica Schilling <jessica@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --000000000000dfcafe058db4c59e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The reason not to use the UnixFSv1 metadata field was that it's in the spec <https://github.com/ipfs/specs/tree/master/unixfs#data-format> but it's not really been implemented. As it stands in v1, you'd have to add explicit metadata types to the spec (executable, owner?, group?, etc) because protobufs need to know about everything ahead of time and each implementation would have update to implement those. This is all possible & not a technical blocker, but since most effort is centred around UnixFSv2 the timescales might not fit with people's requirements. The more pragmatic approach Hector suggested was to wrap a CID that resolves to the UnixFSv1 file in a JSON object that you could use to store application-specific metadata - something similar to the UnixFSv1.5 section <https://github.com/ipfs/camp/blob/master/DEEP_DIVES/package-managers/READM= E.md#unixfs-v15> in our notes from the Package Managers deep dive we did at camp. a. On Fri, Jul 12, 2019 at 9:03 PM Molly Mackinlay <molly@HIDDEN> wrote: > Thanks for the update Pierre! Also adding Alex, Jessica, Eric and Andrew > from the package managers discussions at IPFS Camp as FYI. > > Generating the ipld manifest with the metadata and the tree of files > should also be fine AFAIK - I=E2=80=99m sure Hector and Eric can expand m= ore on how > to compose them, but data storage format shouldn=E2=80=99t make a big dif= ference > for the ipld manifest. > > On Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt <mail@HIDDEN> wrote= : > >> Hi! >> >> (Re-sending to debbugs, sorry for the double email :p) >> >> A little update/recap after many months! :) >> >> I talked with H=C3=A9ctor and some other people from IPFS + I reviewed L= udo's >> patch so now I have a little better understanding of the current state >> of affair. >> >> - We could store the substitutes as tarballs on IPFS, but this has >> some possible downsides: >> >> - We would need to use IPFS' tar chunker to deduplicate the content of >> the tarball. But the tar chunker is not well maintained currently, >> and it's not clear whether it's reproducible at the moment, so it >> would need some more work. >> >> - Tarballs might induce some performance cost. Nix had attempted >> something similar in the past and this may have incurred a significa= nt >> performance penalty, although this remains to be confirmed. >> Lewo? >> >> - Ludo's patch stores all files on IPFS individually. This way we don't >> need to touch the tar chunker, so it's less work :) >> This raises some other issues however: >> >> - Extra metadata: IPFS stores files on UnixFSv1 which does not >> include the executable bit. >> >> - Right now we store a s-exp manifest with a list of files and a >> list of executable bits. But maybe we don't have to roll out our >> own. >> >> - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not >> recommend using it (not sure why though). >> >> - We could use UnixFSv2 but it's not released yet and it's unclear >> when >> it's going to be released. So we can't really count on it right >> now. >> >> - IPLD: As H=C3=A9ctor suggested in the previous email, we could lev= erage >> IPLD and generate a JSON object that references the files with >> their paths together with an "executable?" property. >> A problem would arise if this IPLD object grows over the 2M >> block-size limit because then we would have to shard it (something >> that UnixFS would do automatically for us). >> >> - Flat storage vs. tree storage: Right now we are storing the files >> separately, but this has some shortcomings, namely we need multiple >> "get" requests instead of just one, and that IPFS does >> not "know" that those files are related. (We lose the web view of >> the tree, etc.) Storing them as tree could be better. >> I don't understand if that would work with the "IPLD manifest" >> suggested above. H=C3=A9ctor? >> >> - Pinning: Pinning all files separately incurs an overhead. It's >> enough to just pin the IPLD object since it propagates recursively. >> When adding a tree, then it's no problem since pinning is only done >> once. >> >> - IPFS endpoint calls: instead of adding each file individually, it's >> possible to add them all in one go. Can we add all files at once >> while using a flat storage? (I.e. not adding them all under a common >> root folder.) >> >> To sum up, here is what remains to be done on the current patch: >> >> - Add all files in one go without pinning them. >> - Store as the file tree? Can we still us the IPLD object to reference >> the files in the tree? Else use the "raw-leaves" option to avoid >> wrapping small files in UnixFS blocks. >> - Remove the Scheme manifest if IPLD can do. >> - Generate the IPLD object and pin it. >> >> Any corrections? >> Thoughts? >> >> Cheers! >> >> -- >> Pierre Neidhardt >> https://ambrevar.xyz/ >> >> -- >> You received this message because you are subscribed to the Google Group= s >> "Go IPFS Working Group" group. >> To unsubscribe from this group and stop receiving emails from it, send a= n >> email to go-ipfs-wg+unsubscribe@HIDDEN >> To view this discussion on the web visit >> https://groups.google.com/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40= ambrevar.xyz >> . >> > --000000000000dfcafe058db4c59e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><br></div>The reason not to use the UnixFSv1 metadata= field was that it's <a href=3D"https://github.com/ipfs/specs/tree/mast= er/unixfs#data-format">in the spec</a> but it's not really been impleme= nted.=C2=A0 As it stands in v1, you'd have to add explicit metadata typ= es to the spec (executable, owner?, group?, etc) because protobufs need to = know about everything ahead of time and each implementation would have upda= te to implement those.=C2=A0 This is all possible & not a technical blo= cker, but since most effort is centred around UnixFSv2 the timescales might= not fit with people's requirements.<div><br></div><div>The more pragma= tic approach Hector suggested was to wrap a CID that resolves to the UnixFS= v1 file in a JSON object that you could use to store application-specific m= etadata - something similar to the <a href=3D"https://github.com/ipfs/camp/= blob/master/DEEP_DIVES/package-managers/README.md#unixfs-v15">UnixFSv1.5 se= ction</a> in our notes from the Package Managers deep dive we did at camp.<= /div><div><br></div><div>a.</div><div><br></div><div><br></div><div><br></d= iv><div><br></div><div><br></div></div><br><div class=3D"gmail_quote"><div = dir=3D"ltr" class=3D"gmail_attr">On Fri, Jul 12, 2019 at 9:03 PM Molly Mack= inlay <<a href=3D"mailto:molly@HIDDEN">molly@HIDDEN</a>> wr= ote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px= 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div d= ir=3D"auto">Thanks for the update Pierre! Also adding Alex, Jessica, Eric a= nd Andrew from the package managers discussions at IPFS Camp as FYI.</div><= br></div><div dir=3D"auto">Generating the ipld manifest with the metadata a= nd the tree of files should also be fine AFAIK - I=E2=80=99m sure Hector an= d Eric can expand more on how to compose them, but data storage format shou= ldn=E2=80=99t make a big difference for the ipld manifest.</div><div><br><d= iv class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jul = 1, 2019 at 2:36 PM Pierre Neidhardt <<a href=3D"mailto:mail@HIDDEN= " target=3D"_blank">mail@HIDDEN</a>> wrote:<br></div><blockquote c= lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli= d rgb(204,204,204);padding-left:1ex">Hi!<br> <br> (Re-sending to debbugs, sorry for the double email :p)<br> <br> A little update/recap after many months! :)<br> <br> I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo= 's<br> patch so now I have a little better understanding of the current state<br> of affair.<br> <br> - We could store the substitutes as tarballs on IPFS, but this has<br> =C2=A0 some possible downsides:<br> <br> =C2=A0 - We would need to use IPFS' tar chunker to deduplicate the cont= ent of<br> =C2=A0 =C2=A0 the tarball.=C2=A0 But the tar chunker is not well maintained= currently,<br> =C2=A0 =C2=A0 and it's not clear whether it's reproducible at the m= oment, so it<br> =C2=A0 =C2=A0 would need some more work.<br> <br> =C2=A0 - Tarballs might induce some performance cost.=C2=A0 Nix had attempt= ed<br> =C2=A0 =C2=A0 something similar in the past and this may have incurred a si= gnificant<br> =C2=A0 =C2=A0 performance penalty, although this remains to be confirmed.<b= r> =C2=A0 =C2=A0 Lewo?<br> <br> - Ludo's patch stores all files on IPFS individually.=C2=A0 This way we= don't<br> =C2=A0 need to touch the tar chunker, so it's less work :)<br> =C2=A0 This raises some other issues however:<br> <br> =C2=A0 - Extra metadata:=C2=A0 IPFS stores files on UnixFSv1 which does not= <br> =C2=A0 =C2=A0 include the executable bit.<br> <br> =C2=A0 =C2=A0 - Right now we store a s-exp manifest with a list of files an= d a<br> =C2=A0 =C2=A0 =C2=A0 list of executable bits.=C2=A0 But maybe we don't = have to roll out our own.<br> <br> =C2=A0 =C2=A0 - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex = did not<br> =C2=A0 =C2=A0 =C2=A0 recommend using it (not sure why though).<br> <br> =C2=A0 =C2=A0 - We could use UnixFSv2 but it's not released yet and it&= #39;s unclear when<br> =C2=A0 =C2=A0 =C2=A0 it's going to be released.=C2=A0 So we can't r= eally count on it right now.<br> <br> =C2=A0 =C2=A0 - IPLD: As H=C3=A9ctor suggested in the previous email, we co= uld leverage<br> =C2=A0 =C2=A0 =C2=A0 IPLD and generate a JSON object that references the fi= les with<br> =C2=A0 =C2=A0 =C2=A0 their paths together with an "executable?" p= roperty.<br> =C2=A0 =C2=A0 =C2=A0 A problem would arise if this IPLD object grows over t= he 2M<br> =C2=A0 =C2=A0 =C2=A0 block-size limit because then we would have to shard i= t (something<br> =C2=A0 =C2=A0 =C2=A0 that UnixFS would do automatically for us).<br> <br> =C2=A0 - Flat storage vs. tree storage: Right now we are storing the files<= br> =C2=A0 =C2=A0 separately, but this has some shortcomings, namely we need mu= ltiple<br> =C2=A0 =C2=A0 "get" requests instead of just one, and that IPFS d= oes<br> =C2=A0 =C2=A0 not "know" that those files are related.=C2=A0 (We = lose the web view of<br> =C2=A0 =C2=A0 the tree, etc.)=C2=A0 Storing them as tree could be better.<b= r> =C2=A0 =C2=A0 I don't understand if that would work with the "IPLD= manifest"<br> =C2=A0 =C2=A0 suggested above.=C2=A0 H=C3=A9ctor?<br> <br> =C2=A0 - Pinning: Pinning all files separately incurs an overhead.=C2=A0 It= 's<br> =C2=A0 =C2=A0 enough to just pin the IPLD object since it propagates recurs= ively.<br> =C2=A0 =C2=A0 When adding a tree, then it's no problem since pinning is= only done once.<br> <br> =C2=A0 - IPFS endpoint calls: instead of adding each file individually, it&= #39;s<br> =C2=A0 =C2=A0 possible to add them all in one go.=C2=A0 Can we add all file= s at once<br> =C2=A0 =C2=A0 while using a flat storage? (I.e. not adding them all under a= common<br> =C2=A0 =C2=A0 root folder.)<br> <br> To sum up, here is what remains to be done on the current patch:<br> <br> - Add all files in one go without pinning them.<br> - Store as the file tree?=C2=A0 Can we still us the IPLD object to referenc= e<br> =C2=A0 the files in the tree?=C2=A0 Else use the "raw-leaves" opt= ion to avoid<br> =C2=A0 wrapping small files in UnixFS blocks.<br> - Remove the Scheme manifest if IPLD can do.<br> - Generate the IPLD object and pin it.<br> <br> Any corrections?<br> Thoughts?<br> <br> Cheers!<br> <br> -- <br> Pierre Neidhardt<br> <a href=3D"https://ambrevar.xyz/" rel=3D"noreferrer" target=3D"_blank">http= s://ambrevar.xyz/</a><br> <br> -- <br> You received this message because you are subscribed to the Google Groups &= quot;Go IPFS Working Group" group.<br> To unsubscribe from this group and stop receiving emails from it, send an e= mail to <a href=3D"mailto:go-ipfs-wg%2Bunsubscribe@HIDDEN" target=3D"_blan= k">go-ipfs-wg+unsubscribe@HIDDEN</a>.<br> To view this discussion on the web visit <a href=3D"https://groups.google.c= om/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40ambrevar.xyz" rel=3D"noref= errer" target=3D"_blank">https://groups.google.com/a/ipfs.io/d/msgid/go-ipf= s-wg/87zhlxe8t9.fsf%40ambrevar.xyz</a>.<br> </blockquote></div></div> </blockquote></div> --000000000000dfcafe058db4c59e--
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 10:21:45 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 06:21:45 2019 Received: from localhost ([127.0.0.1]:46475 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hmy77-0007wX-73 for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:21:45 -0400 Received: from mail1.protonmail.ch ([185.70.40.18]:12764) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1hmy75-0007wK-5z for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:21:43 -0400 Date: Mon, 15 Jul 2019 10:21:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1563186095; bh=TIpP7wYPxWbYeJf5L4gugk8l4GYnpT0V2wx9rOANvpI=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=VEvbU3Qm58WF1lDTfG0xttvMUI9OablWV5XHdKmjJHdM60QI3QvhddQnFsRpJ4NjS p8YrCBQ1lQyzk2+ZLpTcTOYTjhtTeiJsrpz3rbfEaHtEqhDTnDj4fNG19o8IEQ00Tk FU/PCbgrxWj05F/ACgUsy5YEpep0LH9Oh+Pjr6Ig= To: Pierre Neidhardt <mail@HIDDEN> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <TbeM3tgSHfcbwcjeZ6QdzDTvWhrkzeTEnWCbfJxJfx4sPZRGupGThXLKTPafQnN2u-MrQ_zcST3TawHVICLzaQAq9gG11BXkNNqgzBkXg6U=@hector.link> In-Reply-To: <87zhlf7glw.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN> <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link> <87ef2rr6om.fsf@HIDDEN> <87zhlf7glw.fsf@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 Cc: Antoine Eiche <lewo@HIDDEN>, =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Monday, July 15, 2019 12:10 PM, Pierre Neidhardt <mail@HIDDEN> wro= te: > H=C3=A9ctor mentioned a possible issue with the IPLD manifest growing too= big > (in case of too many files in a package), that is, above 2MB. > Then we would need to implement some form of sharding. > > H=C3=A9ctor, do you confirm? Any idea on how to tackle this elegantly? > Doing the DAG node the way I proposed it (referencing a single root) should= be ok... Unless you put too many executable files in that list, it should larg= ely stay within the 2MB limit. -- Hector
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 10:11:32 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 06:11:32 2019 Received: from localhost ([127.0.0.1]:46454 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hmxxD-0005VR-Nl for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:11:32 -0400 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:58277) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <mail@HIDDEN>) id 1hmxxA-0005VD-E0 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 06:11:30 -0400 X-Originating-IP: 92.169.116.127 Received: from bababa (lfbn-1-4117-127.w92-169.abo.wanadoo.fr [92.169.116.127]) (Authenticated sender: pierre@HIDDEN) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 29A9E1BF21A; Mon, 15 Jul 2019 10:10:51 +0000 (UTC) From: Pierre Neidhardt <mail@HIDDEN> To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS In-Reply-To: <87ef2rr6om.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN> <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link> <87ef2rr6om.fsf@HIDDEN> Date: Mon, 15 Jul 2019 12:10:51 +0200 Message-ID: <87zhlf7glw.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-Debbugs-Envelope-To: 33899 Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable H=C3=A9ctor mentioned a possible issue with the IPLD manifest growing too b= ig (in case of too many files in a package), that is, above 2MB. Then we would need to implement some form of sharding. H=C3=A9ctor, do you confirm? Any idea on how to tackle this elegantly? =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0sUSsACgkQm9z0l6S7 zH/mJQf+NTkbrr1KF6tXPlj0AoVhvX8pNkzoqt+3elFFZyXYmmAsegpCmM0u1p9T wXijMS+RsKYQkDlq+45gyyXjUFnZFYBlRjY2CVJoYbfpK5BDGTgdxfdch7vTL9A0 4hL7drMpQkjrSpE9mVAwTF7ngGYNzZNyWOE+z3gp/u+bFb69a1SPh6WzCavtdCTs jOnLCJJTFzKl9EaA3VvYFtZ8iS1eunC1qIk5dLjIkbcc0oWK4QQqyC3EZNtDOcyA 3eMwYkVzC8+56XCXT+x8qVgJWo4n73TErviCQEjG6P3ni+JqEPCqkK3tEzncrLw6 I3lUbvMHXWUGWxmMIf38/OKuD00Whg== =bQTd -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 15 Jul 2019 09:25:07 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 15 05:25:07 2019 Received: from localhost ([127.0.0.1]:46393 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hmxEJ-0006GU-6I for submit <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:25:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38340) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1hmxEI-0006Fo-26 for 33899 <at> debbugs.gnu.org; Mon, 15 Jul 2019 05:25:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:43843) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1hmxEB-0008OF-2c; Mon, 15 Jul 2019 05:24:59 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=38884 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1hmxEA-0000G8-JN; Mon, 15 Jul 2019 05:24:58 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN> <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 27 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 15 Jul 2019 11:24:57 +0200 In-Reply-To: <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link> (Hector Sanjuan's message of "Sun, 14 Jul 2019 22:31:43 +0000") Message-ID: <87ef2rr6om.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 33899 Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hello H=C3=A9ctor! :-) Hector Sanjuan <code@HIDDEN> skribis: > On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wr= ote: [...] >> > - Pinning: Pinning all files separately incurs an overhead. It's >> > enough to just pin the IPLD object since it propagates recursively. >> > When adding a tree, then it's no problem since pinning is only don= e once. >> > >> >> Where=E2=80=99s the overhead exactly? > > There are reasons why we are proposing to create a single DAG with an > IPLD object at the root. Pinning has a big overhead because it > involves locking, reading, parsing, and writing an internal pin-DAG. This > is specially relevant when the pinset is very large. > > Doing multiple GET requests also has overhead, like being unable to use > a single bitswap session (which, when downloading something new means a > big overhead since every request will have to find providers). > > And it's not just the web view, it's the ability to walk/traverse all > the object related to a given root natively, which allows also to compare > multiple trees and to be more efficient for some things ("pin update" > for example). Your original idea is to create a manifest with > references to different parts. I'm just asking you to > create that manifest in a format where those references are understood > not only by you, the file creator, but by IPFS and any tool that can > read IPLD, by making this a IPLD object (which is just a json). OK, I see. Put this way, it seems like creating a DAG with an IPLD object as its root is pretty compelling. Thanks for clarifying! Ludo=E2=80=99.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 14 Jul 2019 22:32:03 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 14 18:32:03 2019 Received: from localhost ([127.0.0.1]:46174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hmn2I-0007qp-GE for submit <at> debbugs.gnu.org; Sun, 14 Jul 2019 18:32:03 -0400 Received: from mail1.protonmail.ch ([185.70.40.18]:47974) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1hmn2E-0007jh-W4 for 33899 <at> debbugs.gnu.org; Sun, 14 Jul 2019 18:32:00 -0400 Date: Sun, 14 Jul 2019 22:31:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1563143510; bh=A1ca9BbPvEBnUQhGhk34WkOtYemz98SjsNpEYJUXbZk=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=m29RboLIL4fAf+SslxgfY4xuNdk/Zig/YTJ4VGq3Z5KIZOEBBYS/iSYo5/w5WO1E3 Z9ZUk57jK2C/+D4fBCR+/inlRzz+Dw13Kv3ZRRxQg/giSQrWAhTynXnP3gTZg+bEXP tTJ8iZ+YMz1DiKKomfydSB35KRVLTEjaKvohRxbA= To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <HAv_FWZRa_dCADY6U07oX-E2vTrtcxk8ltkpfvTpYCsIh4O5PuItHGIh6w-g5GGmoqVEfkjDUOplpUErdGEPAiFYdrjjDhwd9RJ4OyjjGgY=@hector.link> In-Reply-To: <87ftnbf1rt.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> <87ftnbf1rt.fsf@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 Cc: Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Hey! Thanks for reviving this discussion! =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wrot= e: > Hello! > > Pierre Neidhardt mail@HIDDEN skribis: > > > A little update/recap after many months! :) > > Thank you, and apologies for the delay! > > > - Extra metadata: IPFS stores files on UnixFSv1 which does not > > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > > list of executable bits. But maybe we don't have to roll out ou= r own. > > > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did = not > > recommend using it (not sure why though). > > > > - We could use UnixFSv2 but it's not released yet and it's unclea= r when > > it's going to be released. So we can't really count on it right= now. > > > > UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2 > would be appropriate, though it stores timestamps that we don=E2=80=99t n= eed > (not necessarily a problem). > > > - Flat storage vs. tree storage: Right now we are storing the files > > separately, but this has some shortcomings, namely we need multiple > > "get" requests instead of just one, and that IPFS does > > not "know" that those files are related. (We lose the web view of > > the tree, etc.) Storing them as tree could be better. > > I don't understand if that would work with the "IPLD manifest" > > suggested above. H=C3=A9ctor? > > > > I don=E2=80=99t consider the web view a strong argument :-) since we coul= d write > tools to deal with whatever format we use. > > Regarding multiple GET requests: we could pipeline them, and it seems > more like an implementation detail to me. The real question is whether > making separate GET requests prevents some optimization in IPFS. > > > - Pinning: Pinning all files separately incurs an overhead. It's > > enough to just pin the IPLD object since it propagates recursively. > > When adding a tree, then it's no problem since pinning is only done= once. > > > > Where=E2=80=99s the overhead exactly? There are reasons why we are proposing to create a single DAG with an IPLD object at the root. Pinning has a big overhead because it involves locking, reading, parsing, and writing an internal pin-DAG. This is specially relevant when the pinset is very large. Doing multiple GET requests also has overhead, like being unable to use a single bitswap session (which, when downloading something new means a big overhead since every request will have to find providers). And it's not just the web view, it's the ability to walk/traverse all the object related to a given root natively, which allows also to compare multiple trees and to be more efficient for some things ("pin update" for example). Your original idea is to create a manifest with references to different parts. I'm just asking you to create that manifest in a format where those references are understood not only by you, the file creator, but by IPFS and any tool that can read IPLD, by making this a IPLD object (which is just a json). The process of adding "something" to ipfs is as follows. ---- 1. Add to IPFS: multipart upload equivalent to "ipfs add -r": ~/ipfstest $ ipfs add -r -q . QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP 2. Add manifest as IPLD object. dag/put a json file like: cat <<EOF | ipfs dag put { "executables": ["ipfstest/1.txt"], "root": { "/": "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP" } } EOF --- That's it "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP" is the root of your package files. "bafyreievcw5qoowhepwskcxybochrui65bbtsliuy7r6kyail4w5lyqnjm" is the root of your manifest file with the list of executables and a pointer to the other root. Hector
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 12 Jul 2019 21:09:26 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jul 12 17:09:26 2019 Received: from localhost ([127.0.0.1]:40806 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hm2nF-0005Na-WA for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 17:09:26 -0400 Received: from mail-io1-f53.google.com ([209.85.166.53]:46436) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <molly@HIDDEN>) id 1hm1lB-0003iJ-6K for 33899 <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:03:13 -0400 Received: by mail-io1-f53.google.com with SMTP id i10so22938897iol.13 for <33899 <at> debbugs.gnu.org>; Fri, 12 Jul 2019 13:03:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protocol.ai; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8czCWGdi1/j1frklq4/BID0J5QGNs9v0xuL0GjsLP/A=; b=PdiQOh+LudNgCjV/pNMfeUvqIHMu81fbF/srzxpcJw7JHtbXZH7vImVu0G5HS3FUUH TDn+nQCtLvEDAE94CCSDqKH3m1/pKWVpihKr14gPHeMmeT0Ig5IC3MJXkZ3Q+W1FCzAA uCmUDhsTGWoo7b0LtQFMaiW1bdAu2LNJEQy+QdHmlsXZwkcU5X3G3KthYiuBhgmbZH6l v7X2MsE28uuNpICshbE8ak6E7o9Lbl/17irrYt+xwr3XkkByyI6/K/Ljj4zct0BTfHaX AVi13EowUyyYemTuDxDk9bX8HUMw5m/tL62cNO6sfk3iLGVknVk4mzYWKILuScK2pf8t 9aAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8czCWGdi1/j1frklq4/BID0J5QGNs9v0xuL0GjsLP/A=; b=FJlTIAJORqrC4U78JCLCIzqd+DrZ/54Ac6sHVnKfRNoSFPiQkRPNPygcXKAXbFeIPH nYxjIHvvO0H6ehhp6EEJ1Ncyl3s4NlDz/vtvFwo5pAC6kMMsqaLh/A3DGLm7uc30HEiz D5Y13roExyWduY7M9/7UufPtYM98yf7VRiY6f10n4k0mu3aalXuVI+Kgd5xacpCd4l90 WZ2Gp4vG5PK9LuA0VO/jqLggf9SmwKF81AddLYFTnXC2P4rwdGqkxyaQdVyjoCRS/d8l kWbXILiumHQyyXcqEVJXEjqoBtHG/SaZoaRBEyPOqEyMO50P7l09joAbD94S8VWTSFyZ ymqw== X-Gm-Message-State: APjAAAUBQAbpc5Cf55pRN4z3EsQEfKP6dgPC6/nvdcNvoDt709/0oKUW OZ+JJMUl0YMzSf6HVNXhQjGXLOBoyHWZiesGjCDMkg== X-Google-Smtp-Source: APXvYqwu2ITNZ/vYX12QdVFcDTU8nGxi3gLK4djlQXHTGzDBHiNSlLVmSOAmXzyElsB6CAZ+QSZCsA20J+Ik5jm9RjU= X-Received: by 2002:a6b:dc17:: with SMTP id s23mr9250638ioc.56.1562961786040; Fri, 12 Jul 2019 13:03:06 -0700 (PDT) MIME-Version: 1.0 References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> In-Reply-To: <87zhlxe8t9.fsf@HIDDEN> From: Molly Mackinlay <molly@HIDDEN> Date: Fri, 12 Jul 2019 13:02:55 -0700 Message-ID: <CA+UaANWE=VKwVaF_uxLTf7jc9wNVRhU105GBqER6fFTwa3zr8Q@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS To: Pierre Neidhardt <mail@HIDDEN> Content-Type: multipart/alternative; boundary="0000000000000e29ad058d8166f7" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 33899 X-Mailman-Approved-At: Fri, 12 Jul 2019 17:09:24 -0400 Cc: Alex Potsides <alex.potsides@HIDDEN>, Hector Sanjuan <code@HIDDEN>, Andrew Nesbitt <andrew@HIDDEN>, Eric Myhre <eric.myhre@HIDDEN>, =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org>, Antoine Eiche <lewo@HIDDEN>, Jessica Schilling <jessica@HIDDEN>, "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) --0000000000000e29ad058d8166f7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks for the update Pierre! Also adding Alex, Jessica, Eric and Andrew from the package managers discussions at IPFS Camp as FYI. Generating the ipld manifest with the metadata and the tree of files should also be fine AFAIK - I=E2=80=99m sure Hector and Eric can expand more on ho= w to compose them, but data storage format shouldn=E2=80=99t make a big differen= ce for the ipld manifest. On Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt <mail@HIDDEN> wrote: > Hi! > > (Re-sending to debbugs, sorry for the double email :p) > > A little update/recap after many months! :) > > I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Lu= do's > patch so now I have a little better understanding of the current state > of affair. > > - We could store the substitutes as tarballs on IPFS, but this has > some possible downsides: > > - We would need to use IPFS' tar chunker to deduplicate the content of > the tarball. But the tar chunker is not well maintained currently, > and it's not clear whether it's reproducible at the moment, so it > would need some more work. > > - Tarballs might induce some performance cost. Nix had attempted > something similar in the past and this may have incurred a significan= t > performance penalty, although this remains to be confirmed. > Lewo? > > - Ludo's patch stores all files on IPFS individually. This way we don't > need to touch the tar chunker, so it's less work :) > This raises some other issues however: > > - Extra metadata: IPFS stores files on UnixFSv1 which does not > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > list of executable bits. But maybe we don't have to roll out our > own. > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not > recommend using it (not sure why though). > > - We could use UnixFSv2 but it's not released yet and it's unclear wh= en > it's going to be released. So we can't really count on it right no= w. > > - IPLD: As H=C3=A9ctor suggested in the previous email, we could leve= rage > IPLD and generate a JSON object that references the files with > their paths together with an "executable?" property. > A problem would arise if this IPLD object grows over the 2M > block-size limit because then we would have to shard it (something > that UnixFS would do automatically for us). > > - Flat storage vs. tree storage: Right now we are storing the files > separately, but this has some shortcomings, namely we need multiple > "get" requests instead of just one, and that IPFS does > not "know" that those files are related. (We lose the web view of > the tree, etc.) Storing them as tree could be better. > I don't understand if that would work with the "IPLD manifest" > suggested above. H=C3=A9ctor? > > - Pinning: Pinning all files separately incurs an overhead. It's > enough to just pin the IPLD object since it propagates recursively. > When adding a tree, then it's no problem since pinning is only done > once. > > - IPFS endpoint calls: instead of adding each file individually, it's > possible to add them all in one go. Can we add all files at once > while using a flat storage? (I.e. not adding them all under a common > root folder.) > > To sum up, here is what remains to be done on the current patch: > > - Add all files in one go without pinning them. > - Store as the file tree? Can we still us the IPLD object to reference > the files in the tree? Else use the "raw-leaves" option to avoid > wrapping small files in UnixFS blocks. > - Remove the Scheme manifest if IPLD can do. > - Generate the IPLD object and pin it. > > Any corrections? > Thoughts? > > Cheers! > > -- > Pierre Neidhardt > https://ambrevar.xyz/ > > -- > You received this message because you are subscribed to the Google Groups > "Go IPFS Working Group" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-ipfs-wg+unsubscribe@HIDDEN > To view this discussion on the web visit > https://groups.google.com/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40a= mbrevar.xyz > . > --0000000000000e29ad058d8166f7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div><div dir=3D"auto">Thanks for the update Pierre! Also adding Alex, Jess= ica, Eric and Andrew from the package managers discussions at IPFS Camp as = FYI.</div><br></div><div dir=3D"auto">Generating the ipld manifest with the= metadata and the tree of files should also be fine AFAIK - I=E2=80=99m sur= e Hector and Eric can expand more on how to compose them, but data storage = format shouldn=E2=80=99t make a big difference for the ipld manifest.</div>= <div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">O= n Mon, Jul 1, 2019 at 2:36 PM Pierre Neidhardt <<a href=3D"mailto:mail@a= mbrevar.xyz">mail@HIDDEN</a>> wrote:<br></div><blockquote class=3D= "gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding= -left:1ex">Hi!<br> <br> (Re-sending to debbugs, sorry for the double email :p)<br> <br> A little update/recap after many months! :)<br> <br> I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo= 's<br> patch so now I have a little better understanding of the current state<br> of affair.<br> <br> - We could store the substitutes as tarballs on IPFS, but this has<br> =C2=A0 some possible downsides:<br> <br> =C2=A0 - We would need to use IPFS' tar chunker to deduplicate the cont= ent of<br> =C2=A0 =C2=A0 the tarball.=C2=A0 But the tar chunker is not well maintained= currently,<br> =C2=A0 =C2=A0 and it's not clear whether it's reproducible at the m= oment, so it<br> =C2=A0 =C2=A0 would need some more work.<br> <br> =C2=A0 - Tarballs might induce some performance cost.=C2=A0 Nix had attempt= ed<br> =C2=A0 =C2=A0 something similar in the past and this may have incurred a si= gnificant<br> =C2=A0 =C2=A0 performance penalty, although this remains to be confirmed.<b= r> =C2=A0 =C2=A0 Lewo?<br> <br> - Ludo's patch stores all files on IPFS individually.=C2=A0 This way we= don't<br> =C2=A0 need to touch the tar chunker, so it's less work :)<br> =C2=A0 This raises some other issues however:<br> <br> =C2=A0 - Extra metadata:=C2=A0 IPFS stores files on UnixFSv1 which does not= <br> =C2=A0 =C2=A0 include the executable bit.<br> <br> =C2=A0 =C2=A0 - Right now we store a s-exp manifest with a list of files an= d a<br> =C2=A0 =C2=A0 =C2=A0 list of executable bits.=C2=A0 But maybe we don't = have to roll out our own.<br> <br> =C2=A0 =C2=A0 - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex = did not<br> =C2=A0 =C2=A0 =C2=A0 recommend using it (not sure why though).<br> <br> =C2=A0 =C2=A0 - We could use UnixFSv2 but it's not released yet and it&= #39;s unclear when<br> =C2=A0 =C2=A0 =C2=A0 it's going to be released.=C2=A0 So we can't r= eally count on it right now.<br> <br> =C2=A0 =C2=A0 - IPLD: As H=C3=A9ctor suggested in the previous email, we co= uld leverage<br> =C2=A0 =C2=A0 =C2=A0 IPLD and generate a JSON object that references the fi= les with<br> =C2=A0 =C2=A0 =C2=A0 their paths together with an "executable?" p= roperty.<br> =C2=A0 =C2=A0 =C2=A0 A problem would arise if this IPLD object grows over t= he 2M<br> =C2=A0 =C2=A0 =C2=A0 block-size limit because then we would have to shard i= t (something<br> =C2=A0 =C2=A0 =C2=A0 that UnixFS would do automatically for us).<br> <br> =C2=A0 - Flat storage vs. tree storage: Right now we are storing the files<= br> =C2=A0 =C2=A0 separately, but this has some shortcomings, namely we need mu= ltiple<br> =C2=A0 =C2=A0 "get" requests instead of just one, and that IPFS d= oes<br> =C2=A0 =C2=A0 not "know" that those files are related.=C2=A0 (We = lose the web view of<br> =C2=A0 =C2=A0 the tree, etc.)=C2=A0 Storing them as tree could be better.<b= r> =C2=A0 =C2=A0 I don't understand if that would work with the "IPLD= manifest"<br> =C2=A0 =C2=A0 suggested above.=C2=A0 H=C3=A9ctor?<br> <br> =C2=A0 - Pinning: Pinning all files separately incurs an overhead.=C2=A0 It= 's<br> =C2=A0 =C2=A0 enough to just pin the IPLD object since it propagates recurs= ively.<br> =C2=A0 =C2=A0 When adding a tree, then it's no problem since pinning is= only done once.<br> <br> =C2=A0 - IPFS endpoint calls: instead of adding each file individually, it&= #39;s<br> =C2=A0 =C2=A0 possible to add them all in one go.=C2=A0 Can we add all file= s at once<br> =C2=A0 =C2=A0 while using a flat storage? (I.e. not adding them all under a= common<br> =C2=A0 =C2=A0 root folder.)<br> <br> To sum up, here is what remains to be done on the current patch:<br> <br> - Add all files in one go without pinning them.<br> - Store as the file tree?=C2=A0 Can we still us the IPLD object to referenc= e<br> =C2=A0 the files in the tree?=C2=A0 Else use the "raw-leaves" opt= ion to avoid<br> =C2=A0 wrapping small files in UnixFS blocks.<br> - Remove the Scheme manifest if IPLD can do.<br> - Generate the IPLD object and pin it.<br> <br> Any corrections?<br> Thoughts?<br> <br> Cheers!<br> <br> -- <br> Pierre Neidhardt<br> <a href=3D"https://ambrevar.xyz/" rel=3D"noreferrer" target=3D"_blank">http= s://ambrevar.xyz/</a><br> <br> -- <br> You received this message because you are subscribed to the Google Groups &= quot;Go IPFS Working Group" group.<br> To unsubscribe from this group and stop receiving emails from it, send an e= mail to <a href=3D"mailto:go-ipfs-wg%2Bunsubscribe@HIDDEN" target=3D"_blan= k">go-ipfs-wg+unsubscribe@HIDDEN</a>.<br> To view this discussion on the web visit <a href=3D"https://groups.google.c= om/a/ipfs.io/d/msgid/go-ipfs-wg/87zhlxe8t9.fsf%40ambrevar.xyz" rel=3D"noref= errer" target=3D"_blank">https://groups.google.com/a/ipfs.io/d/msgid/go-ipf= s-wg/87zhlxe8t9.fsf%40ambrevar.xyz</a>.<br> </blockquote></div></div> --0000000000000e29ad058d8166f7--
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 12 Jul 2019 20:15:24 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jul 12 16:15:24 2019 Received: from localhost ([127.0.0.1]:40775 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hm1wy-00041U-4Y for submit <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:15:24 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45851) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1hm1ww-00041F-BW for 33899 <at> debbugs.gnu.org; Fri, 12 Jul 2019 16:15:22 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:47533) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1hm1wp-0005Rr-8V; Fri, 12 Jul 2019 16:15:15 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52938 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1hm1we-0004RU-PL; Fri, 12 Jul 2019 16:15:06 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Pierre Neidhardt <mail@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 12 Jul 2019 22:15:02 +0200 In-Reply-To: <87zhlxe8t9.fsf@HIDDEN> (Pierre Neidhardt's message of "Mon, 01 Jul 2019 23:36:34 +0200") Message-ID: <87ftnbf1rt.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 33899 Cc: Hector Sanjuan <code@HIDDEN>, Antoine Eiche <lewo@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hello! Pierre Neidhardt <mail@HIDDEN> skribis: > A little update/recap after many months! :) Thank you, and apologies for the delay! > - Extra metadata: IPFS stores files on UnixFSv1 which does not > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > list of executable bits. But maybe we don't have to roll out our o= wn. > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not > recommend using it (not sure why though). > > - We could use UnixFSv2 but it's not released yet and it's unclear wh= en > it's going to be released. So we can't really count on it right no= w. UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2 would be appropriate, though it stores timestamps that we don=E2=80=99t need (not necessarily a problem). > - Flat storage vs. tree storage: Right now we are storing the files > separately, but this has some shortcomings, namely we need multiple > "get" requests instead of just one, and that IPFS does > not "know" that those files are related. (We lose the web view of > the tree, etc.) Storing them as tree could be better. > I don't understand if that would work with the "IPLD manifest" > suggested above. H=C3=A9ctor? I don=E2=80=99t consider the web view a strong argument :-) since we could = write tools to deal with whatever format we use. Regarding multiple GET requests: we could pipeline them, and it seems more like an implementation detail to me. The real question is whether making separate GET requests prevents some optimization in IPFS. > - Pinning: Pinning all files separately incurs an overhead. It's > enough to just pin the IPLD object since it propagates recursively. > When adding a tree, then it's no problem since pinning is only done o= nce. Where=E2=80=99s the overhead exactly? > - IPFS endpoint calls: instead of adding each file individually, it's > possible to add them all in one go. Can we add all files at once > while using a flat storage? (I.e. not adding them all under a common > root folder.) Again, is the concern that we=E2=80=99re making one GET and thus one round = trip per file, or is there some additional cost under the hood? Thanks for the summary and explanations! Ludo=E2=80=99.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 6 Jul 2019 08:44:16 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jul 06 04:44:16 2019 Received: from localhost ([127.0.0.1]:54191 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hjgIq-0003zX-Lg for submit <at> debbugs.gnu.org; Sat, 06 Jul 2019 04:44:16 -0400 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:38019) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <mail@HIDDEN>) id 1hjgIn-0003zL-If for 33899 <at> debbugs.gnu.org; Sat, 06 Jul 2019 04:44:14 -0400 X-Originating-IP: 92.169.116.127 Received: from bababa (lfbn-1-4117-127.w92-169.abo.wanadoo.fr [92.169.116.127]) (Authenticated sender: pierre@HIDDEN) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 266AB1BF206; Sat, 6 Jul 2019 08:44:05 +0000 (UTC) From: Pierre Neidhardt <mail@HIDDEN> To: Hector Sanjuan <code@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Antoine Eiche <lewo@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS In-Reply-To: <87zhlxe8t9.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> <87zhlxe8t9.fsf@HIDDEN> Date: Sat, 06 Jul 2019 10:44:03 +0200 Message-ID: <87imsfd030.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Link to the Nix integration discussion: https://github.com/NixOS/nix/issues/859. =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0gX1MACgkQm9z0l6S7 zH/1mAgAlaFU5i8qcYJsTuIa0XvGq8h0XCSzJmg9iImrYj+zqfxrZrHkl0PwdZGr Z7vGKlxmIyCNRJuVTCi1wdyWcqzKVScJ1YI5FazCTR788pV8Qiu3ytmtcI5Vc6+a 2aYX3dcbYBhSQIYRH0Mr5CB9YfCPy5XhnTYuU8Pz2rkxm0h+fHKjVifzfmfnRHZz NZ9M3+8ZN13mzI0j2nvltsAsVE9zMyD1BkREZ8itvsx9OuYLa9SBQ7sZawyNJKiI LsBBr3AJHl3xCCVYNetY8tMDS98TPEtRiOKp9xro8crjhr/WzEWfCCyYmLwKK7a5 es7eWdkaB7ZY90pWy8gxQE2voTkkAw== =pYr5 -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 1 Jul 2019 21:36:47 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jul 01 17:36:47 2019 Received: from localhost ([127.0.0.1]:46194 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hi3yg-0000w2-Us for submit <at> debbugs.gnu.org; Mon, 01 Jul 2019 17:36:47 -0400 Received: from relay10.mail.gandi.net ([217.70.178.230]:39679) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <mail@HIDDEN>) id 1hi3ye-0000vs-10 for 33899 <at> debbugs.gnu.org; Mon, 01 Jul 2019 17:36:45 -0400 Received: from mimimi (unknown [212.81.186.37]) (Authenticated sender: pierre@HIDDEN) by relay10.mail.gandi.net (Postfix) with ESMTPSA id AB329240007; Mon, 1 Jul 2019 21:36:35 +0000 (UTC) From: Pierre Neidhardt <mail@HIDDEN> To: Hector Sanjuan <code@HIDDEN>, Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Antoine Eiche <lewo@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS In-Reply-To: <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> Date: Mon, 01 Jul 2019 23:36:34 +0200 Message-ID: <87zhlxe8t9.fsf@HIDDEN> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! (Re-sending to debbugs, sorry for the double email :p) A little update/recap after many months! :) I talked with H=C3=A9ctor and some other people from IPFS + I reviewed Ludo= 's patch so now I have a little better understanding of the current state of affair. =2D We could store the substitutes as tarballs on IPFS, but this has some possible downsides: - We would need to use IPFS' tar chunker to deduplicate the content of the tarball. But the tar chunker is not well maintained currently, and it's not clear whether it's reproducible at the moment, so it would need some more work. - Tarballs might induce some performance cost. Nix had attempted something similar in the past and this may have incurred a significant performance penalty, although this remains to be confirmed. Lewo? =2D Ludo's patch stores all files on IPFS individually. This way we don't need to touch the tar chunker, so it's less work :) This raises some other issues however: - Extra metadata: IPFS stores files on UnixFSv1 which does not include the executable bit. - Right now we store a s-exp manifest with a list of files and a list of executable bits. But maybe we don't have to roll out our own. - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did not recommend using it (not sure why though). - We could use UnixFSv2 but it's not released yet and it's unclear when it's going to be released. So we can't really count on it right now. - IPLD: As H=C3=A9ctor suggested in the previous email, we could levera= ge IPLD and generate a JSON object that references the files with their paths together with an "executable?" property. A problem would arise if this IPLD object grows over the 2M block-size limit because then we would have to shard it (something that UnixFS would do automatically for us). - Flat storage vs. tree storage: Right now we are storing the files separately, but this has some shortcomings, namely we need multiple "get" requests instead of just one, and that IPFS does not "know" that those files are related. (We lose the web view of the tree, etc.) Storing them as tree could be better. I don't understand if that would work with the "IPLD manifest" suggested above. H=C3=A9ctor? - Pinning: Pinning all files separately incurs an overhead. It's enough to just pin the IPLD object since it propagates recursively. When adding a tree, then it's no problem since pinning is only done onc= e. - IPFS endpoint calls: instead of adding each file individually, it's possible to add them all in one go. Can we add all files at once while using a flat storage? (I.e. not adding them all under a common root folder.) To sum up, here is what remains to be done on the current patch: =2D Add all files in one go without pinning them. =2D Store as the file tree? Can we still us the IPLD object to reference the files in the tree? Else use the "raw-leaves" option to avoid wrapping small files in UnixFS blocks. =2D Remove the Scheme manifest if IPLD can do. =2D Generate the IPLD object and pin it. Any corrections? Thoughts? Cheers! =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0afOIACgkQm9z0l6S7 zH+oEgf/U76CtGhULJqKEVxBD8+LYT2ggfqVlaC2NnMckdg+JUHCC4dpqMdBLks3 Wye4tFYLfXinaN8SYmLc2ubA0ZiD24xHC5qAO10PPsyiea8VUChJf3jyTkV1TNn/ Knp2Aj8ALk84nEPoTIkG9vBkKgXQj2szzD1lEOTCdyOcLaDFvBzV31uyDMnqbW18 bsvPDmWfoUsDVk8VB/sgrPBK9qA7UF2Hv5qvuAcf1Y5xJ1PTRbWhf2T3cNZe/Wt3 +/hYEwMdEx9UJvW9P71rhA0iATlOjqhpju0HG3MYN95VGYcKL9R9xDJJH1jqUde0 tZyA3prRss1O7/0JMP9+s+4XIuU7uw== =WJum -----END PGP SIGNATURE----- --=-=-=--
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 13 May 2019 18:52:20 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon May 13 14:52:20 2019 Received: from localhost ([127.0.0.1]:46178 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1hQG3f-0008Bw-Uz for submit <at> debbugs.gnu.org; Mon, 13 May 2019 14:52:20 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:59557) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <a@HIDDEN>) id 1hQG3d-0008Bj-B7 for 33899 <at> debbugs.gnu.org; Mon, 13 May 2019 14:52:18 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 0E46524622 for <33899 <at> debbugs.gnu.org>; Mon, 13 May 2019 14:52:12 -0400 (EDT) Received: from imap1 ([10.202.2.51]) by compute6.internal (MEProxy); Mon, 13 May 2019 14:52:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ajgrf.com; h= mime-version:message-id:date:from:to:subject:content-type; s= fm2; bh=YwpmwH8PqqsqeXz7J0/EzZXIVEHJ4n28KJSWyGB79CQ=; b=nNgG37U/ HyW30pWzPn24l6E3oDjnKCD0ffFEwyHtrAOHn22Z2ooNnCnAyXaqmsNLipqc1l4c P7SR6d6KaZBvLTbKLjK/v+t7VhQUCxnThaknZgXRFu9B2PB9dg5oTLfJ8Zws7Msm /Dv0bcIpaVbPDgCEFKiNiKCLpI2NU49xdUII7GJ9jRe2qSG8xAPQjyi6p8zvZzEA arWHzmPZekE9zyZNVXhJzIeF7vjapuWEg7LkO9Wwlx5hPSxff0xQjne2IrNtNEyW rpTOChWFN/eAx7Jo9w2e4y4LOEf0FqRpVSOv9QERj+uq2TGxI+w7+Rd5mSx6t1Ac gglKMbFuT2hJDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=YwpmwH8PqqsqeXz7J0/EzZXIVEHJ4 n28KJSWyGB79CQ=; b=zPnUXWIVcNL92dNDtKF8OWp1c8WwujrH2kMsdxuGVrEds a2AGlb/eKfMKGReyKY7Hxhyvil4BbD9drPjFjsekDZM26XmGhdnMbQM1pbWoxEPK S+V5XWO/VGLk9P90LLcwzZeQsQtgRxiFnsk4OGlQvpzI5Om/G9//mHVEoSEP/lAt pBd+fYQhhaawAWg5P5f+rvf0icJMUjrvrxgv+hb0jHNIEFf9ZjUsm599UeJkUbPR 8MheuTEa3SQsjxui92Xu/+PFsyyAK9g4DAPPL7DLOYuZ8tHA8NG/JpYMFqqjVIpy NYGnAE8vje8Pk7XKUyOfUF33S43dPXle6KobcPWMQ== X-ME-Sender: <xms:27zZXMBsMIdzxhKsG1wyOmdCU60vwuGhK5IfFdsp_Mpgl2_818290A> X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrleeggddufedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsehttdertd erredtnecuhfhrohhmpedftehlvgigucfirhhifhhfihhnfdcuoegrsegrjhhgrhhfrdgt ohhmqeenucffohhmrghinhepghhithhhuhgsrdgtohhmnecurfgrrhgrmhepmhgrihhlfh hrohhmpegrsegrjhhgrhhfrdgtohhmnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: <xmx:27zZXMbTJ0uhw2qZqWHoraR2mDDvpMPvZJzKqipkG-uQf6bbyqHSlA> <xmx:27zZXIqDlmIpMGXZ8dBbrCzerKlScC9JjMn7xicbQjAtjnehDbfnog> <xmx:27zZXEBbjr5dyLR_bu3UTFoVSNBlOWV4cEb-JehsNGpiT2tktBJYRA> <xmx:27zZXH6HLy0BecJilHdKeYxHGNyESfaxn2343S-yFBBvK5reSht0lg> Received: by mailuser.nyi.internal (Postfix, from userid 501) id 7B8C4D4948; Mon, 13 May 2019 14:52:11 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.6-453-g9eaa09e-fmstable-20190430v2bis Mime-Version: 1.0 Message-Id: <5e3fb8d9-1a83-4031-ab02-c4e10e2be1ea@HIDDEN> Date: Mon, 13 May 2019 14:51:42 -0400 From: "Alex Griffin" <a@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Content-Type: text/plain X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) Do I understand correctly that the only reason you don't just store nar files is for deduplication? Reading [this page][1] suggests to me that you might be overthinking it. IPFS already uses a content-driven chunking algorithm that might provide good enough deduplication on its own. It also looks like you can use your own chunker, so a future improvement could be implementing a custom chunker that makes sure to split nar files at the file boundaries within them. [1]: https://github.com/ipfs/archives -- Alex Griffin
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 11:26:31 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 18 06:26:31 2019 Received: from localhost ([127.0.0.1]:35757 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gkSIA-00080t-RZ for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 06:26:31 -0500 Received: from mail-40132.protonmail.ch ([185.70.40.132]:58973) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1gkSI7-00080e-SG for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 06:26:29 -0500 Date: Fri, 18 Jan 2019 11:26:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1547810781; bh=4pQGBXxb3EphmMnCRcuTgN8S67MGMR15cim0fy1tBRA=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=JUHq75Hu99aZzuqyOEOZscwrocjODfW5i1KLzllAUJSK7+zeJrYH3dOcttQwc5yoH O9cITpViXuSiO+t1z1oIOGRit1EdT/mcqVNnOktCHkyrTxYGWoMqQIGsrAHr+8IPVX /eshhWg/uvqWT19jMa1sk+09ClZnODSZ3u/x9ps0= To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <pa258vxfuaaiuhOEdEpZPTqfDHe4hoICI7EPyJ284kk6qVWc_VakU3cwxbx46BVXobg0LMWyrjE_uX6nJ6XoJj78-mfjtsejqHlx3bmkNFw=@hector.link> In-Reply-To: <8736pqthqm.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> <8736pqthqm.fsf@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Friday, January 18, 2019 10:52 AM, Ludovic Court=C3=A8s <ludo@HIDDEN> w= rote: > Hello, > > Hector Sanjuan code@HIDDEN skribis: > > > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Origina= l Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80= =90 > > On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s ludo@HIDDEN = wrote: > > [...] > > > Isn=E2=80=99t there a way, then, to achieve the same behavior with the cu= stom > format? The /api/v0/add entry point has a =E2=80=98pin=E2=80=99 argument;= I suppose we > could leave it to false except when we add the top-level =E2=80=9Cdirecto= ry=E2=80=9D > node? Wouldn=E2=80=99t that give us behavior similar to that of Unixfs? > Yes. What you could do is to add every file flatly/separately (with pin=3Df= alse) and at the end add an IPLD object with references to all the files that you added and including the exec bit information (and size?). This is just a JSON file: { "name": "package name", "contents": [ { "path": "/file/path", # so you know where to extract it later "exec": true, "ipfs": { "/": "Qmhash..." } }, ... } This needs to be added to IPFS with the /api/v0/dag/put endpoint (this converts it to CBOR - IPLD-Cbor is the actual block format used here). When this is pinned (?pin=3Dtrue), this will pin all the things referenced from it recursively in the way we want. So this will be quite similar to unixfs. But note that if this blob ever grows over the 2M block-size limit because you have a package with many files, you will need to start solving problems that unixfs solves automatically now (directory sharding). Because IPLD-cbor is supported, ipfs, the gateway etc will know how to display these manifests, the info in it and their links. > > When the user puts the single root hash in ipfs.io/ipfs/<hash>, it > > will display correctly the underlying files and the people will be > > able to navigate the actual tree with both web and cli. > > Right, though that=E2=80=99s less important in my view. > > > Note that every file added to IPFS is getting wrapped as a Unixfs > > block anyways. You are just saving some "directory" nodes by adding > > them separately. > > Hmm weird. When I do /api/v0/add, I=E2=80=99m really just passing a byte > vector; there=E2=80=99s no notion of a =E2=80=9Cfile=E2=80=9D here, AFAIC= S. Or am I missing > something? They are wrapped in Unixfs blocks anyway by default. From the moment the file is >256K it will get chunked into several pieces and a Unixfs block (or multiple, if a really big file) is necessary to reference them. In this case the root hash will be a Unixfs node with links to the parts. There is a "raw-leaves" option which does not wrap the individual blocks with unixfs, so if the file is small to not be chunked, you can avoid the default unixfs-wrapping this way. > > > > > It will probably need some trial an error to get the multi-part rig= ht > > > > to upload all in a single request. The Go code HTTP Clients doing > > > > this can be found at: > > > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.g= o#L96 > > > > As you see, a directory part in the multipart will have the content= -type Header > > > > set to "application/x-directory". The best way to see how "abspath"= etc is set > > > > is probably to sniff an `ipfs add -r <testfolder>` operation (local= host:5001). > > > > Once UnixFSv2 lands, you will be in a position to just drop the sex= p file > > > > altogether. > > > > > > Yes, that makes sense. In the meantime, I guess we have to keep using > > > our own format. > > > What are the performance implications of adding and retrieving files = one > > > by one like I did? I understand we=E2=80=99re doing N HTTP requests t= o the > > > local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single = request, but this > > > alone can=E2=80=99t be much of a problem since communication is happe= ning > > > locally. Does pinning each file separately somehow incur additional > > > overhead? > > > > Yes, pinning separately is slow and incurs in overhead. Pins are stored > > in a merkle tree themselves so it involves reading, patching and saving= . This > > gets quite slow when you have very large pinsets because your pins bloc= k size > > grow. Your pinset will grow very large if you do this. Additionally the > > pinning operation itself requires global lock making it more slow. > > OK, I see. I should add that even if you want to /add all files separately (and then put the IPLD manifest I described above), you can still add them all in the= same request (it becomes easier as you just need to put more parts in the multip= art and don't have to worry about names/folders/paths). The /add endpoint will forcefully close the HTTP connection for every /add (long story) and small delays might add up to a big one. Specially rel= evant if using IPFS Cluster, where /add might send the blocks somewhere else and = does needs to do some other things. > > > But, even if it was fast, you will not have a way to easily unpin > > anything that becomes obsolete or have an overview of to where things > > belong. It is also unlikely that a single IPFS daemon will be able to > > store everything you build, so you might find yourself using IPFS Clust= er > > soon to distribute the storage across multiple nodes and then you will > > be effectively adding remotely. > > Currently, =E2=80=98guix publish=E2=80=99 stores things as long as they a= re requested, > and then for the duration specified with =E2=80=98--ttl=E2=80=99. I suppo= se we could > have similar behavior with IPFS: if an item hasn=E2=80=99t been requested= for > the specified duration, then we unpin it. > > Does that make sense? Yes, in fact I wanted IPFS Cluster to support a TTL so that things are automatically unpinned when it expires too. > > Thanks for your help! > > Ludo=E2=80=99. Thanks! Hector
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 09:52:56 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 18 04:52:56 2019 Received: from localhost ([127.0.0.1]:35688 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gkQpb-0001c1-OB for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:52:56 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:37936) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gkQpZ-0001br-NQ for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:52:54 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id E070B2341; Fri, 18 Jan 2019 10:52:51 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZQPwYQv1qiwm; Fri, 18 Jan 2019 10:52:51 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id A1334C39; Fri, 18 Jan 2019 10:52:50 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 29 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 18 Jan 2019 10:52:49 +0100 In-Reply-To: <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> (Hector Sanjuan's message of "Fri, 18 Jan 2019 09:08:02 +0000") Message-ID: <8736pqthqm.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.0 (/) Hello, Hector Sanjuan <code@HIDDEN> skribis: > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original = Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 > On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> = wrote: [...] >> Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, a= s I see it, is that it >> stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to sto= re the mtimes or >> permissions; we could ignore them upon reconstruction though), and =E2= =80=9Cnot >> enough=E2=80=9D in another way (the executable bit is lost, IIUC.) > > Actually the only metadata that Unixfs stores is size: > https://github.com/ipfs/go-unixfs/blob/master/pb/unixfs.proto and by all > means the amount of metadata is negligible for the actual data stored > and serves to give you a progress bar when you are downloading. Yes, the format I came up with also store the size so we can eventually display a progress bar. > Having IPFS understand what files are part of a single item is important > because you can pin/unpin,diff,patch all of them as a whole. Unixfs > also takes care of handling the case where the directories need to > be sharded because there are too many entries. Isn=E2=80=99t there a way, then, to achieve the same behavior with the cust= om format? The /api/v0/add entry point has a =E2=80=98pin=E2=80=99 argument; = I suppose we could leave it to false except when we add the top-level =E2=80=9Cdirectory= =E2=80=9D node? Wouldn=E2=80=99t that give us behavior similar to that of Unixfs? > When the user puts the single root hash in ipfs.io/ipfs/<hash>, it > will display correctly the underlying files and the people will be > able to navigate the actual tree with both web and cli. Right, though that=E2=80=99s less important in my view. > Note that every file added to IPFS is getting wrapped as a Unixfs > block anyways. You are just saving some "directory" nodes by adding > them separately. Hmm weird. When I do /api/v0/add, I=E2=80=99m really just passing a byte vector; there=E2=80=99s no notion of a =E2=80=9Cfile=E2=80=9D here, AFAICS.= Or am I missing something? >> > It will probably need some trial an error to get the multi-part right >> > to upload all in a single request. The Go code HTTP Clients doing >> > this can be found at: >> > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L= 96 >> > As you see, a directory part in the multipart will have the content-ty= pe Header >> > set to "application/x-directory". The best way to see how "abspath" et= c is set >> > is probably to sniff an `ipfs add -r <testfolder>` operation (localhos= t:5001). >> > Once UnixFSv2 lands, you will be in a position to just drop the sexp f= ile >> > altogether. >> >> Yes, that makes sense. In the meantime, I guess we have to keep using >> our own format. >> >> What are the performance implications of adding and retrieving files one >> by one like I did? I understand we=E2=80=99re doing N HTTP requests to t= he >> local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single req= uest, but this >> alone can=E2=80=99t be much of a problem since communication is happening >> locally. Does pinning each file separately somehow incur additional >> overhead? >> > > Yes, pinning separately is slow and incurs in overhead. Pins are stored > in a merkle tree themselves so it involves reading, patching and saving. = This > gets quite slow when you have very large pinsets because your pins block = size > grow. Your pinset will grow very large if you do this. Additionally the > pinning operation itself requires global lock making it more slow. OK, I see. > But, even if it was fast, you will not have a way to easily unpin > anything that becomes obsolete or have an overview of to where things > belong. It is also unlikely that a single IPFS daemon will be able to > store everything you build, so you might find yourself using IPFS Cluster > soon to distribute the storage across multiple nodes and then you will > be effectively adding remotely. Currently, =E2=80=98guix publish=E2=80=99 stores things as long as they are= requested, and then for the duration specified with =E2=80=98--ttl=E2=80=99. I suppos= e we could have similar behavior with IPFS: if an item hasn=E2=80=99t been requested f= or the specified duration, then we unpin it. Does that make sense? Thanks for your help! Ludo=E2=80=99.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 18 Jan 2019 09:08:18 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jan 18 04:08:18 2019 Received: from localhost ([127.0.0.1]:35661 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gkQ8P-00072B-OR for submit <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:08:18 -0500 Received: from mail2.protonmail.ch ([185.70.40.22]:18229) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1gkQ8L-00071w-TA for 33899 <at> debbugs.gnu.org; Fri, 18 Jan 2019 04:08:16 -0500 Date: Fri, 18 Jan 2019 09:08:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1547802487; bh=T2CaH9z+ADJ9S9e9RCY4+xyhaJgW2pJel/fjzCSPEXQ=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=jlvynOjlw+RGreZhyTBmiX39eeMW+1XKeOueyx5lFCjmOkWL0f5p3Jy3o2UuI9xu4 rPGBKAk9/NOjOXmJQxfOHLTNMECFChPDt32W4vDbSqYjWlmNeS1XTI24XIgjUipEUB xc2ooFpTQg+foKk4ceF2/Up06S4fnEhP0kZAoxTs= To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <neM1uqJ3yxqbJiTzV6-q6R-8GNGjv7l_7TJhhQIGXpDQbLoS8yIYrJ4KxKYmFwpi1O9YePH3d5i3fknYgv7nfuMrXFgYoxsk_Xxgs9_Sd2U=@hector.link> In-Reply-To: <87r2dfv0nj.fsf@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> <87r2dfv0nj.fsf@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg\\@ipfs.io" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899\\@debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Monday, January 14, 2019 2:17 PM, Ludovic Court=C3=A8s <ludo@HIDDEN> wr= ote: > Hi Hector, > > Happy new year to you too! :-) > > Hector Sanjuan code@HIDDEN skribis: > > > 1. The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-i= pfs > > has a read/write API (on :5001) and a read-only API that we call "g= ateway" > > and which runs on :8080. The gateway, apart from handling most of t= he > > read-only methods from the HTTP API, also handles paths like "/ipfs= /<cid>" > > or "/ipns/<name>" gracefully, and returns an autogenerated webpage = for > > directory-type CIDs. The gateway does not allow to "publish". There= fore I think > > the doc strings should say "IPFS daemon API" rather than "GATEWAY". > > > > Indeed, I=E2=80=99ll change that. > > > 2. I'm not proficient enough in schema to grasp the details of the > > "directory" format. If I understand it right, you keep a separate m= anifest > > object listing the directory structure, the contents and the execut= able bit > > for each. Thus, when adding a store item you add all the files sepa= rately and > > this manifest. And when retrieving a store item you fetch the manif= est and > > reconstruct the tree by fetching the contents in it (and applying t= he > > executable flag). Is this correct? This works, but it can be improv= ed: > > > > That=E2=80=99s correct. > > > You can add all the files/folders in a single request. If I'm > > reading it right, now each files is added separately (and gets pinned > > separately). It would probably make sense to add it all in a single req= uest, > > letting IPFS to store the directory structure as "unixfs". You can > > additionally add the sexp file with the dir-structure and executable fl= ags > > as an extra file to the root folder. This would allow to fetch the whol= e thing > > with a single request too /api/v0/get?arg=3D<hash>. And to pin a single= hash > > recursively (and not each separately). After getting the whole thing, y= ou > > will need to chmod +x things accordingly. > > Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, as= I see it, is that it > stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to stor= e the mtimes or > permissions; we could ignore them upon reconstruction though), and = =E2=80=9Cnot > enough=E2=80=9D in another way (the executable bit is lost, IIUC.) Actually the only metadata that Unixfs stores is size: https://github.com/ipfs/go-unixfs/blob/master/pb/unixfs.proto and by all means the amount of metadata is negligible for the actual data stored and serves to give you a progress bar when you are downloading. Having IPFS understand what files are part of a single item is important because you can pin/unpin,diff,patch all of them as a whole. Unixfs also takes care of handling the case where the directories need to be sharded because there are too many entries. When the user puts the single root hash in ipfs.io/ipfs/<hash>, it will display correctly the underlying files and the people will be able to navigate the actual tree with both web and cli. Note that every file added to IPFS is getting wrapped as a Unixfs block anyways. You are just saving some "directory" nodes by adding them separately. There is an alternative way which is using IPLD to implement a custom block format that carries the executable bit information and nothing else. But I don't see significant advantages at this point for the extra work it requires. > > > It will probably need some trial an error to get the multi-part right > > to upload all in a single request. The Go code HTTP Clients doing > > this can be found at: > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L9= 6 > > As you see, a directory part in the multipart will have the content-typ= e Header > > set to "application/x-directory". The best way to see how "abspath" etc= is set > > is probably to sniff an `ipfs add -r <testfolder>` operation (localhost= :5001). > > Once UnixFSv2 lands, you will be in a position to just drop the sexp fi= le > > altogether. > > Yes, that makes sense. In the meantime, I guess we have to keep using > our own format. > > What are the performance implications of adding and retrieving files one > by one like I did? I understand we=E2=80=99re doing N HTTP requests to th= e > local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single requ= est, but this > alone can=E2=80=99t be much of a problem since communication is happening > locally. Does pinning each file separately somehow incur additional > overhead? > Yes, pinning separately is slow and incurs in overhead. Pins are stored in a merkle tree themselves so it involves reading, patching and saving. Th= is gets quite slow when you have very large pinsets because your pins block si= ze grow. Your pinset will grow very large if you do this. Additionally the pinning operation itself requires global lock making it more slow. But, even if it was fast, you will not have a way to easily unpin anything that becomes obsolete or have an overview of to where things belong. It is also unlikely that a single IPFS daemon will be able to store everything you build, so you might find yourself using IPFS Cluster soon to distribute the storage across multiple nodes and then you will be effectively adding remotely. > Thanks for your feedback! > > Ludo=E2=80=99. Thanks for working on this! Hector
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 14 Jan 2019 13:17:44 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 14 08:17:44 2019 Received: from localhost ([127.0.0.1]:58369 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gj27a-0003gz-RH for submit <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:17:44 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:58392) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gj27Y-0003gp-BJ for 33899 <at> debbugs.gnu.org; Mon, 14 Jan 2019 08:17:41 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id A4FB7D1F; Mon, 14 Jan 2019 14:17:38 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fiz-E9TzlM6x; Mon, 14 Jan 2019 14:17:37 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 9A1E2AFB; Mon, 14 Jan 2019 14:17:37 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN> To: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS References: <20181228231205.8068-1-ludo@HIDDEN> <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 25 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 14 Jan 2019 14:17:36 +0100 In-Reply-To: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> (Hector Sanjuan's message of "Mon, 07 Jan 2019 14:43:56 +0000") Message-ID: <87r2dfv0nj.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.0 (/) Hi Hector, Happy new year to you too! :-) Hector Sanjuan <code@HIDDEN> skribis: > 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs > has a read/write API (on :5001) and a read-only API that we call "gateway" > and which runs on :8080. The gateway, apart from handling most of the > read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>" > or "/ipns/<name>" gracefully, and returns an autogenerated webpage for > directory-type CIDs. The gateway does not allow to "publish". Therefore I= think > the doc strings should say "IPFS daemon API" rather than "GATEWAY". Indeed, I=E2=80=99ll change that. > 2) I'm not proficient enough in schema to grasp the details of the > "directory" format. If I understand it right, you keep a separate manifest > object listing the directory structure, the contents and the executable b= it > for each. Thus, when adding a store item you add all the files separately= and > this manifest. And when retrieving a store item you fetch the manifest and > reconstruct the tree by fetching the contents in it (and applying the > executable flag). Is this correct? This works, but it can be improved: That=E2=80=99s correct. > You can add all the files/folders in a single request. If I'm > reading it right, now each files is added separately (and gets pinned > separately). It would probably make sense to add it all in a single reque= st, > letting IPFS to store the directory structure as "unixfs". You can > additionally add the sexp file with the dir-structure and executable flags > as an extra file to the root folder. This would allow to fetch the whole = thing > with a single request too /api/v0/get?arg=3D<hash>. And to pin a single h= ash > recursively (and not each separately). After getting the whole thing, you > will need to chmod +x things accordingly. Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, as = I see it, is that it stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to store = the mtimes or permissions; we could ignore them upon reconstruction though), and =E2=80= =9Cnot enough=E2=80=9D in another way (the executable bit is lost, IIUC.) > It will probably need some trial an error to get the multi-part right > to upload all in a single request. The Go code HTTP Clients doing > this can be found at: > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 > > As you see, a directory part in the multipart will have the content-type = Header > set to "application/x-directory". The best way to see how "abspath" etc i= s set > is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5= 001). > > Once UnixFSv2 lands, you will be in a position to just drop the sexp file > altogether. Yes, that makes sense. In the meantime, I guess we have to keep using our own format. What are the performance implications of adding and retrieving files one by one like I did? I understand we=E2=80=99re doing N HTTP requests to the local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single reques= t, but this alone can=E2=80=99t be much of a problem since communication is happening locally. Does pinning each file separately somehow incur additional overhead? Thanks for your feedback! Ludo=E2=80=99.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 7 Jan 2019 16:41:51 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 07 11:41:51 2019 Received: from localhost ([127.0.0.1]:48897 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1ggXyH-0000WZ-Rj for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 11:41:50 -0500 Received: from mail2.protonmail.ch ([185.70.40.22]:26380) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1ggW8J-0003bW-9s for 33899 <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:04 -0500 Date: Mon, 07 Jan 2019 14:43:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1546872237; bh=lp764yzH6aORAEQQrqc5Xp6aLNoagBpe0MxADG4AtLk=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=OhgQyfiRy8hSgOA1WJRzOajYeo4JMif6M3sfNh7u1XF3qWu787NWniXdAr9/Q8rUf KtqWQUi3yccrDPcldPCyKMugLzgiPdLjPahb6apDMuzmgr5AvNcpC1XPa0vNyXtKfI 6VfxS5PiC82FCqEFmD8aT1/47qH4PFyVRac1Rr5Q= To: "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 X-Mailman-Approved-At: Mon, 07 Jan 2019 11:41:47 -0500 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "guix-patches@HIDDEN" <guix-patches@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Saturday, December 29, 2018 12:12 AM, Ludovic Court=C3=A8s <ludo@HIDDEN= > wrote: > Hello Guix! > > Here is a first draft adding support to distribute and retrieve substitut= es > over IPFS. This builds on discussions at the R-B Summit with H=C3=A9ctor = Sanjuan > of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian > Paul Schmidt posted on guix-devel last month. > > The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides > bindings to a subset of that API. This module also implements a custom > =E2=80=9Cdirectory=E2=80=9D format to store directory trees in IPFS (IPFS= already provides > =E2=80=9CUnixFS=E2=80=9D and =E2=80=9Ctar=E2=80=9D but they store too man= y or too few file attributes.) > > =E2=80=98guix publish=E2=80=99 and =E2=80=98guix substitute=E2=80=99 use = (guix ipfs) to > store and retrieve store items. Complete directory trees are stored in > IPFS =E2=80=9Cas is=E2=80=9D, rather than as compressed archives (nars). = This allows for > deduplication in IPFS. =E2=80=98guix publish=E2=80=99 adds a new = =E2=80=9CIPFS=E2=80=9D field in > narinfos and =E2=80=98guix substitute=E2=80=99 can then query those objec= ts over IPFS. > So the idea is that you still get narinfos over HTTP(S), and then you > have the option of downloading substitutes over IPFS. > > I=E2=80=99ve pushed these patches in =E2=80=98wip-ipfs-substitutes= =E2=80=99. This is rough on the > edges and probably buggy, but the adventurous among us might want to give > it a spin. :-) > > Thanks, > Ludo=E2=80=99. Hey! Happy new year! This is great news. I'm very glad to see this. I haven't tried this yet but looking at the code there are a couple of things to point out. 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs has a read/write API (on :5001) and a read-only API that we call "gateway" and which runs on :8080. The gateway, apart from handling most of the read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>" or "/ipns/<name>" gracefully, and returns an autogenerated webpage for directory-type CIDs. The gateway does not allow to "publish". Therefore I t= hink the doc strings should say "IPFS daemon API" rather than "GATEWAY". 2) I'm not proficient enough in schema to grasp the details of the "directory" format. If I understand it right, you keep a separate manifest object listing the directory structure, the contents and the executable bit for each. Thus, when adding a store item you add all the files separately a= nd this manifest. And when retrieving a store item you fetch the manifest and reconstruct the tree by fetching the contents in it (and applying the executable flag). Is this correct? This works, but it can be improved: You can add all the files/folders in a single request. If I'm reading it right, now each files is added separately (and gets pinned separately). It would probably make sense to add it all in a single request= , letting IPFS to store the directory structure as "unixfs". You can additionally add the sexp file with the dir-structure and executable flags as an extra file to the root folder. This would allow to fetch the whole th= ing with a single request too /api/v0/get?arg=3D<hash>. And to pin a single has= h recursively (and not each separately). After getting the whole thing, you will need to chmod +x things accordingly. It will probably need some trial an error to get the multi-part right to upload all in a single request. The Go code HTTP Clients doing this can be found at: https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 As you see, a directory part in the multipart will have the content-type He= ader set to "application/x-directory". The best way to see how "abspath" etc is = set is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:500= 1). Once UnixFSv2 lands, you will be in a position to just drop the sexp file altogether. Let me know if you have any doubts, I'll make my best to answer them. In th= e meantime I'll try to get more familiar with Guix. Cheers, Hector PS. There is a place where it says "ifps" instead of "ipfs". A very common = typo.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at submit) by debbugs.gnu.org; 7 Jan 2019 16:41:51 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jan 07 11:41:51 2019 Received: from localhost ([127.0.0.1]:48899 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1ggXyJ-0000Wc-0W for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 11:41:51 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55150) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <code@HIDDEN>) id 1ggW8S-0003cI-GO for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:12 -0500 Received: from lists.gnu.org ([209.51.188.17]:46720) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8M-0005SD-0u for submit <at> debbugs.gnu.org; Mon, 07 Jan 2019 09:44:07 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37113) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8K-0007s6-TI for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:05 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.1 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_LOW autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8J-0005RN-PX for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:04 -0500 Received: from mail1.protonmail.ch ([185.70.40.18]:31631) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from <code@HIDDEN>) id 1ggW8J-0005QN-8L for guix-patches@HIDDEN; Mon, 07 Jan 2019 09:44:03 -0500 Date: Mon, 07 Jan 2019 14:43:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1546872237; bh=lp764yzH6aORAEQQrqc5Xp6aLNoagBpe0MxADG4AtLk=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=OhgQyfiRy8hSgOA1WJRzOajYeo4JMif6M3sfNh7u1XF3qWu787NWniXdAr9/Q8rUf KtqWQUi3yccrDPcldPCyKMugLzgiPdLjPahb6apDMuzmgr5AvNcpC1XPa0vNyXtKfI 6VfxS5PiC82FCqEFmD8aT1/47qH4PFyVRac1Rr5Q= To: "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org> From: Hector Sanjuan <code@HIDDEN> Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: <NIQ7NwCAv476krhArm683GoWozb7fXZrcRzXorY2s-X_QAZ71GhHzrFj7788bRBP3fpklDWfxy-we7928SLdmd-iairDLG_sNDRjbJwxYyo=@hector.link> In-Reply-To: <20181228231205.8068-1-ludo@HIDDEN> References: <20181228231205.8068-1-ludo@HIDDEN> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 185.70.40.18 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Mon, 07 Jan 2019 11:41:48 -0500 Cc: =?UTF-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN>, "go-ipfs-wg@HIDDEN" <go-ipfs-wg@HIDDEN>, "guix-patches@HIDDEN" <guix-patches@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Reply-To: Hector Sanjuan <code@HIDDEN> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.0 (/) =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Saturday, December 29, 2018 12:12 AM, Ludovic Court=C3=A8s <ludo@HIDDEN= > wrote: > Hello Guix! > > Here is a first draft adding support to distribute and retrieve substitut= es > over IPFS. This builds on discussions at the R-B Summit with H=C3=A9ctor = Sanjuan > of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian > Paul Schmidt posted on guix-devel last month. > > The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides > bindings to a subset of that API. This module also implements a custom > =E2=80=9Cdirectory=E2=80=9D format to store directory trees in IPFS (IPFS= already provides > =E2=80=9CUnixFS=E2=80=9D and =E2=80=9Ctar=E2=80=9D but they store too man= y or too few file attributes.) > > =E2=80=98guix publish=E2=80=99 and =E2=80=98guix substitute=E2=80=99 use = (guix ipfs) to > store and retrieve store items. Complete directory trees are stored in > IPFS =E2=80=9Cas is=E2=80=9D, rather than as compressed archives (nars). = This allows for > deduplication in IPFS. =E2=80=98guix publish=E2=80=99 adds a new = =E2=80=9CIPFS=E2=80=9D field in > narinfos and =E2=80=98guix substitute=E2=80=99 can then query those objec= ts over IPFS. > So the idea is that you still get narinfos over HTTP(S), and then you > have the option of downloading substitutes over IPFS. > > I=E2=80=99ve pushed these patches in =E2=80=98wip-ipfs-substitutes= =E2=80=99. This is rough on the > edges and probably buggy, but the adventurous among us might want to give > it a spin. :-) > > Thanks, > Ludo=E2=80=99. Hey! Happy new year! This is great news. I'm very glad to see this. I haven't tried this yet but looking at the code there are a couple of things to point out. 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs has a read/write API (on :5001) and a read-only API that we call "gateway" and which runs on :8080. The gateway, apart from handling most of the read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>" or "/ipns/<name>" gracefully, and returns an autogenerated webpage for directory-type CIDs. The gateway does not allow to "publish". Therefore I t= hink the doc strings should say "IPFS daemon API" rather than "GATEWAY". 2) I'm not proficient enough in schema to grasp the details of the "directory" format. If I understand it right, you keep a separate manifest object listing the directory structure, the contents and the executable bit for each. Thus, when adding a store item you add all the files separately a= nd this manifest. And when retrieving a store item you fetch the manifest and reconstruct the tree by fetching the contents in it (and applying the executable flag). Is this correct? This works, but it can be improved: You can add all the files/folders in a single request. If I'm reading it right, now each files is added separately (and gets pinned separately). It would probably make sense to add it all in a single request= , letting IPFS to store the directory structure as "unixfs". You can additionally add the sexp file with the dir-structure and executable flags as an extra file to the root folder. This would allow to fetch the whole th= ing with a single request too /api/v0/get?arg=3D<hash>. And to pin a single has= h recursively (and not each separately). After getting the whole thing, you will need to chmod +x things accordingly. It will probably need some trial an error to get the multi-part right to upload all in a single request. The Go code HTTP Clients doing this can be found at: https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 As you see, a directory part in the multipart will have the content-type He= ader set to "application/x-directory". The best way to see how "abspath" etc is = set is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:500= 1). Once UnixFSv2 lands, you will be in a position to just drop the sexp file altogether. Let me know if you have any doubts, I'll make my best to answer them. In th= e meantime I'll try to get more familiar with Guix. Cheers, Hector PS. There is a place where it says "ifps" instead of "ipfs". A very common = typo.
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:29 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:16:29 2018 Received: from localhost ([127.0.0.1]:40718 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1Mi-0001Qj-AP for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:28 -0500 Received: from eggs.gnu.org ([208.118.235.92]:56171) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1Mc-0001Q4-54 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:22 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MT-0002LC-0x for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:17 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51204) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MH-00027t-Pg; Fri, 28 Dec 2018 18:16:03 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1MH-00077n-Cc; Fri, 28 Dec 2018 18:16:01 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: [PATCH 3/5] Add (guix ipfs). Date: Sat, 29 Dec 2018 00:15:52 +0100 Message-Id: <20181228231554.8220-3-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN> References: <20181228231554.8220-1-ludo@HIDDEN> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33899 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.0 (------) * guix/ipfs.scm, tests/ipfs.scm: New files. * Makefile.am (MODULES, SCM_TESTS): Add them. --- Makefile.am | 2 + guix/ipfs.scm | 250 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/ipfs.scm | 55 +++++++++++ 3 files changed, 307 insertions(+) create mode 100644 guix/ipfs.scm create mode 100644 tests/ipfs.scm diff --git a/Makefile.am b/Makefile.am index da3720e3a6..975d83db6c 100644 --- a/Makefile.am +++ b/Makefile.am @@ -101,6 +101,7 @@ MODULES = \ guix/cve.scm \ guix/workers.scm \ guix/zlib.scm \ + guix/ipfs.scm \ guix/build-system.scm \ guix/build-system/android-ndk.scm \ guix/build-system/ant.scm \ @@ -384,6 +385,7 @@ SCM_TESTS = \ tests/cve.scm \ tests/workers.scm \ tests/zlib.scm \ + tests/ipfs.scm \ tests/file-systems.scm \ tests/uuid.scm \ tests/system.scm \ diff --git a/guix/ipfs.scm b/guix/ipfs.scm new file mode 100644 index 0000000000..e941feda6f --- /dev/null +++ b/guix/ipfs.scm @@ -0,0 +1,250 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (guix ipfs) + #:use-module (guix json) + #:use-module (guix base64) + #:use-module ((guix build utils) #:select (dump-port)) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-26) + #:use-module (rnrs io ports) + #:use-module (rnrs bytevectors) + #:use-module (ice-9 match) + #:use-module (ice-9 ftw) + #:use-module (web uri) + #:use-module (web client) + #:use-module (web response) + #:export (%ipfs-base-url + add-file + add-file-tree + restore-file-tree + + content? + content-name + content-hash + content-size + + add-empty-directory + add-to-directory + read-contents + publish-name)) + +;;; Commentary: +;;; +;;; This module implements bindings for the HTTP interface of the IPFS +;;; gateway, documented here: <https://docs.ipfs.io/reference/api/http/>. It +;;; allows you to add and retrieve files over IPFS, and a few other things. +;;; +;;; Code: + +(define %ipfs-base-url + ;; URL of the IPFS gateway. + (make-parameter "http://localhost:5001")) + +(define* (call url decode #:optional (method http-post) + #:key body (false-if-404? #t) (headers '())) + "Invoke the endpoint at URL using METHOD. Decode the resulting JSON body +using DECODE, a one-argument procedure that takes an input port; when DECODE +is false, return the input port. When FALSE-IF-404? is true, return #f upon +404 responses." + (let*-values (((response port) + (method url #:streaming? #t + #:body body + + ;; Always pass "Connection: close". + #:keep-alive? #f + #:headers `((connection close) + ,@headers)))) + (cond ((= 200 (response-code response)) + (if decode + (let ((result (decode port))) + (close-port port) + result) + port)) + ((and false-if-404? + (= 404 (response-code response))) + (close-port port) + #f) + (else + (close-port port) + (throw 'ipfs-error url response))))) + +;; Result of a file addition. +(define-json-mapping <content> make-content content? + json->content + (name content-name "Name") + (hash content-hash "Hash") + (bytes content-bytes "Bytes") + (size content-size "Size" string->number)) + +;; Result of a 'patch/add-link' operation. +(define-json-mapping <directory> make-directory directory? + json->directory + (hash directory-hash "Hash") + (links directory-links "Links" json->links)) + +;; A "link". +(define-json-mapping <link> make-link link? + json->link + (name link-name "Name") + (hash link-hash "Hash") + (size link-size "Size" string->number)) + +;; A "binding", also known as a "name". +(define-json-mapping <binding> make-binding binding? + json->binding + (name binding-name "Name") + (value binding-value "Value")) + +(define (json->links json) + (match json + (#f '()) + (links (map json->link links)))) + +(define %multipart-boundary + ;; XXX: We might want to find a more reliable boundary. + (string-append (make-string 24 #\-) "2698127afd7425a6")) + +(define (bytevector->form-data bv port) + "Write to PORT a 'multipart/form-data' representation of BV." + (display (string-append "--" %multipart-boundary "\r\n" + "Content-Disposition: form-data\r\n" + "Content-Type: application/octet-stream\r\n\r\n") + port) + (put-bytevector port bv) + (display (string-append "\r\n--" %multipart-boundary "--\r\n") + port)) + +(define* (add-data data #:key (name "file.txt") recursive?) + "Add DATA, a bytevector, to IPFS. Return a content object representing it." + (call (string-append (%ipfs-base-url) + "/api/v0/add?arg=" (uri-encode name) + "&recursive=" + (if recursive? "true" "false")) + json->content + #:headers + `((content-type + . (multipart/form-data + (boundary . ,%multipart-boundary)))) + #:body + (call-with-bytevector-output-port + (lambda (port) + (bytevector->form-data data port))))) + +(define (not-dot? entry) + (not (member entry '("." "..")))) + +(define (file-tree->sexp file) + "Add FILE, recursively, to the IPFS, and return an sexp representing the +directory's tree structure. + +Unlike IPFS's own \"UnixFS\" structure, this format preserves exactly what we +need: like the nar format, it preserves the executable bit, but does not save +the mtime or other Unixy attributes irrelevant in the store." + ;; The natural approach would be to insert each directory listing as an + ;; object of its own in IPFS. However, this does not buy us much in terms + ;; of deduplication, but it does cause a lot of extra round trips when + ;; fetching it. Thus, this sexp is \"flat\" in that only the leaves are + ;; inserted into the IPFS. + (let ((st (lstat file))) + (match (stat:type st) + ('directory + (let* ((parent file) + (entries (map (lambda (file) + `(entry ,file + ,(file-tree->sexp + (string-append parent "/" file)))) + (scandir file not-dot?))) + (size (fold (lambda (entry total) + (match entry + (('entry name (kind value size)) + (+ total size)))) + 0 + entries))) + `(directory ,entries ,size))) + ('symlink + `(symlink ,(readlink file) 0)) + ('regular + (let ((size (stat:size st))) + (if (zero? (logand (stat:mode st) #o100)) + `(file ,(content-name (add-file file)) ,size) + `(executable ,(content-name (add-file file)) ,size))))))) + +(define (add-file-tree file) + "Add FILE to the IPFS, recursively, using our own canonical directory +format. Return the resulting content object." + (add-data (string->utf8 (object->string + `(file-tree (version 0) + ,(file-tree->sexp file)))))) + +(define (restore-file-tree object file) + "Restore to FILE the tree pointed to by OBJECT." + (let restore ((tree (match (read (read-contents object)) + (('file-tree ('version 0) tree) + tree))) + (file file)) + (match tree + (('file object size) + (call-with-output-file file + (lambda (output) + (dump-port (read-contents object) output)))) + (('executable object size) + (call-with-output-file file + (lambda (output) + (dump-port (read-contents object) output))) + (chmod file #o555)) + (('symlink target size) + (symlink target file)) + (('directory (('entry names entries) ...) size) + (mkdir file) + (for-each restore entries + (map (cut string-append file "/" <>) names)))))) + +(define* (add-file file #:key (name (basename file))) + "Add FILE under NAME to the IPFS and return a content object for it." + (add-data (match (call-with-input-file file get-bytevector-all) + ((? eof-object?) #vu8()) + (bv bv)) + #:name name)) + +(define* (add-empty-directory #:key (name "directory")) + "Return a content object for an empty directory." + (add-data #vu8() #:recursive? #t #:name name)) + +(define* (add-to-directory directory file name) + "Add FILE to DIRECTORY under NAME, and return the resulting directory. +DIRECTORY and FILE must be hashes identifying objects in the IPFS store." + (call (string-append (%ipfs-base-url) + "/api/v0/object/patch/add-link?arg=" + (uri-encode directory) + "&arg=" (uri-encode name) "&arg=" (uri-encode file) + "&create=true") + json->directory)) + +(define* (read-contents object #:key offset length) + "Return an input port to read the content of OBJECT from." + (call (string-append (%ipfs-base-url) + "/api/v0/cat?arg=" object) + #f)) + +(define* (publish-name object) + "Publish OBJECT under the current peer ID." + (call (string-append (%ipfs-base-url) + "/api/v0/name/publish?arg=" object) + json->binding)) diff --git a/tests/ipfs.scm b/tests/ipfs.scm new file mode 100644 index 0000000000..3b662b22bd --- /dev/null +++ b/tests/ipfs.scm @@ -0,0 +1,55 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (test-ipfs) + #:use-module (guix ipfs) + #:use-module ((guix utils) #:select (call-with-temporary-directory)) + #:use-module (guix tests) + #:use-module (web uri) + #:use-module (srfi srfi-64)) + +;; Test the (guix ipfs) module. + +(define (ipfs-gateway-running?) + "Return true if the IPFS gateway is running at %IPFS-BASE-URL." + (let* ((uri (string->uri (%ipfs-base-url))) + (socket (socket AF_INET SOCK_STREAM 0))) + (define connected? + (catch 'system-error + (lambda () + (format (current-error-port) + "probing IPFS gateway at localhost:~a...~%" + (uri-port uri)) + (connect socket AF_INET INADDR_LOOPBACK (uri-port uri)) + #t) + (const #f))) + + (close-port socket) + connected?)) + +(unless (ipfs-gateway-running?) + (test-skip 1)) + +(test-assert "add-file-tree + restore-file-tree" + (call-with-temporary-directory + (lambda (directory) + (let* ((source (dirname (search-path %load-path "guix/base32.scm"))) + (target (string-append directory "/r")) + (content (pk 'content (add-file-tree source)))) + (restore-file-tree (content-name content) target) + (file=? source target))))) -- 2.20.1
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:23 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:16:23 2018 Received: from localhost ([127.0.0.1]:40716 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1Mc-0001QM-HW for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:23 -0500 Received: from eggs.gnu.org ([208.118.235.92]:32852) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1Ma-0001Pm-23 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MP-0002Hs-0F for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:14 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51206) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MI-00028B-GX; Fri, 28 Dec 2018 18:16:03 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1MI-00077n-33; Fri, 28 Dec 2018 18:16:02 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: [PATCH 4/5] publish: Add IPFS support. Date: Sat, 29 Dec 2018 00:15:53 +0100 Message-Id: <20181228231554.8220-4-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN> References: <20181228231554.8220-1-ludo@HIDDEN> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33899 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.0 (------) * guix/scripts/publish.scm (show-help, %options): Add '--ipfs'. (narinfo-string): Add IPFS parameter and honor it. (render-narinfo/cached): Add #:ipfs? and honor it. (bake-narinfo+nar, make-request-handler, run-publish-server): Likewise. (guix-publish): Honor '--ipfs' and parameterize %IPFS-BASE-URL. --- doc/guix.texi | 33 ++++++++++++++++++++ guix/scripts/publish.scm | 67 ++++++++++++++++++++++++++++------------ 2 files changed, 80 insertions(+), 20 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index fcb5b8c088..f2af5a1558 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -8470,6 +8470,15 @@ caching of the archives before they are sent to clients---see below for details. The @command{guix weather} command provides a handy way to check what a server provides (@pxref{Invoking guix weather}). +@cindex peer-to-peer, substitute distribution +@cindex distributed storage, of substitutes +@cindex IPFS, for substitutes +It is also possible to publish substitutes over @uref{https://ipfs.io, IFPS}, +a distributed, peer-to-peer storage mechanism. To enable it, pass the +@option{--ipfs} option alongside @option{--cache}, and make sure you're +running @command{ipfs daemon}. Capable clients will then be able to choose +whether to fetch substitutes over HTTP or over IPFS. + As a bonus, @command{guix publish} also serves as a content-addressed mirror for source files referenced in @code{origin} records (@pxref{origin Reference}). For instance, assuming @command{guix @@ -8560,6 +8569,30 @@ thread per CPU core is created, but this can be customized. See When @option{--ttl} is used, cached entries are automatically deleted when they have expired. +@item --ifps[=@var{gateway}] +When used in conjunction with @option{--cache}, instruct @command{guix +publish} to publish substitutes over the @uref{https://ipfs.io, IPFS +distributed data store} in addition to HTTP. + +@quotation Note +As of version @value{VERSION}, IPFS support is experimental. You're welcome +to share your experience with the developers by emailing +@email{guix-devel@@gnu.org}! +@end quotation + +The IPFS HTTP interface must be reachable at @var{gateway}, by default +@code{localhost:5001}. To get it up and running, it is usually enough to +install IPFS and start the IPFS daemon: + +@example +$ guix package -i go-ipfs +$ ipfs init +$ ipfs daemon +@end example + +For more information on how to get started with IPFS, please refer to the +@uref{https://docs.ipfs.io/introduction/usage/, IPFS documentation}. + @item --workers=@var{N} When @option{--cache} is used, request the allocation of @var{N} worker threads to ``bake'' archives. diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index a236f3e45c..2accd632ab 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -59,6 +59,7 @@ #:use-module ((guix build utils) #:select (dump-port mkdir-p find-files)) #:use-module ((guix build syscalls) #:select (set-thread-name)) + #:use-module ((guix ipfs) #:prefix ipfs:) #:export (%public-key %private-key @@ -78,6 +79,8 @@ Publish ~a over HTTP.\n") %store-directory) compress archives at LEVEL")) (display (G_ " -c, --cache=DIRECTORY cache published items to DIRECTORY")) + (display (G_ " + --ipfs[=GATEWAY] publish items over IPFS via GATEWAY")) (display (G_ " --workers=N use N workers to bake items")) (display (G_ " @@ -168,6 +171,10 @@ compression disabled~%")) (option '(#\c "cache") #t #f (lambda (opt name arg result) (alist-cons 'cache arg result))) + (option '("ipfs") #f #t + (lambda (opt name arg result) + (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url)) + result))) (option '("workers") #t #f (lambda (opt name arg result) (alist-cons 'workers (string->number* arg) @@ -237,12 +244,15 @@ compression disabled~%")) (define* (narinfo-string store store-path key #:key (compression %no-compression) - (nar-path "nar") file-size) + (nar-path "nar") file-size ipfs) "Generate a narinfo key/value string for STORE-PATH; an exception is raised if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs. + Optionally, FILE-SIZE can specify the size in bytes of the compressed NAR; it -informs the client of how much needs to be downloaded." +informs the client of how much needs to be downloaded. + +When IPFS is true, it is the IPFS object identifier for STORE-PATH." (let* ((path-info (query-path-info store store-path)) (compression (actual-compression store-path compression)) (url (encode-and-join-uri-path @@ -295,7 +305,12 @@ References: ~a~%~a" (apply throw args)))))) (signature (base64-encode-string (canonical-sexp->string (signed-string info))))) - (format #f "~aSignature: 1;~a;~a~%" info (gethostname) signature))) + (format #f "~aSignature: 1;~a;~a~%~a" info (gethostname) signature + + ;; Append IPFS info below the signed part. + (if ipfs + (string-append "IPFS: " ipfs "\n") + "")))) (define* (not-found request #:key (phrase "Resource not found") @@ -406,10 +421,12 @@ items. Failing that, we could eventually have to recompute them and return (define* (render-narinfo/cached store request hash #:key ttl (compression %no-compression) (nar-path "nar") - cache pool) + cache pool ipfs?) "Respond to the narinfo request for REQUEST. If the narinfo is available in CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo -requested using POOL." +requested using POOL. + +When IPFS? is true, additionally publish binaries over IPFS." (define (delete-entry narinfo) ;; Delete NARINFO and the corresponding nar from CACHE. (let ((nar (string-append (string-drop-right narinfo @@ -447,7 +464,8 @@ requested using POOL." (bake-narinfo+nar cache item #:ttl ttl #:compression compression - #:nar-path nar-path))) + #:nar-path nar-path + #:ipfs? ipfs?))) (when ttl (single-baker 'cache-cleanup @@ -465,7 +483,7 @@ requested using POOL." (define* (bake-narinfo+nar cache item #:key ttl (compression %no-compression) - (nar-path "/nar")) + (nar-path "/nar") ipfs?) "Write the narinfo and nar for ITEM to CACHE." (let* ((compression (actual-compression item compression)) (nar (nar-cache-file cache item @@ -502,7 +520,11 @@ requested using POOL." #:nar-path nar-path #:compression compression #:file-size (and=> (stat nar #f) - stat:size)) + stat:size) + #:ipfs + (and ipfs? + (ipfs:content-name + (ipfs:add-file-tree item)))) port)))))) ;; XXX: Declare the 'X-Nar-Compression' HTTP header, which is in fact for @@ -766,7 +788,8 @@ blocking." cache pool narinfo-ttl (nar-path "nar") - (compression %no-compression)) + (compression %no-compression) + ipfs?) (define nar-path? (let ((expected (split-and-decode-uri-path nar-path))) (cut equal? expected <>))) @@ -793,7 +816,8 @@ blocking." #:pool pool #:ttl narinfo-ttl #:nar-path nar-path - #:compression compression) + #:compression compression + #:ipfs? ipfs?) (render-narinfo store request hash #:ttl narinfo-ttl #:nar-path nar-path @@ -847,13 +871,14 @@ blocking." (define* (run-publish-server socket store #:key (compression %no-compression) (nar-path "nar") narinfo-ttl - cache pool) + cache pool ipfs?) (run-server (make-request-handler store #:cache cache #:pool pool #:nar-path nar-path #:narinfo-ttl narinfo-ttl - #:compression compression) + #:compression compression + #:ipfs? ipfs?) concurrent-http-server `(#:socket ,socket))) @@ -902,6 +927,7 @@ blocking." (repl-port (assoc-ref opts 'repl)) (cache (assoc-ref opts 'cache)) (workers (assoc-ref opts 'workers)) + (ipfs (assoc-ref opts 'ipfs)) ;; Read the key right away so that (1) we fail early on if we can't ;; access them, and (2) we can then drop privileges. @@ -930,14 +956,15 @@ consider using the '--user' option!~%"))) (set-thread-name "guix publish") (with-store store - (run-publish-server socket store - #:cache cache - #:pool (and cache (make-pool workers - #:thread-name - "publish worker")) - #:nar-path nar-path - #:compression compression - #:narinfo-ttl ttl)))))) + (parameterize ((ipfs:%ipfs-base-url ipfs)) + (run-publish-server socket store + #:cache cache + #:pool (and cache (make-pool workers + #:thread-name + "publish worker")) + #:nar-path nar-path + #:compression compression + #:narinfo-ttl ttl))))))) ;;; Local Variables: ;;; eval: (put 'single-baker 'scheme-indent-function 1) -- 2.20.1
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:22 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:16:22 2018 Received: from localhost ([127.0.0.1]:40713 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1Mb-0001QJ-W9 for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:35129) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1Ma-0001Pn-2Y for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MP-0002Hy-1D for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:14 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51207) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MJ-00028O-3F; Fri, 28 Dec 2018 18:16:03 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1MI-00077n-Pb; Fri, 28 Dec 2018 18:16:02 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: [PATCH 5/5] DRAFT substitute: Add IPFS support. Date: Sat, 29 Dec 2018 00:15:54 +0100 Message-Id: <20181228231554.8220-5-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN> References: <20181228231554.8220-1-ludo@HIDDEN> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33899 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.0 (------) Missing: - documentation - command-line options - progress report when downloading over IPFS - fallback when we fail to fetch from IPFS * guix/scripts/substitute.scm (<narinfo>)[ipfs]: New field. (read-narinfo): Read "IPFS". (process-substitution/http): New procedure, with code formerly in 'process-substitution'. (process-substitution): Check for IPFS and call 'ipfs:restore-file-tree' when IPFS is true. --- guix/scripts/substitute.scm | 106 +++++++++++++++++++++--------------- 1 file changed, 61 insertions(+), 45 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 53b1777241..8be15e4f13 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -42,6 +42,7 @@ #:use-module (guix progress) #:use-module ((guix build syscalls) #:select (set-thread-name)) + #:use-module ((guix ipfs) #:prefix ipfs:) #:use-module (ice-9 rdelim) #:use-module (ice-9 regex) #:use-module (ice-9 match) @@ -281,7 +282,7 @@ failure, return #f and #f." (define-record-type <narinfo> (%make-narinfo path uri uri-base compression file-hash file-size nar-hash nar-size - references deriver system signature contents) + references deriver system ipfs signature contents) narinfo? (path narinfo-path) (uri narinfo-uri) @@ -294,6 +295,7 @@ failure, return #f and #f." (references narinfo-references) (deriver narinfo-deriver) (system narinfo-system) + (ipfs narinfo-ipfs) (signature narinfo-signature) ; canonical sexp ;; The original contents of a narinfo file. This field is needed because we ;; want to preserve the exact textual representation for verification purposes. @@ -335,7 +337,7 @@ s-expression: ~s~%") "Return a narinfo constructor for narinfos originating from CACHE-URL. STR must contain the original contents of a narinfo file." (lambda (path url compression file-hash file-size nar-hash nar-size - references deriver system signature) + references deriver system ipfs signature) "Return a new <narinfo> object." (%make-narinfo path ;; Handle the case where URL is a relative URL. @@ -352,6 +354,7 @@ must contain the original contents of a narinfo file." ((or #f "") #f) (_ deriver)) system + ipfs (false-if-exception (and=> signature narinfo-signature->canonical-sexp)) str))) @@ -386,7 +389,7 @@ No authentication and authorization checks are performed here!" (narinfo-maker str url) '("StorePath" "URL" "Compression" "FileHash" "FileSize" "NarHash" "NarSize" - "References" "Deriver" "System" + "References" "Deriver" "System" "IPFS" "Signature")))) (define (narinfo-sha256 narinfo) @@ -947,13 +950,58 @@ authorized substitutes." (wtf (error "unknown `--query' command" wtf)))) +(define* (process-substitution/http narinfo destination uri + #:key print-build-trace?) + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading ~a...~%") (uri->string uri))) + + (let*-values (((raw download-size) + ;; Note that Hydra currently generates Nars on the fly + ;; and doesn't specify a Content-Length, so + ;; DOWNLOAD-SIZE is #f in practice. + (fetch uri #:buffered? #f #:timeout? #f)) + ((progress) + (let* ((comp (narinfo-compression narinfo)) + (dl-size (or download-size + (and (equal? comp "none") + (narinfo-size narinfo)))) + (reporter (if print-build-trace? + (progress-reporter/trace + destination + (uri->string uri) dl-size + (current-error-port)) + (progress-reporter/file + (uri->string uri) dl-size + (current-error-port) + #:abbreviation nar-uri-abbreviation)))) + (progress-report-port reporter raw))) + ((input pids) + ;; NOTE: This 'progress' port of current process will be + ;; closed here, while the child process doing the + ;; reporting will close it upon exit. + (decompressed-port (and=> (narinfo-compression narinfo) + string->symbol) + progress))) + ;; Unpack the Nar at INPUT into DESTINATION. + (restore-file input destination) + (close-port input) + + ;; Wait for the reporter to finish. + (every (compose zero? cdr waitpid) pids) + + ;; Skip a line after what 'progress-reporter/file' printed, and another + ;; one to visually separate substitutions. + (display "\n\n" (current-error-port)))) + (define* (process-substitution store-item destination #:key cache-urls acl print-build-trace?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL." (let* ((narinfo (lookup-narinfo cache-urls store-item (cut valid-narinfo? <> acl))) - (uri (and=> narinfo narinfo-uri))) + (uri (and=> narinfo narinfo-uri)) + (ipfs (and=> narinfo narinfo-ipfs))) (unless uri (leave (G_ "no valid substitute for '~a'~%") store-item)) @@ -961,47 +1009,15 @@ DESTINATION as a nar file. Verify the substitute against ACL." ;; Tell the daemon what the expected hash of the Nar itself is. (format #t "~a~%" (narinfo-hash narinfo)) - (unless print-build-trace? - (format (current-error-port) - (G_ "Downloading ~a...~%") (uri->string uri))) - - (let*-values (((raw download-size) - ;; Note that Hydra currently generates Nars on the fly - ;; and doesn't specify a Content-Length, so - ;; DOWNLOAD-SIZE is #f in practice. - (fetch uri #:buffered? #f #:timeout? #f)) - ((progress) - (let* ((comp (narinfo-compression narinfo)) - (dl-size (or download-size - (and (equal? comp "none") - (narinfo-size narinfo)))) - (reporter (if print-build-trace? - (progress-reporter/trace - destination - (uri->string uri) dl-size - (current-error-port)) - (progress-reporter/file - (uri->string uri) dl-size - (current-error-port) - #:abbreviation nar-uri-abbreviation)))) - (progress-report-port reporter raw))) - ((input pids) - ;; NOTE: This 'progress' port of current process will be - ;; closed here, while the child process doing the - ;; reporting will close it upon exit. - (decompressed-port (and=> (narinfo-compression narinfo) - string->symbol) - progress))) - ;; Unpack the Nar at INPUT into DESTINATION. - (restore-file input destination) - (close-port input) - - ;; Wait for the reporter to finish. - (every (compose zero? cdr waitpid) pids) - - ;; Skip a line after what 'progress-reporter/file' printed, and another - ;; one to visually separate substitutions. - (display "\n\n" (current-error-port))))) + (if ipfs + (begin + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading from IPFS ~s...~%") ipfs)) + (ipfs:restore-file-tree ipfs destination)) + (process-substitution/http narinfo destination uri + #:print-build-trace? + print-build-trace?)))) ;;; -- 2.20.1
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:17 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:16:16 2018 Received: from localhost ([127.0.0.1]:40709 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1MW-0001Pr-Cf for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:16 -0500 Received: from eggs.gnu.org ([208.118.235.92]:33939) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1MU-0001PV-4K for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MN-0002F9-63 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:09 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51202) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MG-00026W-Eq; Fri, 28 Dec 2018 18:16:01 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1MG-00077n-5l; Fri, 28 Dec 2018 18:16:00 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: [PATCH 1/5] Add (guix json). Date: Sat, 29 Dec 2018 00:15:50 +0100 Message-Id: <20181228231554.8220-1-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33899 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.0 (------) * guix/swh.scm: Use (guix json). (define-json-reader, define-json-mapping): Move to... * guix/json.scm: ... here. New file. * Makefile.am (MODULES): Add it. --- Makefile.am | 1 + guix/json.scm | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++ guix/swh.scm | 35 +--------------------------- 3 files changed, 65 insertions(+), 34 deletions(-) create mode 100644 guix/json.scm diff --git a/Makefile.am b/Makefile.am index 0e5ca02ed3..da3720e3a6 100644 --- a/Makefile.am +++ b/Makefile.am @@ -77,6 +77,7 @@ MODULES = \ guix/discovery.scm \ guix/git-download.scm \ guix/hg-download.scm \ + guix/json.scm \ guix/swh.scm \ guix/monads.scm \ guix/monad-repl.scm \ diff --git a/guix/json.scm b/guix/json.scm new file mode 100644 index 0000000000..d446f6894e --- /dev/null +++ b/guix/json.scm @@ -0,0 +1,63 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2018 Ludovic Courtès <ludo@HIDDEN> +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +(define-module (guix json) + #:use-module (json) + #:use-module (srfi srfi-9) + #:export (define-json-mapping)) + +;;; Commentary: +;;; +;;; This module provides tools to define mappings from JSON objects to SRFI-9 +;;; records. This is useful when writing bindings to HTTP APIs. +;;; +;;; Code: + +(define-syntax-rule (define-json-reader json->record ctor spec ...) + "Define JSON->RECORD as a procedure that converts a JSON representation, +read from a port, string, or hash table, into a record created by CTOR and +following SPEC, a series of field specifications." + (define (json->record input) + (let ((table (cond ((port? input) + (json->scm input)) + ((string? input) + (json-string->scm input)) + ((hash-table? input) + input)))) + (let-syntax ((extract-field (syntax-rules () + ((_ table (field key json->value)) + (json->value (hash-ref table key))) + ((_ table (field key)) + (hash-ref table key)) + ((_ table (field)) + (hash-ref table + (symbol->string 'field)))))) + (ctor (extract-field table spec) ...))))) + +(define-syntax-rule (define-json-mapping rtd ctor pred json->record + (field getter spec ...) ...) + "Define RTD as a record type with the given FIELDs and GETTERs, à la SRFI-9, +and define JSON->RECORD as a conversion from JSON to a record of this type." + (begin + (define-record-type rtd + (ctor field ...) + pred + (field getter) ...) + + (define-json-reader json->record ctor + (field spec ...) ...))) diff --git a/guix/swh.scm b/guix/swh.scm index 89cddb2bdd..c5f2153a22 100644 --- a/guix/swh.scm +++ b/guix/swh.scm @@ -23,6 +23,7 @@ #:use-module (web client) #:use-module (web response) #:use-module (json) + #:use-module (guix json) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) @@ -127,40 +128,6 @@ url (string-append url "/"))) -(define-syntax-rule (define-json-reader json->record ctor spec ...) - "Define JSON->RECORD as a procedure that converts a JSON representation, -read from a port, string, or hash table, into a record created by CTOR and -following SPEC, a series of field specifications." - (define (json->record input) - (let ((table (cond ((port? input) - (json->scm input)) - ((string? input) - (json-string->scm input)) - ((hash-table? input) - input)))) - (let-syntax ((extract-field (syntax-rules () - ((_ table (field key json->value)) - (json->value (hash-ref table key))) - ((_ table (field key)) - (hash-ref table key)) - ((_ table (field)) - (hash-ref table - (symbol->string 'field)))))) - (ctor (extract-field table spec) ...))))) - -(define-syntax-rule (define-json-mapping rtd ctor pred json->record - (field getter spec ...) ...) - "Define RTD as a record type with the given FIELDs and GETTERs, à la SRFI-9, -and define JSON->RECORD as a conversion from JSON to a record of this type." - (begin - (define-record-type rtd - (ctor field ...) - pred - (field getter) ...) - - (define-json-reader json->record ctor - (field spec ...) ...))) - (define %date-regexp ;; Match strings like "2014-11-17T22:09:38+01:00" or ;; "2018-09-30T23:20:07.815449+00:00"". -- 2.20.1
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at 33899) by debbugs.gnu.org; 28 Dec 2018 23:16:16 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:16:16 2018 Received: from localhost ([127.0.0.1]:40707 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1MW-0001Pp-3c for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:16 -0500 Received: from eggs.gnu.org ([208.118.235.92]:47527) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1MS-0001PS-I7 for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1ML-0002Be-Cf for 33899 <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:16:07 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:51203) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1MH-00026v-1g; Fri, 28 Dec 2018 18:16:01 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51692 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1MG-00077n-PH; Fri, 28 Dec 2018 18:16:00 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: 33899 <at> debbugs.gnu.org Subject: [PATCH 2/5] =?UTF-8?q?tests:=20'file=3D=3F'=20now=20recurses=20on?= =?UTF-8?q?=20directories.?= Date: Sat, 29 Dec 2018 00:15:51 +0100 Message-Id: <20181228231554.8220-2-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181228231554.8220-1-ludo@HIDDEN> References: <20181228231554.8220-1-ludo@HIDDEN> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33899 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.0 (------) * guix/tests.scm (not-dot?): New procedure. (file=?)[executable?]: New procedure. In 'regular case, check whether the executable bit is preserved. Add 'directory case. --- guix/tests.scm | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/guix/tests.scm b/guix/tests.scm index f4948148c4..c9ae2718e4 100644 --- a/guix/tests.scm +++ b/guix/tests.scm @@ -26,9 +26,12 @@ #:use-module (gcrypt hash) #:use-module (guix build-system gnu) #:use-module (gnu packages bootstrap) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-64) #:use-module (rnrs bytevectors) + #:use-module (ice-9 ftw) #:use-module (ice-9 binary-ports) #:use-module (web uri) #:export (open-connection-for-tests @@ -138,16 +141,31 @@ too expensive to build entirely in the test store." (loop (1+ i))) bv)))) +(define (not-dot? entry) + (not (member entry '("." "..")))) + (define (file=? a b) - "Return true if files A and B have the same type and same content." + "Return true if files A and B have the same type and same content, +recursively." + (define (executable? file) + (->bool (logand (stat:mode (lstat file)) #o100))) + (and (eq? (stat:type (lstat a)) (stat:type (lstat b))) (case (stat:type (lstat a)) ((regular) - (equal? - (call-with-input-file a get-bytevector-all) - (call-with-input-file b get-bytevector-all))) + (and (eqv? (executable? a) (executable? b)) + (equal? + (call-with-input-file a get-bytevector-all) + (call-with-input-file b get-bytevector-all)))) ((symlink) (string=? (readlink a) (readlink b))) + ((directory) + (let ((lst1 (scandir a not-dot?)) + (lst2 (scandir b not-dot?))) + (and (equal? lst1 lst2) + (every file=? + (map (cut string-append a "/" <>) lst1) + (map (cut string-append b "/" <>) lst2))))) (else (error "what?" (lstat a)))))) -- 2.20.1
guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.Received: (at submit) by debbugs.gnu.org; 28 Dec 2018 23:13:01 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 28 18:13:01 2018 Received: from localhost ([127.0.0.1]:40703 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1gd1JM-0001Jx-Do for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:13:01 -0500 Received: from eggsout.gnu.org ([209.51.188.92]:39980) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1gd1JK-0001Jh-T1 for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:12:59 -0500 Received: from lists.gnu.org ([208.118.235.17]:58459) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1JF-0005rK-EO for submit <at> debbugs.gnu.org; Fri, 28 Dec 2018 18:12:53 -0500 Received: from eggs.gnu.org ([208.118.235.92]:33661) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1J9-0000oR-VG for guix-patches@HIDDEN; Fri, 28 Dec 2018 18:12:53 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1J3-0005bq-Kh for guix-patches@HIDDEN; Fri, 28 Dec 2018 18:12:47 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:50661) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1gd1Ib-00052a-Gm; Fri, 28 Dec 2018 18:12:13 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51292 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1gd1Ib-0006mA-74; Fri, 28 Dec 2018 18:12:13 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN> To: guix-patches@HIDDEN Subject: [PATCH 0/5] Distributing substitutes over IPFS Date: Sat, 29 Dec 2018 00:12:05 +0100 Message-Id: <20181228231205.8068-1-ludo@HIDDEN> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit Cc: Hector Sanjuan <code@HIDDEN>, =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN>, Pierre Neidhardt <mail@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Hello Guix! Here is a first draft adding support to distribute and retrieve substitutes over IPFS. This builds on discussions at the R-B Summit with Héctor Sanjuan of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian Paul Schmidt posted on guix-devel last month. The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides bindings to a subset of that API. This module also implements a custom “directory” format to store directory trees in IPFS (IPFS already provides “UnixFS” and “tar” but they store too many or too few file attributes.) ‘guix publish’ and ‘guix substitute’ use (guix ipfs) to store and retrieve store items. Complete directory trees are stored in IPFS “as is”, rather than as compressed archives (nars). This allows for deduplication in IPFS. ‘guix publish’ adds a new “IPFS” field in narinfos and ‘guix substitute’ can then query those objects over IPFS. So the idea is that you still get narinfos over HTTP(S), and then you have the option of downloading substitutes over IPFS. I’ve pushed these patches in ‘wip-ipfs-substitutes’. This is rough on the edges and probably buggy, but the adventurous among us might want to give it a spin. :-) Thanks, Ludo’. Ludovic Courtès (5): Add (guix json). tests: 'file=?' now recurses on directories. Add (guix ipfs). publish: Add IPFS support. DRAFT substitute: Add IPFS support. Makefile.am | 3 + doc/guix.texi | 33 +++++ guix/ipfs.scm | 250 ++++++++++++++++++++++++++++++++++++ guix/json.scm | 63 +++++++++ guix/scripts/publish.scm | 67 +++++++--- guix/scripts/substitute.scm | 106 ++++++++------- guix/swh.scm | 35 +---- guix/tests.scm | 26 +++- tests/ipfs.scm | 55 ++++++++ 9 files changed, 535 insertions(+), 103 deletions(-) create mode 100644 guix/ipfs.scm create mode 100644 guix/json.scm create mode 100644 tests/ipfs.scm -- 2.20.1
Ludovic Courtès <ludo@HIDDEN>
:guix-patches@HIDDEN
.
Full text available.guix-patches@HIDDEN
:bug#33899
; Package guix-patches
.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.