GNU bug report logs - #46800
[PATCH] Allow defining multiple substituters

Previous Next

To reply to this bug, email your comments to 46800 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH] Allow defining multiple substituters
Date: Fri, 26 Feb 2021 18:41:51 +0100

[Message part 1 (text/plain, inline)]

Hi Guix,

This patch series is my suggestion for allowing
multiple "substitution methods" or "substituters"
as I call them.  Currently, only a method for HTTP/S
is defined, though I hope it will be a good basis
for a common framework for substitutes over GNUnet
and IPFS.

Extending "guix-service-type" to allow configuration
of substitution method is left for later.

Any questions, remarks?

Greetings,
Maxime

[0001-substitute-implement-a-hook-mechanism-for-defining-s.patch (text/x-patch, attachment)]

[0002-doc-Document-how-to-define-new-substituters.patch (text/x-patch, attachment)]

[0003-daemon-Set-the-substitute-methods-option.patch (text/x-patch, attachment)]

[0004-substitute-Unstub-verify-hash-unknown.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 46800 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 46800 <at> debbugs.gnu.org
Subject: Re: bug#46800: [PATCH] Allow defining multiple substituters
Date: Tue, 02 Mar 2021 21:37:36 +0100

Hi Maxime,

Maxime Devos <maximedevos <at> telenet.be> skribis:

> This patch series is my suggestion for allowing
> multiple "substitution methods" or "substituters"
> as I call them.  Currently, only a method for HTTP/S
> is defined, though I hope it will be a good basis
> for a common framework for substitutes over GNUnet
> and IPFS.

Thanks for working on this!

As discussed on IRC, the daemon used to have support for multiple
substituters, but as a built-in C++ interface, which I removed in
f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.

The Scheme interface you propose is of course nicer :-), but I’m still
not sure it’s necessary.  For example, in the IPFS prototype at
<https://issues.guix.gnu.org/33899>, IPFS support goes hand in hand with
HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
over IPFS, or HTTP.  Likewise with “digests”:
<https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html>.

Another issue is that it may be that, instead of letting users choose
one method and stick to it, we’d rather let them choose a policy that
can automatically pick the “best” method, dynamically adjusting choices.

All in all, I would prefer to wait until there’s a clear need for this
abstraction.

WDYT?

Thanks,
Ludo’.

Message #11 received at 46800 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 46800 <at> debbugs.gnu.org
Subject: Re: bug#46800: [PATCH] Allow defining multiple substituters
Date: Thu, 04 Mar 2021 08:48:44 +0100

[Message part 1 (text/plain, inline)]

On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:
> Hi Maxime,
> 
> Maxime Devos <maximedevos <at> telenet.be> skribis:
> 
> > This patch series is my suggestion for allowing
> > multiple "substitution methods" or "substituters"
> > as I call them.  Currently, only a method for HTTP/S
> > is defined, though I hope it will be a good basis
> > for a common framework for substitutes over GNUnet
> > and IPFS.
> 
> Thanks for working on this!
> 
> As discussed on IRC, the daemon used to have support for multiple
> substituters, but as a built-in C++ interface, which I removed in
> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.

Was there any particular reason this support was removed, beyond
moving from C++ to Scheme and the absence of any alternative substituters?

> The Scheme interface you propose is of course nicer :-), but I’m still
> not sure it’s necessary.  For example, in the IPFS prototype at
> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
> over IPFS, or HTTP.

About X going hand-in-hand with Y:

Note that fetching narinfos, or fetching the nar itself are separated
A method can support both procedures, or just one of them (or none,
but that's rather useless.)

Users (well, the system administrator) can choose multiple methods, which
will be each fetch narinfos after each other & combine the
results into
one large list (or maybe some other data structure, I don't recall the
details), and each substituter will be asked to produce
a nar until a substituter
succeeds or all have said "sorry, I don't have that nar".

(That's different from C++ interface for multiple substituters I think, where
the methods are only tried sequentialy, they aren't combined.)

In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
enabled, in that order: "--substitute-methods=ipfs http".  The IPFS substituter
won't be able to produce any narinfos by itself, but that's no problem as
the HTTP substituter can find some.  Then, the IPFS substituter will be asked
first to download a substitute, as it's first in the "--substitute-methods" list.

And what if the narinfo doesn't have a IPFS URI, as the substitute server doesn't
support that?  Then "guix substitute" automatically fall-backs to HTTP.

Summary: some substitution methods can't do everything on their own, but that's ok,
as "guix substitute" will just ask them to try what they can and will see if some
combination of methods works.

About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
at some point.  I suppose it would be possible to define all substitute methods
in (guix scripts substitute), but then you would still end up with a procedure
that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
an "if" structures that tries downloading via IPFS with fall-back to HTTP; this
would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))

Note that there's (guix scripts import X) and (guix build-system X).

> Likewise with “digests”: <https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html>;.

I haven't taken a close look at this yet before (I haven't been around guix
development for long).  To me, this seems compatible with this patch actually.
The HTTP substituter's procedure for downloading the substitute itself
(process-substitution/http in my patch) could be split in two, and look
at the narinfo to see whether the 'digest' or the usual mechanism should be used.

Alternatively, one could define *two* substituters: the ‘standard’ http substituter
‘http’, and the ‘http-digest’ substituter that can't fetch narinfo's, but rather
is an alternative method for downloading the substitute.  The daemon can be started
with "--substitute-methods http-digest http" to prefer downloading via the ‘http-digest’
method when possible, but uses ‘http’ for the narinfos and as a fallback for when the
narinfo does not have a digest.

But what if a non-HTTP substituter wants to use digests?  Well, I don't know any such
substituters (-:.  But for the (hypothetical) GNUnet substituter & the wip IPFS
substituter, I don't think they will use the digests code.

> Another issue is that it may be that, instead of letting users choose
> one method and stick to it,

They (at least the system administrator) can choose a list of substituters,
see above.

>  we’d rather let them choose a policy that
> can automatically pick the “best” method, dynamically adjusting choices.

Who's the user here?
(a) the system administrator, who configuring the daemon to use a certain
    list of substituters and defines a default list of substitute uris.
(b) the ‘user’, that doesn't directly have the capability to modify
    the system's guix daemon (or possibly an administrator that wants to
    to test some things out without the possibility of accidentally messing
    up the ‘real’ system).

If (b), I think it would be ideal to give the (unprivileged) user the
possibility of using their own substituter(s) (under their own capabilities,
not root), albeit at the cost of the guix daemon having to verify the narhash
& narinfo signature.

That could be implemented as a separate patch (though this patch would need
to be rebased then).  WDYT?  Would be useful for developing new substituters
and testing them, I think.

About *automatically* dynamically adjusting choices: would be nice, but how is
this supposed to work?  Any ideas?  The only thing I could think of is a
allowing the user to choose which narinfo to use (e.g. from the list of found
narinfos try to choose a narinfo that has an IPFS URI).

Also, for (a) the shepherd service could use a "set-substitute-methods" option,
and perhaps the user (b) could be allowed to select a subset of these substitute
methods to use when running "guix build PACKAGE" and the like (but only a subset,
as "guix substitute" when invoked by the daemon runs as root and therefore the
potential attack surface shouldn't be increased beyond what the administrator
allows).

> All in all, I would prefer to wait until there’s a clear need for this
> abstraction.

See above responses.

WDYT?

Thanks,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 46800 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 46800 <at> debbugs.gnu.org
Subject: Re: bug#46800: [PATCH] Allow defining multiple substituters
Date: Fri, 05 Mar 2021 21:05:53 +0100

[Message part 1 (text/plain, inline)]

On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:
> Hi Maxime,
> 
> Maxime Devos <maximedevos <at> telenet.be> skribis:
> 
> > This patch series is my suggestion for allowing
> > multiple "substitution methods" or "substituters"
> > as I call them.  Currently, only a method for HTTP/S
> > is defined, though I hope it will be a good basis
> > for a common framework for substitutes over GNUnet
> > and IPFS.
> 
> [Ludovic's reply]

(See previous mail for my responses)

FYI: I've implemented a GNUnet substituter using this patch series
and the "publish hooks" patch (+ an unsubmitted patch that passes
some extra information to the publish hook) here:

https://notabug.org/mdevos/guix-gnunet/src/download-hooks3

(Warning: it does some questionable things with add-to-load-path.
Will hopefully be fixed eventually.  Also requires
<https://notabug.org/mdevos/scheme-gnunet> in a special location.)

Also, there's a bug in fetch-narinfos that causes an error if
the "fetch-narinfos" field of a subtituter is #f.  Also,
recognised-uri-scheme should be removed or reworked, as otherwise
the IPFS and GNUnet substituter won't be used for downloading a
substitute if http and https are not in the list.

I've worked around that for now by setting the latter '(http https file),
and setting the former to (const '()).

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 46800 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 46800 <at> debbugs.gnu.org
Subject: Re: bug#46800: [PATCH] Allow defining multiple substituters
Date: Fri, 12 Mar 2021 18:37:53 +0100

Hi Maxime,

Maxime Devos <maximedevos <at> telenet.be> skribis:

> On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:

[...]

>> As discussed on IRC, the daemon used to have support for multiple
>> substituters, but as a built-in C++ interface, which I removed in
>> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.
>
> Was there any particular reason this support was removed, beyond
> moving from C++ to Scheme and the absence of any alternative substituters?

These were the main reasons, yes.

>> The Scheme interface you propose is of course nicer :-), but I’m still
>> not sure it’s necessary.  For example, in the IPFS prototype at
>> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
>> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
>> over IPFS, or HTTP.
>
> About X going hand-in-hand with Y:
>
> Note that fetching narinfos, or fetching the nar itself are separated
> A method can support both procedures, or just one of them (or none,
> but that's rather useless.)
>
> Users (well, the system administrator) can choose multiple methods, which
> will be each fetch narinfos after each other & combine the
> results into
> one large list (or maybe some other data structure, I don't recall the
> details), and each substituter will be asked to produce
> a nar until a substituter
> succeeds or all have said "sorry, I don't have that nar".

OK.

> (That's different from C++ interface for multiple substituters I think, where
> the methods are only tried sequentialy, they aren't combined.)
>
> In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
> enabled, in that order: "--substitute-methods=ipfs http".  The IPFS substituter
> won't be able to produce any narinfos by itself, but that's no problem as
> the HTTP substituter can find some.  Then, the IPFS substituter will be asked
> first to download a substitute, as it's first in the "--substitute-methods" list.
>
> And what if the narinfo doesn't have a IPFS URI, as the substitute server doesn't
> support that?  Then "guix substitute" automatically fall-backs to HTTP.
>
> Summary: some substitution methods can't do everything on their own, but that's ok,
> as "guix substitute" will just ask them to try what they can and will see if some
> combination of methods works.

Alright.

> About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
> at some point.  I suppose it would be possible to define all substitute methods
> in (guix scripts substitute), but then you would still end up with a procedure
> that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
> an "if" structures that tries downloading via IPFS with fall-back to HTTP; this
> would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))

I guess considerations that are more important to me (and to users, I
suppose) now than a few years back are maintainability and robustness.

Concretely, I wouldn’t want Guix to offer out of the box 4 methods, 3 of
which perform poorly or are downright buggy.  I think it would be more
fruitful if, as a project, we would focus on one or two methods or
method combinations that we have battle-tested, perform well, and a nice
long-term maintenance story, and so on.

[...]

>>  we’d rather let them choose a policy that
>> can automatically pick the “best” method, dynamically adjusting choices.
>
> Who's the user here?
> (a) the system administrator, who configuring the daemon to use a certain
>     list of substituters and defines a default list of substitute uris.
> (b) the ‘user’, that doesn't directly have the capability to modify
>     the system's guix daemon (or possibly an administrator that wants to
>     to test some things out without the possibility of accidentally messing
>     up the ‘real’ system).

I think (b) should be possible, just like users can pass
‘--substitute-urls’.

[...]

> About *automatically* dynamically adjusting choices: would be nice, but how is
> this supposed to work?  Any ideas?  The only thing I could think of is a
> allowing the user to choose which narinfo to use (e.g. from the list of found
> narinfos try to choose a narinfo that has an IPFS URI).

I think it’ll have to be fine-tuned once we have several stable
substitute methods.  After all, we have yet to figure out how to choose
between zstd and lzip for the current substitution mechanism; the
tradeoffs when very different methods are in use may be more complex!

>> All in all, I would prefer to wait until there’s a clear need for this
>> abstraction.
>
> See above responses.

I don’t think my concerns are really addressed :-), but at the same time
I think we need a playground for these things so we can actually grow
new substitute methods like those you’ve been looking at.  Hmmm tricky!

Ludo’.

Message #20 received at 46800 <at> debbugs.gnu.org (full text, mbox):

From: Tony Olagbaiye <me <at> fron.io>
To: "46800 <at> debbugs.gnu.org" <46800 <at> debbugs.gnu.org>
Subject: RE:  [PATCH] Allow defining multiple substituters
Date: Sun, 06 Jun 2021 17:52:40 +0000

[Message part 1 (text/plain, inline)]

Hi, any news on this patch?

Thanks,
ix

[Message part 2 (text/html, inline)]

[publickey - me@fron.io - 0x3026807C.asc (application/pgp-keys, attachment)]

[signature.asc (application/pgp-signature, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.