GNU bug report logs - #51472
substitute servers should be preferred according to their coverage rate

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Reported by: Maxim Cournoyer <maxim.cournoyer@HIDDEN>; dated Fri, 29 Oct 2021 02:08:02 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 51472 <at> debbugs.gnu.org:


Received: (at 51472) by debbugs.gnu.org; 7 Nov 2021 15:11:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Nov 07 10:11:18 2021
Received: from localhost ([127.0.0.1]:54328 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mjjpF-0000Nn-Hb
	for submit <at> debbugs.gnu.org; Sun, 07 Nov 2021 10:11:17 -0500
Received: from eggs.gnu.org ([209.51.188.92]:42824)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1mjjpC-0000NV-Og
 for 51472 <at> debbugs.gnu.org; Sun, 07 Nov 2021 10:11:16 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47078)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mjjp7-0007fC-I1; Sun, 07 Nov 2021 10:11:09 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=nJSxpCipyFyDxVuK2ZXAJmhQPidnqevm8rVWfwgF+40=; b=I3XIZjXC3ifYv6+AOefZ
 +zhysSKzVPLzTHGC+cdDmhk1iWaMcYJuBWAzJJ45NWuwSYgvOIoooCqcZhZGX+eFfH937+yRC/3aG
 zZfV3UtymCIt5jqFdB5GW0lWlFp4U8Svsl4rqO8KnZOhf+998dQWZOOBuENxYauceQZleJEeTgwjH
 pPZruTqSKnVUBL3UnoZWV1ZcXWq0bVTOvX7qEY3ngSwGtMxcwrSRF8sx7PrlEMi38Bh6Dd4oL0yuh
 5LnwRGbRW5CCB6/G0eUk/cdqcZfXviPsvK1ik8QTB1cGkGPzwhIHdhMcZ40yS2Z56JIMCr5Ht8oSv
 mhQ0Y/L9QFSfDg==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:55538
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1mjjp5-0006mg-0v; Sun, 07 Nov 2021 10:11:09 -0500
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: Re: bug#51472: substitute servers should be preferred according to
 their coverage rate
References: <87k0hwa8ic.fsf@HIDDEN>
Date: Sun, 07 Nov 2021 16:11:04 +0100
In-Reply-To: <87k0hwa8ic.fsf@HIDDEN> (Maxim Cournoyer's message of "Thu, 28
 Oct 2021 22:07:39 -0400")
Message-ID: <874k8oknlj.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 51472
Cc: 51472 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> When using substitute servers discovery, I've noticed that if one of the
> substitute servers doesn't have any substitutes available, it'll keep
> getting tried instead of others, leading to a slide-show of substitutes
> updates such as:
>
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.t=
ar.xz.drv...
> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv=
...
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%

We=E2=80=99d need to check why this particular server is checked repeatedly.
The fact that it displays =E2=80=9C0.0%=E2=80=9D doesn=E2=80=99t mean that =
the server lacks
substitutes, but that it does not reply to =E2=80=98GET /xyz.narinfo=E2=80=
=99 requests,
for example because it=E2=80=99s off-line (see
<https://issues.guix.gnu.org/48808>.)

> We should implement some scheme to prefer querying high-substitute
> servers first, instead of wasting time querying servers always failed
> queries; this would greatly improve performance when using substitute
> discovery for example combined with low coverage.

There are several problems with that.  First one is that you can=E2=80=99t =
tell
what substitute coverage is until you=E2=80=99ve actually made those GET
requests.  Second one is that substitute coverage varies and it=E2=80=99s n=
ot an
absolute measure; for example, if a server provides substitutes for only
0.1% of all the packages, but that=E2=80=99s precisely the 0.1% you care ab=
out,
it=E2=80=99s more valuable than the one that has 99% of the packages but la=
cks
those you want.

There are other issues such as the fact that current semantics is to
respect the order of substitute URLs, which is presumably chosen by the
user according to their own criteria: download speed, bandwidth usage,
etc.

I hope this makes sense!

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#51472; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 29 Oct 2021 02:07:53 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Oct 28 22:07:52 2021
Received: from localhost ([127.0.0.1]:53609 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mgHJA-0003c3-Kd
	for submit <at> debbugs.gnu.org; Thu, 28 Oct 2021 22:07:52 -0400
Received: from lists.gnu.org ([209.51.188.17]:32940)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1mgHJ4-0003bp-Sh
 for submit <at> debbugs.gnu.org; Thu, 28 Oct 2021 22:07:51 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:55746)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1mgHJ4-0003ED-MG
 for bug-guix@HIDDEN; Thu, 28 Oct 2021 22:07:46 -0400
Received: from mail-qv1-xf31.google.com ([2607:f8b0:4864:20::f31]:46718)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <maxim.cournoyer@HIDDEN>)
 id 1mgHJ2-0007dl-1f
 for bug-guix@HIDDEN; Thu, 28 Oct 2021 22:07:46 -0400
Received: by mail-qv1-xf31.google.com with SMTP id g25so4410224qvf.13
 for <bug-guix@HIDDEN>; Thu, 28 Oct 2021 19:07:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=from:to:subject:date:message-id:mime-version;
 bh=jDX6pF4VlRqdEYqFOjhNwm9yIQZVztQq9HJ4+7FhTY0=;
 b=cCJQ7QD2bPlZJCigqWcvf1uK3NkueO5biqBJFWSPZOW+4CxyKZmU6cihsq25nhF09m
 aRGIPnSFWkfsqo1xo20Xo8+AwCP5Oby30+xesGkyg1zvWKVM2cYLCJ2L6AZFKHCmq2Xp
 Z19W8TEqbfR8HlbzDg9SBn4f8vLeX7oRl7Q2qmObuOlwGDqcaW3nwdeS4YrIfcLuqBIy
 rYx293yqHBWVR3glEc5xCRvVgrNhpWRpNC0HWzwsh9scCd+vIJat8pfPUUzfM/HkdltO
 EtR0ljlOu4t+NaJBQ/wlG2wgDYRL6KKls4eiJf8dkP4CYRfEy29uo9VvRIEGqbfaq93Y
 FA/A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:from:to:subject:date:message-id:mime-version;
 bh=jDX6pF4VlRqdEYqFOjhNwm9yIQZVztQq9HJ4+7FhTY0=;
 b=SiaOqQN0RkQrkTznlZLSM3z4Ab1V6VURPylEFwiO0+9iE77Gx5/Lj8gbfL1GpOaMqB
 uUyLBwg9zWt2dQNQ3+NSC0uqczx7Yr/NiHm5rmq/thciD4fEY05wcopZvelFwTvWbCfP
 JAZn83sPLqd/XEM7OTUmHAGl0Q2eVmDT9NncV+eV2a7OkH3tFjvcyPMtVJf//ozuaYuE
 1XcqOUH3Pf5znnVuWTfjQOqaqw2BHnzW6Y35yobSS189vhboyRbD2Iy13ge5+PuEc3lE
 sf1nsTSzjIu5GIih+clY2SWezUpA7QqWDC2WzZCOcb0obvi7lYt0Ux+dhdDegZMvLbIB
 E7sA==
X-Gm-Message-State: AOAM533Ne4Evvsz2EB1i/Gt49f3y1+UpjWc2c7AWKepf0FOf5slupseq
 5CVX2ERVEq0wEwt+5CREMO0OknNG/U8=
X-Google-Smtp-Source: ABdhPJymXKBEHEH+xyRHBBrWsQKcVMqgZIayCHNOHDeXSKB5bfMVlsBv7LIVK83LftRJDBZAFfm5pA==
X-Received: by 2002:ad4:5e8d:: with SMTP id jl13mr8085434qvb.61.1635473262633; 
 Thu, 28 Oct 2021 19:07:42 -0700 (PDT)
Received: from hurd (dsl-152-69.b2b2c.ca. [66.158.152.69])
 by smtp.gmail.com with ESMTPSA id bj37sm3107706qkb.49.2021.10.28.19.07.41
 for <bug-guix@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Thu, 28 Oct 2021 19:07:41 -0700 (PDT)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
To: bug-guix <bug-guix@HIDDEN>
Subject: substitute servers should be preferred according to their coverage
 rate
Date: Thu, 28 Oct 2021 22:07:39 -0400
Message-ID: <87k0hwa8ic.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=2607:f8b0:4864:20::f31;
 envelope-from=maxim.cournoyer@HIDDEN; helo=mail-qv1-xf31.google.com
X-Spam_score_int: -8
X-Spam_score: -0.9
X-Spam_bar: /
X-Spam_report: (-0.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 NUMERIC_HTTP_ADDR=1.242, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello,

When using substitute servers discovery, I've noticed that if one of the
substitute servers doesn't have any substitutes available, it'll keep
getting tried instead of others, leading to a slide-show of substitutes
updates such as:

--8<---------------cut here---------------start------------->8---
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'...   0.0%
--8<---------------cut here---------------end--------------->8---

We should implement some scheme to prefer querying high-substitute
servers first, instead of wasting time querying servers always failed
queries; this would greatly improve performance when using substitute
discovery for example combined with low coverage.

Thanks!

Maxim




Acknowledgement sent to Maxim Cournoyer <maxim.cournoyer@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#51472; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sun, 7 Nov 2021 15:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.