Received: (at 65787) by debbugs.gnu.org; 11 Sep 2023 11:24:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 07:24:16 2023
Received: from localhost ([127.0.0.1]:52244 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1qff1b-0007RS-J8
for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:24:16 -0400
Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]:38254)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1qff1X-0007R8-P2
for 65787 <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:24:13 -0400
Received: by mail-wm1-x335.google.com with SMTP id
5b1f17b1804b1-401ef656465so13773925e9.1
for <65787 <at> debbugs.gnu.org>; Mon, 11 Sep 2023 04:24:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20221208; t=1694431442; x=1695036242; darn=debbugs.gnu.org;
h=content-transfer-encoding:mime-version:message-id:date:references
:in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
:reply-to; bh=X0/MgpbFzRUKOjghRU1in+s6M7paHhXd/XPRkQf1kKI=;
b=hzJya5EaU8FZZvQdAhV9upyF17pWcjb1lU1Fmn+0LNFUihFgxMDA/YMXWAbN5apGGP
sbv5lben1W8SEXBto+edSavjb+aeTgoMEgoyWx0ln+7der8zs50inApCvdbWTa5qrSqR
44QIcv0/ik1+R3/+jyCnAFDNwNgon8IkpI0S5S+vNiSH6qp8Z+ypGrHb/7P2HyI3dCIZ
SrzQ8rT/6wwXrqyinx9uk2j0rarmEoAqNgE1uUG8CAVKUg9aRWiFL5LvnzZrmM5oDyqo
9nO0EY/aERE2hMPoeDVAm2whKJjlVBfEkYiIjS0bZkI8aglhDbfkPqnoMAyL7U1h03bt
+/0w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1694431442; x=1695036242;
h=content-transfer-encoding:mime-version:message-id:date:references
:in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=X0/MgpbFzRUKOjghRU1in+s6M7paHhXd/XPRkQf1kKI=;
b=aALBA9q9UzTktInBY8DYgKosmpcOaOhpNXLtxd5zllQtA9CVz2CVYMivNbBiymuRi5
KuzK0WtlCt9N21cgXNPjQ5874oEQBlkQh16cKPxAjtO+5b5FVhow8PUo7BuCRJpfFEgU
gAaXyzemaZyklht4AZkybT8ZeSgvauIlkMk9uZbfhPnXP03fVfbcD3t0mx96zrKyxx8N
TFUCq3v6p/q+o/MYiTnpSOgflCW8DqQk/CmFmmGQz7i0aIDyGdD5Pyth2omVe2WkS/gL
3nwEAIvkXh/yVtLJ/ewhVnPng4Ip155pbWgdbUZ2u8FkBWQ5ekDvEsqXx0aYagMUljAe
uXUQ==
X-Gm-Message-State: AOJu0YzL69xeyHEJZ6QNHlvBqyLibMGExe5/lQGuJL9Lj4jkr7KVkysg
49dXuAE0UKV3pEfniLLzbbttqd7mF+A=
X-Google-Smtp-Source: AGHT+IEQ+qg85YX58lNzGeby/v7ORult2SMP7JhXK5EYTirMALC6Vq6hTP0XN/FEl7c5ZdKRkWvPoQ==
X-Received: by 2002:a05:6000:3c4:b0:31f:a717:f1cc with SMTP id
b4-20020a05600003c400b0031fa717f1ccmr2363727wrg.5.1694431441633;
Mon, 11 Sep 2023 04:24:01 -0700 (PDT)
Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id
q13-20020a05600000cd00b0031ad2f9269dsm9779316wrx.40.2023.09.11.04.24.01
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Mon, 11 Sep 2023 04:24:01 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#65787: time-machine is doing too much network requests
In-Reply-To: <87zg1tvlfn.fsf@HIDDEN>
References: <87wmx3mfo5.fsf@HIDDEN> <87zg1tvlfn.fsf@HIDDEN>
Date: Mon, 11 Sep 2023 11:41:54 +0200
Message-ID: <87tts1jbbx.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 65787
Cc: 65787 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Hi Ludo,
> Maybe this is the purpose of your message:
> reducing Git remote accesses in those cases?=20
Yes. :-)
On Sun, 10 Sep 2023 at 22:10, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
> It would probably be easier if you could come with
> an example where there=E2=80=99s Git-related network traffic where there
> shouldn=E2=80=99t.
Do we agree that the both Guile-Git procedures =E2=80=99remote-fetch=E2=80=
=99 and
=E2=80=99branch-lookup=E2=80=99 are doing network traffic?
If yes, here two examples:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix pull -q -p /tmp/new
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
;;; (remote-fetch NETWORK)
;;; (branch-lookup NETWORK)
Authenticating channel 'guix', commits 9edb3f6 to 2eb6df5 (83 new commits).=
..
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 2eb6df5
--8<---------------cut here---------------end--------------->8---
Well, that=E2=80=99s not perfect=E2=80=A6 it becomes much worse:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix time-machine -q -- describe
;;; (remote-fetch NETWORK)
;;; (branch-lookup NETWORK)
;;; (remote-fetch NETWORK)
;;; (branch-lookup NETWORK)
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
;;; (remote-fetch NETWORK)
;;; (branch-lookup NETWORK)
building /gnu/store/z8jwdgr7z6i8c00msdm2kzfv0n3zp25v-module-import-compiled=
.drv...
--8<---------------cut here---------------end--------------->8---
Do we agree that is suboptimal?
> =E2=80=98cached-channel-instance=E2=80=99 has 3 cases:
>
> 1. Obvious cache hit: This is when CHANNELS specifies the commit of
> each target channel (this happens for example when you type =E2=80=
=98guix
> time-machine -q --commit=3Da4c35c607cfd7d6b0bad90cfcc46188d489e1754)
> *and* the combination of channels is already in
> ~/.cache/guix/inferiors. This is the optimal case: the Git repo
> doesn=E2=80=99t even need to be opened.
Yes.
> 2. Cache hit: CHANNELS are pinned, but refer to tags (like =E2=80=9Cv1.=
2.0=E2=80=9D)
> or short commit IDs (like =E2=80=9Ca4c35c6=E2=80=9D). In that case,
> =E2=80=98channel-full-commit=E2=80=99 opens the Git repo to resolve =
the identifier.
> After that, we go to case #1 or #4.
That=E2=80=99s suboptimal. Currently, it reads:
--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 28 sept. 06 2023 14:54:50 (current)
guix 6113e05
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: 6113e0529d61df7425f64e30a6bf77f7cfdfe5a5
$ ./pre-inst-env guix time-machine -q --commit=3D6113e05 -- describe
;;; (remote-fetch NETWORK)
;;; (remote-fetch NETWORK)
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
;;; (remote-fetch NETWORK)
Computing Guix derivation for 'x86_64-linux'... | C-c C-c
--8<---------------cut here---------------end--------------->8---
Using patch from #65352 [1], it removes these useless traffic:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix time-machine -q --commit=3D6113e05 -- describe
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
Computing Guix derivation for 'x86_64-linux'... | C-c C-c
--8<---------------cut here---------------end--------------->8---
Using the proposed patch, it becomes optimal, IMHO. Well, let discuss
it in #65352 =E2=80=93 comments are welcome. :-)
> 3. Cache hit: CHANNELS are not pinned=E2=80=94i.e., they refer to a bra=
nch,
> not a commit. In that case we first need to perform a =E2=80=98git =
fetch=E2=80=99
> and then we go to #1 or #4.
I hope that I am convincing you that this case is suboptimal: it does 3
=E2=80=99git fetch=E2=80=99 and more 3 others lookup requiring network. So=
it is about
6 network access when only one is necessary, IMHO.
> 4. Cache miss: =E2=80=98reference-available?=E2=80=99 returns #f for th=
e channel
> commits, we got through =E2=80=98remote-fetch=E2=80=99 followed by
> =E2=80=98build-derivations=E2=80=99.
This case is part of #2 and discussed in #65352, IMHO.
Cheers,
simon
1: [bug#65352] [PATCH v2] DRAFT git: Avoid touching the network unless need=
ed in 'reference-available?'.
Simon Tournier <zimon.toutoune@HIDDEN>
Wed, 06 Sep 2023 16:17:08 +0200
id:32d3fb5066e0b20e200dabef0fba897634e21dda.1694009405.git.zimon.toutoune@g=
mail.com
https://yhetil.org/guix/32d3fb5066e0b20e200dabef0fba897634e21dda.1694009405=
.git.zimon.toutoune@HIDDEN
https://issues.guix.gnu.org/msgid/32d3fb5066e0b20e200dabef0fba897634e21dda.=
1694009405.git.zimon.toutoune@HIDDEN
bug-guix@HIDDEN:bug#65787; Package guix.
Full text available.
Received: (at 65787) by debbugs.gnu.org; 10 Sep 2023 20:10:51 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Sep 10 16:10:51 2023
Received: from localhost ([127.0.0.1]:51448 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1qfQlf-0008CA-2k
for submit <at> debbugs.gnu.org; Sun, 10 Sep 2023 16:10:51 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:56114)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <ludo@HIDDEN>) id 1qfQlc-0008Bw-F5
for 65787 <at> debbugs.gnu.org; Sun, 10 Sep 2023 16:10:49 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
id 1qfQlT-0008C8-HW; Sun, 10 Sep 2023 16:10:39 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
From; bh=6cGJEchu8Ozkgc4220loVlhPJzJbXsW6nST188D27FU=; b=TJaolcZQLathJBqhwaiA
gbv22SBz6lFULy2uJvyVUgMQh7lfwFVBSX9UU5B2e3N6OnWf19ntBwQM6ltGNWpJXgUko/6ndSzSR
nz6R5WeKeFGCeYsgTvMcvAO+aRaiuRRhNYrYCfPskxx4IFY/N31sff88Njgm53QGu0vbG2VuA85hR
8tYtN7TrNjVk1jHgWPBILkO4MZj6Vw5u9CJqGqjI0XCnDRfDP1Z1O6a3KsmHNPa0IXkzZ90Iob9l/
+mHM6fMRmWf4Fjq8VCpUhY2v+V5CH0ifGG4yRU4ZyV8b+i3igLXdacAsJK6D08/7eJ0Md+6i5dpWe
rg3nPFIX8jygCw==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Simon Tournier <zimon.toutoune@HIDDEN>
Subject: Re: bug#65787: time-machine is doing too much network requests
References: <87wmx3mfo5.fsf@HIDDEN>
Date: Sun, 10 Sep 2023 22:10:36 +0200
In-Reply-To: <87wmx3mfo5.fsf@HIDDEN> (Simon Tournier's message of "Wed, 06
Sep 2023 18:26:18 +0200")
Message-ID: <87zg1tvlfn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65787
Cc: 65787 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hello Simon,
This is a long message; I agree with the intent (avoiding network
traffic when the required commit is already in cache), but I=E2=80=99m not =
sure
about the analysis. It would probably be easier if you could come with
an example where there=E2=80=99s Git-related network traffic where there
shouldn=E2=80=99t.
Let me give some perspective on what the code intends to do.
=E2=80=98cached-channel-instance=E2=80=99 has 3 cases:
1. Obvious cache hit: This is when CHANNELS specifies the commit of
each target channel (this happens for example when you type =E2=80=98g=
uix
time-machine -q --commit=3Da4c35c607cfd7d6b0bad90cfcc46188d489e1754)
*and* the combination of channels is already in
~/.cache/guix/inferiors. This is the optimal case: the Git repo
doesn=E2=80=99t even need to be opened.
2. Cache hit: CHANNELS are pinned, but refer to tags (like =E2=80=9Cv1.2.=
0=E2=80=9D)
or short commit IDs (like =E2=80=9Ca4c35c6=E2=80=9D). In that case,
=E2=80=98channel-full-commit=E2=80=99 opens the Git repo to resolve th=
e identifier.
After that, we go to case #1 or #4.
3. Cache hit: CHANNELS are not pinned=E2=80=94i.e., they refer to a branc=
h,
not a commit. In that case we first need to perform a =E2=80=98git fe=
tch=E2=80=99
and then we go to #1 or #4.
4. Cache miss: =E2=80=98reference-available?=E2=80=99 returns #f for the =
channel
commits, we got through =E2=80=98remote-fetch=E2=80=99 followed by
=E2=80=98build-derivations=E2=80=99.
As with all caches, what matters is to make sure case #1 is processes as
efficiently as possible. I believe it=E2=80=99s the case since the Git rep=
o is
not even opened.
Of course it would be nice to speed up #2 and #3 too (as long as it=E2=80=
=99s
not at the expense of #1). Maybe this is the purpose of your message:
reducing Git remote accesses in those cases? (Apologies, I just
realized this might have been what you had in mind. :-))
Thanks,
Ludo=E2=80=99.
bug-guix@HIDDEN:bug#65787; Package guix.
Full text available.
Received: (at submit) by debbugs.gnu.org; 6 Sep 2023 16:26:39 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Sep 06 12:26:39 2023
Received: from localhost ([127.0.0.1]:37238 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1qdvMV-0006cO-4P
for submit <at> debbugs.gnu.org; Wed, 06 Sep 2023 12:26:39 -0400
Received: from lists.gnu.org ([2001:470:142::17]:39894)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <zimon.toutoune@HIDDEN>) id 1qdvMR-0006c9-4U
for submit <at> debbugs.gnu.org; Wed, 06 Sep 2023 12:26:38 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <zimon.toutoune@HIDDEN>)
id 1qdvMK-0003GL-EF
for bug-guix@HIDDEN; Wed, 06 Sep 2023 12:26:28 -0400
Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <zimon.toutoune@HIDDEN>)
id 1qdvMH-0001mg-TD
for bug-guix@HIDDEN; Wed, 06 Sep 2023 12:26:28 -0400
Received: by mail-wm1-x329.google.com with SMTP id
5b1f17b1804b1-401d61e9fecso136975e9.0
for <bug-guix@HIDDEN>; Wed, 06 Sep 2023 09:26:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20221208; t=1694017584; x=1694622384; darn=gnu.org;
h=content-transfer-encoding:mime-version:message-id:date:subject:to
:from:from:to:cc:subject:date:message-id:reply-to;
bh=9zptltsTTkXLwFc/tMyTfKfrxto0K8jnUb/771Bd28I=;
b=NImgWX8s11Wtvl6uvhiB+DJdg9LbqQ5Ug6z3OS4+i3xif+xGhGb3MDHNUjhWcG+d+U
jj4eHOxb5hxV3NCVD6rcH+Nk2rp6P0q0zC6Xw/ZAd3sKJpbdOB15K/XIwYUTymtqK8FF
uk2wjiJpYZOAiP4KzJZ5ZJqHKm5//80YWdIv4bF0GZx6mWdCG7UTBAFCbPFs4uirj7E9
oNlQxeTJH/BKuNRsyFwp2UkEmAasCPb+3kqY8KXJ81bCIwJQ+/No4pLudpRYHLmEStd/
nCO/D72WsUDLGi/s+N23ANeNp13RPrCYEHG90elI2WxBNirtFw8NwJV6qgZTganGmt1i
1Ctg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20221208; t=1694017584; x=1694622384;
h=content-transfer-encoding:mime-version:message-id:date:subject:to
:from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
bh=9zptltsTTkXLwFc/tMyTfKfrxto0K8jnUb/771Bd28I=;
b=NibgwUkhd8wlhmOk0irV0N+KPlI6qz0xbMCnuUmzbTSoG6qBzlAOzZxBxKGk5SOHNx
jOk2Wp/meFpDKiM1tDLKj6dFvY0277OkEyshw5SyqJGJv1iup439gTSERAvxjy/HSvgP
KyjrKXQE7IwNS39J1VjadtKi9UGLCLFc8q+YTXlrZ1oiAzUODUS/6Hnxn9oSXkCYlhEZ
6T063OZfrkqGoFOca8nM67fKvbYUCuO8RZsTSILcqDyb0ufr1eo1TvluTtQu+6h6e22u
KeD/dHqBU8R6rNdKUpDjj2bPc3t4/OqTN9fD6D+qw3TjDdff2kjKs8XtHd3LvkyUNWJQ
rgZg==
X-Gm-Message-State: AOJu0Yx0S3io0h6sqKwOieXFQlAC3c+zHh8mhOXmpfQHhJk/bh/vBdHt
wDupMRvMHgaiIX3e/wradT9FdBgpPvs=
X-Google-Smtp-Source: AGHT+IEtsAIihGaYSh+4jpPJc06NDoBNGJjpLaZbohaECNLJJvm2nxPqEqWn/EM2F9RIK2L5C/x+uA==
X-Received: by 2002:adf:d08d:0:b0:317:3d36:b2c1 with SMTP id
y13-20020adfd08d000000b003173d36b2c1mr12102068wrh.7.1694017583833;
Wed, 06 Sep 2023 09:26:23 -0700 (PDT)
Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id
x9-20020adfdd89000000b003196b1bb528sm21052494wrl.64.2023.09.06.09.26.22
for <bug-guix@HIDDEN>
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 06 Sep 2023 09:26:23 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: bug-guix@HIDDEN
Subject: time-machine is doing too much network requests
Date: Wed, 06 Sep 2023 18:26:18 +0200
Message-ID: <87wmx3mfo5.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=2a00:1450:4864:20::329;
envelope-from=zimon.toutoune@HIDDEN; helo=mail-wm1-x329.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)
Hi,
Well, I am in a quest to make Guix more robust for the worst-case
scenario: Savannah is unreachable (as well as other servers). For
context, it matters when using Guix for reproducing scientific
production. An example of such worst-case, see [1]. Well, this quote:
The first annoyance is that guix time-machine needs an access to the
server git.savannah.gnu.org, although the Git repository is already
cloned and already contains the required commit.=20
is almost tackled by #65352; at least tracked. :-)
Investigating that, I am noticed that the current design is suboptimal,
IMHO. I am reporting here and I hope to improve the situation by
reducing the number of network requests.
It matters in worst-case scenario of scientific production. And it also
matters for people with poor or unstable network link.
Sorry if the report is hard to follow, I did my best for being clear.
To keep the discussion simple, I only consider the Git reference
specification =E2=80=99branch=E2=80=99 and =E2=80=99tag-or-commit=E2=80=99.=
These Git reference
specification that various internal procedures are using is poorly
documented. See the docstring of the procedure =E2=80=99update-cached-chec=
kout=E2=80=99
from (guix git) for an idea or the implementation of =E2=80=99resolve-refer=
ence=E2=80=99
for the complete list.
Let consider only the Git reference specifications:
(branch . "string")
(tag-or-commit . "string")
because that are what =E2=80=9Cguix time-machine=E2=80=9D sets from the CLI=
or reads
from channels.scm files, IIUC.
The command =E2=80=9Cguix time-machine=E2=80=9D starts to call =E2=80=99cac=
hed-channel-instance=E2=80=99
passing as argument the procedure =E2=80=99validate-guix-channel=E2=80=99.
This procedure =E2=80=99cached-channel-instance=E2=80=99 starts by collecti=
ng all the
commits for each channel. It maps the channels list using the procedure
=E2=80=99channel-full-commit=E2=80=99. And that procedure calls
=E2=80=99update-cached-checkout. (1)
Then, =E2=80=99cached-channel-instance=E2=80=99 calls =E2=80=99validate-gui=
x-channel=E2=80=99. And this
procedure also calls =E2=80=99update-cached-checkout=E2=80=99. (2)
Then, =E2=80=99cached-channel-instance=E2=80=99 calls =E2=80=99latest-chann=
el-instances=E2=80=99 which
calls =E2=80=99latest-channel-instance=E2=80=99. And guess what, this proc=
edure also
calls =E2=80=99update-cached-checkout=E2=80=99. (3)
Ok, let give a look at =E2=80=99update-cached-checkout=E2=80=99.
This procedure =E2=80=99update-cached-checkout=E2=80=99 first looks if the =
Git reference
specification is already in the cached Git checkout using the procedure
=E2=80=99reference-available?=E2=80=99.
Consider that the Git reference specification is (branch . "some"), then
=E2=80=99reference-available?=E2=80=99 returns #false, so it triggers =E2=
=80=99remote-fetch=E2=80=99
from Guile-Git. If I read correctly, this generates network traffic and
Savannah needs to be reachable. (I)
Hum, I am not convinced someone is following. Who knows? :-)
Let continue. =E2=80=99update-cached-checkout=E2=80=99 starts to check som=
e commit
relation and friends. There is an if-branch calling then
=E2=80=99switch-to-ref=E2=80=99 else =E2=80=99resolve-reference=E2=80=99. =
Under the hood, the procedure
=E2=80=99switch-to-ref=E2=80=99 is calling =E2=80=99resolve-reference=E2=80=
=99.
For the case (branch . "some"), this =E2=80=99resolve-reference=E2=80=99 pr=
ocedure calls
=E2=80=99branch-lookup=E2=80=99 from Guile-Git. If I read correctly, this =
generates
network traffic because of BRANCH-REMOTE and Savannah needs to be
reachable. (II)
Summary: ( (1) + (2) + (3) ) * ( (I) + (II) ) =3D 6.
If I am correct and if I am not missing something, the current design
requires 6 network traffic with Savannah and most of this traffic is
useless because it had already be done, somehow.
Well, (branch . "some") is the worst case, IMHO. And the short commit
ID (tag-or-commit . "1234abc") or the tag (tag-or-commit . "v1.4.0")
too.
Applying my proposal from #65352 (DRAFT v2), it removes some useless
=E2=80=99remote-fetch=E2=80=99 calls.
Well, let me know if this diagnostic is correct.
To be continued=E2=80=A6
Cheers,
simon
1: https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html
Simon Tournier <zimon.toutoune@HIDDEN>:bug-guix@HIDDEN.
Full text available.bug-guix@HIDDEN:bug#65787; Package guix.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.