GNU logs - #65787, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#65787: time-machine is doing too much network requests
Resent-From: Simon Tournier <zimon.toutoune@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 06 Sep 2023 16:27:01 +0000
Resent-Message-ID: <handler.65787.B.169401759925449 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 65787 <at> debbugs.gnu.org
From: Simon Tournier <zimon.toutoune@HIDDEN>
Date: Wed, 06 Sep 2023 18:26:18 +0200
Message-ID: <87wmx3mfo5.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=2a00:1450:4864:20::329;
 envelope-from=zimon.toutoune@HIDDEN; helo=mail-wm1-x329.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi,

Well, I am in a quest to make Guix more robust for the worst-case
scenario: Savannah is unreachable (as well as other servers).  For
context, it matters when using Guix for reproducing scientific
production.  An example of such worst-case, see [1].  Well, this quote:

        The first annoyance is that guix time-machine needs an access to the
        server git.savannah.gnu.org, although the Git repository is already
        cloned and already contains the required commit.=20

is almost tackled by #65352; at least tracked. :-)

Investigating that, I am noticed that the current design is suboptimal,
IMHO.  I am reporting here and I hope to improve the situation by
reducing the number of network requests.

It matters in worst-case scenario of scientific production.  And it also
matters for people with poor or unstable network link.

Sorry if the report is hard to follow, I did my best for being clear.

To keep the discussion simple, I only consider the Git reference
specification =E2=80=99branch=E2=80=99 and =E2=80=99tag-or-commit=E2=80=99.=
  These Git reference
specification that various internal procedures are using is poorly
documented.  See the docstring of the procedure =E2=80=99update-cached-chec=
kout=E2=80=99
from (guix git) for an idea or the implementation of =E2=80=99resolve-refer=
ence=E2=80=99
for the complete list.

Let consider only the Git reference specifications:

    (branch        . "string")
    (tag-or-commit . "string")

because that are what =E2=80=9Cguix time-machine=E2=80=9D sets from the CLI=
 or reads
from channels.scm files, IIUC.

The command =E2=80=9Cguix time-machine=E2=80=9D starts to call =E2=80=99cac=
hed-channel-instance=E2=80=99
passing as argument the procedure =E2=80=99validate-guix-channel=E2=80=99.

This procedure =E2=80=99cached-channel-instance=E2=80=99 starts by collecti=
ng all the
commits for each channel.  It maps the channels list using the procedure
=E2=80=99channel-full-commit=E2=80=99.  And that procedure calls
=E2=80=99update-cached-checkout. (1)

Then, =E2=80=99cached-channel-instance=E2=80=99 calls =E2=80=99validate-gui=
x-channel=E2=80=99.  And this
procedure also calls =E2=80=99update-cached-checkout=E2=80=99. (2)

Then, =E2=80=99cached-channel-instance=E2=80=99 calls =E2=80=99latest-chann=
el-instances=E2=80=99 which
calls =E2=80=99latest-channel-instance=E2=80=99.  And guess what, this proc=
edure also
calls =E2=80=99update-cached-checkout=E2=80=99. (3)

Ok, let give a look at =E2=80=99update-cached-checkout=E2=80=99.

This procedure =E2=80=99update-cached-checkout=E2=80=99 first looks if the =
Git reference
specification is already in the cached Git checkout using the procedure
=E2=80=99reference-available?=E2=80=99.

Consider that the Git reference specification is (branch . "some"), then
=E2=80=99reference-available?=E2=80=99 returns #false, so it triggers =E2=
=80=99remote-fetch=E2=80=99
from Guile-Git.  If I read correctly, this generates network traffic and
Savannah needs to be reachable. (I)

Hum, I am not convinced someone is following.  Who knows? :-)

Let continue.  =E2=80=99update-cached-checkout=E2=80=99 starts to check som=
e commit
relation and friends.  There is an if-branch calling then
=E2=80=99switch-to-ref=E2=80=99 else =E2=80=99resolve-reference=E2=80=99.  =
Under the hood, the procedure
=E2=80=99switch-to-ref=E2=80=99 is calling =E2=80=99resolve-reference=E2=80=
=99.

For the case (branch . "some"), this =E2=80=99resolve-reference=E2=80=99 pr=
ocedure calls
=E2=80=99branch-lookup=E2=80=99 from Guile-Git.  If I read correctly, this =
generates
network traffic because of BRANCH-REMOTE and Savannah needs to be
reachable. (II)

Summary: ( (1) + (2) + (3) ) * ( (I) + (II) ) =3D 6.

If I am correct and if I am not missing something, the current design
requires 6 network traffic with Savannah and most of this traffic is
useless because it had already be done, somehow.

Well, (branch . "some") is the worst case, IMHO.  And the short commit
ID (tag-or-commit . "1234abc") or the tag (tag-or-commit . "v1.4.0")
too.

Applying my proposal from #65352 (DRAFT v2), it removes some useless
=E2=80=99remote-fetch=E2=80=99 calls.

Well, let me know if this diagnostic is correct.

To be continued=E2=80=A6

Cheers,
simon

1: https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Simon Tournier <zimon.toutoune@HIDDEN>
Subject: bug#65787: Acknowledgement (time-machine is doing too much
 network requests)
Message-ID: <handler.65787.B.169401759925449.ack <at> debbugs.gnu.org>
References: <87wmx3mfo5.fsf@HIDDEN>
X-Gnu-PR-Message: ack 65787
X-Gnu-PR-Package: guix
Reply-To: 65787 <at> debbugs.gnu.org
Date: Wed, 06 Sep 2023 16:27:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 65787 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
65787: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D65787
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#65787: time-machine is doing too much network requests
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 10 Sep 2023 20:11:01 +0000
Resent-Message-ID: <handler.65787.B65787.169437665131511 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Simon Tournier <zimon.toutoune@HIDDEN>
Cc: 65787 <at> debbugs.gnu.org
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87wmx3mfo5.fsf@HIDDEN>
Date: Sun, 10 Sep 2023 22:10:36 +0200
In-Reply-To: <87wmx3mfo5.fsf@HIDDEN> (Simon Tournier's message of "Wed, 06
 Sep 2023 18:26:18 +0200")
Message-ID: <87zg1tvlfn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello Simon,

This is a long message; I agree with the intent (avoiding network
traffic when the required commit is already in cache), but I=E2=80=99m not =
sure
about the analysis.  It would probably be easier if you could come with
an example where there=E2=80=99s Git-related network traffic where there
shouldn=E2=80=99t.

Let me give some perspective on what the code intends to do.

=E2=80=98cached-channel-instance=E2=80=99 has 3 cases:

  1. Obvious cache hit: This is when CHANNELS specifies the commit of
     each target channel (this happens for example when you type =E2=80=98g=
uix
     time-machine -q --commit=3Da4c35c607cfd7d6b0bad90cfcc46188d489e1754)
     *and* the combination of channels is already in
     ~/.cache/guix/inferiors.  This is the optimal case: the Git repo
     doesn=E2=80=99t even need to be opened.

  2. Cache hit: CHANNELS are pinned, but refer to tags (like =E2=80=9Cv1.2.=
0=E2=80=9D)
     or short commit IDs (like =E2=80=9Ca4c35c6=E2=80=9D).  In that case,
     =E2=80=98channel-full-commit=E2=80=99 opens the Git repo to resolve th=
e identifier.
     After that, we go to case #1 or #4.

  3. Cache hit: CHANNELS are not pinned=E2=80=94i.e., they refer to a branc=
h,
     not a commit.  In that case we first need to perform a =E2=80=98git fe=
tch=E2=80=99
     and then we go to #1 or #4.

  4. Cache miss: =E2=80=98reference-available?=E2=80=99 returns #f for the =
channel
     commits, we got through =E2=80=98remote-fetch=E2=80=99 followed by
     =E2=80=98build-derivations=E2=80=99.

As with all caches, what matters is to make sure case #1 is processes as
efficiently as possible.  I believe it=E2=80=99s the case since the Git rep=
o is
not even opened.

Of course it would be nice to speed up #2 and #3 too (as long as it=E2=80=
=99s
not at the expense of #1).  Maybe this is the purpose of your message:
reducing Git remote accesses in those cases?  (Apologies, I just
realized this might have been what you had in mind.  :-))

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

Subject: bug#65787: time-machine is doing too much network requests
Resent-From: Simon Tournier <zimon.toutoune@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 11 Sep 2023 11:25:03 +0000
Resent-Message-ID: <handler.65787.B65787.169443145628615 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 65787 <at> debbugs.gnu.org
From: Simon Tournier <zimon.toutoune@HIDDEN>
In-Reply-To: <87zg1tvlfn.fsf@HIDDEN>
References: <87wmx3mfo5.fsf@HIDDEN> <87zg1tvlfn.fsf@HIDDEN>
Date: Mon, 11 Sep 2023 11:41:54 +0200
Message-ID: <87tts1jbbx.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Ludo,

>                             Maybe this is the purpose of your message:
> reducing Git remote accesses in those cases?=20

Yes. :-)

On Sun, 10 Sep 2023 at 22:10, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

>                      It would probably be easier if you could come with
> an example where there=E2=80=99s Git-related network traffic where there
> shouldn=E2=80=99t.

Do we agree that the both Guile-Git procedures =E2=80=99remote-fetch=E2=80=
=99 and
=E2=80=99branch-lookup=E2=80=99 are doing network traffic?

If yes, here two examples:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix pull -q -p /tmp/new
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...

;;; (remote-fetch NETWORK)

;;; (branch-lookup NETWORK)
Authenticating channel 'guix', commits 9edb3f6 to 2eb6df5 (83 new commits).=
..
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git	2eb6df5
--8<---------------cut here---------------end--------------->8---

Well, that=E2=80=99s not perfect=E2=80=A6 it becomes much worse:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix time-machine -q -- describe

;;; (remote-fetch NETWORK)

;;; (branch-lookup NETWORK)

;;; (remote-fetch NETWORK)

;;; (branch-lookup NETWORK)
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...

;;; (remote-fetch NETWORK)

;;; (branch-lookup NETWORK)
building /gnu/store/z8jwdgr7z6i8c00msdm2kzfv0n3zp25v-module-import-compiled=
.drv...
--8<---------------cut here---------------end--------------->8---

Do we agree that is suboptimal?


> =E2=80=98cached-channel-instance=E2=80=99 has 3 cases:
>
>   1. Obvious cache hit: This is when CHANNELS specifies the commit of
>      each target channel (this happens for example when you type =E2=80=
=98guix
>      time-machine -q --commit=3Da4c35c607cfd7d6b0bad90cfcc46188d489e1754)
>      *and* the combination of channels is already in
>      ~/.cache/guix/inferiors.  This is the optimal case: the Git repo
>      doesn=E2=80=99t even need to be opened.

Yes.


>   2. Cache hit: CHANNELS are pinned, but refer to tags (like =E2=80=9Cv1.=
2.0=E2=80=9D)
>      or short commit IDs (like =E2=80=9Ca4c35c6=E2=80=9D).  In that case,
>      =E2=80=98channel-full-commit=E2=80=99 opens the Git repo to resolve =
the identifier.
>      After that, we go to case #1 or #4.

That=E2=80=99s suboptimal.  Currently, it reads:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 28	sept. 06 2023 14:54:50	(current)
  guix 6113e05
    repository URL: https://git.savannah.gnu.org/git/guix.git
    commit: 6113e0529d61df7425f64e30a6bf77f7cfdfe5a5

$ ./pre-inst-env guix time-machine -q --commit=3D6113e05 -- describe

;;; (remote-fetch NETWORK)

;;; (remote-fetch NETWORK)
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...

;;; (remote-fetch NETWORK)
Computing Guix derivation for 'x86_64-linux'... |  C-c C-c
--8<---------------cut here---------------end--------------->8---

Using patch from #65352 [1], it removes these useless traffic:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix time-machine -q --commit=3D6113e05 -- describe
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.or=
g/git/guix.git'...
Computing Guix derivation for 'x86_64-linux'... |  C-c C-c
--8<---------------cut here---------------end--------------->8---

Using the proposed patch, it becomes optimal, IMHO.  Well, let discuss
it in #65352 =E2=80=93 comments are welcome. :-)

>   3. Cache hit: CHANNELS are not pinned=E2=80=94i.e., they refer to a bra=
nch,
>      not a commit.  In that case we first need to perform a =E2=80=98git =
fetch=E2=80=99
>      and then we go to #1 or #4.

I hope that I am convincing you that this case is suboptimal: it does 3
=E2=80=99git fetch=E2=80=99 and more 3 others lookup requiring network.  So=
 it is about
6 network access when only one is necessary, IMHO.


>   4. Cache miss: =E2=80=98reference-available?=E2=80=99 returns #f for th=
e channel
>      commits, we got through =E2=80=98remote-fetch=E2=80=99 followed by
>      =E2=80=98build-derivations=E2=80=99.

This case is part of #2 and discussed in #65352, IMHO.


Cheers,
simon


1: [bug#65352] [PATCH v2] DRAFT git: Avoid touching the network unless need=
ed in 'reference-available?'.
Simon Tournier <zimon.toutoune@HIDDEN>
Wed, 06 Sep 2023 16:17:08 +0200
id:32d3fb5066e0b20e200dabef0fba897634e21dda.1694009405.git.zimon.toutoune@g=
mail.com
https://yhetil.org/guix/32d3fb5066e0b20e200dabef0fba897634e21dda.1694009405=
.git.zimon.toutoune@HIDDEN
https://issues.guix.gnu.org/msgid/32d3fb5066e0b20e200dabef0fba897634e21dda.=
1694009405.git.zimon.toutoune@HIDDEN

Last modified: Mon, 11 Sep 2023 11:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.