GNU bug report logs - #65720
Guile-Git-managed checkouts grow way too much

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Ludovic Courtès <ludo@HIDDEN>; dated Sun, 3 Sep 2023 20:45:02 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 13 Sep 2023 22:52:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Sep 13 18:52:33 2023
Received: from localhost ([127.0.0.1]:36301 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgYim-0005MA-Sv
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 18:52:33 -0400
Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]:41143)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1qgYih-0005La-UV
 for 65720 <at> debbugs.gnu.org; Wed, 13 Sep 2023 18:52:29 -0400
Received: by mail-wm1-x330.google.com with SMTP id
 5b1f17b1804b1-3fe4f3b5f25so867035e9.0
 for <65720 <at> debbugs.gnu.org>; Wed, 13 Sep 2023 15:52:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694645536; x=1695250336; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=8/Ddynd1Q5u7aJf6LPAQyd1FEGPttudrblI+TDhDRAc=;
 b=k769VeRGXZWuc9m7cbX/X9TmncOtapPVnQBIXTji6+Eb0HkMFjPiz11rqvEhKhHYlP
 VjGqSc1K3ynkNwIDQS9svy/g/g0zCQ1m6rdZMZ5ssT59wLJr17ICLKPXT7jVSQZnoy1a
 TYqIF58/vMRG7ndn+YN9OAf0TagXvdm/p9CgkiCc+6qiKl4LBqGFV0DoaHyBPA8lBGmP
 2kbugskd6wAuEYvTfRFzGLqRKmkHz6Ry72p5ljAt7pCHmtW0/Jc4Jwj4Iio7SintPjjJ
 zDgdzxHob9erUQYlzWQunD0jCKbrBV2GVVR2BwxobYB7gd0W7K1ltaPFvThSqQtBzfkO
 6fDA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1694645536; x=1695250336;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=8/Ddynd1Q5u7aJf6LPAQyd1FEGPttudrblI+TDhDRAc=;
 b=ocuDaebx2iIhL2hlV4huM0CqHGUNJh0JELdOs0VqHfpRQ2GfnSu+ecDGKE3rOoyMIB
 4YG/8/CecX9GLna1XdeIZ9dLGHbdokwnDEKXO+5qerE8VdQta57zhbRv7YZOYR0j+W4u
 Sit8/+//FJcc9iQoXPY2QsgtaKVIkSBPA5RJAFY7FvQGQ7Et96FDyIgFfC0DufI284o9
 U+SBNLVbcBnuOp0gXRqECkY2Ffnkidf8IEIX8svGeN70jg5wrsPNqJVl5TiTWVNrJCuh
 dEaZle/kNXu/pZYXSv/i8qV0HiZXXYFjeNHJAK1+XE8IP4jMqxYLB1uwT/4vde1xabMN
 C+9w==
X-Gm-Message-State: AOJu0YxK/RLs4Mc7h/LTfVlzVUXaNvpU62lyl+UuwsWMKHAgO5y3bZde
 Ur7R5TDzWxiekMdMJi7ocL4def+UYAo=
X-Google-Smtp-Source: AGHT+IEQyzrX4pqn8PTZYG+0dGN0WVMtzuH+hEQ610+sX5ZmCz9heI6iwhfQTfSb6CRLI5ZsL0j7wA==
X-Received: by 2002:a05:600c:34ce:b0:401:bb6e:453a with SMTP id
 d14-20020a05600c34ce00b00401bb6e453amr3280276wmq.0.1694645536274; 
 Wed, 13 Sep 2023 15:52:16 -0700 (PDT)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id
 z25-20020a1c4c19000000b003fe29f6b61bsm291851wmf.46.2023.09.13.15.52.15
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Wed, 13 Sep 2023 15:52:15 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, wolf <wolf@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <874jjylza9.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
 <ZP8nc1m8rN_34XV-@ws> <874jjylza9.fsf@HIDDEN>
Date: Thu, 14 Sep 2023 00:36:17 +0200
Message-ID: <86o7i5wvj2.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludo,

On Wed, 13 Sep 2023 at 20:10, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

> =E2=80=98get-internal-run-time=E2=80=99 returns =E2=80=9Cunits of process=
or time=E2=80=9D used by the
> current process (info "(guile) Time").  When shelling out, the process
> calls waitpid(2) and does nothing, so naturally its processor time is
> close to zero.
>
> =E2=80=98get-internal-real-time=E2=80=99 should give something closer to =
elapsed time.

Well, let avoid to mix unrelated discussion. :-)  For discussing that
specific part, I reported on guix-devel my timing using ,time.

        comparing commit-relation using Scheme+libgit2 vs shellout plumbing=
 Git
        Simon Tournier <zimon.toutoune@HIDDEN>
        Tue, 12 Sep 2023 00:48:30 +0200
        id:865y4gz5q9.fsf@HIDDEN
        https://lists.gnu.org/archive/html/guix-devel/2023-09
        https://yhetil.org/guix/865y4gz5q9.fsf@HIDDEN

The result is still significantly less and discussion is welcome
overthere. :-)

Cheers,
simon




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 13 Sep 2023 18:10:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Sep 13 14:10:54 2023
Received: from localhost ([127.0.0.1]:35997 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgUKD-0003s5-Tp
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 14:10:54 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:55598)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qgUKB-0003rr-Fn
 for 65720 <at> debbugs.gnu.org; Wed, 13 Sep 2023 14:10:52 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qgUJz-0000vz-VB; Wed, 13 Sep 2023 14:10:39 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=3thjA8phpPJbXWv6yUX3y+eTpXZIAV+/cobGCrMybw8=; b=CrAXPm3gUTQVSoIHo0It
 dujVEAB9jcGUjfAteAjNhmleILF7r1mN8eI6CcyouDv7K0/Og/6EeZun2KZQhISUNgksfr7syNG+3
 27jHO8gy4CpeNwsPfGUmOK/c/F/ryiqRkh0wM3Ki4Me7jUxSISy23BGe+Pz5uwJj7w5ciktokA8OT
 AmzxJ03Lmawxqn8azP3FGpKaJp6RznDFMljQPr40Mm5gCk+HaC7HasyxqTX8i0S8dL6oNgolUy+zw
 AI61eCtvrZVy4KM673AK3KG/yqqmAhfsZrxBzbmqjd8ma95nNNOZNHbQGJhOOyLyA5HghFC+d3Fa6
 cqGP7FB0plg9hA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: wolf <wolf@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
 <ZP8nc1m8rN_34XV-@ws>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Septidi 27 Fructidor an 231 de la =?utf-8?Q?R=C3=A9v?=
 =?utf-8?Q?olution=2C?= jour de la Verge-d'or
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 13 Sep 2023 20:10:38 +0200
In-Reply-To: <ZP8nc1m8rN_34XV-@ws> (wolf@HIDDEN's message of "Mon, 11 Sep
 2023 16:42:59 +0200")
Message-ID: <874jjylza9.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

wolf <wolf@HIDDEN> skribis:

> (define (time proc)
>   (let* ((start (get-internal-run-time))
>          (_ (proc))
>          (end   (get-internal-run-time)))
>     (exact->inexact (* 1000 (/ (- end start) internal-time-units-per-seco=
nd)))))
>
> (format #t "Guix: ~ams\nGit:  ~ams\n"
>         (time (=CE=BB () (commit-relation c1 c2)))
>         (time (=CE=BB () (shelling-commit-relation c1 c2))))

=E2=80=98get-internal-run-time=E2=80=99 returns =E2=80=9Cunits of processor=
 time=E2=80=9D used by the
current process (info "(guile) Time").  When shelling out, the process
calls waitpid(2) and does nothing, so naturally its processor time is
close to zero.

=E2=80=98get-internal-real-time=E2=80=99 should give something closer to el=
apsed time.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 14:43:11 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 10:43:11 2023
Received: from localhost ([127.0.0.1]:54353 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfi86-00078r-Rb
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 10:43:11 -0400
Received: from wolfsden.cz ([37.205.8.62]:49238)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ws@HIDDEN>) id 1qfi82-00078e-Kf
 for 65720 <at> debbugs.gnu.org; Mon, 11 Sep 2023 10:43:08 -0400
Received: by wolfsden.cz (Postfix, from userid 104)
 id 55CFE25A352; Mon, 11 Sep 2023 14:43:00 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail;
 t=1694443380; bh=9UzFheRlBpqwhqYLwxRvcCIW6Z1Q4wMYjKudYbhoL0k=;
 h=Date:From:To:Cc:Subject:References:In-Reply-To;
 b=bTC939dBzFFk6esxnKSOdRbxsN5LTCAxeP+eUlfD6WXcMK1fPf+c/OMD3TJE9JWA0
 V8YqtIuX2v7qqB89+jlxymepApZVyeZ7Wmx8KIFblCz3mp0xyBQDZHKu1e8H00TIak
 OW/PStKUBtzR554J/Q9rLtq6H8yKWbiAovHXG/A4nhLajO9+HQMNcC/e+ylMIrKVdy
 /UmiT1xTV8gMEw7Exngp0L283yRGtMMtiEmwhWNVKDxFpEo87xKvQ4aLmyyG0QB/f+
 BhuIBpNggW9jk28mNB6Kk5Asyuow9HNWHKb+HVVTL3xmcaYFadGIT/quzNvgC0lLZj
 JiWf/jNTmRn6VDaCRi6ajjahLrCPMQiz7mTsYBDTjPv5Wr8i/5oYrYs+QXBKIBsvqY
 MT8pkNdbw/lVnM4hvHLiCJE8sCE70/1/clKzTRvn6r0H2s+J+ZSi4iTfhZvWD+r+6R
 ccpQERSnXMD41vYkc2Y5WLpn//7B7YSkdtw/JzE7kHkr6SN+MThp+kx2iJ1zGfc4lB
 nuHT3phEsdDNEcneitKWh94rPJRts0e+4GgFEyHaXF5ftnL86Sh0lYNDtPAKbPE5K+
 1BDl/U/kivM0/KiFoK/vF+8MPCgORoiwvA/azIgShwwU9OvMqPdid84h3N5gMojtSp
 GVKm6eEfYHasfGQwNWdFF92s=
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on wolfsden
X-Spam-Level: 
X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED,DKIM_SIGNED,
 DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,URIBL_BLOCKED
 autolearn=no autolearn_force=no version=3.4.6
Received: from localhost (unknown [193.32.127.154])
 by wolfsden.cz (Postfix) with ESMTPSA id BB86725B039;
 Mon, 11 Sep 2023 14:42:59 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail;
 t=1694443379; bh=9UzFheRlBpqwhqYLwxRvcCIW6Z1Q4wMYjKudYbhoL0k=;
 h=Date:From:To:Cc:Subject:References:In-Reply-To;
 b=qW4/Mjo471X6YMVfJ3cUHTAk8/ewQ6oCyMfjBcToj1Mv2H4UhilOAq1NCf1lSotaT
 y2uyBO/bSj67hec6z/vW4AKC/LgD2tNhMoF8f9JY279CfZvnk5fYPiyyCthi5/47rG
 XoamN6Xl8UofI4jdT3viwzZPvVEfMQmftv1RtzAu656kXEKOK9+SGrLOa/DSrV/s6w
 Eatt+StL0NiiO5ViFmo7gRhgivSan2kHyg5eJElLwigstsapE2ufuo/KclZFsMIwF3
 CuRjyo/kBdHwntelqHWRtVF7fVT7EhiRjk5kRHPsOl71/WeuyVIH6w27GenRJ44Vpp
 F7L7on1u+T3oJX/LiBM4+ngE/FY4V8Vfv8P+P9HMBwTES5gjMso6V8DOEQeRfudaE1
 Ty+skOfuJFR6VSOHARKjYgOjFUcdNeHHAr3KzEahVG+HWANkX2SMNLAhD1JFO/f5oI
 unnqL/o4rEPogGsXbMGQvU6SWRMBDSv4dFO6EesebJB9n/Vqmyg4WneRi8u9SZhZeR
 q//58iTfvXhyUgMIwOxztlSg7ZRTSvBEZyodhehLcVNDPGDtsJqTYZK9+WifadKXvs
 TtZz7d4GUFPSBg11ahInlTs5nZfaXzz/P2HvM5UZ+lu6eW7nddAdiH82I9fM0VujUR
 tknbqMsHpZgDuE9KxmKPTTpo=
Received: from localhost (localhost [local])
 by localhost (OpenSMTPD) with ESMTPA id 99c8a95f;
 Mon, 11 Sep 2023 14:42:59 +0000 (UTC)
Date: Mon, 11 Sep 2023 16:42:59 +0200
From: wolf <wolf@HIDDEN>
To: Ludovic =?iso-8859-1?Q?Court=E8s?= <ludo@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Message-ID: <ZP8nc1m8rN_34XV-@ws>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature"; boundary="38x2NUBZrcLd/9Sh"
Content-Disposition: inline
In-Reply-To: <87pm2s385m.fsf@HIDDEN>
X-Spam-Score: 0.2 (/)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)


--38x2NUBZrcLd/9Sh
Content-Type: multipart/mixed; boundary="ZmJCXHxmcj67AztS"
Content-Disposition: inline


--ZmJCXHxmcj67AztS
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2023-09-08 19:08:05 +0200, Ludovic Court=C3=A8s wrote:
> Hello!
>=20
> Josselin Poiret <dev@HIDDEN> skribis:
>=20
> > Right, although I wouldn't necessarily say that the former doesn't have
> > a proper API, but rather that it has a Unix-oriented API.  That leads to
> > performance issues on e.g. Windows but on Linux I'm not sure there's
> > much of a difference.
>=20
> [...]
>=20
> > We could consider replacing the guile-git dependency with another
> > library built directly on top of git-minimal, and have this be a
> > dependency of Guix.  Not ideal though, and not really scalable either:
> > we can't just add every VCS as direct dependencies.
>=20
> I cannot imagine a viable implementation of things like =E2=80=98commit-c=
losure=E2=80=99
> and =E2=80=98commit-relation=E2=80=99 from (guix git) done by shelling ou=
t to =E2=80=98git=E2=80=99.

I am sure I must be missing some part of the contract of the function, but =
at
least the commit-relation seems fairly straightforward:

    (define (shelling-commit-relation old new)
      (let ((h-old (oid->string (commit-id old)))
            (h-new (oid->string (commit-id new))))
        (cond ((eq? old new)
               'self)
              ((zero? (git-C %repo "merge-base" "--is-ancestor" h-old h-new=
))
               'ancestor)
              ((zero? (git-C %repo "merge-base" "--is-ancestor" h-new h-old=
))
               'descendant)
              (else
               'unrelated))))

I would argue it is even somewhat more readable than the current implementa=
tion.

> I=E2=80=99m quite confident this would be slow

My version is ~2000x faster compared to (guix git):

    Guix: 1048.620992ms
    Git:  0.532143ms

Again, I am sure I must have miss something, either in the implementation o=
r in
the measurements, because it is pretty hard to believe there is so much roo=
m for
improvement.

The full script I used is attached to this email.

> and brittle.

In general git plumbing command are design to have stable CLI interface in =
order
to be usable in scripting.  So I am not sure where the brittleness would co=
me
=66rom.

>=20
> It looks like there=E2=80=99s no option other than carrying the two
> implementations.

Assuming I made no mistake (hard to believe), it is probably worth explorin=
g the
feasibility of just shelling out to the git binary some more.

>=20
> ~~~
>=20
> Years ago, Andy Wingo sketched a plan for GNU hackers to implement Git
> in pure Scheme.  That was on April 1st though, so people mistakenly
> assumed it was a joke and the project was never carried out.
>=20
> I digress, but I wonder: is there not even a viable Haskell or OCaml
> implementation of Git?
>=20
> Thanks,
> Ludo=E2=80=99.
>

W.

--=20
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

--ZmJCXHxmcj67AztS
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="test.scm"
Content-Transfer-Encoding: quoted-printable

#!/bin/sh
# -*-scheme-*-
exec guile -s "$0" "$@"
!#

(use-modules (git)
             (guix git))

(define %repo "/tmp/guix-fork")

(define h1 "72745172d155e489936f694d6b9013cb76272370")
(define h2 "6d60d7ccba5a8e06c17d55a1772fa7f4529b5eff")
(define h3 "c3db650680f995f0556d3ddce567cdc1c33e4603")

;;; r has to still be defined when the commit-relation is called.  There is=
 *no*
;;; error, but it always returns 'unrelated.  Quite a footgun.
(define r (repository-open %repo))
(define c1 (commit-lookup r (string->oid h1)))
(define c2 (commit-lookup r (string->oid h2)))
(define c3 (commit-lookup r (string->oid h3)))

(define (git-C dir . args)
  (apply system* "git" "-C" dir args))

(define (shelling-commit-relation old new)
  (let ((h-old (oid->string (commit-id old)))
        (h-new (oid->string (commit-id new))))
    (cond ((eq? old new)
           'self)
          ;; In real code, git-C should probably return #t (for 0), #f (for=
 1)
          ;; or raise (for anything else).
          ((zero? (git-C %repo "merge-base" "--is-ancestor" h-old h-new))
           'ancestor)
          ((zero? (git-C %repo "merge-base" "--is-ancestor" h-new h-old))
           'descendant)
          (else
           'unrelated))))

;;; Make sure it actually works.
(let ((tests `((,c1 . ,c1)
               (,c1 . ,c2)
               (,c2 . ,c1)
               (,c1 . ,c3))))
  (for-each (=CE=BB (c)
              (format #t "Guix: ~a\nGit:  ~a\n\n"
                      (commit-relation (car c) (cdr c))
                      (shelling-commit-relation (car c) (cdr c))))
            tests))

(define (time proc)
  (let* ((start (get-internal-run-time))
         (_ (proc))
         (end   (get-internal-run-time)))
    (exact->inexact (* 1000 (/ (- end start) internal-time-units-per-second=
)))))

(format #t "Guix: ~ams\nGit:  ~ams\n"
        (time (=CE=BB () (commit-relation c1 c2)))
        (time (=CE=BB () (shelling-commit-relation c1 c2))))

--ZmJCXHxmcj67AztS--

--38x2NUBZrcLd/9Sh
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEt4NJs4wUfTYpiGikL7/ufbZ/wakFAmT/J20ACgkQL7/ufbZ/
walA7BAAioswpeyaAYJlo/HjxXOUviMoZ49RJ0vjoWkcdKDJyZCvY1bSaa6E+o38
4mjw+8qT2VH3Su+GKTWgYJ66O6PT2IZh7kybzqPCdIFFXAK3KHNP2cQlweDgl6jG
YhktsUBWalhzk06rEy3JXNPqrIinGHmMqm/pIxMQXPcOLN5/d90TMB304YqbjAio
J5sCeNNYNhVL0A1jY7rZMefUcHISKX8B3XvsNr2A0AvofGv6OQrftf3OMEX4OeE1
5KFeukwv9FRZ38Cc6+Ob3Jw+Atmz5WrOutTPMXAbp4fxxXHQguG9/fIP3JinAtd1
3ruwT7Q4V5n6pGcz81vMYTR+24Tfbcs4thDqKfIM2uoPOvCh1c6dQ3ap2hI4uvls
DlCSviISQkjjCqR30jj2ZhHIHF3kPDl+DnaDCn/LIKBRwEbLDJJ+eW9Bv9JJLG2h
6TCouuRrJzCZ7OpkTg6psZI7mhzwYNdJO2wIkGib8eI2U+/GxFDgWTi8U9HHQFiR
Z8/97ph5AdoIObDz0R/hezyvpWOJMYuhI0IhKvBksyx8UYOnpM0lIaSASQt2DqU8
xmRztjNazvoUbTASBg9l4MedSejPcDVn6FFFQ+QpkBORXTMYJ5E572BVxOOmnTbu
s2K2nZZMczHKbOgWyJt4rafRzSZeJRY6fr062Cu+TrHs9TO0i70=
=UIrN
-----END PGP SIGNATURE-----

--38x2NUBZrcLd/9Sh--




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 14:38:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 10:38:05 2023
Received: from localhost ([127.0.0.1]:54341 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfi3B-00071H-0k
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 10:38:05 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:58936)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qfi38-00070l-2v
 for 65720 <at> debbugs.gnu.org; Mon, 11 Sep 2023 10:38:02 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qfi2y-0005Kc-Kx; Mon, 11 Sep 2023 10:37:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=slniUQC9+028PxuFEpuMCgSmgL9gNYctBBnUkHGl+Yw=; b=sf+0dp0sJRNtHMqTh3ih
 /idbj+n0lEPqG2VR0ik70c6apt0jY0eLCNwLX2h56fGQlv/Xm3uCKeRFpYt8EDeG0TlI0j6PrL/1q
 lmQEVEaRRbKaiWiV58ZYIhLhidxr1hdjZ8CvN/F+quY+uqmAZkGJDsgHwWY89FQN1AqGhCVgGtugc
 1tOuJQoIUR9K+g4e3KhaJxf/IMHGd6oq1+1fDkIFRW2wByB8M6vavu+XKrtPooVQFulJpR/TmuSca
 vMnCPAnoSlQn4+aQvVRHbLDXKGU2+F4bDfwzHkj2WspJY5F1gXa1bAmpMS7sEMdrGpo2v5dHSNJZH
 VTeCYnVtU1u0DQ==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Josselin Poiret <dev@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
Date: Mon, 11 Sep 2023 16:37:50 +0200
In-Reply-To: <87msy0ln4m.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?=
 =?utf-8?Q?s?= message of "Tue, 05 Sep 2023 16:18:17 +0200")
Message-ID: <87jzswsrlt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> It would also be pretty bad for closure size:
>
> $ guix size guile-git | tail -1
> total: 106.6 MiB
> $ guix size guile-git git-minimal | tail -1
> total: 169.8 MiB
>
> It=E2=80=99s also not clear concretely how we=E2=80=99d add that dependen=
cy.  Try
> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it doesn=
=E2=80=99t work?

A solution to this particular problem is coming:

  https://issues.guix.gnu.org/65866

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 11:24:16 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 07:24:16 2023
Received: from localhost ([127.0.0.1]:52246 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qff1c-0007RU-4b
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:24:16 -0400
Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:39077)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1qff1W-0007R6-O6
 for 65720 <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:24:14 -0400
Received: by mail-wr1-x433.google.com with SMTP id
 ffacd0b85a97d-31f2ab21231so655497f8f.0
 for <65720 <at> debbugs.gnu.org>; Mon, 11 Sep 2023 04:24:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694431441; x=1695036241; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=C8OuW9Kc/Bl4fezquUaFGZUMX86ENwX4OYueChGdRGI=;
 b=G+mlTAwXUNAeLYJ36WBqUOa6RNy4wwHwiL02IEmXpKmxWM+rI3zX8zorKnI0iCmofH
 69zmMk718qaaLe8gO1naMAEIAnNlYKrmuH1C5YFNfLUDgMCHJ3JCWasgiAAgI/sEdGZB
 b6kQ6lnt+K2tTpqT9s6abh3Hzk6jscGlp94y/QTgVjrv6uYiHNvecA1X74Tl5jG9PlRJ
 IvGuFNJKcwOwpxxQktK4vFYqLPu9MwNP+ngOCV6i0FJxa86zKHmfsh2XwKPYSz0tKcfp
 nhUYQO4owx0UoTI7FGF1Ivem7C6YGk1QTNXZSvuQT1yBOTtX95WOxTmczf2wKGLIhauV
 k7VQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1694431441; x=1695036241;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=C8OuW9Kc/Bl4fezquUaFGZUMX86ENwX4OYueChGdRGI=;
 b=Sxn1Ut1/T+hy66up0Sfv6LKq+Oj9goelZ5McDFQg8UTBA4OQ3ecLTbz4ahMvzy8JGw
 hyIymOy2P37+D+wW7Pnht7obN7Tto/irShKQZI+nr+scnGc7m8wbub9LrCkzDqJehQ5+
 ebscD/EEmrnePzVZTmUZe5n55XM7KA0hUy7OK4XTvkyJJNS8hVecHU5kyfcipjMJD/HE
 uvHSWcm4gNDbZYyvlL2SeFJ58hjKqZR32NP6fJWJFQJgqosfEQl+FVaf0ul1Oqo5B5SU
 1au73q8LurJZQLNVFjfEE4B5tPL0AMKC1pJTfFARZqv7L6Zq21nfoCzYoadHl+mgcaeI
 kdbg==
X-Gm-Message-State: AOJu0YzPmYPBsjV1a6vkp6iiDxAepvWYuDuhyzQgJSQ4ellq82y/s9GR
 3DI9cIl4lzxXJ/asgT1lCSAcFvr1PqM=
X-Google-Smtp-Source: AGHT+IEm53aH9z6OxtH/I6npDph9yH7BvLZyuJ9ib2+pNIYApogUVArOxNt9CXz3Y8I58fR/1FfdBQ==
X-Received: by 2002:a5d:5707:0:b0:317:3da0:7606 with SMTP id
 a7-20020a5d5707000000b003173da07606mr7133550wrv.4.1694431440677; 
 Mon, 11 Sep 2023 04:24:00 -0700 (PDT)
Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id
 y19-20020a05600c365300b003fef5e76f2csm3498262wmq.0.2023.09.11.04.24.00
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 11 Sep 2023 04:24:00 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Josselin Poiret
 <dev@HIDDEN>
Subject: Digression about Git implementations (was Re: bug#65720:
 Guile-Git-managed checkouts grow way too much)
In-Reply-To: <87pm2s385m.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
Date: Mon, 11 Sep 2023 10:42:35 +0200
Message-ID: <87zg1tje2s.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludo,

On Fri, 08 Sep 2023 at 19:08, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

> Years ago, Andy Wingo sketched a plan for GNU hackers to implement Git
> in pure Scheme.  That was on April 1st though, so people mistakenly
> assumed it was a joke and the project was never carried out.

Well, that is a piece of work. :-)

Maybe there is an hope with: git-std-lib.

    Subject: Proposal/Discussion: Turning parts of Git into libraries
    From: Emily Shaffer <nasamuffin@HIDDEN>
    To: Git List <git@HIDDEN>
    Date: Fri, 17 Feb 2023 13:12:23 -0800=09=20=20=20=20
    https://lore.kernel.org/git/CAJoAoZ=3DCig_kLocxKGax31sU7Xe4=3D=3DBGzC__=
Bg2_pr7krNq6MA@HIDDEN/

And some patches are starting to float around.
https://public-inbox.org/git/20230810163346.274132-1-calvinwan@HIDDEN/


> I digress, but I wonder: is there not even a viable Haskell or OCaml
> implementation of Git?

It depends on what means =E2=80=9Cviable=E2=80=9D. :-)

https://github.com/mirage/ocaml-git
https://hackage.haskell.org/package/git

Irmin [1] is an OCaml library for building mergeable, branchable
distributed data stores =E2=80=93 A Distributed Database Built on the Same
Principles as Git.  And irmin relies on ocaml-git.

1: https://github.com/mirage/irmin

Then there is a pure Go implementation and another using Java.

https://git-scm.com/book/en/v2/Appendix-B%3A-Embedding-Git-in-your-Applicat=
ions-go-git
https://git-scm.com/book/en/v2/Appendix-B%3A-Embedding-Git-in-your-Applicat=
ions-JGit

I do not know all that are =E2=80=9Cviable=E2=80=9D.  Well, I do not know i=
f =E2=80=99git gc=E2=80=99 is
implemented.  And I do not know which plumbing is implemented and which
porcelain is available.

Last, SWH uses dulwich [2] which is a pure Python implementation of Git.

2: https://www.dulwich.io/

To my knowledge, there is no =E2=80=9Cdulwich gc=E2=80=9D but they implemen=
t =E2=80=9Cdulwich
fsck=E2=80=9D and =E2=80=9Cdulwich repack=E2=80=9D.

Back on 10 Years of Guix or at UNESCO on February =E2=80=93 I do not rememb=
er
exactly when =E2=80=93 we were discussing about implementation of Git.  And=
 we
mentioned an implementation in Rust.  Maybe this one:

https://github.com/Byron/gitoxide

Cheers,
simon





Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 07:10:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 03:10:08 2023
Received: from localhost ([127.0.0.1]:51995 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfb3f-0006mI-S6
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:10:08 -0400
Received: from mx1.riseup.net ([198.252.153.129]:55064)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raingloom@HIDDEN>) id 1qfb3e-0006lg-2o
 for 65720 <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:10:06 -0400
Received: from fews02-sea.riseup.net (fews02-sea-pn.riseup.net [10.0.1.112])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by mx1.riseup.net (Postfix) with ESMTPS id 4RkdBr30skzDqpZ;
 Mon, 11 Sep 2023 07:09:56 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak;
 t=1694416196; bh=jCBwtrOdu4IjIK1DNoGSobWo/knpWw03ag0O4gDJiso=;
 h=References:From:To:Cc:Subject:Date:In-reply-to:From;
 b=p8OnFoG5FSZ4qVyuqU7eoO1NHJb8FeMLpDQu9pEtagI3uq2ku2qWHNqtfOFkZMiSk
 P9inRFGTA0r8s30MFTbGMkkBqwFCiediO2TsptW5zYwKk0QKIZglisSWlbs+QkSsCG
 CDxEWNaOUY4BssSxgZaE29mtnumAa53HgUXUmdhU=
X-Riseup-User-ID: 558C9CB603439A5DAFC0FDC808E10CF6C3CA907716EB7D430B23A41F042139CA
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by fews02-sea.riseup.net (Postfix) with ESMTPSA id 4RkdBb6bTLzFrwy;
 Mon, 11 Sep 2023 07:09:43 +0000 (UTC)
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <86il8mn7al.fsf@HIDDEN> <87jzt0382l.fsf@HIDDEN>
 <86cyyrskmj.fsf@HIDDEN>
From: Csepp <raingloom@HIDDEN>
To: Simon Tournier <zimon.toutoune@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Date: Mon, 11 Sep 2023 09:06:58 +0200
In-reply-to: <86cyyrskmj.fsf@HIDDEN>
Message-ID: <cuca5tt6va2.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 65720
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 65720 <at> debbugs.gnu.org,
 bug-guix@HIDDEN, Josselin Poiret <dev@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)


Simon Tournier <zimon.toutoune@HIDDEN> writes:

> Hi,
>
> On Fri, 08 Sep 2023 at 19:09, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>
>>>> It would also be pretty bad for closure size:
>>>>
>>>> --8<---------------cut here---------------start------------->8---
>>>> $ guix size guile-git | tail -1
>>>> total: 106.6 MiB
>>>> $ guix size guile-git git-minimal | tail -1
>>>> total: 169.8 MiB
>>>> --8<---------------cut here---------------end--------------->8---
>>>>
>>>> It=E2=80=99s also not clear concretely how we=E2=80=99d add that depen=
dency.  Try
>>>> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it do=
esn=E2=80=99t work?
>>>> But then, what about applications like Cuirass and hpcguix-web?
>>>
>>> I think we can rely on something like,
>>>
>>>     guix shell -C git-minimal -- git gc
>>
>> We=E2=80=99re talking about the implementation of a cache (meant to spee=
d up
>> operations), that would actually fill said cache plus do a whole bunch
>> of expensive operations?  Nah.  :-)
>
> I do not think.  If I understand correctly, we need to run =E2=80=9Cgit g=
c=E2=80=9D at
> some point, therefore git-minimal needs to me around.  The question is
> how and when.
>
> Well, maybe I am missing what the bug is about.  For me, it is about
> running =E2=80=98git gc=E2=80=99 for cleaning the Git checkout cache, no?
>
>
> Solution #1.  Add git-minimal as inputs.  It increases the closure and
> the extra load (on average) is about the ratio between the rate of =E2=80=
=9Cguix
> pull=E2=80=9D and the rate of the git-minimal changes.
>
> Assuming, that people are running =E2=80=9Cguix pull=E2=80=9D once per we=
ek and say =E2=80=9Cgit
> gc=E2=80=9D is run after 50 pulls.  (These both number are totally arbitr=
ary and
> based on my personal estimate).
>
> Data Service [1] tells:
>
>         2023-07-07 15:45:22 2023-09-08 21:22:08
>         2023-05-11 16:10:48 2023-07-07 14:21:45
>         2023-05-01 16:40:08 2023-05-11 14:36:16
>         2023-04-25 13:34:54 2023-05-01 15:19:55
>         2023-04-25 13:34:54 2023-09-08 21:22:08=20=20=20=20=20=20=20=20
>         2023-03-06 17:22:28 2023-04-25 12:27:33
>         2023-01-17 23:49:19 2023-03-06 16:48:43
>         2022-11-08 13:06:42 2023-01-17 15:11:47
>         2022-10-08 05:14:46 2022-11-08 09:56:31
>         2022-09-06 15:00:08 2022-10-08 04:15:43
>         2022-08-13 22:02:31 2022-09-06 12:58:52
>         =E2=80=A6
>
> It means that an user will download ~10 times git-minimal for nothing.
>
>
> Solution #2.  The one I am proposing. :-)  Download git-minimal only
> when Guix needs it for running =E2=80=9Cgit gc=E2=80=9D.  Yeah, there is =
probably a
> small overload with some operations.  But, I bet this overload is much
> smaller than the one of solution #1.
>
> Well, it depends on the number of times people are updating the cache vs
> the rate of change of git-minimal.
>
> For sure, if one updates 100 times per week the cache, having
> git-minimal as inputs is far better.  But I do not think that the
> regular usage on average. :-)
>
> That=E2=80=99s why I am proposing to have an option for turning off this =
=E2=80=9Cgit
> gc=E2=80=9C operation.
>
> Well, we have lived since years without running =E2=80=98git gc=E2=80=99 =
so running it
> once per year on average is probably enough to keep the cache size
> reasonable.  And git-minimal is changing every month.
>
>
> Maybe, there is some solution #3. ;-)
>
> Cheers,
> simon
>
>
> 1: https://data.guix.gnu.org/repository/1/branch/master/package/git-minim=
al/output-history

Please don't create another situation like with guix system roll-back,
where a crucial sysadmin operation doesn't work without network access.
Or at least make it configurable, so things that are likely to be needed
for future operations are pre-fetched.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 11 Sep 2023 07:10:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 03:10:13 2023
Received: from localhost ([127.0.0.1]:51998 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfb3l-0006ma-Aw
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:10:13 -0400
Received: from lists.gnu.org ([2001:470:142::17]:57050)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raingloom@HIDDEN>) id 1qfb3j-0006m6-Qu
 for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:10:12 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <raingloom@HIDDEN>)
 id 1qfb3Z-0001oZ-4L
 for bug-guix@HIDDEN; Mon, 11 Sep 2023 03:10:01 -0400
Received: from mx1.riseup.net ([198.252.153.129])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <raingloom@HIDDEN>)
 id 1qfb3W-0000fI-Hk; Mon, 11 Sep 2023 03:10:00 -0400
Received: from fews02-sea.riseup.net (fews02-sea-pn.riseup.net [10.0.1.112])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by mx1.riseup.net (Postfix) with ESMTPS id 4RkdBr30skzDqpZ;
 Mon, 11 Sep 2023 07:09:56 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak;
 t=1694416196; bh=jCBwtrOdu4IjIK1DNoGSobWo/knpWw03ag0O4gDJiso=;
 h=References:From:To:Cc:Subject:Date:In-reply-to:From;
 b=p8OnFoG5FSZ4qVyuqU7eoO1NHJb8FeMLpDQu9pEtagI3uq2ku2qWHNqtfOFkZMiSk
 P9inRFGTA0r8s30MFTbGMkkBqwFCiediO2TsptW5zYwKk0QKIZglisSWlbs+QkSsCG
 CDxEWNaOUY4BssSxgZaE29mtnumAa53HgUXUmdhU=
X-Riseup-User-ID: 558C9CB603439A5DAFC0FDC808E10CF6C3CA907716EB7D430B23A41F042139CA
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by fews02-sea.riseup.net (Postfix) with ESMTPSA id 4RkdBb6bTLzFrwy;
 Mon, 11 Sep 2023 07:09:43 +0000 (UTC)
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <86il8mn7al.fsf@HIDDEN> <87jzt0382l.fsf@HIDDEN>
 <86cyyrskmj.fsf@HIDDEN>
From: Csepp <raingloom@HIDDEN>
To: Simon Tournier <zimon.toutoune@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Date: Mon, 11 Sep 2023 09:06:58 +0200
In-reply-to: <86cyyrskmj.fsf@HIDDEN>
Message-ID: <cuca5tt6va2.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=198.252.153.129;
 envelope-from=raingloom@HIDDEN; helo=mx1.riseup.net
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: submit
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 65720 <at> debbugs.gnu.org,
 bug-guix@HIDDEN, Josselin Poiret <dev@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Simon Tournier <zimon.toutoune@HIDDEN> writes:

> Hi,
>
> On Fri, 08 Sep 2023 at 19:09, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>
>>>> It would also be pretty bad for closure size:
>>>>
>>>> --8<---------------cut here---------------start------------->8---
>>>> $ guix size guile-git | tail -1
>>>> total: 106.6 MiB
>>>> $ guix size guile-git git-minimal | tail -1
>>>> total: 169.8 MiB
>>>> --8<---------------cut here---------------end--------------->8---
>>>>
>>>> It=E2=80=99s also not clear concretely how we=E2=80=99d add that depen=
dency.  Try
>>>> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it do=
esn=E2=80=99t work?
>>>> But then, what about applications like Cuirass and hpcguix-web?
>>>
>>> I think we can rely on something like,
>>>
>>>     guix shell -C git-minimal -- git gc
>>
>> We=E2=80=99re talking about the implementation of a cache (meant to spee=
d up
>> operations), that would actually fill said cache plus do a whole bunch
>> of expensive operations?  Nah.  :-)
>
> I do not think.  If I understand correctly, we need to run =E2=80=9Cgit g=
c=E2=80=9D at
> some point, therefore git-minimal needs to me around.  The question is
> how and when.
>
> Well, maybe I am missing what the bug is about.  For me, it is about
> running =E2=80=98git gc=E2=80=99 for cleaning the Git checkout cache, no?
>
>
> Solution #1.  Add git-minimal as inputs.  It increases the closure and
> the extra load (on average) is about the ratio between the rate of =E2=80=
=9Cguix
> pull=E2=80=9D and the rate of the git-minimal changes.
>
> Assuming, that people are running =E2=80=9Cguix pull=E2=80=9D once per we=
ek and say =E2=80=9Cgit
> gc=E2=80=9D is run after 50 pulls.  (These both number are totally arbitr=
ary and
> based on my personal estimate).
>
> Data Service [1] tells:
>
>         2023-07-07 15:45:22 2023-09-08 21:22:08
>         2023-05-11 16:10:48 2023-07-07 14:21:45
>         2023-05-01 16:40:08 2023-05-11 14:36:16
>         2023-04-25 13:34:54 2023-05-01 15:19:55
>         2023-04-25 13:34:54 2023-09-08 21:22:08=20=20=20=20=20=20=20=20
>         2023-03-06 17:22:28 2023-04-25 12:27:33
>         2023-01-17 23:49:19 2023-03-06 16:48:43
>         2022-11-08 13:06:42 2023-01-17 15:11:47
>         2022-10-08 05:14:46 2022-11-08 09:56:31
>         2022-09-06 15:00:08 2022-10-08 04:15:43
>         2022-08-13 22:02:31 2022-09-06 12:58:52
>         =E2=80=A6
>
> It means that an user will download ~10 times git-minimal for nothing.
>
>
> Solution #2.  The one I am proposing. :-)  Download git-minimal only
> when Guix needs it for running =E2=80=9Cgit gc=E2=80=9D.  Yeah, there is =
probably a
> small overload with some operations.  But, I bet this overload is much
> smaller than the one of solution #1.
>
> Well, it depends on the number of times people are updating the cache vs
> the rate of change of git-minimal.
>
> For sure, if one updates 100 times per week the cache, having
> git-minimal as inputs is far better.  But I do not think that the
> regular usage on average. :-)
>
> That=E2=80=99s why I am proposing to have an option for turning off this =
=E2=80=9Cgit
> gc=E2=80=9C operation.
>
> Well, we have lived since years without running =E2=80=98git gc=E2=80=99 =
so running it
> once per year on average is probably enough to keep the cache size
> reasonable.  And git-minimal is changing every month.
>
>
> Maybe, there is some solution #3. ;-)
>
> Cheers,
> simon
>
>
> 1: https://data.guix.gnu.org/repository/1/branch/master/package/git-minim=
al/output-history

Please don't create another situation like with guix system roll-back,
where a crucial sysadmin operation doesn't work without network access.
Or at least make it configurable, so things that are likely to be needed
for future operations are pre-fetched.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 07:06:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 03:06:09 2023
Received: from localhost ([127.0.0.1]:51987 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfazo-0006gN-LU
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:06:08 -0400
Received: from mx0.riseup.net ([198.252.153.6]:52400)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raingloom@HIDDEN>) id 1qfazk-0006fq-TB
 for 65720 <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:06:06 -0400
Received: from fews02-sea.riseup.net (fews02-sea-pn.riseup.net [10.0.1.112])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by mx0.riseup.net (Postfix) with ESMTPS id 4Rkd6C1GS0z9t0C;
 Mon, 11 Sep 2023 07:05:55 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak;
 t=1694415955; bh=DvgxFBmsqF7lmWFKoPyuRScC1d3IK0LLKh3DGJfCs7w=;
 h=References:From:To:Cc:Subject:Date:In-reply-to:From;
 b=sgMvSPrS8Pqcb73dlwo1+I2JXpvmCx1F0Tt0CTVbKQnkF7lWAczBAiDV6jyhC1sCT
 zfuyja8n50BiNer/VIBbZQtZRK/F2f0KqQ9v3JkgROW0dP+ALQ8y9V8WBh3knp4cUh
 fR8lYF8fzNw66grNOuDMoXis7/WCR9JYn1gNymyY=
X-Riseup-User-ID: A0079F29EF7EC8FC5519C967AEF101960BA8D34E079E6B32377221C8FCF49627
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by fews02-sea.riseup.net (Postfix) with ESMTPSA id 4Rkd5z3M0dzFqjr;
 Mon, 11 Sep 2023 07:05:43 +0000 (UTC)
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
From: Csepp <raingloom@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Date: Mon, 11 Sep 2023 09:00:07 +0200
In-reply-to: <87pm2s385m.fsf@HIDDEN>
Message-ID: <cuch6o16vgh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org, bug-guix@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Hello!
>
> Josselin Poiret <dev@HIDDEN> skribis:
>
>> Right, although I wouldn't necessarily say that the former doesn't have
>> a proper API, but rather that it has a Unix-oriented API.  That leads to
>> performance issues on e.g. Windows but on Linux I'm not sure there's
>> much of a difference.
>
> [...]
>
>> We could consider replacing the guile-git dependency with another
>> library built directly on top of git-minimal, and have this be a
>> dependency of Guix.  Not ideal though, and not really scalable either:
>> we can't just add every VCS as direct dependencies.
>
> I cannot imagine a viable implementation of things like =E2=80=98commit-c=
losure=E2=80=99
> and =E2=80=98commit-relation=E2=80=99 from (guix git) done by shelling ou=
t to =E2=80=98git=E2=80=99.
> I=E2=80=99m quite confident this would be slow and brittle.
>
> It looks like there=E2=80=99s no option other than carrying the two
> implementations.
>
> ~~~
>
> Years ago, Andy Wingo sketched a plan for GNU hackers to implement Git
> in pure Scheme.  That was on April 1st though, so people mistakenly
> assumed it was a joke and the project was never carried out.
>
> I digress, but I wonder: is there not even a viable Haskell or OCaml
> implementation of Git?
>
> Thanks,
> Ludo=E2=80=99.

For sake of completeness:
There is an alternative implentation in C for Plan 9 that I've used and
is now mature enough that the 9front project switched to it from
Mercurial.
It might be possible to compile it with the plan9port compiler wrapper.

There is also a Git implementation in OCaml that some MirageOS
unikernels use to serve static content from a git repository.
Also the Irmin "database" is based on git and is written in OCaml.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 11 Sep 2023 07:06:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 03:06:17 2023
Received: from localhost ([127.0.0.1]:51990 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfazw-0006gh-2H
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:06:17 -0400
Received: from lists.gnu.org ([2001:470:142::17]:43114)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raingloom@HIDDEN>) id 1qfazu-0006gO-03
 for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 03:06:14 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <raingloom@HIDDEN>)
 id 1qfazh-0007YJ-O5
 for bug-guix@HIDDEN; Mon, 11 Sep 2023 03:06:02 -0400
Received: from mx0.riseup.net ([198.252.153.6])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <raingloom@HIDDEN>)
 id 1qfaze-0008N9-6O; Mon, 11 Sep 2023 03:06:01 -0400
Received: from fews02-sea.riseup.net (fews02-sea-pn.riseup.net [10.0.1.112])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by mx0.riseup.net (Postfix) with ESMTPS id 4Rkd6C1GS0z9t0C;
 Mon, 11 Sep 2023 07:05:55 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak;
 t=1694415955; bh=DvgxFBmsqF7lmWFKoPyuRScC1d3IK0LLKh3DGJfCs7w=;
 h=References:From:To:Cc:Subject:Date:In-reply-to:From;
 b=sgMvSPrS8Pqcb73dlwo1+I2JXpvmCx1F0Tt0CTVbKQnkF7lWAczBAiDV6jyhC1sCT
 zfuyja8n50BiNer/VIBbZQtZRK/F2f0KqQ9v3JkgROW0dP+ALQ8y9V8WBh3knp4cUh
 fR8lYF8fzNw66grNOuDMoXis7/WCR9JYn1gNymyY=
X-Riseup-User-ID: A0079F29EF7EC8FC5519C967AEF101960BA8D34E079E6B32377221C8FCF49627
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by fews02-sea.riseup.net (Postfix) with ESMTPSA id 4Rkd5z3M0dzFqjr;
 Mon, 11 Sep 2023 07:05:43 +0000 (UTC)
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN> <87pm2s385m.fsf@HIDDEN>
From: Csepp <raingloom@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Date: Mon, 11 Sep 2023 09:00:07 +0200
In-reply-to: <87pm2s385m.fsf@HIDDEN>
Message-ID: <cuch6o16vgh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=198.252.153.6; envelope-from=raingloom@HIDDEN;
 helo=mx0.riseup.net
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: submit
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org, bug-guix@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Hello!
>
> Josselin Poiret <dev@HIDDEN> skribis:
>
>> Right, although I wouldn't necessarily say that the former doesn't have
>> a proper API, but rather that it has a Unix-oriented API.  That leads to
>> performance issues on e.g. Windows but on Linux I'm not sure there's
>> much of a difference.
>
> [...]
>
>> We could consider replacing the guile-git dependency with another
>> library built directly on top of git-minimal, and have this be a
>> dependency of Guix.  Not ideal though, and not really scalable either:
>> we can't just add every VCS as direct dependencies.
>
> I cannot imagine a viable implementation of things like =E2=80=98commit-c=
losure=E2=80=99
> and =E2=80=98commit-relation=E2=80=99 from (guix git) done by shelling ou=
t to =E2=80=98git=E2=80=99.
> I=E2=80=99m quite confident this would be slow and brittle.
>
> It looks like there=E2=80=99s no option other than carrying the two
> implementations.
>
> ~~~
>
> Years ago, Andy Wingo sketched a plan for GNU hackers to implement Git
> in pure Scheme.  That was on April 1st though, so people mistakenly
> assumed it was a joke and the project was never carried out.
>
> I digress, but I wonder: is there not even a viable Haskell or OCaml
> implementation of Git?
>
> Thanks,
> Ludo=E2=80=99.

For sake of completeness:
There is an alternative implentation in C for Plan 9 that I've used and
is now mature enough that the 9front project switched to it from
Mercurial.
It might be possible to compile it with the plan9port compiler wrapper.

There is also a Git implementation in OCaml that some MirageOS
unikernels use to serve static content from a git repository.
Also the Irmin "database" is based on git and is written in OCaml.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 9 Sep 2023 12:33:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Sep 09 08:33:13 2023
Received: from localhost ([127.0.0.1]:46359 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qex9F-0000SZ-9p
	for submit <at> debbugs.gnu.org; Sat, 09 Sep 2023 08:33:13 -0400
Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]:39422)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1qex9A-0000S5-2e
 for 65720 <at> debbugs.gnu.org; Sat, 09 Sep 2023 08:33:08 -0400
Received: by mail-wr1-x434.google.com with SMTP id
 ffacd0b85a97d-3175e1bb38cso481221f8f.1
 for <65720 <at> debbugs.gnu.org>; Sat, 09 Sep 2023 05:33:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694262779; x=1694867579; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=d6ewT6orRq96ryi4+ofr8KXNoRevq8PABtO1IA0+1mQ=;
 b=XYIjXe3ZYWc1X3QQqNofjFpv+cs0GRZcYFG+vrEmoZI4S0ZejGdSI7rTQLDhfta8nd
 GuJlldccjO31QGuVVr2d3RcwrXCDXWBzk0vSDAC8udAnKbb1EqnWg3JzX8idhf8IMfeT
 AwWSr5h1X/srL8/EaAcPcn8vBROodeep8WlioHgiZjIUuLr4qydGfxakAC8x8AKZs38Y
 CTQMar/tY9JnRm36GlMz2mfPLcLjRulh+3iAlj3YuDKVU0iBRZWvjlMnYvrvpAGyJIDh
 judu0S7+3+hKWeRe1p09oiMtm31mDx513cyBqXr513KW1JfbNmOMU/HlOqXB4lESd+9h
 BLGA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1694262779; x=1694867579;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=d6ewT6orRq96ryi4+ofr8KXNoRevq8PABtO1IA0+1mQ=;
 b=UvCXaSGn0+f27zOtL9JBqs93VJM6vBA0KfvEu8lNq8Uiaxo+ZQgiLEXYogPJrYHHdo
 ZvtxwfMmOa4Fwj1k0SCyQNNNyjGvzR6T7j7YbrS0emU6ikG8PV7MywWgsJnIuWnZywyM
 4FNNGQdzQxF5T50rlAITbviiZDG163JYwc81zNsCdOZw2B3xwVWRaasjX/DIHEr2nC4g
 WGbKs21O3Yvi6UZLBR9Klj/mqbCqBOkcyXSO4voWd1kXJ7Hw0pG2dqgKT998cPSH+HUb
 +yl+ZtvrLJqF5/Cl/CGri8rkajAwAaYsQg5TL0Eph2LRATFXq7uuv6dKwW/GA40o3LkB
 5DAQ==
X-Gm-Message-State: AOJu0Yy9Rv8GKkSCkD8BLhWNCJGs9WmeZiUZMlNKYFcbdcHewsjJ5c7s
 MC9MAl2mUP9KJ6GxowdF1OeJhmAUlhg=
X-Google-Smtp-Source: AGHT+IHapoioggDK+q/PoFsj4M5AULl2AXhOseWEAOC9iCQL+ZAdc32q2RdfLcHAQ5DvZ0ZWZcEDtQ==
X-Received: by 2002:a5d:620f:0:b0:319:7624:4ca2 with SMTP id
 y15-20020a5d620f000000b0031976244ca2mr4014016wru.0.1694262779211; 
 Sat, 09 Sep 2023 05:32:59 -0700 (PDT)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id
 x15-20020a5d60cf000000b00317a04131c5sm4704478wrt.57.2023.09.09.05.32.58
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Sat, 09 Sep 2023 05:32:58 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <87jzt0382l.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <86il8mn7al.fsf@HIDDEN> <87jzt0382l.fsf@HIDDEN>
Date: Sat, 09 Sep 2023 12:31:48 +0200
Message-ID: <86cyyrskmj.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

On Fri, 08 Sep 2023 at 19:09, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

>>> It would also be pretty bad for closure size:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> $ guix size guile-git | tail -1
>>> total: 106.6 MiB
>>> $ guix size guile-git git-minimal | tail -1
>>> total: 169.8 MiB
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> It=E2=80=99s also not clear concretely how we=E2=80=99d add that depend=
ency.  Try
>>> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it doe=
sn=E2=80=99t work?
>>> But then, what about applications like Cuirass and hpcguix-web?
>>
>> I think we can rely on something like,
>>
>>     guix shell -C git-minimal -- git gc
>
> We=E2=80=99re talking about the implementation of a cache (meant to speed=
 up
> operations), that would actually fill said cache plus do a whole bunch
> of expensive operations?  Nah.  :-)

I do not think.  If I understand correctly, we need to run =E2=80=9Cgit gc=
=E2=80=9D at
some point, therefore git-minimal needs to me around.  The question is
how and when.

Well, maybe I am missing what the bug is about.  For me, it is about
running =E2=80=98git gc=E2=80=99 for cleaning the Git checkout cache, no?


Solution #1.  Add git-minimal as inputs.  It increases the closure and
the extra load (on average) is about the ratio between the rate of =E2=80=
=9Cguix
pull=E2=80=9D and the rate of the git-minimal changes.

Assuming, that people are running =E2=80=9Cguix pull=E2=80=9D once per week=
 and say =E2=80=9Cgit
gc=E2=80=9D is run after 50 pulls.  (These both number are totally arbitrar=
y and
based on my personal estimate).

Data Service [1] tells:

        2023-07-07 15:45:22 2023-09-08 21:22:08
        2023-05-11 16:10:48 2023-07-07 14:21:45
        2023-05-01 16:40:08 2023-05-11 14:36:16
        2023-04-25 13:34:54 2023-05-01 15:19:55
        2023-04-25 13:34:54 2023-09-08 21:22:08=20=20=20=20=20=20=20=20
        2023-03-06 17:22:28 2023-04-25 12:27:33
        2023-01-17 23:49:19 2023-03-06 16:48:43
        2022-11-08 13:06:42 2023-01-17 15:11:47
        2022-10-08 05:14:46 2022-11-08 09:56:31
        2022-09-06 15:00:08 2022-10-08 04:15:43
        2022-08-13 22:02:31 2022-09-06 12:58:52
        =E2=80=A6

It means that an user will download ~10 times git-minimal for nothing.


Solution #2.  The one I am proposing. :-)  Download git-minimal only
when Guix needs it for running =E2=80=9Cgit gc=E2=80=9D.  Yeah, there is pr=
obably a
small overload with some operations.  But, I bet this overload is much
smaller than the one of solution #1.

Well, it depends on the number of times people are updating the cache vs
the rate of change of git-minimal.

For sure, if one updates 100 times per week the cache, having
git-minimal as inputs is far better.  But I do not think that the
regular usage on average. :-)

That=E2=80=99s why I am proposing to have an option for turning off this =
=E2=80=9Cgit
gc=E2=80=9C operation.

Well, we have lived since years without running =E2=80=98git gc=E2=80=99 so=
 running it
once per year on average is probably enough to keep the cache size
reasonable.  And git-minimal is changing every month.


Maybe, there is some solution #3. ;-)

Cheers,
simon


1: https://data.guix.gnu.org/repository/1/branch/master/package/git-minimal=
/output-history




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 8 Sep 2023 17:10:06 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Sep 08 13:10:06 2023
Received: from localhost ([127.0.0.1]:45457 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeeze-0005Hz-43
	for submit <at> debbugs.gnu.org; Fri, 08 Sep 2023 13:10:06 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:37014)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qeezc-0005HO-Vj
 for 65720 <at> debbugs.gnu.org; Fri, 08 Sep 2023 13:10:05 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qeezV-0002fC-7a; Fri, 08 Sep 2023 13:09:57 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=dXf8knm9Rru+7HimoEi0QYUzsGr+djsbxH5qp3mxxis=; b=MEXysHIhl9LU6xc/ryAr
 wsN5uY0LVxBQsOPTkyMVaH2/0E/xemumCJbEtSYpDE47wSpNymyuSM8AeRxepDIGf3jEE6lvdR+Nd
 dIdmcFOtqHiRBCXySBGkZgXi4kOwEw2+hyk+rTqO31H8hBfhw0yvHY5OzwQUqfTxsTR8Fl4EJD7+n
 jl4Wcx13VcxJaKh9nbYyo/Zkem2SJAx6n7LwKlNupzkXMJ5Ca5CzbobXVADiO8m0mnWAtu5HTjD+m
 +N5cCIVFXhfUfipaP7ff6yX36shIQcPYxfZqB5sS6oEZLcR3daNYcSLv6a3kBGs9PeihxQJ66SjJx
 o5Z0IfQX6ZJtCg==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Simon Tournier <zimon.toutoune@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <86il8mn7al.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Duodi 22 Fructidor an 231 de la =?utf-8?Q?R=C3=A9vol?=
 =?utf-8?Q?ution=2C?= jour de la Noisette
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 08 Sep 2023 19:09:54 +0200
In-Reply-To: <86il8mn7al.fsf@HIDDEN> (Simon Tournier's message of "Thu, 07
 Sep 2023 02:41:54 +0200")
Message-ID: <87jzt0382l.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi!

Simon Tournier <zimon.toutoune@HIDDEN> skribis:

> On Tue, 05 Sep 2023 at 16:18, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>
>> It would also be pretty bad for closure size:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix size guile-git | tail -1
>> total: 106.6 MiB
>> $ guix size guile-git git-minimal | tail -1
>> total: 169.8 MiB
>> --8<---------------cut here---------------end--------------->8---
>>
>> It=E2=80=99s also not clear concretely how we=E2=80=99d add that depende=
ncy.  Try
>> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it does=
n=E2=80=99t work?
>> But then, what about applications like Cuirass and hpcguix-web?
>
> I think we can rely on something like,
>
>     guix shell -C git-minimal -- git gc

We=E2=80=99re talking about the implementation of a cache (meant to speed up
operations), that would actually fill said cache plus do a whole bunch
of expensive operations?  Nah.  :-)

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 8 Sep 2023 17:08:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Sep 08 13:08:18 2023
Received: from localhost ([127.0.0.1]:45452 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeexu-0005F7-Jp
	for submit <at> debbugs.gnu.org; Fri, 08 Sep 2023 13:08:18 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:33820)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qeexs-0005Es-6M
 for 65720 <at> debbugs.gnu.org; Fri, 08 Sep 2023 13:08:17 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qeexk-0001Zu-7i; Fri, 08 Sep 2023 13:08:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=Lhl/xDj71JknxWuiD+ZkuXeTLLdfQqIwQpKUD7oloTI=; b=BU2IfNbirdykSUNnWi08
 6jnLCqvdp7Ds6yb0zLjunNjuqe42RJ3gPulzG3WEEpWLdIzmgkiSo9PWAo2lJs1rcuF17Z/Q3DUWK
 Wo0nUWUGaUCRXUV8kjYkOtCjRKaHVRzbnE48QlA7TcO2oYMGeCz9mNKMLqDOUfEhlCLRRas0Q8ScT
 vxCxPp7uO9oSRFM5ZuTACDAx3CGgjJElVyk6RFMPY83x9y8ZCqFe3NgMtBK3wlLfq8HFdTTc035SA
 ifiQxoXFzkOMHH2s5XAOzvo7wrnihk+BFeIrZ62yfKOFYRAXkP6UFXneLYDPdmutAaFLXdc+o7e8T
 D8DYM6qXN3ywvQ==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Josselin Poiret <dev@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
 <87pm2vvibo.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Duodi 22 Fructidor an 231 de la =?utf-8?Q?R=C3=A9vol?=
 =?utf-8?Q?ution=2C?= jour de la Noisette
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 08 Sep 2023 19:08:05 +0200
In-Reply-To: <87pm2vvibo.fsf@HIDDEN> (Josselin Poiret's message of "Wed, 
 06 Sep 2023 10:04:11 +0200")
Message-ID: <87pm2s385m.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello!

Josselin Poiret <dev@HIDDEN> skribis:

> Right, although I wouldn't necessarily say that the former doesn't have
> a proper API, but rather that it has a Unix-oriented API.  That leads to
> performance issues on e.g. Windows but on Linux I'm not sure there's
> much of a difference.

[...]

> We could consider replacing the guile-git dependency with another
> library built directly on top of git-minimal, and have this be a
> dependency of Guix.  Not ideal though, and not really scalable either:
> we can't just add every VCS as direct dependencies.

I cannot imagine a viable implementation of things like =E2=80=98commit-clo=
sure=E2=80=99
and =E2=80=98commit-relation=E2=80=99 from (guix git) done by shelling out =
to =E2=80=98git=E2=80=99.
I=E2=80=99m quite confident this would be slow and brittle.

It looks like there=E2=80=99s no option other than carrying the two
implementations.

~~~

Years ago, Andy Wingo sketched a plan for GNU hackers to implement Git
in pure Scheme.  That was on April 1st though, so people mistakenly
assumed it was a joke and the project was never carried out.

I digress, but I wonder: is there not even a viable Haskell or OCaml
implementation of Git?

Thanks,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 7 Sep 2023 00:43:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Sep 06 20:43:43 2023
Received: from localhost ([127.0.0.1]:38163 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qe37X-0004h6-9x
	for submit <at> debbugs.gnu.org; Wed, 06 Sep 2023 20:43:43 -0400
Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]:41365)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1qe37T-0004gS-N9
 for 65720 <at> debbugs.gnu.org; Wed, 06 Sep 2023 20:43:40 -0400
Received: by mail-wm1-x32e.google.com with SMTP id
 5b1f17b1804b1-401e6ce2d9fso1031245e9.1
 for <65720 <at> debbugs.gnu.org>; Wed, 06 Sep 2023 17:43:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694047412; x=1694652212; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=hMZsS0KKScfuGIZ10c6RNyODKqltKAVWn2apINlpol0=;
 b=m6oBDHt7M6ajRVArZs+ixrIWhJfiwlY6vh8olqLpIeyp/XvBV9+5YhyDY/1ia3XHc5
 IR84Z6seEgfUM7zKjFVAS8RRUMt/195EXH9Sac1d1lsl1ToZzwk3gPjwsCJZY4M+ihEW
 HokuC/bNeS2bhGEGVrZu6OobxIsUrrSWkYOQNaEQ4rG96LhbFaY7ScW6WUURAGs6NcQ4
 nhKokFqLnExUlbct+JRQaEOpZb8TzS2eSdsIL2Ozhdbi4pGU62P2H1n0IoKBKpyISY00
 Ays54sQPCT4JJLo+1Kax5MVCUxX2jbJc7oVRrD1paROP5w6LwU+9Dyz2LeoDENdJzJ77
 DuUg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1694047412; x=1694652212;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=hMZsS0KKScfuGIZ10c6RNyODKqltKAVWn2apINlpol0=;
 b=OAB6G43AoHpOSn89n+tnuxzBGwm9gCDRASrptNuoQ+Iotpbh19LnbzkeuBfF80PI+X
 wjBF7ePsirZy2tnP3PSZMni5y6mifkW/qgeogKolpt5CLy/NTpwxxouDfwtgMw7oJQBA
 KXkWinKwwofUmF8hB6KTdr3z2HgOgvROsaIgfN/r+Nm2Opn071BTxjHBl+fnZvFnmTGZ
 IrsznP8Sxo7E6CAnB/QhXvHo5sJaoTUHfv3mn0zIfFCZtncByR5ibrhsSi25vHpUSwMk
 Td6UqcSze8a7ffYn5EJ58IQJh1OMKOOfsNn2KKupQAm2K5tr+wa5pfb3wJmmn8OnjR4J
 hksA==
X-Gm-Message-State: AOJu0YxeMuZOHAQSl6+kev6NBHERiBEJKN+AaZWjvv5ROMTqxakDapLr
 BOtvTsd7M6Mdwcn5mVnqXQIS52vC3F0=
X-Google-Smtp-Source: AGHT+IGp9HK6PcjPwF89K2Tuc+A8RlkJd9zGQlDjGUfWgtXaB4XR4uhZs1naGE3781v4/pHe19yn0A==
X-Received: by 2002:a5d:4b87:0:b0:31a:ea18:c516 with SMTP id
 b7-20020a5d4b87000000b0031aea18c516mr13157801wrt.3.1694047412002; 
 Wed, 06 Sep 2023 17:43:32 -0700 (PDT)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id
 y1-20020adff6c1000000b00317a29af4b2sm21595830wrp.68.2023.09.06.17.43.31
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Wed, 06 Sep 2023 17:43:31 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Josselin Poiret
 <dev@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <87msy0ln4m.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
Date: Thu, 07 Sep 2023 02:41:54 +0200
Message-ID: <86il8mn7al.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

On Tue, 05 Sep 2023 at 16:18, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

> It would also be pretty bad for closure size:
>
> --8<---------------cut here---------------start------------->8---
> $ guix size guile-git | tail -1
> total: 106.6 MiB
> $ guix size guile-git git-minimal | tail -1
> total: 169.8 MiB
> --8<---------------cut here---------------end--------------->8---
>
> It=E2=80=99s also not clear concretely how we=E2=80=99d add that dependen=
cy.  Try
> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it doesn=
=E2=80=99t work?
> But then, what about applications like Cuirass and hpcguix-web?

I think we can rely on something like,

    guix shell -C git-minimal -- git gc

It would be invoked internally using the Scheme API for inferiors and
friends.  Doing so, it would add nothing to the closure size.

It appears to me safe to assume that this command can be run from any
Guix installation.  Since the Git GC would only be done once every X Git
fetches, the overhead would be much lower.

Hum, am I repeating myself [1]? :-)

And I would run this =E2=80=9Cgit gc=E2=80=9D via =E2=80=9Cguix gc=E2=80=9D=
, not via =E2=80=9Cguix pull=E2=80=9D.  Well,
I do not like all these automatic removals happening based on date
(last-expiry-cleanup) with some usual commands.  It always happens when
I do not want. ;-) Contrary to =E2=80=9Cguix gc=E2=80=9D.  Bah, another sto=
ry. :-)

Cheers,
simon


1: bug#65720: Guile-Git-managed checkouts grow way too much
Simon Tournier <zimon.toutoune@HIDDEN>
Tue, 05 Sep 2023 20:59:07 +0200
id:86edjcqwec.fsf@HIDDEN
https://issues.guix.gnu.org//65720
https://issues.guix.gnu.org/msgid/86edjcqwec.fsf@HIDDEN
https://yhetil.org/guix/86edjcqwec.fsf@HIDDEN






Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 6 Sep 2023 08:04:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Sep 06 04:04:26 2023
Received: from localhost ([127.0.0.1]:60737 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdnWU-0004DF-36
	for submit <at> debbugs.gnu.org; Wed, 06 Sep 2023 04:04:26 -0400
Received: from jpoiret.xyz ([206.189.101.64]:34952)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dev@HIDDEN>) id 1qdnWR-0004D6-9H
 for 65720 <at> debbugs.gnu.org; Wed, 06 Sep 2023 04:04:24 -0400
Received: from authenticated-user (jpoiret.xyz [206.189.101.64])
 by jpoiret.xyz (Postfix) with ESMTPA id 91F69184D5F;
 Wed,  6 Sep 2023 08:04:20 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jpoiret.xyz; s=dkim;
 t=1693987460;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
 in-reply-to:in-reply-to:references:references;
 bh=gyi8zkcyvxV3/1pT9N7jwCyamv5CshT2blTVJG3Xnrc=;
 b=fQg34g6JyiP+/me0aYN7zBWmsq5wxDC+grcS0a3n7Du2UVpugp4VzeUthbr+yLvznPY/6G
 C3U4KKoK9LcFooMUDcARSzBrmBOO4iR/rQbocaerR7nCoJ9Q4YnEu00vKyUYItgdTeNudo
 tN86jN9oE7QlfvXjImUROsHF0B0EBnNqz6zR58ljB8ztH4iH9LtE2tAK4cSUuizuePgQix
 KwJsjMGGcw6o1tSBvDi2OiZcz24nW7b1sqU70Aaoa0yTu7k1+nBIMBhtcPYUtFPizu1iZN
 X57Oaq9rFdJFZnGuf0mVP5h4JMvLbZx+U/BNFHo1ingazKEaqviwAS/DCFH/hA==
From: Josselin Poiret <dev@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <87msy0ln4m.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN> <87msy0ln4m.fsf@HIDDEN>
Date: Wed, 06 Sep 2023 10:04:11 +0200
Message-ID: <87pm2vvibo.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spamd-Bar: --
Authentication-Results: jpoiret.xyz;
 auth=pass smtp.auth=jpoiret@HIDDEN smtp.mailfrom=dev@HIDDEN
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Ludo,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Surely you=E2=80=99d agree that it would suck though: depending on two Git
> implementations because one doesn=E2=80=99t have a proper API and the oth=
er one
> lacks a bunch of features.

Right, although I wouldn't necessarily say that the former doesn't have
a proper API, but rather that it has a Unix-oriented API.  That leads to
performance issues on e.g. Windows but on Linux I'm not sure there's
much of a difference.

> It would also be pretty bad for closure size:
>
> --8<---------------cut here---------------start------------->8---
> $ guix size guile-git | tail -1
> total: 106.6 MiB
> $ guix size guile-git git-minimal | tail -1
> total: 169.8 MiB
> --8<---------------cut here---------------end--------------->8---
>
> It=E2=80=99s also not clear concretely how we=E2=80=99d add that dependen=
cy.  Try
> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it doesn=
=E2=80=99t work?
> But then, what about applications like Cuirass and hpcguix-web?
>
> Tricky, tricky.

We could consider replacing the guile-git dependency with another
library built directly on top of git-minimal, and have this be a
dependency of Guix.  Not ideal though, and not really scalable either:
we can't just add every VCS as direct dependencies.

From=20what I've seen, people are now scaling back on their use of
libgit2 because of the impedence mismatch and are resorting more and
more to git plumbing.  From a pragmatic point of view, I'd prefer the
latter, since it is more stable and feature-complete.

Best,
=2D-=20
Josselin Poiret

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQHEBAEBCgAuFiEEOSSM2EHGPMM23K8vUF5AuRYXGooFAmT4MnsQHGRldkBqcG9p
cmV0Lnh5egAKCRBQXkC5FhcairUvDACAJZdGUEBC2qVWbsms7Xk6OUTUgfjucIC3
XOLFH74Ewo4OdrUJUrADcWP4GKjrEmglO1hQjRlTwpo60TB7CkFyZIC39Dkm0MPm
R7Oc8BYnPByFHihy3RJwrtk0zH1jOaRd/A6cvdIXCrXk1rnlTBn9EEAKpYlA1OrG
7al4FfxKfFkea48xZsGVM8uc1fsqiHrycZH3gLCbT8V0O4BtNY2rhYLf1eTjFkQl
PbcAdHmkOcnQaZR2WzUHUUH/9GGrHwcXkqSZtgnJ8y/zauig5nyWzX3Zgej4K+VJ
nN0l6QCVtkaAIgCN2+8zTg2ml+WeXjWcE1gGjVyv7748ICmpP3jXE0uGTSGvM0MX
ZFfi+TKU4wJUl8fCZLf41P5v7P1jSy2TbthqbAHkRKlsasYMf/KZMhSh6jypVIN0
VOFPWMUIJhZNQMP2DtVOMC/thc+O8BcfkbFSORNJ5XYaLiNvC4/ODwsKCGxYOa2h
4XlDMtm7YgAewG19hO9fYvbS1oXKLZM=
=olI3
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 19:01:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 15:01:20 2023
Received: from localhost ([127.0.0.1]:59287 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdbId-0003sQ-RT
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 15:01:20 -0400
Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]:38129)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>) id 1qdbIZ-0003rn-Lg
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 15:01:19 -0400
Received: by mail-wm1-x330.google.com with SMTP id
 5b1f17b1804b1-401ce65dfc4so9350215e9.0
 for <65720 <at> debbugs.gnu.org>; Tue, 05 Sep 2023 12:01:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1693940469; x=1694545269; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=mCfMYEy946HhJdJ3oyKFtf36lO0XRBfdG9cIIRIKuwQ=;
 b=gqPXDMQTtZd2XIEOandnyWc2j/Rp+6Q8Um0a5zCjPWBKWmHEVzOzDTtybVgmW8bQky
 8jaqroq/U3ZUakSPLHPx8EYYeAJmYturVNXyaDBicIgiUpjEXIxA3xEwaLBSxFJf+Or6
 vLtCJX9Ev4eS1Tanv0S85odW88jPbuC2TzMJE9tvEGT8bG3mzilDKofBWAPyCtJRNODW
 agnLUXihkZC6b6wr/5nqE3xA+NN5WxtzX7nHJ1UOltt05N9+5ME/pLqmd6SWi0+mcgKa
 k38qnW/v/f2yZ+u86+TFgOeX+ohq5xfknn3QLpypyS65nDswhfpztWVWrEJwSIOORzWm
 DWZw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1693940469; x=1694545269;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:to:from:x-gm-message-state:from:to:cc:subject
 :date:message-id:reply-to;
 bh=mCfMYEy946HhJdJ3oyKFtf36lO0XRBfdG9cIIRIKuwQ=;
 b=fNUEAENc6OWDaEoyMkH4+ltgXOqkX6kNquEARTuEY6q9uTYyONVl9yP9omveB2IRp3
 oAGnNZvJORdkoKXUxtWp5w2OsMjQozrgPmpqv4fqz68DBVGFkwRn9g18eXKF95WlUZYv
 AWdpnRzRa0W99XFwqBTs9s4Go6dF66hQrm9tTzKTQ0foCh1Aubt1a9/HMI2yzWPKFqaz
 jzzcHbQSfH6I+R2PdVIXTiNECO1io3kV5dYHYj/Bhje7l8Yq7A4b/0liStUBwRlsTJso
 dtRlvpTKTEfIUFXM0QLRDGDL+eGVa9vZ7Z3hKjAKIVVT9uljEY3n+r5tFwd4c1PiMVJC
 n0Xw==
X-Gm-Message-State: AOJu0Yw4cAQSirTYFkQOfP1peKs8lIEBunhaQwxXPdgnIhXKiaXwgJ0B
 K0r5fIbUFelB8SX+dE55dSPDU+oRwQQ=
X-Google-Smtp-Source: AGHT+IGqfVbO7szTpGsEGwU/DGl9YdLE1sHWu/1G247m6RjDE09ds/itQgaLLpMAwm/cMo4OcywgsQ==
X-Received: by 2002:a05:6000:141:b0:31a:e29f:5eab with SMTP id
 r1-20020a056000014100b0031ae29f5eabmr9325264wrx.7.1693940468913; 
 Tue, 05 Sep 2023 12:01:08 -0700 (PDT)
Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e])
 by smtp.gmail.com with ESMTPSA id
 bt9-20020a056000080900b0031c5ce91ad6sm14297502wrb.97.2023.09.05.12.01.08
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 05 Sep 2023 12:01:08 -0700 (PDT)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 65720 <at> debbugs.gnu.org
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <87fs3tobju.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
Date: Tue, 05 Sep 2023 20:59:07 +0200
Message-ID: <86edjcqwec.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 65720
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

On Mon, 04 Sep 2023 at 23:47, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

>> It would seem that libgit2 doesn=E2=80=99t do the equivalent of =E2=80=
=98git gc=E2=80=99.
>
> Confirmed: <https://github.com/libgit2/libgit2/issues/3247>.

Ouch!

        The goals of the project haven't changed, and neither have the
        tradeoffs. If one were to rewrite git-gc on top of libgit2, the
        best-case scenario is ending up with what we already had.

        If you want to use regular maintenance on some repostories, use
        git gc, that's what it's there for.

        https://github.com/libgit2/libgit2/issues/3247#issuecomment-1525080=
40

> My inclination for the short term would be to work around this
> limitation by (1) finding a heuristic to determine is a checkout has
> likely accumulated too much cruft, and (2) considering such checkouts
> as expired (thereby forcing a re-clone) or running =E2=80=98git gc=E2=80=
=99 on them if
> =E2=80=98git=E2=80=99 is available.

About (1) maybe we could add a =E2=80=9Ccounter=E2=80=9D and teach after X =
updates of
the checkout then let run (2).  Well, I guess the number of crufts is
more or less proportional with the number of checkout updates; that=E2=80=
=99s
the heuristic I would use.

The most annoying is (2).  Because forcing a re-clone does not appear to
me a solution; I prefer to waste disk space (and probably run myself and
manually =E2=80=98git gc=E2=80=99) than re-clone=E2=80=A6 Somehow this re-c=
lone would always
happen when I am using a poor network.

Moreover, assuming this clean-up (2) would be run once every while, we
could imagine to invoke something like,

    guix shell -C git-minimal
         -- git
         -C ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr2=
7shj7apsnalwq
         gc

when the checkout is updated.  And maybe we could provide another =E2=80=9C=
guix
pull=E2=80=9D command-line option for turning off this and mark it as done
(reset the =E2=80=9Ccounter=E2=80=9D).

Well, that=E2=80=99s a poor solution but we can assume that git-minimal is =
at
worse available using =E2=80=9Cguix shell git-minimal=E2=80=9D.  Note that =
the closure
of git-minimal is far less than re-cloning the full Guix repository.

Cheers,
simon




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 14:21:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 10:21:27 2023
Received: from localhost ([127.0.0.1]:57493 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdWvn-00029Y-Iz
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:21:27 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:49908)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qdWvk-00029K-5Q
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:21:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qdWvZ-0007oP-6U; Tue, 05 Sep 2023 10:21:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=Bt5Z+dmoKEFfEqZncGEW60V+4NqlHWt73Qz997Jox3U=; b=YpIru6c1+oKA4CaaAvWl
 sWF1RBZXf+anUYk/Dhb19VU/MDJwQix9LFo/RBJKwFloZzaoW8WY7CaPsa4cvdt2VDoS/RI+7ca1y
 jnK2l0EO67wrU8Zqc1AMdJTyOkjrKOTSxC0iEsvKgHDyXGJm8L7O1F58D3w8Oo6a5ZzVu792pUFgz
 yABKzFSmGn1xSrYfvXpLP8cN6unB4DOCq5Kpq+tLZe06dc5UEs2MGc9Hq5KFEjy51CdULuERBjS9h
 PwQH4x7eOhFYIhhyunZIPHY8eOU5qCMkutzz2FgVcoYiytXxHmB9CVqE14l4x4npzW+Aa9wimW8lN
 P1C6Yt02569ZKA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Jelle Licht <jlicht@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87fs3tobju.fsf@HIDDEN>
 <CE9E1465-187B-462B-B9E2-94E6A43B86EC@HIDDEN>
Date: Tue, 05 Sep 2023 16:20:57 +0200
In-Reply-To: <CE9E1465-187B-462B-B9E2-94E6A43B86EC@HIDDEN> (Jelle Licht's
 message of "Tue, 5 Sep 2023 08:22:48 +0000")
Message-ID: <87il8oln06.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Jelle Licht <jlicht@HIDDEN> skribis:

>> On 4 Sep 2023, at 23:49, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>>=20
>> Of course having to re-clone entire repositories every 9 months is
>> ridiculous, but storing gigabytes of packs is worse IMO (I=E2=80=99m
>> specifically thinking about the Guix repo, which every users copies via
>> =E2=80=98guix pull=E2=80=99).
>
> Please ignore if it doesn=E2=80=99t make sense, or would not make a pract=
ical difference for the current issue, but wouldn=E2=80=99t a local clone d=
o the trick here? As in, clone from the =E2=80=98clogged=E2=80=99 local rep=
o, move over fresh clone to old location.

Good question.

--8<---------------cut here---------------start------------->8---
scheme@(guix git)> ,use(git)
scheme@(guix git)> (clone "/home/ludo/.cache/guix/checkouts/pjmkglp4t7znuug=
eurpurzikxq3tnlaywmisyr27shj7apsnalwq/" "/tmp/fresh-clone")
$7 =3D #<git-repository ba4240>
scheme@(guix git)> (system* "du" "-hs" "/tmp/fresh-clone")
6.7G	/tmp/fresh-clone
$8 =3D 0
scheme@(guix git)> (system* "du" "-hs" "/tmp/fresh-clone/.git")
6.6G	/tmp/fresh-clone/.git
$9 =3D 0
scheme@(guix git)> (system* "du" "-hs" "/home/ludo/.cache/guix/checkouts/pj=
mkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq/")
6.7G	/home/ludo/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisy=
r27shj7apsnalwq/
$10 =3D 0
--8<---------------cut here---------------end--------------->8---

Conclusion: it makes no difference.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 14:18:30 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 10:18:30 2023
Received: from localhost ([127.0.0.1]:57485 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdWsw-00024t-0q
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:18:30 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:37652)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qdWss-00024d-PT
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:18:27 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qdWsm-00075u-MQ; Tue, 05 Sep 2023 10:18:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=gZEO0byQQf3bv9yNUhHvKI8UoLiI2n6yEJlAHLep+v4=; b=WrTkjU1vm18VaVFLfJxK
 oes9R9tjL00VRNzF8RvCpCHnfa0Ksjqa2Cp+VJW1waO3RmmL1luFIhoIRwd9WKO35cuoqvjLi31be
 3iTS/6GoK1vGVHH8z7Pdd4gvnC2G0hCRbq6dkXEVuIYBegoA5CEbRLBZ7EWNO/zQuZ7KBs9l0vuYZ
 Muq4z43NPF6L1VZXKIEWrl9rTUcvilQjTf9rjp7DlDC+V3dXDoVivDCqUDnQG2bkzny7I6z2wILB5
 LINW4/Qj2J2q+Ro3rM0wcgB0J+6Lt7NR1FYy58g7ueNrCZcxxITAQmnpi5kG2Y7RSz4fiJSuTuvwN
 P3VHhV1vXRCmaQ==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Josselin Poiret <dev@HIDDEN>
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
 <87tts9uj6x.fsf@HIDDEN>
Date: Tue, 05 Sep 2023 16:18:17 +0200
In-Reply-To: <87tts9uj6x.fsf@HIDDEN> (Josselin Poiret's message of "Tue, 
 05 Sep 2023 10:18:30 +0200")
Message-ID: <87msy0ln4m.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Josselin Poiret <dev@HIDDEN> skribis:

> I think using the git binary instead of libgit2 as a workaround is a
> good idea.  We can consider building it directly as well, so that people
> who don't have it in their profiles can still benefit from it.  We could
> even consider using git commands in most places and using libgit2 only
> where we really need the tight coupling.

Surely you=E2=80=99d agree that it would suck though: depending on two Git
implementations because one doesn=E2=80=99t have a proper API and the other=
 one
lacks a bunch of features.

It would also be pretty bad for closure size:

--8<---------------cut here---------------start------------->8---
$ guix size guile-git | tail -1
total: 106.6 MiB
$ guix size guile-git git-minimal | tail -1
total: 169.8 MiB
--8<---------------cut here---------------end--------------->8---

It=E2=80=99s also not clear concretely how we=E2=80=99d add that dependency=
.  Try
invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it doesn=
=E2=80=99t work?
But then, what about applications like Cuirass and hpcguix-web?

Tricky, tricky.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 14:12:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 10:12:03 2023
Received: from localhost ([127.0.0.1]:57465 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdWmh-0001vH-6E
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:12:03 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:58188)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qdWmd-0001ul-O6
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:12:01 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1qdWmX-00063x-P7
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 10:11:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=Gq6HBLNSzUAf5+rzaCwbVT8YuN3r8ctRRGblQ3J+5X4=; b=UZsaQ744GAFUzmxGVZFj
 0iSOUWoAHNugEitWhVLK/ru+d/Qj6Qhxa4rzE40Kb3XrYNW4zuvegyuJs+snjCDkbrDIyzw1WnZ0P
 VQ7OnRds3wWPcV7wfoF0PAYlJYVmYkn2ygyZEMnBeyAtP2M9661GP7W5r3a+cWyKtYP3ixPDYc+KP
 VLZe3m4XX3L5c3Lk0nzx2/zyqeKHi+suJZj6Zx6zt07nwtY5OvBpNE5trYJ4ZYpBLd/qi6dh+arGS
 g3bRDGy7IpavDvKhJ0GJG4XGNkvIREoKdRza/Cyhs+eHGHjjbFlb5jELZ8FdqUNZOvducPviMbwvU
 +wjclxceVmYbqw==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: 65720 <at> debbugs.gnu.org
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN>
Date: Tue, 05 Sep 2023 16:11:33 +0200
In-Reply-To: <87bkejc7go.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?=
 =?utf-8?Q?'s?= message of "Sun, 03 Sep 2023 22:44:39 +0200")
Message-ID: <87wmx4lnfu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> $ du -hs ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27=
shj7apsnalwq
> 6.7G    /home/ludo/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlay=
wmisyr27shj7apsnalwq

Another data point, with Cuirass instances:

--8<---------------cut here---------------start------------->8---
ludo@berlin ~$ sudo du -hs /var/lib/cuirass/.cache/guix/checkouts/pjmkglp4t=
7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
65G     /var/lib/cuirass/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3t=
nlaywmisyr27shj7apsnalwq
ludo@berlin ~$ sudo stat /var/lib/cuirass/.cache/guix/checkouts/pjmkglp4t7z=
nuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq | tail -1
 Birth: 2022-07-30 23:15:45.582559879 +0200
--8<---------------cut here---------------end--------------->8---

=E2=80=A6 and:

--8<---------------cut here---------------start------------->8---
ludo@guix-hpc4 ~$ sudo du -hs /var/lib/cuirass/.cache/guix/checkouts/pjmkgl=
p4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
86G     /var/lib/cuirass/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3t=
nlaywmisyr27shj7apsnalwq
ludo@guix-hpc4 ~$ sudo stat /var/lib/cuirass/.cache/guix/checkouts/pjmkglp4=
t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq | tail -1
  Cr=C3=A9=C3=A9=C2=A0: 2021-06-01 11:48:48.854669310 +0200
--8<---------------cut here---------------end--------------->8---

So yeah, problem we have.

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 08:23:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 04:23:13 2023
Received: from localhost ([127.0.0.1]:53475 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdRL7-0003no-5m
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 04:23:13 -0400
Received: from mout01.posteo.de ([185.67.36.65]:59791)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jlicht@HIDDEN>) id 1qdRL1-0003nE-Tl
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 04:23:11 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id AAA06240027
 for <65720 <at> debbugs.gnu.org>; Tue,  5 Sep 2023 10:23:01 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1693902181; bh=6gQz5IO3eRHSw9H7uOYRWnTisVSy8sLhtyfs7TlisDI=;
 h=Content-Transfer-Encoding:From:Mime-Version:Subject:Date:
 Message-Id:Cc:To:From;
 b=eWuu7e2M/zsNmRcz/5KpaLuRDYhWhTYqUi8CBZxys5Tcgnli7MfG7rrZtpBisXTto
 i3Tz1PMoBRndFdC2nY9WFOK1L5gmUYO0XTiFWJ9WQn8U3EfBLaJLyWWFidRgy7AhRN
 WuFH/4OMfyDPjMP9zy4TfkfCeY7kUKB2RjcK3Lt/XEz7tbu77nK7vcXicYNIwsskkI
 RAt4O90Y8Rs0RHCV1Zis9sbo8LNuwI0ziEW4mHvwPqzbT/fclTAIsuFO40CqKBv6Da
 sAWLcraM5RSveomfX61b+XQr6UvWDLAgLL7/8a+3+JOP9Tn45DfgjrfVtosOxHwarl
 ABJ1I71WFDTEg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4Rfz5x0bcyz6trs;
 Tue,  5 Sep 2023 10:23:00 +0200 (CEST)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Jelle Licht <jlicht@HIDDEN>
Mime-Version: 1.0 (1.0)
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
Date: Tue,  5 Sep 2023 08:22:48 +0000
Message-Id: <CE9E1465-187B-462B-B9E2-94E6A43B86EC@HIDDEN>
References: <87fs3tobju.fsf@HIDDEN>
In-Reply-To: <87fs3tobju.fsf@HIDDEN>
To: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 65720
Cc: 65720 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hi Ludo,

>=20
> On 4 Sep 2023, at 23:49, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:
>=20
> Of course having to re-clone entire repositories every 9 months is
> ridiculous, but storing gigabytes of packs is worse IMO (I=E2=80=99m
> specifically thinking about the Guix repo, which every users copies via
> =E2=80=98guix pull=E2=80=99).

Please ignore if it doesn=E2=80=99t make sense, or would not make a practica=
l difference for the current issue, but wouldn=E2=80=99t a local clone do th=
e trick here? As in, clone from the =E2=80=98clogged=E2=80=99 local repo, mo=
ve over fresh clone to old location.

Kr, Jelle=





Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 5 Sep 2023 08:18:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Sep 05 04:18:55 2023
Received: from localhost ([127.0.0.1]:53458 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdRGw-0003gJ-Os
	for submit <at> debbugs.gnu.org; Tue, 05 Sep 2023 04:18:55 -0400
Received: from jpoiret.xyz ([206.189.101.64]:55990)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dev@HIDDEN>) id 1qdRGt-0003g9-0b
 for 65720 <at> debbugs.gnu.org; Tue, 05 Sep 2023 04:18:53 -0400
Received: from authenticated-user (jpoiret.xyz [206.189.101.64])
 by jpoiret.xyz (Postfix) with ESMTPA id 968F91851DF;
 Tue,  5 Sep 2023 08:18:48 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jpoiret.xyz; s=dkim;
 t=1693901929;
 h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
 to:to:cc:mime-version:mime-version:content-type:content-type:
 in-reply-to:in-reply-to:references:references;
 bh=8pM8T1310gSR8edOn+Ojn9vP6FDTUMbm5DyEBYbX+RI=;
 b=E5fqAWKR36KGu3A3dWgbPYvmxOS6zdgYXQaFn0sY6toc3M6Vl8n7iH8o9OO9Y2qm/iwym1
 zgSUgNHbcungycJPm0MLBLOVu6Wkk6wyduVNr3ikIRY6sp/YSZrp2Bgd0Gj+HKWFvb0t7O
 qN5EBIvXJ7Tod6fj+RSHbv01NsGsQhzxEB5RL8Pf34mzEvDFgqYlRcdG5LhBzq3ddd0U3s
 FmV8aXD/mYuE3lOpePWx8AOFY9f9prlilxl81LlFBR9of/nFj6XW1rWXohZRIr3G3dclES
 TRqSO8aixv/w4wYwV44Sass8VOvHX5CE6ZrzHNCbtcjkEDiaWtpzUcZrh1e34g==
From: Josselin Poiret <dev@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, 65720 <at> debbugs.gnu.org
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
In-Reply-To: <87fs3tobju.fsf@HIDDEN>
References: <87bkejc7go.fsf@HIDDEN> <87fs3tobju.fsf@HIDDEN>
Date: Tue, 05 Sep 2023 10:18:30 +0200
Message-ID: <87tts9uj6x.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spamd-Bar: --
Authentication-Results: jpoiret.xyz;
 auth=pass smtp.auth=jpoiret@HIDDEN smtp.mailfrom=dev@HIDDEN
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 65720
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Ludo,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> My inclination for the short term would be to work around this
> limitation by (1) finding a heuristic to determine is a checkout has
> likely accumulated too much cruft, and (2) considering such checkouts as
> expired (thereby forcing a re-clone) or running =E2=80=98git gc=E2=80=99 =
on them if
> =E2=80=98git=E2=80=99 is available.

I think using the git binary instead of libgit2 as a workaround is a
good idea.  We can consider building it directly as well, so that people
who don't have it in their profiles can still benefit from it.  We could
even consider using git commands in most places and using libgit2 only
where we really need the tight coupling.  IIUC, libgit2 is eternally
trying to catch up to git and often performs in a counter-intuitive way
(I expect the various bugs with stale deleted files in checkouts to be
caused by this).  Maybe it could also let us use bare repository and
directly extract the refs we want without having to mess with checkouts?

Best,
=2D-=20
Josselin Poiret

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQHEBAEBCgAuFiEEOSSM2EHGPMM23K8vUF5AuRYXGooFAmT25FYQHGRldkBqcG9p
cmV0Lnh5egAKCRBQXkC5Fhcain93DACHuEyLuP52K5rHucB5+rmiiaHAqwh6U7Us
GdD98bt8ggLcGkuJviQJKAL7sWrbLZLZGoFvGOVSIFU71zixL0aDy0vLHdLrr9kw
nhlp9FBrsTE1WJ87n6cqN7QEGAKdecX8QSHzAPSgOiPniby8DDML/EZ5qkJA8HeA
x8atrhamPs/j9lUiWMR94O9eTAo0iOrZ3V+o6Phc7711vRWXiSOneIkcrCcylLfs
gtCjJUaVp2AwZXNvvCj8Lf17wWcEacsdvi4e6hTzhnT75xiDcCCc5O682F+lLPy2
XzaDc6GAql8y2tt+/zuNUa2S+anBXFGaqqz5Dxzmm3Vi/AEPA24dbCUyjmw+gX+z
yzCbSSdt2wWKT9U3Us6jQWbtzztwRjHEqNLFS7NqZvCRZA2UiDF9XxZSLlB6jZB8
01biNYDABcz1SDgEFj007l2iBoxtTWwDeOt1bTdrYP3pmTZpojf0mCox4RXkviHB
z6d2UwRQq8C9aGdHVCM0n6FSzs6a7l4=
=6KMj
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.

Message received at 65720 <at> debbugs.gnu.org:


Received: (at 65720) by debbugs.gnu.org; 4 Sep 2023 21:48:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 04 17:48:03 2023
Received: from localhost ([127.0.0.1]:52859 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qdHQQ-0003u2-Hy
	for submit <at> debbugs.gnu.org; Mon, 04 Sep 2023 17:48:03 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:43088)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qdHQM-0003sn-7m
 for 65720 <at> debbugs.gnu.org; Mon, 04 Sep 2023 17:48:01 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1qdHQG-0006vJ-Lm
 for 65720 <at> debbugs.gnu.org; Mon, 04 Sep 2023 17:47:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=BurYVm1rImaTmYgUS5+9Jz3dIfEvZ64VE5iDtpY6cmI=; b=Ec4/3vhf5zNdNW4lKvpy
 4XoBmWZ7JEUZt0CvLm9yNWuCgOJxEBWGa3ratQojSKu6/QiCo0xWe2muZVgsAQCCTWfyVNtLqAHKF
 DAOJ/s5K4mWL5ZASq59Fa8Lq+8O8w72B7rdh4yMk1RbmjXQm80F3a/25EHfP/buVA5QMtN8s6oRye
 U3YnB3ZZ7e+NwO+ZS62u/3SJCaOV9xUvkoVHCoLIODb+GfJn+2nIktYkugnmVUBXREel7MpSCI4fq
 Sk/ncFUDJgI6LMCvtoV2iMcZzu0OJHGXg0By+5B2qk9criXQf+mLk1DFLvHKLk2NyuLBbcs59Q1yq
 kO4HjFO8jDdi6w==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: 65720 <at> debbugs.gnu.org
Subject: Re: bug#65720: Guile-Git-managed checkouts grow way too much
References: <87bkejc7go.fsf@HIDDEN>
Date: Mon, 04 Sep 2023 23:47:49 +0200
In-Reply-To: <87bkejc7go.fsf@HIDDEN> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?=
 =?utf-8?Q?'s?= message of "Sun, 03 Sep 2023 22:44:39 +0200")
Message-ID: <87fs3tobju.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 65720
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> As reported by Tobias on IRC (in the context of =E2=80=98hpcguix-web=E2=
=80=99),
> checkouts managed by Guile-Git appear to grow beyond reason.  As an
> example, here=E2=80=99s the same =E2=80=98.git=E2=80=99 managed with Guil=
e-Git and with Git:
>
> $ du -hs ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27=
shj7apsnalwq
> 6.7G    /home/ludo/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlay=
wmisyr27shj7apsnalwq
> $ du -hs .git
> 517M    .git

Unsurprisingly, GC makes a big difference:

--8<---------------cut here---------------start------------->8---
$ cp -r ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj=
7apsnalwq /tmp/checkout
$ (cd /tmp/checkout/; git gc)
Enumerating objects: 717785, done.
Counting objects: 100% (717785/717785), done.
Delta compression using up to 4 threads
Compressing objects: 100% (154644/154644), done.
Writing objects: 100% (717785/717785), done.
Total 717785 (delta 569440), reused 710535 (delta 562274), pack-reused 0
Enumerating cruft objects: 103412, done.
Traversing cruft objects: 81753, done.
Counting objects: 100% (64171/64171), done.
Delta compression using up to 4 threads
Compressing objects: 100% (17379/17379), done.
Writing objects: 100% (64171/64171), done.
Total 64171 (delta 52330), reused 58296 (delta 46792), pack-reused 0
Expanding reachable commits in commit graph: 133730, done.
$ du -hs /tmp/checkout
539M    /tmp/checkout
--8<---------------cut here---------------end--------------->8---

> It would seem that libgit2 doesn=E2=80=99t do the equivalent of =E2=80=98=
git gc=E2=80=99.

Confirmed: <https://github.com/libgit2/libgit2/issues/3247>.

My inclination for the short term would be to work around this
limitation by (1) finding a heuristic to determine is a checkout has
likely accumulated too much cruft, and (2) considering such checkouts as
expired (thereby forcing a re-clone) or running =E2=80=98git gc=E2=80=99 on=
 them if
=E2=80=98git=E2=80=99 is available.

I can=E2=80=99t think of a good heuristic for (1).  Birth time could be one=
, but
we=E2=80=99d need statx(2):

--8<---------------cut here---------------start------------->8---
$ stat ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7=
apsnalwq | tail -4
Access: 2023-09-04 23:13:54.668279105 +0200
Modify: 2023-09-04 11:34:41.665385000 +0200
Change: 2023-09-04 11:34:41.661629102 +0200
 Birth: 2021-08-09 10:48:17.748722151 +0200
--8<---------------cut here---------------end--------------->8---

Lacking statx(2), we can approximate creation time by looking at
=E2=80=98.git/config=E2=80=99:

--8<---------------cut here---------------start------------->8---
$ stat ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7=
apsnalwq/.git/config | tail -3
Modify: 2021-08-09 10:50:28.031760953 +0200
Change: 2021-08-09 10:50:28.031760953 +0200
 Birth: 2021-08-09 10:50:28.031760953 +0200
--8<---------------cut here---------------end--------------->8---

This strategy can be implemented like this:


--=-=-=
Content-Type: text/x-patch
Content-Disposition: inline

diff --git a/guix/git.scm b/guix/git.scm
index ebe2600209..ed3fa56bc8 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -405,7 +405,16 @@ (define cached-checkout-expiration
 
   ;; Use the mtime rather than the atime to cope with file systems mounted
   ;; with 'noatime'.
-  (file-expiration-time (* 90 24 3600) stat:mtime))
+  (let ((ttl (* 90 24 3600))
+        (max-checkout-retention (* 9 30 24 3600)))
+    (lambda (file)
+      (match (false-if-exception (lstat file))
+        (#f 0)                     ;FILE may have been deleted in the meantime
+        (st (min (pk 'ttl (+ (stat:mtime st) ttl))
+                 (pk 'maxttl (match (false-if-exception
+                          (lstat (in-vicinity file ".git/config")))
+                    (#f +inf.0)
+                    (st (+ (stat:mtime st) max-checkout-retention))))))))))
 
 (define %checkout-cache-cleanup-period
   ;; Period for the removal of expired cached checkouts.

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Namely, a cached checkout as considered as =E2=80=9Cexpired=E2=80=9D after =
9 months.  In
my case, it gives this:

--8<---------------cut here---------------start------------->8---
scheme@(guix git)> (cached-checkout-expiration "/home/ludo/.cache/guix/chec=
kouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq/")

;;; (ttl 1701596081)

;;; (maxttl 1651827028)
$6 =3D 1651827028
--8<---------------cut here---------------end--------------->8---

Of course having to re-clone entire repositories every 9 months is
ridiculous, but storing gigabytes of packs is worse IMO (I=E2=80=99m
specifically thinking about the Guix repo, which every users copies via
=E2=80=98guix pull=E2=80=99).

Thoughts?

Thanks,
Ludo=E2=80=99.

--=-=-=--




Information forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.
Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 3 Sep 2023 20:44:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Sep 03 16:44:52 2023
Received: from localhost ([127.0.0.1]:47021 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qctxj-000252-Nn
	for submit <at> debbugs.gnu.org; Sun, 03 Sep 2023 16:44:52 -0400
Received: from lists.gnu.org ([2001:470:142::17]:45444)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qctxf-00024l-3P
 for submit <at> debbugs.gnu.org; Sun, 03 Sep 2023 16:44:50 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1qctxa-0005j4-46
 for bug-guix@HIDDEN; Sun, 03 Sep 2023 16:44:42 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1qctxZ-0000zE-RW
 for bug-guix@HIDDEN; Sun, 03 Sep 2023 16:44:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
 references; bh=Jkb+5XJZ/cpgVmsVrZkaJedRMIuJpZjLE4M6ySVFbqo=; b=NRPelYOW3/og8H
 1G8z8V2RBe7B5QmopsEMT7R1gOPOPj2MRata4axsYyyj5rqcowJAhWvjCxU3v2ZMNyu7L4BktujqV
 dPngiUPsF43aAXM9QJUmLGhJN6SBDLV+RGitx3il/dkCCwfuW/byz0UWWISq9sKfbUJEghY8au/Os
 wbLjTLITTlND7RrqeX3urkNJoOxD0TTqQyDOEOzUfWjNpR9uiNutDUzjeFdE2xYv75imRhDSxsOAx
 JnsM8RLB9m9aKk6qqm6phnuSKK26JqXvwuepCJOh+8Xq/+3cfMQ0Y0w7W6/pug5ki685Sz97D2Vqv
 bfUnShKQ+fGxZ9U+1LGA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: bug-guix@HIDDEN
Subject: Guile-Git-managed checkouts grow way too much
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Septidi 17 Fructidor an 231 de la =?utf-8?Q?R=C3=A9v?=
 =?utf-8?Q?olution=2C?= jour de
 la =?utf-8?Q?Card=C3=A8re?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Sun, 03 Sep 2023 22:44:39 +0200
Message-ID: <87bkejc7go.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello!

As reported by Tobias on IRC (in the context of =E2=80=98hpcguix-web=E2=80=
=99),
checkouts managed by Guile-Git appear to grow beyond reason.  As an
example, here=E2=80=99s the same =E2=80=98.git=E2=80=99 managed with Guile-=
Git and with Git:

--8<---------------cut here---------------start------------->8---
$ du -hs ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27sh=
j7apsnalwq
6.7G    /home/ludo/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywm=
isyr27shj7apsnalwq
$ du -hs .git
517M    .git
--8<---------------cut here---------------end--------------->8---

It would seem that libgit2 doesn=E2=80=99t do the equivalent of =E2=80=98gi=
t gc=E2=80=99.

Ludo=E2=80=99.




Acknowledgement sent to Ludovic Courtès <ludo@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#65720; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 13 Sep 2023 23:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.