GNU bug report logs - #66650
[PATCH] git: Shell out to ‘git gc’ when necessary.

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Reported by: Ludovic Courtès <ludo@HIDDEN>; Keywords: patch; dated Fri, 20 Oct 2023 16:17:01 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 66650 <at> debbugs.gnu.org:


Received: (at 66650) by debbugs.gnu.org; 14 Nov 2023 12:04:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 14 07:04:26 2023
Received: from localhost ([127.0.0.1]:60186 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1r2s9Z-0005LP-O3
	for submit <at> debbugs.gnu.org; Tue, 14 Nov 2023 07:04:26 -0500
Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]:39148)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <zimon.toutoune@HIDDEN>)
 id 1r2s9X-0005Kr-ND; Tue, 14 Nov 2023 07:04:24 -0500
Received: by mail-wr1-x42c.google.com with SMTP id
 ffacd0b85a97d-32f831087c6so1028145f8f.0; 
 Tue, 14 Nov 2023 04:03:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1699963416; x=1700568216; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id
 :reply-to; bh=ylTF2Qwpjb3oXQkbeS9e3AnLlKkJ1OUyQRtwgU+E1+Q=;
 b=AwP+SsOPMQHvKYvI/o4aU2byBlVjQ/CaHhUssWI6cpQiahhd1zCpvl6U2/utawpQBx
 ZTZvA+Vk2sSqMnvz/OHFVRQsTUHRPNqJ0Iyf7sQVS7FbXNtxnL5f3zmXfSnqv8fw7uRO
 9S8lXYSKh66vvSv7jcDKKU4XqDUTUoms6CoyLcIx3xyXhCY2eZ3azAt8b9Y5EtTuLEUz
 A/zdHK2nJ5BWAxqjUccSJ5LQz3yvypZcd2bZ1pjHB/3Yf9cxz3PW7AdYrhdqIzlBtWTN
 iX3kqQ9rD8TgASpm7v/AZBgUPeOemaIPBgrEHVwTajcVa0ktjskYEx2SY0CAqPoWjSwc
 38iw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1699963416; x=1700568216;
 h=content-transfer-encoding:mime-version:message-id:date:references
 :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc
 :subject:date:message-id:reply-to;
 bh=ylTF2Qwpjb3oXQkbeS9e3AnLlKkJ1OUyQRtwgU+E1+Q=;
 b=lH24OIockUIOGanmqFgbxrGRc7zttqbK3ESKamxrsLW7PnpTCsdHXI0phtt6zvCLml
 8yBP7XfUHD4onbzgiItQW/c7qSXWqsVg4mYvFbVf+GNl5BYpADFPkhwsGx5AgC+L01Iy
 rCkmsMgmlVhHCZxn1EeZ5fyBNlP88KEXX9n84Na/mKgS/y29/dJeYMQQ+NOyRq1UbwHR
 inOwBbax1uVPo/df/xqlNEXppNOm2756cw9ZyG+0fvYL6ixJEkhHZBINOuEHdJC88ZBp
 UO5PPoi8tLHIOxQLUtDpdf6MjbJlb5yWVCH+/JGyTytulOszcRIm6qUqQVwQmotqrNTK
 dS8w==
X-Gm-Message-State: AOJu0YxmDycvu0Kzn756BoKs/IPx6TdQsZTlu7/M2iYMK2IL5k9fZlHf
 sMvVouVj9jniSrUgVTcBwkM2c3I4ZV4=
X-Google-Smtp-Source: AGHT+IGxrX8kNQ98k5MVjas+2EFIinZ9YvkPaQ15MARAjwxnTfVMWW20wlN2YlpNmNhmPTWnsvkRBQ==
X-Received: by 2002:a5d:648c:0:b0:331:3cec:214f with SMTP id
 o12-20020a5d648c000000b003313cec214fmr1209144wri.7.1699963415483; 
 Tue, 14 Nov 2023 04:03:35 -0800 (PST)
Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id
 u30-20020adfa19e000000b00318147fd2d3sm7718993wru.41.2023.11.14.04.03.34
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Tue, 14 Nov 2023 04:03:35 -0800 (PST)
From: Simon Tournier <zimon.toutoune@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Christopher Baines
 <mail@HIDDEN>
Subject: Re: bug#65720: [bug#66650] [PATCH] git: Shell out to
 =?utf-8?B?4oCYZ2l0IGdj4oCZ?= when necessary.
In-Reply-To: <87o7fwae0q.fsf@HIDDEN>
References: <87jzswsrlt.fsf@HIDDEN>
 <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@HIDDEN>
 <87sf5swc3j.fsf@HIDDEN> <87o7fwae0q.fsf@HIDDEN>
Date: Tue, 14 Nov 2023 10:32:41 +0100
Message-ID: <87v8a4el3a.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 66650
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org,
 66650 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi,

On Tue, 14 Nov 2023 at 10:19, Ludovic Court=C3=A8s <ludo@HIDDEN> wrote:

> Any other comments?  I=E2=80=99d like to push the patch within a few days=
 if
> there are no objections.

As mentioned in [1],

>> * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New
>> procedures.
>> (update-cached-checkout): Use it.
>> ---
>>  guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 36 insertions(+), 3 deletions(-)

LGTM.  Just two colors for the bikeshed. :-)


>> +  (when (> (packs-in-git-repository directory) 25)

Why 25?  And not 10 or 50 or 100?


>>  (define* (update-cached-checkout url
>>                                   #:key
>>                                   (ref '())
>> @@ -515,6 +545,9 @@ (define* (update-cached-checkout url
>>                     seconds seconds
>>                     nanoseconds nanoseconds))))
>>=20=20
>> +       ;; Run 'git gc' if needed.
>> +       (maybe-run-git-gc cache-directory)

Why not trigger it by =E2=80=9Cguix gc=E2=80=9D?

Well, I expect =E2=80=9Cguix gc=E2=80=9D to take some time and I choose whe=
n.  However,
I want =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2=80=9D t=
o be as fast as possible and
here some extra time is added, and I cannot control exactly when.


Cheers,
simon


1: bug#65720: [PATCH] git: Shell out to =E2=80=98git gc=E2=80=99 when neces=
sary.
Simon Tournier <zimon.toutoune@HIDDEN>
Mon, 23 Oct 2023 12:08:07 +0200
id:87il6xlkhk.fsf@HIDDEN
https://issues.guix.gnu.org/65720
https://issues.guix.gnu.org/msgid/87il6xlkhk.fsf@HIDDEN
https://yhetil.org/guix/87il6xlkhk.fsf@HIDDEN




Information forwarded to guix-patches@HIDDEN:
bug#66650; Package guix-patches. Full text available.

Message received at 66650 <at> debbugs.gnu.org:


Received: (at 66650) by debbugs.gnu.org; 14 Nov 2023 09:19:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Nov 14 04:19:56 2023
Received: from localhost ([127.0.0.1]:60019 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1r2paO-0000lH-23
	for submit <at> debbugs.gnu.org; Tue, 14 Nov 2023 04:19:56 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10]:39730)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>)
 id 1r2paL-0000kr-HX; Tue, 14 Nov 2023 04:19:54 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1r2pZY-0004TY-Te; Tue, 14 Nov 2023 04:19:04 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=JZDU1hOSj2ITSd6DjETVBVkYN5+QtFH0XZM216c9Cuc=; b=kdEbrjIYkhUUgzl1O0BG
 z4mkDThAEOWVKdLJu5dwi6F0U0RBk6hj78YdTDQqIERlIJXAW6DRj9+d0Cio1OrcxTvQiAbOdQnPG
 ret1TpZDdGdeUS0uRurMQqjzddTomVBoMEStUEpTLBDX64ibAgXqimdmMkPnUDsndrKbPiLS43wOi
 8JiRM3pHJr1hR1NT762nRI3csilJ3L7OUtI87viVA/P5DaKWZiGlANo9Lz5Cm7gFRAVLqJEZpiwud
 +l6U3JGnsQiCMFH6CxaGDPeX9ING1cJHWPlzUr/eGPQdsfh5j6dJ0GZspypgY5uktRkbOBxHQkGPY
 VDKqnJgU+seV6A==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: Christopher Baines <mail@HIDDEN>
Subject: Re: [bug#66650] [PATCH] git: Shell out to =?utf-8?B?4oCYZ2l0IGdj?=
 =?utf-8?B?4oCZ?= when necessary.
In-Reply-To: <87sf5swc3j.fsf@HIDDEN> (Christopher Baines's message of
 "Mon, 30 Oct 2023 12:02:59 +0000")
References: <87jzswsrlt.fsf@HIDDEN>
 <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@HIDDEN>
 <87sf5swc3j.fsf@HIDDEN>
Date: Tue, 14 Nov 2023 10:19:01 +0100
Message-ID: <87o7fwae0q.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 66650
Cc: Josselin Poiret <dev@HIDDEN>, 65720 <at> debbugs.gnu.org,
 66650 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

Christopher Baines <mail@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:
>
>> Fixes <https://issues.guix.gnu.org/65720>.
>>
>> This fixes a bug whereby libgit2-managed checkouts would keep growing as
>> we fetch.

[...]

> This sounds good to me, the data service has this problem as well of
> cached checkouts that grow to be too large and this sounds like it'll
> address it.

Thanks for your input, Chris.

Any other comments?  I=E2=80=99d like to push the patch within a few days if
there are no objections.

  https://issues.guix.gnu.org/66650

Ludo=E2=80=99.




Information forwarded to guix-patches@HIDDEN:
bug#66650; Package guix-patches. Full text available.

Message received at 66650 <at> debbugs.gnu.org:


Received: (at 66650) by debbugs.gnu.org; 30 Oct 2023 12:04:59 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 30 08:04:59 2023
Received: from localhost ([127.0.0.1]:44442 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qxR0t-00085W-4I
	for submit <at> debbugs.gnu.org; Mon, 30 Oct 2023 08:04:59 -0400
Received: from mira.cbaines.net ([212.71.252.8]:42990)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>)
 id 1qxR0p-00085H-Nd; Mon, 30 Oct 2023 08:04:56 -0400
Received: from localhost (pd95b13b0.dip0.t-ipconnect.de [217.91.19.176])
 by mira.cbaines.net (Postfix) with ESMTPSA id 5E33E27BBE2;
 Mon, 30 Oct 2023 12:04:21 +0000 (GMT)
Received: from felis (localhost [127.0.0.1])
 by localhost (OpenSMTPD) with ESMTP id 955346f3;
 Mon, 30 Oct 2023 12:04:18 +0000 (UTC)
References: <87jzswsrlt.fsf@HIDDEN>
 <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@HIDDEN>
User-agent: mu4e 1.10.5; emacs 28.2
From: Christopher Baines <mail@HIDDEN>
To: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Subject: Re: [bug#66650] [PATCH] git: Shell out to =?utf-8?B?4oCYZ2l0IGdj?=
 =?utf-8?B?4oCZ?= when necessary.
Date: Mon, 30 Oct 2023 12:02:59 +0000
In-reply-to: <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@HIDDEN>
Message-ID: <87sf5swc3j.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: 66650
Cc: 65720 <at> debbugs.gnu.org, 66650 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Fixes <https://issues.guix.gnu.org/65720>.
>
> This fixes a bug whereby libgit2-managed checkouts would keep growing as
> we fetch.
>
> * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New
> procedures.
> (update-cached-checkout): Use it.
> ---
>  guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++---
>  1 file changed, 36 insertions(+), 3 deletions(-)
>
> Hi!
>
> This is a radical fix/workaround for the unbounded Git checkout growth
> problem, shelling out to =E2=80=98git gc=E2=80=99 when it=E2=80=99s likel=
y needed (=E2=80=9Ctoo many=E2=80=9D
> pack files around).
>
> I thought we might be able to implement a =E2=80=98git gc=E2=80=99 approx=
imation using
> the libgit2 =E2=80=9Cpackbuilder=E2=80=9D interface, but I haven=E2=80=99=
t got around to doing
> it: <https://libgit2.org/libgit2/#HEAD/search/pack>.
>
> Once again, shelling out is not my favorite option, but it=E2=80=99s a bu=
g we
> should fix sooner rather than later, hence this compromise.
>
> Thoughts?

This sounds good to me, the data service has this problem as well of
cached checkouts that grow to be too large and this sounds like it'll
address it.

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmU/m8BfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xcx2A//f3qqeG4+4J5uvVPcP/26SMtpzvNCUsRs
5hzUgJherxg4U25tlUdXlpjSVTNDSc0qN7RpUQWy6rca/S/ro2NL/KcR7VAdyumu
v39ldNwbq0W2YC/eZ9fxS7SzeCnWV2oOpO5X5sy69TGvjE/plWNStttvOF/HFy6Y
CtH9GNXZE9xwL5PbRK1Lxun5JmamP3Lxk+oivN3ZC7AnhoYxGxJ4xHD9HfN970b/
6kBbc0Vcf0wHtHRVIEuhBw01JkklchBhTTzbYoi3SUFZeovbkm4Ys5g3s7nDVVZD
5XNMkdp2YcgdUtsfZN1jhgFXTsa6XyfFnQS/1qMfPg3U1niAj0nKIqjEfmUNnGD1
PpQbQ5WvOZm5S70HHDG9Cg58BVIcH0hrHqfVYyghhttf2yUvdKp6CtqNyuBzsr7D
276K8EAeTMcfQwtArxKaFfFG/ggInMvPy1UA1FoN2j0EIIxeND/7vcejqqIssjZm
jsU716+s9bP1JCf0s/gJPWSw7Iph7gOs4CKFUdQSeEqNawyXyetxc5PjI6K7NKzq
QSa7SJlTe0Lv8maRIZ7LV8t08n3PPFO0sFC7MQMVTCFbkwqFwaVGGdeFiF8dYXfN
m4eigk0nl9Poq7gQ79r0igy/rfkZW8mVKRucqVdUJ/znykJfiyBbadSmPcCZP6MS
XzvO0Cgwki8=
=sesm
-----END PGP SIGNATURE-----
--=-=-=--




Information forwarded to guix-patches@HIDDEN:
bug#66650; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 20 Oct 2023 16:16:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 20 12:16:13 2023
Received: from localhost ([127.0.0.1]:41305 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qtsAW-0004fs-VW
	for submit <at> debbugs.gnu.org; Fri, 20 Oct 2023 12:16:13 -0400
Received: from lists.gnu.org ([2001:470:142::17]:49828)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1qtsAV-0004f0-F5
 for submit <at> debbugs.gnu.org; Fri, 20 Oct 2023 12:16:11 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1qts9t-0008NU-Dy
 for guix-patches@HIDDEN; Fri, 20 Oct 2023 12:15:33 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1qts9r-0004Rf-PD; Fri, 20 Oct 2023 12:15:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To:
 From; bh=guykA3Jje4B/fdh8d1YKTVE8F+9iXXIl3FUKarfygvY=; b=jF0iDL/GbSiYleynVOTG
 /8r/H3EkXts9gzZHBgd9R8nNRQDQE9k6G5S40965qhyaaKSTNmvZmEP05W378rPNL287CkVM+Ibc7
 7E32AmsdJM3Am6miqMoPX9YkU5o+Q5z+orHS8Uflfk+Q6GcnHJFgV54Bg6XcAw2I1w6n8BhwLD8XH
 Xc6+syD3NXx+2uTgwx7sgIiSqx8Q8TyYiO+e+VCDOrgg9PMq1m8eYZ6M32W5h/fNQYw3CoJwQsjTq
 N7T/JL+bxKLs4dAkgwLr8izbta894kjMEC5GTNjZ2vBkHDJJ991iz+BbSwyZrUfYucrv//2aKtBun
 iam1wsZbcvK/hA==;
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN>
To: guix-patches@HIDDEN
Subject: [PATCH] =?UTF-8?q?git:=20Shell=20out=20to=20=E2=80=98git=20gc?=
 =?UTF-8?q?=E2=80=99=20when=20necessary.?=
Date: Fri, 20 Oct 2023 18:15:12 +0200
Message-ID: <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@HIDDEN>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <87jzswsrlt.fsf@HIDDEN>
References: <87jzswsrlt.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-Debbugs-Cc: Christopher Baines <guix@HIDDEN>, Josselin Poiret <dev@HIDDEN>, Ludovic Courtès <ludo@HIDDEN>, Mathieu Othacehe <othacehe@HIDDEN>, Ricardo Wurmus <rekado@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>, Tobias Geerinckx-Rice <me@HIDDEN>
Content-Transfer-Encoding: 8bit
X-Spam-Score: -0.0 (/)
X-Debbugs-Envelope-To: submit
Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@HIDDEN>, 65720 <at> debbugs.gnu.org,
 Josselin Poiret <dev@HIDDEN>, Simon Tournier <zimon.toutoune@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Fixes <https://issues.guix.gnu.org/65720>.

This fixes a bug whereby libgit2-managed checkouts would keep growing as
we fetch.

* guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New
procedures.
(update-cached-checkout): Use it.
---
 guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

Hi!

This is a radical fix/workaround for the unbounded Git checkout growth
problem, shelling out to ‘git gc’ when it’s likely needed (“too many”
pack files around).

I thought we might be able to implement a ‘git gc’ approximation using
the libgit2 “packbuilder” interface, but I haven’t got around to doing
it: <https://libgit2.org/libgit2/#HEAD/search/pack>.

Once again, shelling out is not my favorite option, but it’s a bug we
should fix sooner rather than later, hence this compromise.

Thoughts?

Ludo’.

diff --git a/guix/git.scm b/guix/git.scm
index b7182305cf..d704b62333 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -1,6 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017, 2020 Mathieu Othacehe <m.othacehe@HIDDEN>
-;;; Copyright © 2018-2022 Ludovic Courtès <ludo@HIDDEN>
+;;; Copyright © 2018-2023 Ludovic Courtès <ludo@HIDDEN>
 ;;; Copyright © 2021 Kyle Meyer <kyle@HIDDEN>
 ;;; Copyright © 2021 Marius Bakke <marius@HIDDEN>
 ;;; Copyright © 2022 Maxime Devos <maximedevos@HIDDEN>
@@ -29,15 +29,16 @@ (define-module (guix git)
   #:use-module (guix cache)
   #:use-module (gcrypt hash)
   #:use-module ((guix build utils)
-                #:select (mkdir-p delete-file-recursively))
+                #:select (mkdir-p delete-file-recursively invoke/quiet))
   #:use-module (guix store)
   #:use-module (guix utils)
   #:use-module (guix records)
   #:use-module (guix gexp)
   #:autoload   (guix git-download)
   (git-reference-url git-reference-commit git-reference-recursive?)
+  #:autoload   (guix config) (%git)
   #:use-module (guix sets)
-  #:use-module ((guix diagnostics) #:select (leave warning))
+  #:use-module ((guix diagnostics) #:select (leave warning info))
   #:use-module (guix progress)
   #:autoload   (guix swh) (swh-download commit-id?)
   #:use-module (rnrs bytevectors)
@@ -428,6 +429,35 @@ (define (delete-checkout directory)
     (rename-file directory trashed)
     (delete-file-recursively trashed)))
 
+(define (packs-in-git-repository directory)
+  "Return the number of pack files under DIRECTORY, a Git checkout."
+  (catch 'system-error
+    (lambda ()
+      (let ((directory (opendir (in-vicinity directory ".git/objects/pack"))))
+        (let loop ((count 0))
+          (match (readdir directory)
+            ((? eof-object?)
+             (closedir directory)
+             count)
+            (str
+             (loop (if (string-suffix? ".pack" str)
+                       (+ 1 count)
+                       count)))))))
+    (const 0)))
+
+(define (maybe-run-git-gc directory)
+  "Run 'git gc' in DIRECTORY if needed."
+  ;; XXX: As of libgit2 1.3.x (used by Guile-Git), there's no support for GC.
+  ;; Each time a checkout is pulled, a new pack is created, which eventually
+  ;; takes up a lot of space (lots of small, poorly-compressed packs).  As a
+  ;; workaround, shell out to 'git gc' when the number of packs in a
+  ;; repository has become "too large", potentially wasting a lot of space.
+  ;; See <https://issues.guix.gnu.org/65720>.
+  (when (> (packs-in-git-repository directory) 25)
+    (info (G_ "compressing cached Git repository at '~a'...~%")
+          directory)
+    (invoke/quiet %git "-C" directory "gc")))
+
 (define* (update-cached-checkout url
                                  #:key
                                  (ref '())
@@ -515,6 +545,9 @@ (define* (update-cached-checkout url
                    seconds seconds
                    nanoseconds nanoseconds))))
 
+       ;; Run 'git gc' if needed.
+       (maybe-run-git-gc cache-directory)
+
        ;; When CACHE-DIRECTORY is a sub-directory of the default cache
        ;; directory, remove expired checkouts that are next to it.
        (let ((parent (dirname cache-directory)))

base-commit: 6b0a32196982a0a2f4dbb59d35e55833a5545ac6
-- 
2.41.0





Acknowledgement sent to Ludovic Courtès <ludo@HIDDEN>:
New bug report received and forwarded. Copy sent to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN. Full text available.
Report forwarded to guix@HIDDEN, dev@HIDDEN, ludo@HIDDEN, othacehe@HIDDEN, rekado@HIDDEN, zimon.toutoune@HIDDEN, me@HIDDEN, guix-patches@HIDDEN:
bug#66650; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Tue, 14 Nov 2023 12:15:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.