GNU bug report logs - #77606
‘fdatasync’ on cached narinfo files is too expensive

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 7 Apr 2025 10:20:45 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Apr 07 06:20:44 2025
Received: from localhost ([127.0.0.1]:52869 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1u1jau-0004eP-6D
	for submit <at> debbugs.gnu.org; Mon, 07 Apr 2025 06:20:44 -0400
Received: from lists.gnu.org ([2001:470:142::17]:60896)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <ludovic.courtes@HIDDEN>)
 id 1u1jar-0004dm-Hy
 for submit <at> debbugs.gnu.org; Mon, 07 Apr 2025 06:20:42 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludovic.courtes@HIDDEN>)
 id 1u1jal-00043e-UK
 for bug-guix@HIDDEN; Mon, 07 Apr 2025 06:20:35 -0400
Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludovic.courtes@HIDDEN>)
 id 1u1jaj-0000ti-1x
 for bug-guix@HIDDEN; Mon, 07 Apr 2025 06:20:35 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc;
 h=from:to:subject:date:message-id:mime-version:
 content-transfer-encoding;
 bh=uwDl8Y3p9PYBSmwhEfkzjbSDGdahr46mdfxOK3pKDPQ=;
 b=DQumi73DbqdWnhBfBwmX0bF7ArAKMHO82GYN3dOX7RuTp1DSti5IwP4g
 SnOw955ebjKSIzCfOAmdo8djorkpUpkf76UdxuUFRkcpNEkmlsUSmW99j
 idVXpDYEXoNrzTlQtw9Nt6/kCOnnji9KTlfVu3jhOuXHAFhp9T1cvRs3s g=;
Authentication-Results: mail2-relais-roc.national.inria.fr;
 dkim=none (message not signed) header.i=none;
 spf=SoftFail smtp.mailfrom=ludovic.courtes@HIDDEN;
 dmarc=fail (p=none dis=none) d=inria.fr
X-IronPort-AV: E=Sophos;i="6.15,194,1739833200"; d="scan'208";a="216664827"
Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201])
 by mail2-relais-roc.national.inria.fr with
 ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2025 12:20:26 +0200
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludovic.courtes@HIDDEN>
To: bug-guix@HIDDEN
Subject: =?utf-8?B?4oCYZmRhdGFzeW5j4oCZ?= on cached narinfo files is too
 expensive
X-Debbugs-CC: Christopher Baines <guix@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Octidi 18 Germinal an 233 de la =?utf-8?Q?R=C3=A9vol?=
 =?utf-8?Q?ution=2C?= jour de la =?utf-8?Q?Cigu=C3=AB?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Mon, 07 Apr 2025 12:20:26 +0200
Message-ID: <87frik9st1.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=192.134.164.83;
 envelope-from=ludovic.courtes@HIDDEN;
 helo=mail2-relais-roc.national.inria.fr
X-Spam_score_int: -43
X-Spam_score: -4.4
X-Spam_bar: ----
X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.01,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hello,

As Christopher Baines reported on IRC, calling =E2=80=98fdatasync=E2=80=99 =
on each
cached narinfo under /var/guix/substitute/cache can be unreasonably
expensive on spinning disks.

On bayfront, when tracing =E2=80=98guix substitute=E2=80=99 processes, we o=
bserve
=E2=80=98fdatasync=E2=80=99 occasionally taking more than a second, and typ=
ically taking
more than 0.2s:

--8<---------------cut here---------------start------------->8---
mkdir("/var", 0777)                     =3D -1 EEXIST (File exists) <0.0000=
48>
mkdir("/var/guix", 0777)                =3D -1 EEXIST (File exists) <0.0000=
42>
mkdir("/var/guix/substitute", 0777)     =3D -1 EEXIST (File exists) <0.0000=
40>
mkdir("/var/guix/substitute/cache", 0777) =3D -1 EEXIST (File exists) <0.00=
0042>
mkdir("/var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw2=
2hgajbhfy3q", 0777) =3D -1 EEXIST (File exists
) <0.000041>
clock_gettime(CLOCK_MONOTONIC, {tv_sec=3D153237, tv_nsec=3D595042938}) =3D =
0 <0.000037>
openat(AT_FDCWD, "/var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7=
tqjmewzcyw22hgajbhfy3q/nfmn2nkfjz5kzklw165aj
6ilwphw9zrz.3kAGMM", O_RDWR|O_CREAT|O_EXCL, 0600) =3D 15 <0.000123>
fstat(15, {st_mode=3DS_IFREG|0600, st_size=3D0, ...}) =3D 0 <0.000037>
lseek(15, 0, SEEK_CUR)                  =3D 0 <0.000038>
write(15, "(narinfo (version 2) (cache-uri "..., 1066) =3D 1066 <0.000071>
fdatasync(15)                           =3D 0 <1.860604>
--8<---------------cut here---------------end--------------->8---

A longer sample:

--8<---------------cut here---------------start------------->8---
ludo@bayfront ~$ grep ^fdatasync log.strace=20
fdatasync(15)                           =3D 0 <1.860604>
fdatasync(15)                           =3D 0 <1.101868>
fdatasync(15)                           =3D 0 <0.464357>
fdatasync(15)                           =3D 0 <0.363363>
fdatasync(15)                           =3D 0 <0.307759>
fdatasync(15)                           =3D 0 <0.619979>
fdatasync(15)                           =3D 0 <0.269880>
fdatasync(15)                           =3D 0 <0.274093>
fdatasync(15)                           =3D 0 <0.247207>
fdatasync(15)                           =3D 0 <0.367507>
fdatasync(15)                           =3D 0 <0.212277>
fdatasync(15)                           =3D 0 <0.234929>
fdatasync(15)                           =3D 0 <0.330697>
fdatasync(15)                           =3D 0 <0.193925>
fdatasync(15)                           =3D 0 <0.251554>
fdatasync(15)                           =3D 0 <0.379571>
--8<---------------cut here---------------end--------------->8---

That call comes from =E2=80=98with-atomic-file-output=E2=80=99, commit
1752a17a1e6f7138892eeeb4806cd04ccb3ca1b0, which refers to this
discussion:

  https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00456.html

It probably makes sense in some uses of =E2=80=98with-atomic-file-output=E2=
=80=99, but
nor for a cache as is the case here.

Ludo=E2=80=99.