GNU bug report logs - #77680
[PATCH 0/4] Avoid 'fdatasync' calls for cache files

Previous Next

To reply to this bug, email your comments to 77680 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 0/4] Avoid 'fdatasync' calls for cache files
Date: Wed,  9 Apr 2025 16:21:27 +0200

Hello,

As Chris found out, calling ‘fdatasync’ for each and every narinfo
cache file created by ‘guix substitute’ (and likewise for other caches)
is too expensive and pointless.  This patch series fixes that.

Thoughts?

Ludo’.

Ludovic Courtès (4):
  cache: Remove unused import.
  utils: Add #:sync? parameter to ‘with-atomic-file-output’.
  substitutes: Ignore corrupt cached narinfos.
  guix: Avoid ‘fdatasync’ call for caches and regular files.

 guix/cache.scm            |  1 -
 guix/git-authenticate.scm |  5 +++--
 guix/http-client.scm      |  5 +++--
 guix/scripts/discover.scm |  3 ++-
 guix/scripts/style.scm    |  5 +++--
 guix/substitutes.scm      |  8 +++++---
 guix/utils.scm            | 14 ++++++++++----
 7 files changed, 26 insertions(+), 15 deletions(-)


base-commit: c88f98bb3ca2616baea6b1e452cc900cc9c87503
-- 
2.49.0

Message #8 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 1/4] cache: Remove unused import.
Date: Wed,  9 Apr 2025 16:23:36 +0200

* guix/cache.scm: Remove unused import.

Change-Id: I06956534aa5342a084ebf71b61518c515ce9288c
---
 guix/cache.scm | 1 -
 1 file changed, 1 deletion(-)

diff --git a/guix/cache.scm b/guix/cache.scm
index 8b12312c77..5d8a0edbaa 100644
--- a/guix/cache.scm
+++ b/guix/cache.scm
@@ -18,7 +18,6 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (guix cache)
-  #:use-module ((guix utils) #:select (with-atomic-file-output))
   #:autoload   (guix build syscalls) (lock-file unlock-file)
   #:use-module (srfi srfi-19)
   #:use-module (srfi srfi-26)
-- 
2.49.0

Message #11 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 2/4] utils: Add #:sync? parameter to ‘with-atomic-file-output’.
Date: Wed,  9 Apr 2025 16:23:37 +0200

* guix/utils.scm (with-atomic-file-output): Add #:sync? and honor it.
Add ‘force-output’ call.

Change-Id: I2479778ae55360c0fab3389ac9249045a27b3568
---
 guix/utils.scm | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/guix/utils.scm b/guix/utils.scm
index c7c23d9d5b..7ae98096c2 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2022, 2024 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012-2022, 2024-2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2013, 2014, 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2014 Eric Bavier <bavier <at> member.fsf.org>
 ;;; Copyright © 2014 Ian Denhardt <ian <at> zenhack.net>
@@ -1057,16 +1057,22 @@ (define (call-with-temporary-directory proc)
       (lambda ()
         (false-if-exception (delete-file-recursively tmp-dir))))))
 
-(define (with-atomic-file-output file proc)
+(define* (with-atomic-file-output file proc #:key (sync? #t))
   "Call PROC with an output port for the file that is going to replace FILE.
 Upon success, FILE is atomically replaced by what has been written to the
-output port, and PROC's result is returned."
+output port, and PROC's result is returned.
+
+When SYNC? is true, call 'fdatasync' on the temporary file before renaming it
+to FILE; set it to #false for caches and temporary files to improve
+performance."
   (let* ((template (string-append file ".XXXXXX"))
          (out      (mkstemp! template)))
     (with-throw-handler #t
       (lambda ()
         (let ((result (proc out)))
-          (fdatasync out)
+          (when sync?
+            (force-output out)
+            (fdatasync out))
           (close-port out)
           (rename-file template file)
           result))
-- 
2.49.0

Message #14 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Christopher Baines <mail <at> cbaines.net>
Subject: [PATCH 3/3] guix: Avoid ‘fdatasync’ call for caches and regular files.
Date: Wed,  9 Apr 2025 16:23:38 +0200

Fixes <https://issues.guix.gnu.org/77606>.

Calling ‘fdatasync’ for each and every narinfo file created by ‘guix
substitute’ proved to be too expensive on spinning HDDs and/or under
load (from 0.1s to 1.3s for the ‘fdatasync’ call alone).

* guix/git-authenticate.scm (cache-authenticated-commit): Pass #:sync? #f.
* guix/http-client.scm (http-fetch/cached): Likewise.
* guix/scripts/discover.scm (write-publish-file): Likewise.
* guix/scripts/style.scm (format-whole-file): Likewise.
* guix/substitutes.scm (cache-narinfo!): Likewise.

Reported-by: Christopher Baines <mail <at> cbaines.net>
Change-Id: I82297eae737bc5aae8a3f7604119e9f3d4b625bf
---
 guix/git-authenticate.scm | 5 +++--
 guix/http-client.scm      | 5 +++--
 guix/scripts/discover.scm | 3 ++-
 guix/scripts/style.scm    | 5 +++--
 guix/substitutes.scm      | 3 ++-
 5 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/guix/git-authenticate.scm b/guix/git-authenticate.scm
index 37c69d0880..3cd1175c32 100644
--- a/guix/git-authenticate.scm
+++ b/guix/git-authenticate.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019, 2020, 2021, 2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2019-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -344,7 +344,8 @@ (define (cache-authenticated-commit key commit-id)
           (chmod port #o600)
           (display ";; List of previously-authenticated commits.\n\n"
                    port)
-          (pretty-print lst port))))))
+          (pretty-print lst port)))
+      #:sync? #f)))
 
 
 ;;;
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 9138a627ac..4e0cc59e91 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2018, 2020-2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012-2018, 2020-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me <at> tobias.gr>
@@ -346,7 +346,8 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
           (when cache-port
             (close-port cache-port))
           (with-atomic-file-output file
-            (cut write-cache port <>))
+            (cut write-cache port <>)
+            #:sync? #f)
           (close-port port)
           (open-input-file file))))
 
diff --git a/guix/scripts/discover.scm b/guix/scripts/discover.scm
index 32bf6085a5..cbe01638f5 100644
--- a/guix/scripts/discover.scm
+++ b/guix/scripts/discover.scm
@@ -87,7 +87,8 @@ (define* (write-publish-file #:key (file (%publish-file)))
          (format port "http://~a:~a~%"
                  (avahi-service-address service)
                  (avahi-service-port service)))
-       %publish-services)))
+       %publish-services))
+    #:sync? #f)
   (chmod file #o644))
 
 (define* (read-substitute-urls #:key (file (%publish-file)))
diff --git a/guix/scripts/style.scm b/guix/scripts/style.scm
index 4801529f7e..c45bdd4458 100644
--- a/guix/scripts/style.scm
+++ b/guix/scripts/style.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021-2024 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2021-2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2024 Herman Rimm <herman <at> rimm.ee>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -554,7 +554,8 @@ (define* (format-whole-file file order? #:rest rest)
           (apply pretty-print-with-comments/splice port lst
                  #:format-comment canonicalize-comment
                  #:format-vertical-space canonicalize-vertical-space
-                 rest))))))
+                 rest))
+        #:sync? #f))))
 
 
 ;;;
diff --git a/guix/substitutes.scm b/guix/substitutes.scm
index 86b9f5472a..a7a2dda0dc 100644
--- a/guix/substitutes.scm
+++ b/guix/substitutes.scm
@@ -127,7 +127,8 @@ (define (cache-narinfo! cache-url path narinfo ttl)
     (mkdir-p (dirname file))
     (with-atomic-file-output file
       (lambda (out)
-        (write (cache-entry cache-url narinfo) out))))
+        (write (cache-entry cache-url narinfo) out))
+      #:sync? #f))
 
   narinfo)
 
-- 
2.49.0

Message #17 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 3/4] substitutes: Ignore corrupt cached narinfos.
Date: Wed,  9 Apr 2025 16:23:39 +0200

* guix/substitutes.scm (cached-narinfo): Wrap ‘read’ call in
‘false-if-exception’.  Add catch-all ‘match’ clause.

Change-Id: I2d776f64b6521f778b4ab3f956b35cdef2ddb383
---
 guix/substitutes.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/guix/substitutes.scm b/guix/substitutes.scm
index 86b9f5472a..45c165e03e 100644
--- a/guix/substitutes.scm
+++ b/guix/substitutes.scm
@@ -300,7 +300,7 @@ (define (cached-narinfo cache-url path)
     (lambda ()
       (call-with-input-file cache-file
         (lambda (p)
-          (match (read p)
+          (match (false-if-exception (read p))
             (('narinfo ('version 2)
                        ('cache-uri cache-uri)
                        ('date date) ('ttl ttl) ('value #f))
@@ -315,7 +315,8 @@ (define (cached-narinfo cache-url path)
              (if (obsolete? date now ttl)
                  (values #f #f)
                  (values #t (string->narinfo value cache-uri))))
-            (('narinfo ('version v) _ ...)
+            (_
+             ;; Corrupt or incompatible cached entry.
              (values #f #f))))))
     (lambda _
       (values #f #f))))
-- 
2.49.0

Message #20 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Christopher Baines <mail <at> cbaines.net>
Subject: [PATCH 4/4] guix: Avoid ‘fdatasync’ call for caches and regular files.
Date: Wed,  9 Apr 2025 16:23:40 +0200

Fixes <https://issues.guix.gnu.org/77606>.

Calling ‘fdatasync’ for each and every narinfo file created by ‘guix
substitute’ proved to be too expensive on spinning HDDs and/or under
load (from 0.1s to 1.3s for the ‘fdatasync’ call alone).

* guix/git-authenticate.scm (cache-authenticated-commit): Pass #:sync? #f.
* guix/http-client.scm (http-fetch/cached): Likewise.
* guix/scripts/discover.scm (write-publish-file): Likewise.
* guix/scripts/style.scm (format-whole-file): Likewise.
* guix/substitutes.scm (cache-narinfo!): Likewise.

Reported-by: Christopher Baines <mail <at> cbaines.net>
Change-Id: I82297eae737bc5aae8a3f7604119e9f3d4b625bf
---
 guix/git-authenticate.scm | 5 +++--
 guix/http-client.scm      | 5 +++--
 guix/scripts/discover.scm | 3 ++-
 guix/scripts/style.scm    | 5 +++--
 guix/substitutes.scm      | 3 ++-
 5 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/guix/git-authenticate.scm b/guix/git-authenticate.scm
index 37c69d0880..3cd1175c32 100644
--- a/guix/git-authenticate.scm
+++ b/guix/git-authenticate.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019, 2020, 2021, 2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2019-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -344,7 +344,8 @@ (define (cache-authenticated-commit key commit-id)
           (chmod port #o600)
           (display ";; List of previously-authenticated commits.\n\n"
                    port)
-          (pretty-print lst port))))))
+          (pretty-print lst port)))
+      #:sync? #f)))
 
 
 ;;;
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 9138a627ac..4e0cc59e91 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2018, 2020-2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012-2018, 2020-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me <at> tobias.gr>
@@ -346,7 +346,8 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
           (when cache-port
             (close-port cache-port))
           (with-atomic-file-output file
-            (cut write-cache port <>))
+            (cut write-cache port <>)
+            #:sync? #f)
           (close-port port)
           (open-input-file file))))
 
diff --git a/guix/scripts/discover.scm b/guix/scripts/discover.scm
index 32bf6085a5..cbe01638f5 100644
--- a/guix/scripts/discover.scm
+++ b/guix/scripts/discover.scm
@@ -87,7 +87,8 @@ (define* (write-publish-file #:key (file (%publish-file)))
          (format port "http://~a:~a~%"
                  (avahi-service-address service)
                  (avahi-service-port service)))
-       %publish-services)))
+       %publish-services))
+    #:sync? #f)
   (chmod file #o644))
 
 (define* (read-substitute-urls #:key (file (%publish-file)))
diff --git a/guix/scripts/style.scm b/guix/scripts/style.scm
index 4801529f7e..c45bdd4458 100644
--- a/guix/scripts/style.scm
+++ b/guix/scripts/style.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021-2024 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2021-2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2024 Herman Rimm <herman <at> rimm.ee>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -554,7 +554,8 @@ (define* (format-whole-file file order? #:rest rest)
           (apply pretty-print-with-comments/splice port lst
                  #:format-comment canonicalize-comment
                  #:format-vertical-space canonicalize-vertical-space
-                 rest))))))
+                 rest))
+        #:sync? #f))))
 
 
 ;;;
diff --git a/guix/substitutes.scm b/guix/substitutes.scm
index 45c165e03e..44d666f851 100644
--- a/guix/substitutes.scm
+++ b/guix/substitutes.scm
@@ -127,7 +127,8 @@ (define (cache-narinfo! cache-url path narinfo ttl)
     (mkdir-p (dirname file))
     (with-atomic-file-output file
       (lambda (out)
-        (write (cache-entry cache-url narinfo) out))))
+        (write (cache-entry cache-url narinfo) out))
+      #:sync? #f))
 
   narinfo)
 
-- 
2.49.0

Message #23 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77680 <at> debbugs.gnu.org
Subject: Re: [bug#77680] [PATCH 3/4] substitutes: Ignore corrupt cached
 narinfos.
Date: Thu, 10 Apr 2025 11:31:27 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> * guix/substitutes.scm (cached-narinfo): Wrap ‘read’ call in
> ‘false-if-exception’.  Add catch-all ‘match’ clause.
>
> Change-Id: I2d776f64b6521f778b4ab3f956b35cdef2ddb383
> ---
>  guix/substitutes.scm | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/guix/substitutes.scm b/guix/substitutes.scm
> index 86b9f5472a..45c165e03e 100644
> --- a/guix/substitutes.scm
> +++ b/guix/substitutes.scm
> @@ -300,7 +300,7 @@ (define (cached-narinfo cache-url path)
>      (lambda ()
>        (call-with-input-file cache-file
>          (lambda (p)
> -          (match (read p)
> +          (match (false-if-exception (read p))
>              (('narinfo ('version 2)
>                         ('cache-uri cache-uri)
>                         ('date date) ('ttl ttl) ('value #f))
> @@ -315,7 +315,8 @@ (define (cached-narinfo cache-url path)
>               (if (obsolete? date now ttl)
>                   (values #f #f)
>                   (values #t (string->narinfo value cache-uri))))
> -            (('narinfo ('version v) _ ...)
> +            (_
> +             ;; Corrupt or incompatible cached entry.
>               (values #f #f))))))
>      (lambda _
>        (values #f #f))))

I believe the particular failure case we're introducing here is that the
cache file is empty, so maybe we could match that specific case here and
handle it silently.

I'm a bit wary of handling any and all cache issues silently since that
could make it harder to spot that the caching is entirely broken.

[signature.asc (application/pgp-signature, inline)]

Message #26 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77680 <at> debbugs.gnu.org
Subject: Re: [PATCH 4/4] guix: Avoid ‘fdatasync’ call
 for caches and regular files.
Date: Thu, 10 Apr 2025 11:34:55 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Fixes <https://issues.guix.gnu.org/77606>.
>
> Calling ‘fdatasync’ for each and every narinfo file created by ‘guix
> substitute’ proved to be too expensive on spinning HDDs and/or under
> load (from 0.1s to 1.3s for the ‘fdatasync’ call alone).
>
> * guix/git-authenticate.scm (cache-authenticated-commit): Pass #:sync? #f.
> * guix/http-client.scm (http-fetch/cached): Likewise.
> * guix/scripts/discover.scm (write-publish-file): Likewise.
> * guix/scripts/style.scm (format-whole-file): Likewise.
> * guix/substitutes.scm (cache-narinfo!): Likewise.
>
> Reported-by: Christopher Baines <mail <at> cbaines.net>
> Change-Id: I82297eae737bc5aae8a3f7604119e9f3d4b625bf
> ---
>  guix/git-authenticate.scm | 5 +++--
>  guix/http-client.scm      | 5 +++--
>  guix/scripts/discover.scm | 3 ++-
>  guix/scripts/style.scm    | 5 +++--
>  guix/substitutes.scm      | 3 ++-
>  5 files changed, 13 insertions(+), 8 deletions(-)
>
> diff --git a/guix/git-authenticate.scm b/guix/git-authenticate.scm
> index 37c69d0880..3cd1175c32 100644
> --- a/guix/git-authenticate.scm
> +++ b/guix/git-authenticate.scm
> @@ -1,5 +1,5 @@
>  ;;; GNU Guix --- Functional package management for GNU
> -;;; Copyright © 2019, 2020, 2021, 2022 Ludovic Courtès <ludo <at> gnu.org>
> +;;; Copyright © 2019-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -344,7 +344,8 @@ (define (cache-authenticated-commit key commit-id)
>            (chmod port #o600)
>            (display ";; List of previously-authenticated commits.\n\n"
>                     port)
> -          (pretty-print lst port))))))
> +          (pretty-print lst port)))
> +      #:sync? #f)))

This introduces (or maybe increases) the risk that this file will be
empty, does this happen often enough to justify the change?

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 77680 <at> debbugs.gnu.org
Subject: Re: [PATCH 4/4] guix: Avoid ‘fdatasync’ call
 for caches and regular files.
Date: Thu, 10 Apr 2025 13:08:02 +0200

Christopher Baines <mail <at> cbaines.net> skribis:

>> @@ -344,7 +344,8 @@ (define (cache-authenticated-commit key commit-id)
>>            (chmod port #o600)
>>            (display ";; List of previously-authenticated commits.\n\n"
>>                     port)
>> -          (pretty-print lst port))))))
>> +          (pretty-print lst port)))
>> +      #:sync? #f)))
>
> This introduces (or maybe increases) the risk that this file will be
> empty, does this happen often enough to justify the change?

My thinking is that it’s just a cache, so it shouldn’t need special
treatment as if it were a precious file (which is what the ‘fdatasync’
call was initially added for); it’s OK if we lose this file.

But it’s rarely modified so from that perspective keeping the
‘fdatasync’ call cannot hurt.

Ludo’.

Message #32 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v2 0/4] Avoid 'fdatasync' calls for cache files
Date: Fri, 11 Apr 2025 15:54:56 +0200

Changes since v1:

  • Patch #3 only tolerates empty narinfo files instead of
    doing (false-if-exception (read p)), as suggested by Chris.

Ludo'.

Ludovic Courtès (4):
  cache: Remove unused import.
  utils: Add #:sync? parameter to ‘with-atomic-file-output’.
  substitutes: Ignore corrupt cached narinfos.
  guix: Avoid ‘fdatasync’ call for caches and regular files.

 guix/cache.scm            |  1 -
 guix/git-authenticate.scm |  5 +++--
 guix/http-client.scm      |  5 +++--
 guix/scripts/discover.scm |  3 ++-
 guix/scripts/style.scm    |  5 +++--
 guix/substitutes.scm      |  5 ++++-
 guix/utils.scm            | 14 ++++++++++----
 7 files changed, 25 insertions(+), 13 deletions(-)


base-commit: 772b70455d0d5972fdad80d8529647dce20f409a
-- 
2.49.0

Message #35 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v2 1/4] cache: Remove unused import.
Date: Fri, 11 Apr 2025 15:54:57 +0200

* guix/cache.scm: Remove unused import.

Change-Id: I06956534aa5342a084ebf71b61518c515ce9288c
---
 guix/cache.scm | 1 -
 1 file changed, 1 deletion(-)

diff --git a/guix/cache.scm b/guix/cache.scm
index 8b12312c77..5d8a0edbaa 100644
--- a/guix/cache.scm
+++ b/guix/cache.scm
@@ -18,7 +18,6 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (guix cache)
-  #:use-module ((guix utils) #:select (with-atomic-file-output))
   #:autoload   (guix build syscalls) (lock-file unlock-file)
   #:use-module (srfi srfi-19)
   #:use-module (srfi srfi-26)
-- 
2.49.0

Message #38 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v2 2/4] utils: Add #:sync? parameter to ‘with-atomic-file-output’.
Date: Fri, 11 Apr 2025 15:54:58 +0200

* guix/utils.scm (with-atomic-file-output): Add #:sync? and honor it.
Add ‘force-output’ call.

Change-Id: I2479778ae55360c0fab3389ac9249045a27b3568
---
 guix/utils.scm | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/guix/utils.scm b/guix/utils.scm
index c7c23d9d5b..7ae98096c2 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2022, 2024 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012-2022, 2024-2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2013, 2014, 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2014 Eric Bavier <bavier <at> member.fsf.org>
 ;;; Copyright © 2014 Ian Denhardt <ian <at> zenhack.net>
@@ -1057,16 +1057,22 @@ (define (call-with-temporary-directory proc)
       (lambda ()
         (false-if-exception (delete-file-recursively tmp-dir))))))
 
-(define (with-atomic-file-output file proc)
+(define* (with-atomic-file-output file proc #:key (sync? #t))
   "Call PROC with an output port for the file that is going to replace FILE.
 Upon success, FILE is atomically replaced by what has been written to the
-output port, and PROC's result is returned."
+output port, and PROC's result is returned.
+
+When SYNC? is true, call 'fdatasync' on the temporary file before renaming it
+to FILE; set it to #false for caches and temporary files to improve
+performance."
   (let* ((template (string-append file ".XXXXXX"))
          (out      (mkstemp! template)))
     (with-throw-handler #t
       (lambda ()
         (let ((result (proc out)))
-          (fdatasync out)
+          (when sync?
+            (force-output out)
+            (fdatasync out))
           (close-port out)
           (rename-file template file)
           result))
-- 
2.49.0

Message #41 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v2 3/4] substitutes: Ignore corrupt cached narinfos.
Date: Fri, 11 Apr 2025 15:54:59 +0200

* guix/substitutes.scm (cached-narinfo): Add ‘eof-object?’ clause.

Change-Id: I2d776f64b6521f778b4ab3f956b35cdef2ddb383
---
 guix/substitutes.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/guix/substitutes.scm b/guix/substitutes.scm
index 86b9f5472a..9edce5b2f8 100644
--- a/guix/substitutes.scm
+++ b/guix/substitutes.scm
@@ -316,6 +316,8 @@ (define (cached-narinfo cache-url path)
                  (values #f #f)
                  (values #t (string->narinfo value cache-uri))))
             (('narinfo ('version v) _ ...)
+             (values #f #f))
+            ((? eof-object?)                      ;corrupt file
              (values #f #f))))))
     (lambda _
       (values #f #f))))
-- 
2.49.0

Message #44 received at 77680 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 77680 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Christopher Baines <mail <at> cbaines.net>
Subject: [PATCH v2 4/4] guix: Avoid ‘fdatasync’ call for caches and regular files.
Date: Fri, 11 Apr 2025 15:55:00 +0200

Fixes <https://issues.guix.gnu.org/77606>.

Calling ‘fdatasync’ for each and every narinfo file created by ‘guix
substitute’ proved to be too expensive on spinning HDDs and/or under
load (from 0.1s to 1.3s for the ‘fdatasync’ call alone).

* guix/git-authenticate.scm (cache-authenticated-commit): Pass #:sync? #f.
* guix/http-client.scm (http-fetch/cached): Likewise.
* guix/scripts/discover.scm (write-publish-file): Likewise.
* guix/scripts/style.scm (format-whole-file): Likewise.
* guix/substitutes.scm (cache-narinfo!): Likewise.

Reported-by: Christopher Baines <mail <at> cbaines.net>
Change-Id: I82297eae737bc5aae8a3f7604119e9f3d4b625bf
---
 guix/git-authenticate.scm | 5 +++--
 guix/http-client.scm      | 5 +++--
 guix/scripts/discover.scm | 3 ++-
 guix/scripts/style.scm    | 5 +++--
 guix/substitutes.scm      | 3 ++-
 5 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/guix/git-authenticate.scm b/guix/git-authenticate.scm
index 37c69d0880..3cd1175c32 100644
--- a/guix/git-authenticate.scm
+++ b/guix/git-authenticate.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019, 2020, 2021, 2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2019-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -344,7 +344,8 @@ (define (cache-authenticated-commit key commit-id)
           (chmod port #o600)
           (display ";; List of previously-authenticated commits.\n\n"
                    port)
-          (pretty-print lst port))))))
+          (pretty-print lst port)))
+      #:sync? #f)))
 
 
 ;;;
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 9138a627ac..4e0cc59e91 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2018, 2020-2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012-2018, 2020-2022, 2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me <at> tobias.gr>
@@ -346,7 +346,8 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
           (when cache-port
             (close-port cache-port))
           (with-atomic-file-output file
-            (cut write-cache port <>))
+            (cut write-cache port <>)
+            #:sync? #f)
           (close-port port)
           (open-input-file file))))
 
diff --git a/guix/scripts/discover.scm b/guix/scripts/discover.scm
index 32bf6085a5..cbe01638f5 100644
--- a/guix/scripts/discover.scm
+++ b/guix/scripts/discover.scm
@@ -87,7 +87,8 @@ (define* (write-publish-file #:key (file (%publish-file)))
          (format port "http://~a:~a~%"
                  (avahi-service-address service)
                  (avahi-service-port service)))
-       %publish-services)))
+       %publish-services))
+    #:sync? #f)
   (chmod file #o644))
 
 (define* (read-substitute-urls #:key (file (%publish-file)))
diff --git a/guix/scripts/style.scm b/guix/scripts/style.scm
index 4801529f7e..c45bdd4458 100644
--- a/guix/scripts/style.scm
+++ b/guix/scripts/style.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021-2024 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2021-2025 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2024 Herman Rimm <herman <at> rimm.ee>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -554,7 +554,8 @@ (define* (format-whole-file file order? #:rest rest)
           (apply pretty-print-with-comments/splice port lst
                  #:format-comment canonicalize-comment
                  #:format-vertical-space canonicalize-vertical-space
-                 rest))))))
+                 rest))
+        #:sync? #f))))
 
 
 ;;;
diff --git a/guix/substitutes.scm b/guix/substitutes.scm
index 9edce5b2f8..24b7873ce2 100644
--- a/guix/substitutes.scm
+++ b/guix/substitutes.scm
@@ -127,7 +127,8 @@ (define (cache-narinfo! cache-url path narinfo ttl)
     (mkdir-p (dirname file))
     (with-atomic-file-output file
       (lambda (out)
-        (write (cache-entry cache-url narinfo) out))))
+        (write (cache-entry cache-url narinfo) out))
+      #:sync? #f))
 
   narinfo)
 
-- 
2.49.0

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.