GNU bug report logs - #50040
[PATCH 0/2] publish: Always render nar/narinfo during backing.

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix-patches; Reported by: Mathieu Othacehe <othacehe@HIDDEN>; Keywords: patch; dated Fri, 13 Aug 2021 10:29:02 UTC; Maintainer for guix-patches is guix-patches@HIDDEN.

Message received at 50040 <at> debbugs.gnu.org:


Received: (at 50040) by debbugs.gnu.org; 13 Aug 2021 10:30:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:30:46 2021
Received: from localhost ([127.0.0.1]:40541 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEUSb-00072G-Or
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:46 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54804)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mEUSX-00071e-V6
 for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:42 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51806)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mEUSS-00015y-P2
 for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:36 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44574
 helo=localhost.localdomain)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mEUSS-0008GJ-5B; Fri, 13 Aug 2021 06:30:36 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: 50040 <at> debbugs.gnu.org
Subject: [PATCH 2/2] publish: Remove cache bypass support.
Date: Fri, 13 Aug 2021 12:30:30 +0200
Message-Id: <20210813103030.1017-2-othacehe@HIDDEN>
X-Mailer: git-send-email 2.32.0
In-Reply-To: <20210813103030.1017-1-othacehe@HIDDEN>
References: <20210813103030.1017-1-othacehe@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 50040
Cc: Mathieu Othacehe <othacehe@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Unconditionally render narinfo and nar files during baking. This avoids the
need for pre-baking nar with a CI system. It also prevents receiving 404
errors during baking.

* guix/scripts/publish.scm (%options): Remove the cache-bypass-threshold
option.
(show-help): Ditto.
(cache-bypass-threshold, bypass-cache?): Remove them.
(render-narinfo/cached): Render a temporary narinfo unconditionally during
baking.
(render-nar/cached): Render a live nar unconditionally during baking.
(guix-publish): Remove cache bypass option.
* gnu/services/base.scm (<guix-publish-configuration>)
[cache-bypass-threshold]: Remove it
(guix-publish-shepherd-service): Adapt it.
* doc/guix.texi (Base services, Invoking guix publish): Document it.
* tests/publish.scm ("with cache", "with cache, lzip + gzip", "with cache,
uncompressed"): Adapt them.
("with cache, cache bypass", "with cache, cache bypass, unmapped hash part"):
Remove them.
---
 doc/guix.texi            | 28 -------------
 gnu/services/base.scm    | 11 +----
 guix/scripts/publish.scm | 54 ++++++------------------
 tests/publish.scm        | 88 ++++------------------------------------
 4 files changed, 24 insertions(+), 157 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 78c1c09858..adb2ada4e8 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12752,13 +12752,6 @@ archive is cached in @var{directory}, subsequent requests succeed and
 are served directly from the cache, which guarantees that clients get
 the best possible bandwidth.
 
-That first @code{.narinfo} request nonetheless returns 200, provided the
-requested store item is ``small enough'', below the cache bypass
-threshold---see @option{--cache-bypass-threshold} below.  That way,
-clients do not have to wait until the archive is baked.  For larger
-store items, the first @code{.narinfo} request returns 404, meaning that
-clients have to wait until the archive is baked.
-
 The ``baking'' process is performed by worker threads.  By default, one
 thread per CPU core is created, but this can be customized.  See
 @option{--workers} below.
@@ -12794,21 +12787,6 @@ This parameter can help adjust server load and substitute latency by
 instructing cooperating clients to be more or less patient when a store
 item is missing.
 
-@item --cache-bypass-threshold=@var{size}
-When used in conjunction with @option{--cache}, store items smaller than
-@var{size} are immediately available, even when they are not yet in
-cache.  @var{size} is a size in bytes, or it can be suffixed by @code{M}
-for megabytes and so on.  The default is @code{10M}.
-
-``Cache bypass'' allows you to reduce the publication delay for clients
-at the expense of possibly additional I/O and CPU use on the server
-side: depending on the client access patterns, those store items can end
-up being baked several times until a copy is available in cache.
-
-Increasing the threshold may be useful for sites that have few users, or
-to guarantee that users get substitutes even for store items that are
-not popular.
-
 @item --nar-path=@var{path}
 Use @var{path} as the prefix for the URLs of ``nar'' files
 (@pxref{Invoking guix archive, normalized archives}).
@@ -15818,12 +15796,6 @@ When it is an integer, this is the number of worker threads used for
 caching; when @code{#f}, the number of processors is used.
 @xref{Invoking guix publish, @option{--workers}}, for more information.
 
-@item @code{cache-bypass-threshold} (default: 10 MiB)
-When @code{cache} is true, this is the maximum size in bytes of a store
-item for which @command{guix publish} may bypass its cache in case of a
-cache miss.  @xref{Invoking guix publish,
-@option{--cache-bypass-threshold}}, for more information.
-
 @item @code{ttl} (default: @code{#f})
 When it is an integer, this denotes the @dfn{time-to-live} in seconds
 of the published archives.  @xref{Invoking guix publish, @option{--ttl}},
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index c784d312b1..ac469abf8c 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1775,8 +1775,6 @@ proxy of 'guix-daemon'...~%")
                (default "nar"))
   (cache       guix-publish-configuration-cache   ;#f | string
                (default #f))
-  (cache-bypass-threshold guix-publish-configuration-cache-bypass-threshold
-                          (default (* 10 (expt 2 20)))) ;integer
   (workers     guix-publish-configuration-workers ;#f | integer
                (default #f))
   (ttl         guix-publish-configuration-ttl     ;#f | integer
@@ -1815,8 +1813,7 @@ raise a deprecation warning if the 'compression-level' field was used."
                    lst))))
 
   (match-record config <guix-publish-configuration>
-    (guix port host nar-path cache workers ttl cache-bypass-threshold
-          advertise?)
+    (guix port host nar-path cache workers ttl advertise?)
     (list (shepherd-service
            (provision '(guix-publish))
            (requirement `(user-processes
@@ -1843,11 +1840,7 @@ raise a deprecation warning if the 'compression-level' field was used."
                                                     "s"))
                                   #~())
                            #$@(if cache
-                                  #~((string-append "--cache=" #$cache)
-                                     #$(string-append
-                                        "--cache-bypass-threshold="
-                                        (number->string
-                                         cache-bypass-threshold)))
+                                  #~((string-append "--cache=" #$cache))
                                   #~()))
 
                      ;; Make sure we run in a UTF-8 locale so we can produce
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 981ef8d267..64f444e901 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -97,9 +97,6 @@ Publish ~a over HTTP.\n") %store-directory)
                          compress archives with METHOD at LEVEL"))
   (display (G_ "
   -c, --cache=DIRECTORY  cache published items to DIRECTORY"))
-  (display (G_ "
-      --cache-bypass-threshold=SIZE
-                         serve store items below SIZE even when not cached"))
   (display (G_ "
       --workers=N        use N workers to bake items"))
   (display (G_ "
@@ -214,10 +211,6 @@ usage."
         (option '(#\c "cache") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'cache arg result)))
-        (option '("cache-bypass-threshold") #t #f
-                (lambda (opt name arg result)
-                  (alist-cons 'cache-bypass-threshold (size->number arg)
-                              result)))
         (option '("workers") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'workers (string->number* arg)
@@ -513,18 +506,6 @@ vanished from the store in the meantime."
                result))
             (apply throw args))))))
 
-(define cache-bypass-threshold
-  ;; Maximum size of a store item that may be served by the '/cached' handlers
-  ;; below even when not in cache.
-  (make-parameter (* 10 (expt 2 20))))
-
-(define (bypass-cache? store item)
-  "Return true if we allow ITEM to be downloaded before it is cached.  ITEM is
-interpreted as the basename of a store item."
-  (guard (c ((store-error? c) #f))
-    (< (path-info-nar-size (query-path-info store item))
-       (cache-bypass-threshold))))
-
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compressions (list %no-compression))
                                 (nar-path "nar") negative-ttl
@@ -584,19 +565,14 @@ requested using POOL."
                                                      #:delete-entry delete-entry
                                                      #:cleanup-period ttl))))
 
-           ;; If ITEM passes 'bypass-cache?', render a temporary narinfo right
-           ;; away, with a short TTL.  The narinfo is temporary because it
-           ;; lacks 'FileSize', for instance, which the cached narinfo will
-           ;; have.  Chances are that the nar will be baked by the time the
-           ;; client asks for it.
-           (if (bypass-cache? store item)
-               (render-narinfo store request hash
-                               #:ttl 300          ;temporary
-                               #:nar-path nar-path
-                               #:compressions compressions)
-               (not-found request
-                          #:phrase "We're baking it"
-                          #:ttl 300)))          ;should be available within 5m
+           ;; Render a temporary narinfo right away, with a short TTL.  The
+           ;; narinfo is temporary because it lacks 'FileSize', for instance,
+           ;; which the cached narinfo will have.  Chances are that the nar
+           ;; will be baked by the time the client asks for it.
+           (render-narinfo store request hash
+                           #:ttl 300          ;temporary
+                           #:nar-path nar-path
+                           #:compressions compressions))
           (else
            (not-found request #:phrase "" #:ttl negative-ttl)))))
 
@@ -740,9 +716,9 @@ return it; otherwise, return 404.  When TTL is true, use it as the
                            `((cache-control (max-age . ,ttl)))
                            '())
 
-                     ;; XXX: We're not returning the actual contents, deferring
-                     ;; instead to 'http-write'.  This is a hack to work around
-                     ;; <http://bugs.gnu.org/21093>.
+                     ;; XXX: We're not returning the actual contents,
+                     ;; deferring instead to 'http-write'.  This is a hack to
+                     ;; work around <http://bugs.gnu.org/21093>.
                      (x-raw-file . ,cached))
                    #f))
           ((let* ((hash (and=> (string-index store-item #\-)
@@ -750,8 +726,7 @@ return it; otherwise, return 404.  When TTL is true, use it as the
                   (item (and hash
                              (guard (c ((store-error? c) #f))
                                (hash-part->path store hash)))))
-             (and item (not (string-null? item))
-                  (bypass-cache? store item)))
+             (and item (not (string-null? item))))
            ;; Render STORE-ITEM live.  We reach this because STORE-ITEM is
            ;; being baked but clients are already asking for it.  Thus, we're
            ;; duplicating work, but doing so allows us to reduce delays.
@@ -1306,10 +1281,7 @@ headers."
 consider using the '--user' option!~%")))
 
       (parameterize ((%public-key public-key)
-                     (%private-key private-key)
-                     (cache-bypass-threshold
-                      (or (assoc-ref opts 'cache-bypass-threshold)
-                          (cache-bypass-threshold))))
+                     (%private-key private-key))
         (info (G_ "publishing ~a on ~a, port ~d~%")
               %store-directory
               (inet-ntop (sockaddr:fam address) (sockaddr:addr address))
diff --git a/tests/publish.scm b/tests/publish.scm
index c3d086995a..43871a2a93 100644
--- a/tests/publish.scm
+++ b/tests/publish.scm
@@ -422,15 +422,14 @@ References: ~%"
         200                                       ;nar/gzip/…
         #t                                        ;Content-Length
         #t                                        ;FileSize
-        404)                                      ;nar/…
+        200)                                      ;nar/…
   (call-with-temporary-directory
    (lambda (cache)
      (let ((thread (with-separate-output-ports
                     (call-with-new-thread
                      (lambda ()
                        (guix-publish "--port=6797" "-C2"
-                                     (string-append "--cache=" cache)
-                                     "--cache-bypass-threshold=0"))))))
+                                     (string-append "--cache=" cache)))))))
        (wait-until-ready 6797)
        (let* ((base     "http://localhost:6797/")
               (part     (store-path-hash-part %item))
@@ -441,9 +440,9 @@ References: ~%"
               (nar      (string-append cache "/gzip/"
                                        (basename %item) ".nar"))
               (response (http-get url)))
-         (and (= 404 (response-code response))
+         (and (= 200 (response-code response))
 
-              ;; We should get an explicitly short TTL for 404 in this case
+              ;; We should get an explicitly short TTL for 200 in this case
               ;; because it's going to become 200 shortly.
               (match (assq-ref (response-headers response) 'cache-control)
                 ((('max-age . ttl))
@@ -477,15 +476,14 @@ References: ~%"
                       (response-code uncompressed)))))))))
 
 (test-equal "with cache, lzip + gzip"
-  '(200 200 404)
+  '(200 200 200)
   (call-with-temporary-directory
    (lambda (cache)
      (let ((thread (with-separate-output-ports
                     (call-with-new-thread
                      (lambda ()
                        (guix-publish "--port=6794" "-Cgzip:2" "-Clzip:2"
-                                     (string-append "--cache=" cache)
-                                     "--cache-bypass-threshold=0"))))))
+                                     (string-append "--cache=" cache)))))))
        (wait-until-ready 6794)
        (let* ((base     "http://localhost:6794/")
               (part     (store-path-hash-part %item))
@@ -533,15 +531,14 @@ References: ~%"
           (* 42 3600)                             ;TTL on nar/…
           (path-info-nar-size
            (query-path-info %store item))         ;FileSize
-          404)                                    ;nar/gzip/…
+          200)                                    ;nar/gzip/…
     (call-with-temporary-directory
      (lambda (cache)
        (let ((thread (with-separate-output-ports
                       (call-with-new-thread
                        (lambda ()
                          (guix-publish "--port=6796" "-C2" "--ttl=42h"
-                                       (string-append "--cache=" cache)
-                                       "--cache-bypass-threshold=0"))))))
+                                       (string-append "--cache=" cache)))))))
          (wait-until-ready 6796)
          (let* ((base     "http://localhost:6796/")
                 (part     (store-path-hash-part item))
@@ -551,7 +548,7 @@ References: ~%"
                 (nar      (string-append cache "/none/"
                                          (basename item) ".nar"))
                 (response (http-get url)))
-           (and (= 404 (response-code response))
+           (and (= 200 (response-code response))
 
                 (wait-for-file cached)
                 (let* ((response     (http-get url))
@@ -611,73 +608,6 @@ References: ~%"
                 (delete-paths %store (list item))
                 (response-code (pk 'response (http-get url))))))))))
 
-(test-equal "with cache, cache bypass"
-  200
-  (call-with-temporary-directory
-   (lambda (cache)
-     (let ((thread (with-separate-output-ports
-                    (call-with-new-thread
-                     (lambda ()
-                       (guix-publish "--port=6788" "-C" "gzip"
-                                     (string-append "--cache=" cache)))))))
-       (wait-until-ready 6788)
-
-       (let* ((base     "http://localhost:6788/")
-              (item     (add-text-to-store %store "random" (random-text)))
-              (part     (store-path-hash-part item))
-              (narinfo  (string-append base part ".narinfo"))
-              (nar      (string-append base "nar/gzip/" (basename item)))
-              (cached   (string-append cache "/gzip/" (basename item)
-                                       ".narinfo")))
-         ;; We're below the default cache bypass threshold, so NAR and NARINFO
-         ;; should immediately return 200.  The NARINFO request should trigger
-         ;; caching, and the next request to NAR should return 200 as well.
-         (and (let ((response (pk 'r1 (http-get nar))))
-                (and (= 200 (response-code response))
-                     (not (response-content-length response)))) ;not known
-              (= 200 (response-code (http-get narinfo)))
-              (begin
-                (wait-for-file cached)
-                (let ((response (pk 'r2 (http-get nar))))
-                  (and (> (response-content-length response)
-                          (stat:size (stat item)))
-                       (response-code response))))))))))
-
-(test-equal "with cache, cache bypass, unmapped hash part"
-  200
-
-  ;; This test reproduces the bug described in <https://bugs.gnu.org/44442>:
-  ;; the daemon connection would be closed as a side effect of a nar request
-  ;; for a non-existing file name.
-  (call-with-temporary-directory
-   (lambda (cache)
-     (let ((thread (with-separate-output-ports
-                    (call-with-new-thread
-                     (lambda ()
-                       (guix-publish "--port=6787" "-C" "gzip"
-                                     (string-append "--cache=" cache)))))))
-       (wait-until-ready 6787)
-
-       (let* ((base     "http://localhost:6787/")
-              (item     (add-text-to-store %store "random" (random-text)))
-              (part     (store-path-hash-part item))
-              (narinfo  (string-append base part ".narinfo"))
-              (nar      (string-append base "nar/gzip/" (basename item)))
-              (cached   (string-append cache "/gzip/" (basename item)
-                                       ".narinfo")))
-         ;; The first response used to be 500 and to terminate the daemon
-         ;; connection as a side effect.
-         (and (= (response-code
-                  (http-get (string-append base "nar/gzip/"
-                                           (make-string 32 #\e)
-                                           "-does-not-exist")))
-                 404)
-              (= 200 (response-code (http-get nar)))
-              (= 200 (response-code (http-get narinfo)))
-              (begin
-                (wait-for-file cached)
-                (response-code (http-get nar)))))))))
-
 (test-equal "/log/NAME"
   `(200 #t application/x-bzip2)
   (let ((drv (run-with-store %store
-- 
2.32.0





Information forwarded to guix-patches@HIDDEN:
bug#50040; Package guix-patches. Full text available.

Message received at 50040 <at> debbugs.gnu.org:


Received: (at 50040) by debbugs.gnu.org; 13 Aug 2021 10:30:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:30:42 2021
Received: from localhost ([127.0.0.1]:40539 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEUSY-00071z-9R
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54802)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mEUSX-00071U-4g
 for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51804)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mEUSR-00014j-Sk
 for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:35 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44574
 helo=localhost.localdomain)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mEUSR-0008GJ-Hq; Fri, 13 Aug 2021 06:30:35 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: 50040 <at> debbugs.gnu.org
Subject: [PATCH 1/2] publish: Defer narinfo string creation to the http-write.
Date: Fri, 13 Aug 2021 12:30:29 +0200
Message-Id: <20210813103030.1017-1-othacehe@HIDDEN>
X-Mailer: git-send-email 2.32.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 50040
Cc: Mathieu Othacehe <othacehe@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

The "narinfo-string" procedure is expensive in term of IO operations and can
take a while under IO pressure, such a GC collecting. Defer its call to a new
thread created in the http-write procedure.

Fixes: <https://issues.guix.gnu.org/48468>
Partially fixes: <https://issues.guix.gnu.org/49089>

* guix/scripts/publish.scm (render-narinfo): Defer the narinfo string creation
to the http-write procedure.
(compression->sexp, sexp->compression): New procedures.
("X-Nar-Compression"): Use them.
("X-Narinfo-Compressions"): New custom header.
(strip-headers): Add the x-nar-path header.
(http-write): Add narinfo on-the-fly creation support. It happens in a
separated thread to prevent blocking the main thread.
---
 guix/scripts/publish.scm | 82 +++++++++++++++++++++++++++++++++-------
 1 file changed, 69 insertions(+), 13 deletions(-)

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 913cbd4fda..981ef8d267 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -24,6 +24,7 @@
   #:use-module ((system repl server) #:prefix repl:)
   #:use-module (ice-9 binary-ports)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 iconv)
   #:use-module (ice-9 match)
   #:use-module (ice-9 poll)
   #:use-module (ice-9 regex)
@@ -409,15 +410,18 @@ appropriate duration.  NAR-PATH specifies the prefix for nar URLs."
   (let ((store-path (hash-part->path store hash)))
     (if (string-null? store-path)
         (not-found request #:phrase "" #:ttl negative-ttl)
-        (values `((content-type . (application/x-nix-narinfo))
+        (values `((content-type . (application/x-nix-narinfo
+                                   (charset . "UTF-8")))
+                  (x-nar-path . ,nar-path)
+                  (x-narinfo-compressions . ,compressions)
                   ,@(if ttl
                         `((cache-control (max-age . ,ttl)))
                         '()))
-                (cut display
-                  (narinfo-string store store-path
-                                  #:nar-path nar-path
-                                  #:compressions compressions)
-                  <>)))))
+                ;; Do not call narinfo-string directly here as it is an
+                ;; expensive call that could potentially block the main
+                ;; thread.  Instead, create the narinfo string in the
+                ;; http-write procedure.
+                store-path))))
 
 (define* (nar-cache-file directory item
                              #:key (compression %no-compression))
@@ -672,19 +676,38 @@ requested using POOL."
                        (link narinfo other)))
                    others))))))
 
+(define (compression->sexp compression)
+  "Return the SEXP representation of COMPRESSION."
+  (match compression
+    (($ <compression> type level)
+     `(compression ,type ,level))))
+
+(define (sexp->compression sexp)
+  "Turn the given SEXP into a <compression> record and return it."
+  (match sexp
+    (('compression type level)
+     (compression type level))))
+
 ;; XXX: Declare the 'X-Nar-Compression' HTTP header, which is in fact for
 ;; internal consumption: it allows us to pass the compression info to
 ;; 'http-write', as part of the workaround to <http://bugs.gnu.org/21093>.
 (declare-header! "X-Nar-Compression"
                  (lambda (str)
-                   (match (call-with-input-string str read)
-                     (('compression type level)
-                      (compression type level))))
+                   (sexp->compression
+                    (call-with-input-string str read)))
                  compression?
                  (lambda (compression port)
-                   (match compression
-                     (($ <compression> type level)
-                      (write `(compression ,type ,level) port)))))
+                   (write (compression->sexp compression) port)))
+
+;; This header is used to pass the supported compressions to http-write in
+;; order to format on-the-fly narinfo responses.
+(declare-header! "X-Narinfo-Compressions"
+                 (lambda (str)
+                   (map sexp->compression
+                        (call-with-input-string str read)))
+                 (cut every compression? <>)
+                 (lambda (compressions port)
+                   (write (map compression->sexp compressions) port)))
 
 (define* (render-nar store request store-item
                      #:key (compression %no-compression))
@@ -839,7 +862,8 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")."
   "Return RESPONSE's headers minus 'Content-Length' and our internal headers."
   (fold alist-delete
         (response-headers response)
-        '(content-length x-raw-file x-nar-compression)))
+        '(content-length x-raw-file x-nar-compression
+                         x-narinfo-compressions x-nar-path)))
 
 (define (sans-content-length response)
   "Return RESPONSE without its 'content-length' header."
@@ -973,6 +997,38 @@ blocking."
              (unless keep-alive?
                (close-port client)))
             (values))))))
+    (('application/x-nix-narinfo . _)
+     (let ((compressions (assoc-ref (response-headers response)
+                                    'x-narinfo-compressions))
+           (nar-path (assoc-ref (response-headers response)
+                                'x-nar-path)))
+       (if nar-path
+           (begin
+             (when (keep-alive? response)
+               (keep-alive client))
+             (call-with-new-thread
+              (lambda ()
+                (set-thread-name "publish narinfo")
+                (let* ((narinfo
+                        (with-store store
+                          (narinfo-string store (utf8->string body)
+                                          #:nar-path nar-path
+                                          #:compressions compressions)))
+                       (narinfo-bv (string->bytevector narinfo "UTF-8"))
+                       (narinfo-length
+                        (bytevector-length narinfo-bv))
+                       (response (write-response
+                                  (with-content-length response
+                                                       narinfo-length)
+                                  client))
+                       (output (response-port response)))
+                  (configure-socket client)
+                  (put-bytevector output narinfo-bv)
+                  (force-output output)
+                  (unless (keep-alive? response)
+                    (close-port output))
+                  (values)))))
+           (%http-write server client response body))))
     (_
      (match (assoc-ref (response-headers response) 'x-raw-file)
        ((? string? file)
-- 
2.32.0





Information forwarded to guix-patches@HIDDEN:
bug#50040; Package guix-patches. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 13 Aug 2021 10:28:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:28:10 2021
Received: from localhost ([127.0.0.1]:40526 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mEUQ6-0006uy-Ds
	for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:28:10 -0400
Received: from lists.gnu.org ([209.51.188.17]:37340)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0006up-HE
 for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:28:08 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42384)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0000MP-9X
 for guix-patches@HIDDEN; Fri, 13 Aug 2021 06:28:06 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51410)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0007Lh-1F
 for guix-patches@HIDDEN; Fri, 13 Aug 2021 06:28:06 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44572
 helo=localhost.localdomain)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1mEUQ1-0003LR-Ny; Fri, 13 Aug 2021 06:28:05 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: guix-patches@HIDDEN
Subject: [PATCH 0/2] publish: Always render nar/narinfo during backing.
Date: Fri, 13 Aug 2021 12:28:00 +0200
Message-Id: <20210813102800.805-1-othacehe@HIDDEN>
X-Mailer: git-send-email 2.32.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
Cc: Mathieu Othacehe <othacehe@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hello,

This patchset does two things:

* It ensures that the narinfo creation is always done in a dedicated thread to
  prevent blocking the main publish thread and cause the timeout connection
  errors that some users experimented.

* It removes the cache bypass support and always renders temporary narinfo and
  nar files during baking. This way, the first user asking for a substitute
  won't get a 404 error. It also removes the need for Cuirass to pre-bake nar
  files. Finally it will make way easier to use a publish server with caching
  in a CI context.

The downside is that it makes the publish server code harder to understand and
that it consumes more server resources by duplicating the nar/narinfo file
creation.

In the long run, spawning new threads constantly won't scale. I think it would
be nice to rewrite the publish server using Fiber or an HTTP server framework.

Thanks,

Mathieu

Mathieu Othacehe (2):
  publish: Defer narinfo string creation to the http-write.
  publish: Remove cache bypass support.

 doc/guix.texi            |  15 -----
 gnu/services/base.scm    |  11 +---
 guix/scripts/publish.scm | 136 +++++++++++++++++++++++----------------
 tests/publish.scm        |  88 +++----------------------
 4 files changed, 93 insertions(+), 157 deletions(-)

-- 
2.32.0





Acknowledgement sent to Mathieu Othacehe <othacehe@HIDDEN>:
New bug report received and forwarded. Copy sent to guix-patches@HIDDEN. Full text available.
Report forwarded to guix-patches@HIDDEN:
bug#50040; Package guix-patches. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Fri, 13 Aug 2021 10:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.