Received: (at 50040) by debbugs.gnu.org; 13 Aug 2021 10:30:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:30:46 2021
Received: from localhost ([127.0.0.1]:40541 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1mEUSb-00072G-Or
for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:46 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54804)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <othacehe@HIDDEN>) id 1mEUSX-00071e-V6
for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:42 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51806)
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <othacehe@HIDDEN>) id 1mEUSS-00015y-P2
for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:36 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44574
helo=localhost.localdomain)
by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
id 1mEUSS-0008GJ-5B; Fri, 13 Aug 2021 06:30:36 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: 50040 <at> debbugs.gnu.org
Subject: [PATCH 2/2] publish: Remove cache bypass support.
Date: Fri, 13 Aug 2021 12:30:30 +0200
Message-Id: <20210813103030.1017-2-othacehe@HIDDEN>
X-Mailer: git-send-email 2.32.0
In-Reply-To: <20210813103030.1017-1-othacehe@HIDDEN>
References: <20210813103030.1017-1-othacehe@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 50040
Cc: Mathieu Othacehe <othacehe@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Unconditionally render narinfo and nar files during baking. This avoids the
need for pre-baking nar with a CI system. It also prevents receiving 404
errors during baking.
* guix/scripts/publish.scm (%options): Remove the cache-bypass-threshold
option.
(show-help): Ditto.
(cache-bypass-threshold, bypass-cache?): Remove them.
(render-narinfo/cached): Render a temporary narinfo unconditionally during
baking.
(render-nar/cached): Render a live nar unconditionally during baking.
(guix-publish): Remove cache bypass option.
* gnu/services/base.scm (<guix-publish-configuration>)
[cache-bypass-threshold]: Remove it
(guix-publish-shepherd-service): Adapt it.
* doc/guix.texi (Base services, Invoking guix publish): Document it.
* tests/publish.scm ("with cache", "with cache, lzip + gzip", "with cache,
uncompressed"): Adapt them.
("with cache, cache bypass", "with cache, cache bypass, unmapped hash part"):
Remove them.
---
doc/guix.texi | 28 -------------
gnu/services/base.scm | 11 +----
guix/scripts/publish.scm | 54 ++++++------------------
tests/publish.scm | 88 ++++------------------------------------
4 files changed, 24 insertions(+), 157 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 78c1c09858..adb2ada4e8 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12752,13 +12752,6 @@ archive is cached in @var{directory}, subsequent requests succeed and
are served directly from the cache, which guarantees that clients get
the best possible bandwidth.
-That first @code{.narinfo} request nonetheless returns 200, provided the
-requested store item is ``small enough'', below the cache bypass
-threshold---see @option{--cache-bypass-threshold} below. That way,
-clients do not have to wait until the archive is baked. For larger
-store items, the first @code{.narinfo} request returns 404, meaning that
-clients have to wait until the archive is baked.
-
The ``baking'' process is performed by worker threads. By default, one
thread per CPU core is created, but this can be customized. See
@option{--workers} below.
@@ -12794,21 +12787,6 @@ This parameter can help adjust server load and substitute latency by
instructing cooperating clients to be more or less patient when a store
item is missing.
-@item --cache-bypass-threshold=@var{size}
-When used in conjunction with @option{--cache}, store items smaller than
-@var{size} are immediately available, even when they are not yet in
-cache. @var{size} is a size in bytes, or it can be suffixed by @code{M}
-for megabytes and so on. The default is @code{10M}.
-
-``Cache bypass'' allows you to reduce the publication delay for clients
-at the expense of possibly additional I/O and CPU use on the server
-side: depending on the client access patterns, those store items can end
-up being baked several times until a copy is available in cache.
-
-Increasing the threshold may be useful for sites that have few users, or
-to guarantee that users get substitutes even for store items that are
-not popular.
-
@item --nar-path=@var{path}
Use @var{path} as the prefix for the URLs of ``nar'' files
(@pxref{Invoking guix archive, normalized archives}).
@@ -15818,12 +15796,6 @@ When it is an integer, this is the number of worker threads used for
caching; when @code{#f}, the number of processors is used.
@xref{Invoking guix publish, @option{--workers}}, for more information.
-@item @code{cache-bypass-threshold} (default: 10 MiB)
-When @code{cache} is true, this is the maximum size in bytes of a store
-item for which @command{guix publish} may bypass its cache in case of a
-cache miss. @xref{Invoking guix publish,
-@option{--cache-bypass-threshold}}, for more information.
-
@item @code{ttl} (default: @code{#f})
When it is an integer, this denotes the @dfn{time-to-live} in seconds
of the published archives. @xref{Invoking guix publish, @option{--ttl}},
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index c784d312b1..ac469abf8c 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1775,8 +1775,6 @@ proxy of 'guix-daemon'...~%")
(default "nar"))
(cache guix-publish-configuration-cache ;#f | string
(default #f))
- (cache-bypass-threshold guix-publish-configuration-cache-bypass-threshold
- (default (* 10 (expt 2 20)))) ;integer
(workers guix-publish-configuration-workers ;#f | integer
(default #f))
(ttl guix-publish-configuration-ttl ;#f | integer
@@ -1815,8 +1813,7 @@ raise a deprecation warning if the 'compression-level' field was used."
lst))))
(match-record config <guix-publish-configuration>
- (guix port host nar-path cache workers ttl cache-bypass-threshold
- advertise?)
+ (guix port host nar-path cache workers ttl advertise?)
(list (shepherd-service
(provision '(guix-publish))
(requirement `(user-processes
@@ -1843,11 +1840,7 @@ raise a deprecation warning if the 'compression-level' field was used."
"s"))
#~())
#$@(if cache
- #~((string-append "--cache=" #$cache)
- #$(string-append
- "--cache-bypass-threshold="
- (number->string
- cache-bypass-threshold)))
+ #~((string-append "--cache=" #$cache))
#~()))
;; Make sure we run in a UTF-8 locale so we can produce
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 981ef8d267..64f444e901 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -97,9 +97,6 @@ Publish ~a over HTTP.\n") %store-directory)
compress archives with METHOD at LEVEL"))
(display (G_ "
-c, --cache=DIRECTORY cache published items to DIRECTORY"))
- (display (G_ "
- --cache-bypass-threshold=SIZE
- serve store items below SIZE even when not cached"))
(display (G_ "
--workers=N use N workers to bake items"))
(display (G_ "
@@ -214,10 +211,6 @@ usage."
(option '(#\c "cache") #t #f
(lambda (opt name arg result)
(alist-cons 'cache arg result)))
- (option '("cache-bypass-threshold") #t #f
- (lambda (opt name arg result)
- (alist-cons 'cache-bypass-threshold (size->number arg)
- result)))
(option '("workers") #t #f
(lambda (opt name arg result)
(alist-cons 'workers (string->number* arg)
@@ -513,18 +506,6 @@ vanished from the store in the meantime."
result))
(apply throw args))))))
-(define cache-bypass-threshold
- ;; Maximum size of a store item that may be served by the '/cached' handlers
- ;; below even when not in cache.
- (make-parameter (* 10 (expt 2 20))))
-
-(define (bypass-cache? store item)
- "Return true if we allow ITEM to be downloaded before it is cached. ITEM is
-interpreted as the basename of a store item."
- (guard (c ((store-error? c) #f))
- (< (path-info-nar-size (query-path-info store item))
- (cache-bypass-threshold))))
-
(define* (render-narinfo/cached store request hash
#:key ttl (compressions (list %no-compression))
(nar-path "nar") negative-ttl
@@ -584,19 +565,14 @@ requested using POOL."
#:delete-entry delete-entry
#:cleanup-period ttl))))
- ;; If ITEM passes 'bypass-cache?', render a temporary narinfo right
- ;; away, with a short TTL. The narinfo is temporary because it
- ;; lacks 'FileSize', for instance, which the cached narinfo will
- ;; have. Chances are that the nar will be baked by the time the
- ;; client asks for it.
- (if (bypass-cache? store item)
- (render-narinfo store request hash
- #:ttl 300 ;temporary
- #:nar-path nar-path
- #:compressions compressions)
- (not-found request
- #:phrase "We're baking it"
- #:ttl 300))) ;should be available within 5m
+ ;; Render a temporary narinfo right away, with a short TTL. The
+ ;; narinfo is temporary because it lacks 'FileSize', for instance,
+ ;; which the cached narinfo will have. Chances are that the nar
+ ;; will be baked by the time the client asks for it.
+ (render-narinfo store request hash
+ #:ttl 300 ;temporary
+ #:nar-path nar-path
+ #:compressions compressions))
(else
(not-found request #:phrase "" #:ttl negative-ttl)))))
@@ -740,9 +716,9 @@ return it; otherwise, return 404. When TTL is true, use it as the
`((cache-control (max-age . ,ttl)))
'())
- ;; XXX: We're not returning the actual contents, deferring
- ;; instead to 'http-write'. This is a hack to work around
- ;; <http://bugs.gnu.org/21093>.
+ ;; XXX: We're not returning the actual contents,
+ ;; deferring instead to 'http-write'. This is a hack to
+ ;; work around <http://bugs.gnu.org/21093>.
(x-raw-file . ,cached))
#f))
((let* ((hash (and=> (string-index store-item #\-)
@@ -750,8 +726,7 @@ return it; otherwise, return 404. When TTL is true, use it as the
(item (and hash
(guard (c ((store-error? c) #f))
(hash-part->path store hash)))))
- (and item (not (string-null? item))
- (bypass-cache? store item)))
+ (and item (not (string-null? item))))
;; Render STORE-ITEM live. We reach this because STORE-ITEM is
;; being baked but clients are already asking for it. Thus, we're
;; duplicating work, but doing so allows us to reduce delays.
@@ -1306,10 +1281,7 @@ headers."
consider using the '--user' option!~%")))
(parameterize ((%public-key public-key)
- (%private-key private-key)
- (cache-bypass-threshold
- (or (assoc-ref opts 'cache-bypass-threshold)
- (cache-bypass-threshold))))
+ (%private-key private-key))
(info (G_ "publishing ~a on ~a, port ~d~%")
%store-directory
(inet-ntop (sockaddr:fam address) (sockaddr:addr address))
diff --git a/tests/publish.scm b/tests/publish.scm
index c3d086995a..43871a2a93 100644
--- a/tests/publish.scm
+++ b/tests/publish.scm
@@ -422,15 +422,14 @@ References: ~%"
200 ;nar/gzip/…
#t ;Content-Length
#t ;FileSize
- 404) ;nar/…
+ 200) ;nar/…
(call-with-temporary-directory
(lambda (cache)
(let ((thread (with-separate-output-ports
(call-with-new-thread
(lambda ()
(guix-publish "--port=6797" "-C2"
- (string-append "--cache=" cache)
- "--cache-bypass-threshold=0"))))))
+ (string-append "--cache=" cache)))))))
(wait-until-ready 6797)
(let* ((base "http://localhost:6797/")
(part (store-path-hash-part %item))
@@ -441,9 +440,9 @@ References: ~%"
(nar (string-append cache "/gzip/"
(basename %item) ".nar"))
(response (http-get url)))
- (and (= 404 (response-code response))
+ (and (= 200 (response-code response))
- ;; We should get an explicitly short TTL for 404 in this case
+ ;; We should get an explicitly short TTL for 200 in this case
;; because it's going to become 200 shortly.
(match (assq-ref (response-headers response) 'cache-control)
((('max-age . ttl))
@@ -477,15 +476,14 @@ References: ~%"
(response-code uncompressed)))))))))
(test-equal "with cache, lzip + gzip"
- '(200 200 404)
+ '(200 200 200)
(call-with-temporary-directory
(lambda (cache)
(let ((thread (with-separate-output-ports
(call-with-new-thread
(lambda ()
(guix-publish "--port=6794" "-Cgzip:2" "-Clzip:2"
- (string-append "--cache=" cache)
- "--cache-bypass-threshold=0"))))))
+ (string-append "--cache=" cache)))))))
(wait-until-ready 6794)
(let* ((base "http://localhost:6794/")
(part (store-path-hash-part %item))
@@ -533,15 +531,14 @@ References: ~%"
(* 42 3600) ;TTL on nar/…
(path-info-nar-size
(query-path-info %store item)) ;FileSize
- 404) ;nar/gzip/…
+ 200) ;nar/gzip/…
(call-with-temporary-directory
(lambda (cache)
(let ((thread (with-separate-output-ports
(call-with-new-thread
(lambda ()
(guix-publish "--port=6796" "-C2" "--ttl=42h"
- (string-append "--cache=" cache)
- "--cache-bypass-threshold=0"))))))
+ (string-append "--cache=" cache)))))))
(wait-until-ready 6796)
(let* ((base "http://localhost:6796/")
(part (store-path-hash-part item))
@@ -551,7 +548,7 @@ References: ~%"
(nar (string-append cache "/none/"
(basename item) ".nar"))
(response (http-get url)))
- (and (= 404 (response-code response))
+ (and (= 200 (response-code response))
(wait-for-file cached)
(let* ((response (http-get url))
@@ -611,73 +608,6 @@ References: ~%"
(delete-paths %store (list item))
(response-code (pk 'response (http-get url))))))))))
-(test-equal "with cache, cache bypass"
- 200
- (call-with-temporary-directory
- (lambda (cache)
- (let ((thread (with-separate-output-ports
- (call-with-new-thread
- (lambda ()
- (guix-publish "--port=6788" "-C" "gzip"
- (string-append "--cache=" cache)))))))
- (wait-until-ready 6788)
-
- (let* ((base "http://localhost:6788/")
- (item (add-text-to-store %store "random" (random-text)))
- (part (store-path-hash-part item))
- (narinfo (string-append base part ".narinfo"))
- (nar (string-append base "nar/gzip/" (basename item)))
- (cached (string-append cache "/gzip/" (basename item)
- ".narinfo")))
- ;; We're below the default cache bypass threshold, so NAR and NARINFO
- ;; should immediately return 200. The NARINFO request should trigger
- ;; caching, and the next request to NAR should return 200 as well.
- (and (let ((response (pk 'r1 (http-get nar))))
- (and (= 200 (response-code response))
- (not (response-content-length response)))) ;not known
- (= 200 (response-code (http-get narinfo)))
- (begin
- (wait-for-file cached)
- (let ((response (pk 'r2 (http-get nar))))
- (and (> (response-content-length response)
- (stat:size (stat item)))
- (response-code response))))))))))
-
-(test-equal "with cache, cache bypass, unmapped hash part"
- 200
-
- ;; This test reproduces the bug described in <https://bugs.gnu.org/44442>:
- ;; the daemon connection would be closed as a side effect of a nar request
- ;; for a non-existing file name.
- (call-with-temporary-directory
- (lambda (cache)
- (let ((thread (with-separate-output-ports
- (call-with-new-thread
- (lambda ()
- (guix-publish "--port=6787" "-C" "gzip"
- (string-append "--cache=" cache)))))))
- (wait-until-ready 6787)
-
- (let* ((base "http://localhost:6787/")
- (item (add-text-to-store %store "random" (random-text)))
- (part (store-path-hash-part item))
- (narinfo (string-append base part ".narinfo"))
- (nar (string-append base "nar/gzip/" (basename item)))
- (cached (string-append cache "/gzip/" (basename item)
- ".narinfo")))
- ;; The first response used to be 500 and to terminate the daemon
- ;; connection as a side effect.
- (and (= (response-code
- (http-get (string-append base "nar/gzip/"
- (make-string 32 #\e)
- "-does-not-exist")))
- 404)
- (= 200 (response-code (http-get nar)))
- (= 200 (response-code (http-get narinfo)))
- (begin
- (wait-for-file cached)
- (response-code (http-get nar)))))))))
-
(test-equal "/log/NAME"
`(200 #t application/x-bzip2)
(let ((drv (run-with-store %store
--
2.32.0
guix-patches@HIDDEN:bug#50040; Package guix-patches.
Full text available.
Received: (at 50040) by debbugs.gnu.org; 13 Aug 2021 10:30:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:30:42 2021
Received: from localhost ([127.0.0.1]:40539 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1mEUSY-00071z-9R
for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:54802)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <othacehe@HIDDEN>) id 1mEUSX-00071U-4g
for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51804)
by eggs.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <othacehe@HIDDEN>) id 1mEUSR-00014j-Sk
for 50040 <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:30:35 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44574
helo=localhost.localdomain)
by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
id 1mEUSR-0008GJ-Hq; Fri, 13 Aug 2021 06:30:35 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: 50040 <at> debbugs.gnu.org
Subject: [PATCH 1/2] publish: Defer narinfo string creation to the http-write.
Date: Fri, 13 Aug 2021 12:30:29 +0200
Message-Id: <20210813103030.1017-1-othacehe@HIDDEN>
X-Mailer: git-send-email 2.32.0
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 50040
Cc: Mathieu Othacehe <othacehe@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
The "narinfo-string" procedure is expensive in term of IO operations and can
take a while under IO pressure, such a GC collecting. Defer its call to a new
thread created in the http-write procedure.
Fixes: <https://issues.guix.gnu.org/48468>
Partially fixes: <https://issues.guix.gnu.org/49089>
* guix/scripts/publish.scm (render-narinfo): Defer the narinfo string creation
to the http-write procedure.
(compression->sexp, sexp->compression): New procedures.
("X-Nar-Compression"): Use them.
("X-Narinfo-Compressions"): New custom header.
(strip-headers): Add the x-nar-path header.
(http-write): Add narinfo on-the-fly creation support. It happens in a
separated thread to prevent blocking the main thread.
---
guix/scripts/publish.scm | 82 +++++++++++++++++++++++++++++++++-------
1 file changed, 69 insertions(+), 13 deletions(-)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 913cbd4fda..981ef8d267 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -24,6 +24,7 @@
#:use-module ((system repl server) #:prefix repl:)
#:use-module (ice-9 binary-ports)
#:use-module (ice-9 format)
+ #:use-module (ice-9 iconv)
#:use-module (ice-9 match)
#:use-module (ice-9 poll)
#:use-module (ice-9 regex)
@@ -409,15 +410,18 @@ appropriate duration. NAR-PATH specifies the prefix for nar URLs."
(let ((store-path (hash-part->path store hash)))
(if (string-null? store-path)
(not-found request #:phrase "" #:ttl negative-ttl)
- (values `((content-type . (application/x-nix-narinfo))
+ (values `((content-type . (application/x-nix-narinfo
+ (charset . "UTF-8")))
+ (x-nar-path . ,nar-path)
+ (x-narinfo-compressions . ,compressions)
,@(if ttl
`((cache-control (max-age . ,ttl)))
'()))
- (cut display
- (narinfo-string store store-path
- #:nar-path nar-path
- #:compressions compressions)
- <>)))))
+ ;; Do not call narinfo-string directly here as it is an
+ ;; expensive call that could potentially block the main
+ ;; thread. Instead, create the narinfo string in the
+ ;; http-write procedure.
+ store-path))))
(define* (nar-cache-file directory item
#:key (compression %no-compression))
@@ -672,19 +676,38 @@ requested using POOL."
(link narinfo other)))
others))))))
+(define (compression->sexp compression)
+ "Return the SEXP representation of COMPRESSION."
+ (match compression
+ (($ <compression> type level)
+ `(compression ,type ,level))))
+
+(define (sexp->compression sexp)
+ "Turn the given SEXP into a <compression> record and return it."
+ (match sexp
+ (('compression type level)
+ (compression type level))))
+
;; XXX: Declare the 'X-Nar-Compression' HTTP header, which is in fact for
;; internal consumption: it allows us to pass the compression info to
;; 'http-write', as part of the workaround to <http://bugs.gnu.org/21093>.
(declare-header! "X-Nar-Compression"
(lambda (str)
- (match (call-with-input-string str read)
- (('compression type level)
- (compression type level))))
+ (sexp->compression
+ (call-with-input-string str read)))
compression?
(lambda (compression port)
- (match compression
- (($ <compression> type level)
- (write `(compression ,type ,level) port)))))
+ (write (compression->sexp compression) port)))
+
+;; This header is used to pass the supported compressions to http-write in
+;; order to format on-the-fly narinfo responses.
+(declare-header! "X-Narinfo-Compressions"
+ (lambda (str)
+ (map sexp->compression
+ (call-with-input-string str read)))
+ (cut every compression? <>)
+ (lambda (compressions port)
+ (write (map compression->sexp compressions) port)))
(define* (render-nar store request store-item
#:key (compression %no-compression))
@@ -839,7 +862,8 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")."
"Return RESPONSE's headers minus 'Content-Length' and our internal headers."
(fold alist-delete
(response-headers response)
- '(content-length x-raw-file x-nar-compression)))
+ '(content-length x-raw-file x-nar-compression
+ x-narinfo-compressions x-nar-path)))
(define (sans-content-length response)
"Return RESPONSE without its 'content-length' header."
@@ -973,6 +997,38 @@ blocking."
(unless keep-alive?
(close-port client)))
(values))))))
+ (('application/x-nix-narinfo . _)
+ (let ((compressions (assoc-ref (response-headers response)
+ 'x-narinfo-compressions))
+ (nar-path (assoc-ref (response-headers response)
+ 'x-nar-path)))
+ (if nar-path
+ (begin
+ (when (keep-alive? response)
+ (keep-alive client))
+ (call-with-new-thread
+ (lambda ()
+ (set-thread-name "publish narinfo")
+ (let* ((narinfo
+ (with-store store
+ (narinfo-string store (utf8->string body)
+ #:nar-path nar-path
+ #:compressions compressions)))
+ (narinfo-bv (string->bytevector narinfo "UTF-8"))
+ (narinfo-length
+ (bytevector-length narinfo-bv))
+ (response (write-response
+ (with-content-length response
+ narinfo-length)
+ client))
+ (output (response-port response)))
+ (configure-socket client)
+ (put-bytevector output narinfo-bv)
+ (force-output output)
+ (unless (keep-alive? response)
+ (close-port output))
+ (values)))))
+ (%http-write server client response body))))
(_
(match (assoc-ref (response-headers response) 'x-raw-file)
((? string? file)
--
2.32.0
guix-patches@HIDDEN:bug#50040; Package guix-patches.
Full text available.Received: (at submit) by debbugs.gnu.org; 13 Aug 2021 10:28:10 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Aug 13 06:28:10 2021 Received: from localhost ([127.0.0.1]:40526 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1mEUQ6-0006uy-Ds for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:28:10 -0400 Received: from lists.gnu.org ([209.51.188.17]:37340) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0006up-HE for submit <at> debbugs.gnu.org; Fri, 13 Aug 2021 06:28:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42384) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0000MP-9X for guix-patches@HIDDEN; Fri, 13 Aug 2021 06:28:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51410) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mEUQ2-0007Lh-1F for guix-patches@HIDDEN; Fri, 13 Aug 2021 06:28:06 -0400 Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44572 helo=localhost.localdomain) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1mEUQ1-0003LR-Ny; Fri, 13 Aug 2021 06:28:05 -0400 From: Mathieu Othacehe <othacehe@HIDDEN> To: guix-patches@HIDDEN Subject: [PATCH 0/2] publish: Always render nar/narinfo during backing. Date: Fri, 13 Aug 2021 12:28:00 +0200 Message-Id: <20210813102800.805-1-othacehe@HIDDEN> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: Mathieu Othacehe <othacehe@HIDDEN> X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) Hello, This patchset does two things: * It ensures that the narinfo creation is always done in a dedicated thread to prevent blocking the main publish thread and cause the timeout connection errors that some users experimented. * It removes the cache bypass support and always renders temporary narinfo and nar files during baking. This way, the first user asking for a substitute won't get a 404 error. It also removes the need for Cuirass to pre-bake nar files. Finally it will make way easier to use a publish server with caching in a CI context. The downside is that it makes the publish server code harder to understand and that it consumes more server resources by duplicating the nar/narinfo file creation. In the long run, spawning new threads constantly won't scale. I think it would be nice to rewrite the publish server using Fiber or an HTTP server framework. Thanks, Mathieu Mathieu Othacehe (2): publish: Defer narinfo string creation to the http-write. publish: Remove cache bypass support. doc/guix.texi | 15 ----- gnu/services/base.scm | 11 +--- guix/scripts/publish.scm | 136 +++++++++++++++++++++++---------------- tests/publish.scm | 88 +++---------------------- 4 files changed, 93 insertions(+), 157 deletions(-) -- 2.32.0
Mathieu Othacehe <othacehe@HIDDEN>:guix-patches@HIDDEN.
Full text available.guix-patches@HIDDEN:bug#50040; Package guix-patches.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.