GNU logs - #48468, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 16 May 2021 17:58:02 +0000
Resent-Message-ID: <handler.48468.B.16211878767959 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 48468 <at> debbugs.gnu.org
From: Mathieu Othacehe <othacehe@HIDDEN>
Date: Sun, 16 May 2021 19:57:49 +0200
Message-ID: <87lf8e4l42.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>


Hello,

We recently have a lot of those errors on Cuirass:

--8<---------------cut here---------------start------------->8---
guix substitute: warning: while fetching http://141.80.167.131:5557/nar/g7ka09613k5v1vlznh87yg35905ggw51-python2-scipy-1.2.2-guile-builder: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: connect*: Connection timed out
--8<---------------cut here---------------end--------------->8---

which means that the workers are failing to connect to the Cuirass
remote-server publish server on berlin at 141.80.167.131:5557.

Stracing this publish server shows that connection reuse seems to be
broken:

--8<---------------cut here---------------start------------->8---
accept4(9, {sa_family=AF_INET, sin_port=htons(41742), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41744), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41746), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
accept4(9, {sa_family=AF_INET, sin_port=htons(41748), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
accept4(9, {sa_family=AF_INET, sin_port=htons(41750), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41752), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41754), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
accept4(9, {sa_family=AF_INET, sin_port=htons(41756), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41758), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 26
accept4(9, {sa_family=AF_INET, sin_port=htons(41760), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
accept4(9, {sa_family=AF_INET, sin_port=htons(41762), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41764), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41766), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41768), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 22
accept4(9, {sa_family=AF_INET, sin_port=htons(41770), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41772), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41774), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41776), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41778), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41780), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
accept4(9, {sa_family=AF_INET, sin_port=htons(41782), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
--8<---------------cut here---------------end--------------->8---

Investigating it, I found that the connection is closed and opened
multiple times in the call-with-cached-connection procedure of the (guix
script substitute) module.

It looks like its because a 'bad-headers exception is raised when trying
to parse an eof object:

--8<---------------cut here---------------start------------->8---
;;; (error bad-header (read-header-line #<eof>))
--8<---------------cut here---------------end--------------->8---

I'm not sure where this eof comes from. There is this comment in the
http-multiple-get procedure in (guix http-client):

--8<---------------cut here---------------start------------->8---
;; Swallow networking errors that could occur due to connection reuse
;; and the like; they will be handled down the road when trying to
;; read responses.
(false-if-networking-error
 (begin
   (for-each (cut write-request <> buffer) batch)
   (put-bytevector p (get))
   (force-output p))))
--8<---------------cut here---------------end--------------->8---

which would suggest that connection reuse could cause networking errors?

What also puzzles me it that the main guix publish server on berlin does
not seem to present this issue. That would indicate that this error is
caused by how the Cuirass remote-server publish server is started or
configured.

Ludo, Chris, any idea?

I will keep searching anyway :)

Thanks,

Mathieu

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Mathieu Othacehe <othacehe@HIDDEN>
Subject: bug#48468: Acknowledgement (substitute server connection timeout)
Message-ID: <handler.48468.B.16211878767959.ack <at> debbugs.gnu.org>
References: <87lf8e4l42.fsf@HIDDEN>
X-Gnu-PR-Message: ack 48468
X-Gnu-PR-Package: guix
Reply-To: 48468 <at> debbugs.gnu.org
Date: Sun, 16 May 2021 17:58:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 48468 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
48468: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D48468
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#48468: substitute server connection timeout
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 16 May 2021 18:27:01 +0000
Resent-Message-ID: <handler.48468.B.162118959210739 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
References: <87lf8e4l42.fsf@HIDDEN>
User-agent: mu4e 1.4.15; emacs 27.1
From: Christopher Baines <mail@HIDDEN>
In-reply-to: <87lf8e4l42.fsf@HIDDEN>
Date: Sun, 16 May 2021 19:26:22 +0100
Message-ID: <87eee6cz75.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@HIDDEN;
 helo=mira.cbaines.net
X-Spam_score_int: -6
X-Spam_score: -0.7
X-Spam_bar: /
X-Spam_report: (-0.7 / 5.0 requ) BAYES_00=-1.9, NORMAL_HTTP_TO_IP=0.001,
 NUMERIC_HTTP_ADDR=1.242, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001,
 WEIRD_PORT=0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

--=-=-=
Content-Type: text/plain


Mathieu Othacehe <othacehe@HIDDEN> writes:

> Hello,
>
> We recently have a lot of those errors on Cuirass:
>
> --8<---------------cut here---------------start------------->8---
> guix substitute: warning: while fetching http://141.80.167.131:5557/nar/g7ka09613k5v1vlznh87yg35905ggw51-python2-scipy-1.2.2-guile-builder: server is somewhat slow
> guix substitute: warning: try `--no-substitutes' if the problem persists
> guix substitute: error: connect*: Connection timed out
> --8<---------------cut here---------------end--------------->8---
>
> which means that the workers are failing to connect to the Cuirass
> remote-server publish server on berlin at 141.80.167.131:5557.
>
> Stracing this publish server shows that connection reuse seems to be
> broken:
>
> --8<---------------cut here---------------start------------->8---
> accept4(9, {sa_family=AF_INET, sin_port=htons(41742), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41744), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41746), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
> accept4(9, {sa_family=AF_INET, sin_port=htons(41748), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
> accept4(9, {sa_family=AF_INET, sin_port=htons(41750), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41752), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41754), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
> accept4(9, {sa_family=AF_INET, sin_port=htons(41756), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41758), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 26
> accept4(9, {sa_family=AF_INET, sin_port=htons(41760), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
> accept4(9, {sa_family=AF_INET, sin_port=htons(41762), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41764), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41766), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41768), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 22
> accept4(9, {sa_family=AF_INET, sin_port=htons(41770), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41772), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41774), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41776), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41778), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41780), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41782), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> --8<---------------cut here---------------end--------------->8---
>
> Investigating it, I found that the connection is closed and opened
> multiple times in the call-with-cached-connection procedure of the (guix
> script substitute) module.
>
> It looks like its because a 'bad-headers exception is raised when trying
> to parse an eof object:
>
> --8<---------------cut here---------------start------------->8---
> ;;; (error bad-header (read-header-line #<eof>))
> --8<---------------cut here---------------end--------------->8---
>
> I'm not sure where this eof comes from. There is this comment in the
> http-multiple-get procedure in (guix http-client):
>
> --8<---------------cut here---------------start------------->8---
> ;; Swallow networking errors that could occur due to connection reuse
> ;; and the like; they will be handled down the road when trying to
> ;; read responses.
> (false-if-networking-error
>  (begin
>    (for-each (cut write-request <> buffer) batch)
>    (put-bytevector p (get))
>    (force-output p))))
> --8<---------------cut here---------------end--------------->8---
>
> which would suggest that connection reuse could cause networking errors?
>
> What also puzzles me it that the main guix publish server on berlin does
> not seem to present this issue. That would indicate that this error is
> caused by how the Cuirass remote-server publish server is started or
> configured.
>
> Ludo, Chris, any idea?

While I've been working in this area, I've actually been trying to pick
apart the connection caching, since the single thread assumption doesn't
hold in the Guix Build Coordinator.

Anyway, I do have a theory. Assuming I'm correct in saying that there's
no nginx between the client and publish server here, I think that's your
configuration difference.

For ci.guix.gnu.org, as well as data.guix.gnu.org, it's NGinx which is
keeping connections alive. I'm not sure the Guile code for the publish
server does similarly, so talking to it directly might be different.

That's on the server side, the actual problem is probably on the client
side, as I guess there are possibly places where closed connections
aren't handled properly. This reminds me I sent some patches relating to
closing connections, this could well be related [1].

1: https://issues.guix.gnu.org/47174

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmChY89fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfG9w/+KvzCHwfnB+eeEd6i0xTvTfc8+oLxb+GS
j1TSbXSGR02ByGMGIbJeAMmNPV20V9Lzi24a49451fskUZbCGeg6aS2rO3c9QpUX
jbF7//NW8zwgmhO/Hh9dvB4v+q01+VwrzE3c6ssAVO6772ZrTJqGA7oCU9UqD1w6
dQYUto8iSxa95UhFcNFMtGcBAevdoN4Ku9OpkqIt8uViCED+YM6a9Ej4d9f9oC8h
vxw6EU0YsCrobJQJ8KW56N9uR4oeI4IqWevou2LbSAmcHEWmGHAF6XDgxu0594Ip
YOFuqso4nuRxU3ffzgEupMbJA07yrwf0EB4lreqsSfU8gtZupFMLgyc8PviyXjw3
XPpo3PAukKhsVZyQlnTc6ldatWWVHIeTPy8vxdr6GvSQnzliaQmzpHj7xVqkfS3f
R1gz5hEXoS0o/CXpVkpnaYMC8m+Y2ujM5AbEUBVo6itR6mmUCCF5D9q/BtosjQvu
8wrLes42iAqwlnNeg8UGKra/cwJ0YGVrOInsOyXX97jz3UVjkxV7jq53+vkw3fGZ
eyHDugAJ8DKw9js8BX88AmXaV/Art4M0qG6D6Zp+MvIUbU9z+oNeTjAXgGewxntw
sfIdxHju+DNs5x5je3pTzyRSDPEfle8Xvd58DbGyAIoP8v7C7LYPmxVID7R0D/cs
ccIQKCj/kqg=
=yVQQ
-----END PGP SIGNATURE-----
--=-=-=--

Message sent to bug-guix@HIDDEN:

Subject: bug#48468: substitute server connection timeout
Resent-From: Christopher Baines <mail@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 16 May 2021 18:27:02 +0000
Resent-Message-ID: <handler.48468.B48468.162118959010724 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
References: <87lf8e4l42.fsf@HIDDEN>
User-agent: mu4e 1.4.15; emacs 27.1
From: Christopher Baines <mail@HIDDEN>
In-reply-to: <87lf8e4l42.fsf@HIDDEN>
Date: Sun, 16 May 2021 19:26:22 +0100
Message-ID: <87eee6cz75.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain


Mathieu Othacehe <othacehe@HIDDEN> writes:

> Hello,
>
> We recently have a lot of those errors on Cuirass:
>
> --8<---------------cut here---------------start------------->8---
> guix substitute: warning: while fetching http://141.80.167.131:5557/nar/g7ka09613k5v1vlznh87yg35905ggw51-python2-scipy-1.2.2-guile-builder: server is somewhat slow
> guix substitute: warning: try `--no-substitutes' if the problem persists
> guix substitute: error: connect*: Connection timed out
> --8<---------------cut here---------------end--------------->8---
>
> which means that the workers are failing to connect to the Cuirass
> remote-server publish server on berlin at 141.80.167.131:5557.
>
> Stracing this publish server shows that connection reuse seems to be
> broken:
>
> --8<---------------cut here---------------start------------->8---
> accept4(9, {sa_family=AF_INET, sin_port=htons(41742), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41744), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41746), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
> accept4(9, {sa_family=AF_INET, sin_port=htons(41748), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
> accept4(9, {sa_family=AF_INET, sin_port=htons(41750), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41752), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41754), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 25
> accept4(9, {sa_family=AF_INET, sin_port=htons(41756), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41758), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 26
> accept4(9, {sa_family=AF_INET, sin_port=htons(41760), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 24
> accept4(9, {sa_family=AF_INET, sin_port=htons(41762), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41764), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41766), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41768), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 22
> accept4(9, {sa_family=AF_INET, sin_port=htons(41770), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41772), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41774), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41776), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41778), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41780), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> accept4(9, {sa_family=AF_INET, sin_port=htons(41782), sin_addr=inet_addr("141.80.167.185")}, [112->16], 0) = 21
> --8<---------------cut here---------------end--------------->8---
>
> Investigating it, I found that the connection is closed and opened
> multiple times in the call-with-cached-connection procedure of the (guix
> script substitute) module.
>
> It looks like its because a 'bad-headers exception is raised when trying
> to parse an eof object:
>
> --8<---------------cut here---------------start------------->8---
> ;;; (error bad-header (read-header-line #<eof>))
> --8<---------------cut here---------------end--------------->8---
>
> I'm not sure where this eof comes from. There is this comment in the
> http-multiple-get procedure in (guix http-client):
>
> --8<---------------cut here---------------start------------->8---
> ;; Swallow networking errors that could occur due to connection reuse
> ;; and the like; they will be handled down the road when trying to
> ;; read responses.
> (false-if-networking-error
>  (begin
>    (for-each (cut write-request <> buffer) batch)
>    (put-bytevector p (get))
>    (force-output p))))
> --8<---------------cut here---------------end--------------->8---
>
> which would suggest that connection reuse could cause networking errors?
>
> What also puzzles me it that the main guix publish server on berlin does
> not seem to present this issue. That would indicate that this error is
> caused by how the Cuirass remote-server publish server is started or
> configured.
>
> Ludo, Chris, any idea?

While I've been working in this area, I've actually been trying to pick
apart the connection caching, since the single thread assumption doesn't
hold in the Guix Build Coordinator.

Anyway, I do have a theory. Assuming I'm correct in saying that there's
no nginx between the client and publish server here, I think that's your
configuration difference.

For ci.guix.gnu.org, as well as data.guix.gnu.org, it's NGinx which is
keeping connections alive. I'm not sure the Guile code for the publish
server does similarly, so talking to it directly might be different.

That's on the server side, the actual problem is probably on the client
side, as I guess there are possibly places where closed connections
aren't handled properly. This reminds me I sent some patches relating to
closing connections, this could well be related [1].

1: https://issues.guix.gnu.org/47174

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmChY89fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfG9w/+KvzCHwfnB+eeEd6i0xTvTfc8+oLxb+GS
j1TSbXSGR02ByGMGIbJeAMmNPV20V9Lzi24a49451fskUZbCGeg6aS2rO3c9QpUX
jbF7//NW8zwgmhO/Hh9dvB4v+q01+VwrzE3c6ssAVO6772ZrTJqGA7oCU9UqD1w6
dQYUto8iSxa95UhFcNFMtGcBAevdoN4Ku9OpkqIt8uViCED+YM6a9Ej4d9f9oC8h
vxw6EU0YsCrobJQJ8KW56N9uR4oeI4IqWevou2LbSAmcHEWmGHAF6XDgxu0594Ip
YOFuqso4nuRxU3ffzgEupMbJA07yrwf0EB4lreqsSfU8gtZupFMLgyc8PviyXjw3
XPpo3PAukKhsVZyQlnTc6ldatWWVHIeTPy8vxdr6GvSQnzliaQmzpHj7xVqkfS3f
R1gz5hEXoS0o/CXpVkpnaYMC8m+Y2ujM5AbEUBVo6itR6mmUCCF5D9q/BtosjQvu
8wrLes42iAqwlnNeg8UGKra/cwJ0YGVrOInsOyXX97jz3UVjkxV7jq53+vkw3fGZ
eyHDugAJ8DKw9js8BX88AmXaV/Art4M0qG6D6Zp+MvIUbU9z+oNeTjAXgGewxntw
sfIdxHju+DNs5x5je3pTzyRSDPEfle8Xvd58DbGyAIoP8v7C7LYPmxVID7R0D/cs
ccIQKCj/kqg=
=yVQQ
-----END PGP SIGNATURE-----
--=-=-=--

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 17 May 2021 14:50:01 +0000
Resent-Message-ID: <handler.48468.B.162126299931033 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: bug-guix@HIDDEN, 48468 <at> debbugs.gnu.org
Received: via spool by submit <at> debbugs.gnu.org id=B.162126299931033
          (code B ref -1); Mon, 17 May 2021 14:50:01 +0000
Received: (at submit) by debbugs.gnu.org; 17 May 2021 14:49:59 +0000
Received: from localhost ([127.0.0.1]:53395 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lieZD-00084T-38
	for submit <at> debbugs.gnu.org; Mon, 17 May 2021 10:49:59 -0400
Received: from lists.gnu.org ([209.51.188.17]:36806)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lieZ9-00084K-Q9
 for submit <at> debbugs.gnu.org; Mon, 17 May 2021 10:49:57 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34242)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1lieZ9-0007kn-Dt
 for bug-guix@HIDDEN; Mon, 17 May 2021 10:49:55 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45226)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lieZ8-0003sN-6g; Mon, 17 May 2021 10:49:55 -0400
Received: from [109.190.253.11] (port=55430 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1lieZ7-0002KA-Mq; Mon, 17 May 2021 10:49:54 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
Date: Mon, 17 May 2021 16:49:49 +0200
In-Reply-To: <87eee6cz75.fsf@HIDDEN> (Christopher Baines's message of
 "Sun, 16 May 2021 19:26:22 +0100")
Message-ID: <87eee59zzm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> That's on the server side, the actual problem is probably on the client
> side, as I guess there are possibly places where closed connections
> aren't handled properly. This reminds me I sent some patches relating to
> closing connections, this could well be related [1].

Oh, you're right, the Nginx server probably makes the difference
here. The http-write procedure of the Guile (web server http) module
seems to handle keep-alive connections. However, the wrapping http-write
procedure of (guix scripts publish) may not.

I'll have a closer look, thanks for your help.

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 17 May 2021 14:51:02 +0000
Resent-Message-ID: <handler.48468.B48468.162126301031088 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: bug-guix@HIDDEN, 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.162126301031088
          (code B ref 48468); Mon, 17 May 2021 14:51:02 +0000
Received: (at 48468) by debbugs.gnu.org; 17 May 2021 14:50:10 +0000
Received: from localhost ([127.0.0.1]:53399 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lieZK-00085F-9k
	for submit <at> debbugs.gnu.org; Mon, 17 May 2021 10:50:10 -0400
Received: from eggs.gnu.org ([209.51.188.92]:40592)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lieZE-00084J-FE
 for 48468 <at> debbugs.gnu.org; Mon, 17 May 2021 10:50:04 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45226)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lieZ8-0003sN-6g; Mon, 17 May 2021 10:49:55 -0400
Received: from [109.190.253.11] (port=55430 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1lieZ7-0002KA-Mq; Mon, 17 May 2021 10:49:54 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
Date: Mon, 17 May 2021 16:49:49 +0200
In-Reply-To: <87eee6cz75.fsf@HIDDEN> (Christopher Baines's message of
 "Sun, 16 May 2021 19:26:22 +0100")
Message-ID: <87eee59zzm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> That's on the server side, the actual problem is probably on the client
> side, as I guess there are possibly places where closed connections
> aren't handled properly. This reminds me I sent some patches relating to
> closing connections, this could well be related [1].

Oh, you're right, the Nginx server probably makes the difference
here. The http-write procedure of the Guile (web server http) module
seems to handle keep-alive connections. However, the wrapping http-write
procedure of (guix scripts publish) may not.

I'll have a closer look, thanks for your help.

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 18 May 2021 14:36:01 +0000
Resent-Message-ID: <handler.48468.B.162134853029824 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: ludo@HIDDEN, 48468 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, bug-guix@HIDDEN, 48468 <at> debbugs.gnu.org
Received: via spool by submit <at> debbugs.gnu.org id=B.162134853029824
          (code B ref -1); Tue, 18 May 2021 14:36:01 +0000
Received: (at submit) by debbugs.gnu.org; 18 May 2021 14:35:30 +0000
Received: from localhost ([127.0.0.1]:56844 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lj0ok-0007kx-5f
	for submit <at> debbugs.gnu.org; Tue, 18 May 2021 10:35:30 -0400
Received: from lists.gnu.org ([209.51.188.17]:35134)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lj0og-0007km-Ml
 for submit <at> debbugs.gnu.org; Tue, 18 May 2021 10:35:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:51018)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1lj0og-0003Ay-F8
 for bug-guix@HIDDEN; Tue, 18 May 2021 10:35:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37814)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lj0of-0005tX-Go; Tue, 18 May 2021 10:35:25 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=34756 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lj0oe-0002Rn-CL; Tue, 18 May 2021 10:35:24 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
 <87eee59zzm.fsf@HIDDEN>
Date: Tue, 18 May 2021 16:35:17 +0200
In-Reply-To: <87eee59zzm.fsf@HIDDEN> (Mathieu Othacehe's message of "Mon, 17
 May 2021 16:49:49 +0200")
Message-ID: <878s4c85zu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> I'll have a closer look, thanks for your help.

So this snippet in the http-write procedure of the (guix scripts
publish) module:

--8<---------------cut here---------------start------------->8---
          (swallow-zlib-error
           (close-port port))
--8<---------------cut here---------------end--------------->8---

is closing the client port unconditionally, which means that guix
publish cannot keep connections alive, unless sitting behind an Nginx
proxy.

I'm trying to turn the close-port call into a maybe-close-port with the
following procedure:

--8<---------------cut here---------------start------------->8---
(define (maybe-close-port port)
    (cond
     ((keep-alive? response)
      (poll-set-add! (http-poll-set server) port *events*))
     (else
      (close-port port))))
--8<---------------cut here---------------end--------------->8---

however this is terribly hacky, as I need to access the private poll-set
from (web server http).

Ludo, do you have a better idea?

Thanks,

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 18 May 2021 14:36:02 +0000
Resent-Message-ID: <handler.48468.B48468.162134853429839 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: ludo@HIDDEN, 48468 <at> debbugs.gnu.org
X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, bug-guix@HIDDEN, 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.162134853429839
          (code B ref 48468); Tue, 18 May 2021 14:36:02 +0000
Received: (at 48468) by debbugs.gnu.org; 18 May 2021 14:35:34 +0000
Received: from localhost ([127.0.0.1]:56847 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lj0oo-0007lD-C0
	for submit <at> debbugs.gnu.org; Tue, 18 May 2021 10:35:34 -0400
Received: from eggs.gnu.org ([209.51.188.92]:57368)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lj0ol-0007kl-Bf
 for 48468 <at> debbugs.gnu.org; Tue, 18 May 2021 10:35:31 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37814)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lj0of-0005tX-Go; Tue, 18 May 2021 10:35:25 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=34756 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lj0oe-0002Rn-CL; Tue, 18 May 2021 10:35:24 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
 <87eee59zzm.fsf@HIDDEN>
Date: Tue, 18 May 2021 16:35:17 +0200
In-Reply-To: <87eee59zzm.fsf@HIDDEN> (Mathieu Othacehe's message of "Mon, 17
 May 2021 16:49:49 +0200")
Message-ID: <878s4c85zu.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> I'll have a closer look, thanks for your help.

So this snippet in the http-write procedure of the (guix scripts
publish) module:

--8<---------------cut here---------------start------------->8---
          (swallow-zlib-error
           (close-port port))
--8<---------------cut here---------------end--------------->8---

is closing the client port unconditionally, which means that guix
publish cannot keep connections alive, unless sitting behind an Nginx
proxy.

I'm trying to turn the close-port call into a maybe-close-port with the
following procedure:

--8<---------------cut here---------------start------------->8---
(define (maybe-close-port port)
    (cond
     ((keep-alive? response)
      (poll-set-add! (http-poll-set server) port *events*))
     (else
      (close-port port))))
--8<---------------cut here---------------end--------------->8---

however this is terribly hacky, as I need to access the private poll-set
from (web server http).

Ludo, do you have a better idea?

Thanks,

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 21 May 2021 13:31:02 +0000
Resent-Message-ID: <handler.48468.B48468.16216038316691 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: ludo@HIDDEN, 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16216038316691
          (code B ref 48468); Fri, 21 May 2021 13:31:02 +0000
Received: (at 48468) by debbugs.gnu.org; 21 May 2021 13:30:31 +0000
Received: from localhost ([127.0.0.1]:36065 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lk5EU-0001jq-Jk
	for submit <at> debbugs.gnu.org; Fri, 21 May 2021 09:30:30 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38876)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lk5EM-0001jV-EN
 for 48468 <at> debbugs.gnu.org; Fri, 21 May 2021 09:30:28 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:43390)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lk5EH-00057D-04; Fri, 21 May 2021 09:30:17 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=50402 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lk5EG-0003ny-MQ; Fri, 21 May 2021 09:30:16 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
 <87eee59zzm.fsf@HIDDEN> <878s4c85zu.fsf@HIDDEN>
Date: Fri, 21 May 2021 15:30:15 +0200
In-Reply-To: <878s4c85zu.fsf@HIDDEN> (Mathieu Othacehe's message of "Tue, 18
 May 2021 16:35:17 +0200")
Message-ID: <87cztkqkns.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

I posted a patchset adding keep-alive support to guix publish earlier:
https://issues.guix.gnu.org/48556.

Thanks,

Mathieu

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 29 May 2021 21:04:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat May 29 17:04:55 2021
Received: from localhost ([127.0.0.1]:57815 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ln68c-0001Ai-Vh
	for submit <at> debbugs.gnu.org; Sat, 29 May 2021 17:04:55 -0400
Received: from eggs.gnu.org ([209.51.188.92]:35536)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1ln68b-0001AV-Ps
 for control <at> debbugs.gnu.org; Sat, 29 May 2021 17:04:54 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:54224)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>) id 1ln68W-0001sw-LS
 for control <at> debbugs.gnu.org; Sat, 29 May 2021 17:04:48 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41512 helo=ribbon)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>) id 1ln68W-0005ub-DB
 for control <at> debbugs.gnu.org; Sat, 29 May 2021 17:04:48 -0400
Date: Sat, 29 May 2021 23:04:47 +0200
Message-Id: <877djh5k0g.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Subject: control message for bug #48468
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

severity 48468 important
quit

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 29 May 2021 21:45:02 +0000
Resent-Message-ID: <handler.48468.B48468.162232468316650 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Christopher Baines <mail@HIDDEN>
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.162232468316650
          (code B ref 48468); Sat, 29 May 2021 21:45:02 +0000
Received: (at 48468) by debbugs.gnu.org; 29 May 2021 21:44:43 +0000
Received: from localhost ([127.0.0.1]:57852 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ln6l8-0004KU-TX
	for submit <at> debbugs.gnu.org; Sat, 29 May 2021 17:44:43 -0400
Received: from eggs.gnu.org ([209.51.188.92]:40360)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1ln6l8-0004KI-BF
 for 48468 <at> debbugs.gnu.org; Sat, 29 May 2021 17:44:42 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:55892)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1ln6l3-0001I6-1J; Sat, 29 May 2021 17:44:37 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41666 helo=ribbon)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1ln6l2-0003Ey-Nx; Sat, 29 May 2021 17:44:36 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87eee6cz75.fsf@HIDDEN>
Date: Sat, 29 May 2021 23:44:35 +0200
In-Reply-To: <87eee6cz75.fsf@HIDDEN> (Christopher Baines's message of
 "Sun, 16 May 2021 19:26:22 +0100")
Message-ID: <87v97143lo.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Christopher Baines <mail@HIDDEN> skribis:

> Mathieu Othacehe <othacehe@HIDDEN> writes:

[...]

>> Stracing this publish server shows that connection reuse seems to be
>> broken:
>>
>> --8<---------------cut here---------------start------------->8---
>> accept4(9, {sa_family=3DAF_INET, sin_port=3Dhtons(41742), sin_addr=3Dine=
t_addr("141.80.167.185")}, [112->16], 0) =3D 21
>> accept4(9, {sa_family=3DAF_INET, sin_port=3Dhtons(41744), sin_addr=3Dine=
t_addr("141.80.167.185")}, [112->16], 0) =3D 21
>> accept4(9, {sa_family=3DAF_INET, sin_port=3Dhtons(41746), sin_addr=3Dine=
t_addr("141.80.167.185")}, [112->16], 0) =3D 25
>> accept4(9, {sa_family=3DAF_INET, sin_port=3Dhtons(41748), sin_addr=3Dine=
t_addr("141.80.167.185")}, [112->16], 0) =3D 24
>> accept4(9, {sa_family=3DAF_INET, sin_port=3Dhtons(41750), sin_addr=3Dine=
t_addr("141.80.167.185")}, [112->16], 0) =3D 21

Ouch.

>> Investigating it, I found that the connection is closed and opened
>> multiple times in the call-with-cached-connection procedure of the (guix
>> script substitute) module.
>>
>> It looks like its because a 'bad-headers exception is raised when trying
>> to parse an eof object:
>>
>> --8<---------------cut here---------------start------------->8---
>> ;;; (error bad-header (read-header-line #<eof>))
>> --8<---------------cut here---------------end--------------->8---
>>
>> I'm not sure where this eof comes from. There is this comment in the
>> http-multiple-get procedure in (guix http-client):
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; Swallow networking errors that could occur due to connection reuse
>> ;; and the like; they will be handled down the road when trying to
>> ;; read responses.
>> (false-if-networking-error
>>  (begin
>>    (for-each (cut write-request <> buffer) batch)
>>    (put-bytevector p (get))
>>    (force-output p))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> which would suggest that connection reuse could cause networking errors?

[...]

> That's on the server side, the actual problem is probably on the client
> side, as I guess there are possibly places where closed connections
> aren't handled properly.

D=E2=80=99oh.  So we should be able to use it by talking to a bare =E2=80=
=98guix
publish=E2=80=99?  I tried and failed to reproduce it, but that=E2=80=99s p=
robably not
deterministic.

> This reminds me I sent some patches relating to closing connections,
> this could well be related [1].
>
> 1: https://issues.guix.gnu.org/47174

This one will properly close connections when the servers asks for it,
but would that make a difference?

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 18 Jun 2021 12:34:02 +0000
Resent-Message-ID: <handler.48468.B48468.162401961111909 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.162401961111909
          (code B ref 48468); Fri, 18 Jun 2021 12:34:02 +0000
Received: (at 48468) by debbugs.gnu.org; 18 Jun 2021 12:33:31 +0000
Received: from localhost ([127.0.0.1]:56007 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1luDgh-000360-EP
	for submit <at> debbugs.gnu.org; Fri, 18 Jun 2021 08:33:31 -0400
Received: from eggs.gnu.org ([209.51.188.92]:58296)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1luDgf-00035o-Ah
 for 48468 <at> debbugs.gnu.org; Fri, 18 Jun 2021 08:33:29 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45758)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>) id 1luDga-0002no-4Q
 for 48468 <at> debbugs.gnu.org; Fri, 18 Jun 2021 08:33:24 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54544 helo=ribbon)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <ludo@HIDDEN>)
 id 1luDgZ-0004v0-0B; Fri, 18 Jun 2021 08:33:23 -0400
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN>
Date: Fri, 18 Jun 2021 14:33:21 +0200
In-Reply-To: <87lf8e4l42.fsf@HIDDEN> (Mathieu Othacehe's message of "Sun, 16
 May 2021 19:57:49 +0200")
Message-ID: <87im2bfj3i.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Mathieu Othacehe <othacehe@HIDDEN> skribis:

> Investigating it, I found that the connection is closed and opened
> multiple times in the call-with-cached-connection procedure of the (guix
> script substitute) module.
>
> It looks like its because a 'bad-headers exception is raised when trying
> to parse an eof object:
>
> ;;; (error bad-header (read-header-line #<eof>))
>
>
> I'm not sure where this eof comes from. There is this comment in the
> http-multiple-get procedure in (guix http-client):
>
> ;; Swallow networking errors that could occur due to connection reuse
> ;; and the like; they will be handled down the road when trying to
> ;; read responses.
> (false-if-networking-error
>  (begin
>    (for-each (cut write-request <> buffer) batch)
>    (put-bytevector p (get))
>    (force-output p))))
>
> which would suggest that connection reuse could cause networking errors?

Trying to see exactly which bit is at fault here.  First,
=E2=80=98http-multiple-get=E2=80=99 seems to be working as expected when pa=
ssed a fresh
connection:

--8<---------------cut here---------------start------------->8---
scheme@(guix http-client)> (http-multiple-get (string->uri "https://ci.guix=
.gnu.org")
					      (lambda (request response port result)
						(get-bytevector-n port (response-content-length response))
						(cons #t result))
					      '()
					      (make-list
					       5000
					       (build-request (build-uri 'https
										   #:host "ci.guix.gnu.org"
										   #:path "/d7gpr41qpsifri54vp2lzjs99zkylscq.narinfo"))))
connecting (5000 requests left)...connecting (4400 requests left)...connect=
ing (3800 requests left)...connecting (3200 requests left)...connecting (26=
00 requests left)...connecting (2000 requests left)...connecting (1400 requ=
ests left)...connecting (800 requests left)...connecting (200 requests left=
)...$143 =3D (#t #t =E2=80=A6)
scheme@(guix http-client)> (length $143)
$144 =3D 5000
--8<---------------cut here---------------end--------------->8---

But maybe I=E2=80=99m not looking at the right thing.

Do you have evidence or a reproducer?

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Tue, 29 Jun 2021 16:50:01 +0000
Resent-Message-ID: <handler.48468.B48468.16249853777979 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16249853777979
          (code B ref 48468); Tue, 29 Jun 2021 16:50:01 +0000
Received: (at 48468) by debbugs.gnu.org; 29 Jun 2021 16:49:37 +0000
Received: from localhost ([127.0.0.1]:55196 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyGvY-00024d-RT
	for submit <at> debbugs.gnu.org; Tue, 29 Jun 2021 12:49:37 -0400
Received: from eggs.gnu.org ([209.51.188.92]:34882)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1lyGvY-00024S-3x
 for 48468 <at> debbugs.gnu.org; Tue, 29 Jun 2021 12:49:36 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34964)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lyGvQ-0001sO-LC; Tue, 29 Jun 2021 12:49:29 -0400
Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=51100 helo=meije)
 by fencepost.gnu.org with esmtpsa
 (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1)
 (envelope-from <othacehe@HIDDEN>)
 id 1lyGvQ-0002tg-DC; Tue, 29 Jun 2021 12:49:28 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
Date: Tue, 29 Jun 2021 18:49:26 +0200
In-Reply-To: <87im2bfj3i.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 18 Jun 2021 14:33:21 +0200")
Message-ID: <87sg108vl5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hey,

> But maybe I=E2=80=99m not looking at the right thing.
>
> Do you have evidence or a reproducer?

Yes, adding the following debug message here:

--8<---------------cut here---------------start------------->8---
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -416,6 +416,7 @@ server certificates."
         ;; and retry.  We might also get 'bad-response or a similar
         ;; exception from (web response) later on, once we've sent the
         ;; request, or a ERROR/INVALID-SESSION from GnuTLS.
+        (pk key args)
         (if (or (and (eq? key 'system-error)
                      (=3D EPIPE (system-error-errno `(,key ,@args))))
                 (and (eq? key 'gnutls-error)
--8<---------------cut here---------------end--------------->8---

and using substitutes from a publish server without keep alive support
(before 0b8fa24), prints something like:

--8<---------------cut here---------------start------------->8---
downloading from http://192.168.1.51:8080/nar/gzip/lkmiyfzi1bq571yn4gypbcs5=
vn4fpma7-texlive-cm-51265 ...
 texlive-cm-51265                                                          =
                                                                           =
                                                                           =
                      7.8MiB/s 00:00 | 1.7MiB transferred

substituting /gnu/store/l788x07ska5vffayz0gayv4hsx5flxal-module-import-comp=
iled...

;;; (bad-header (read-header-line #<eof>))
downloading from http://192.168.1.51:8080/nar/gzip/l788x07ska5vffayz0gayv4h=
sx5flxal-module-import-compiled ...
 module-import-compiled                                                    =
                                                                           =
                                                                           =
                       7.5MiB/s 00:00 | 85KiB transferred

substituting /gnu/store/1s1lrnxlkjwxshk5q2w97ig3clc6n06f-ruby-2.6.5...

;;; (bad-header (read-header-line #<eof>))
--8<---------------cut here---------------end--------------->8---

That's because the connection is closed by the publish server each time
a NAR is sent. This particular behaviour is fixed by:
https://issues.guix.gnu.org/48556.

I hoped that it would decrease the load of the Cuirass publish server
that was flooded by worker connections. While the situation is better,
there are still a lot of substitute timeout errors on Cuirass.

I have also observed those timeouts using the publish server behind
ci.guix.gnu.org. This makes me think that there's something else that is
preventing the server from honoring every connection request in less
than 5 seconds (the timeout duration).

Thanks,

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 05 Dec 2022 13:22:02 +0000
Resent-Message-ID: <handler.48468.B48468.16702464963175 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16702464963175
          (code B ref 48468); Mon, 05 Dec 2022 13:22:02 +0000
Received: (at 48468) by debbugs.gnu.org; 5 Dec 2022 13:21:36 +0000
Received: from localhost ([127.0.0.1]:35994 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p2BPc-0000p9-4z
	for submit <at> debbugs.gnu.org; Mon, 05 Dec 2022 08:21:36 -0500
Received: from eggs.gnu.org ([209.51.188.92]:42810)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1p2BPa-0000p3-M6
 for 48468 <at> debbugs.gnu.org; Mon, 05 Dec 2022 08:21:35 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1p2BPS-0007KC-PN
 for 48468 <at> debbugs.gnu.org; Mon, 05 Dec 2022 08:21:29 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=QdgzazPdBaky6oWPFshw7JxEXLHN3x9Z1jEgar2OWp4=; b=Tq3PRLGnbvwXUXkn+Lk9
 B5HPonNF5S2wASbMfDEfvPQUEYjRYcGs/Vb2UiTKVQVErsa4yeKa947UwMyAdKHYTaU6aUFRUjv8q
 3lc1SYUgsZhpII9RmacW81iqvxh/3CQB0XrS1mipklFkd2P8v0hizfjcnhVjq7J/VzQ5cx4ID5Suq
 Pl+Ot0mHbAlFJ+4M/gHEWdW5TJpQtA4um4Gdow3HWzUzaiarW+PWfQ5ZH78ews2btAbkmYlIAhH1U
 mGgOT5B/0W2wFCaWAdlhv3XEq8uT/Ap1CARcP/lUF6LgrC939OUpe5z7+qWhAouyjr8CK3YnwJC6k
 HZWJ+mNW5gO9sw==;
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1p2BPR-0006qF-UV; Mon, 05 Dec 2022 08:21:26 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN>
Date: Mon, 05 Dec 2022 14:21:23 +0100
In-Reply-To: <87sg108vl5.fsf@HIDDEN> (Mathieu Othacehe's message of "Tue, 29
 Jun 2021 18:49:26 +0200")
Message-ID: <87lenmrnos.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Mathieu Othacehe <othacehe@HIDDEN> skribis:

> I hoped that it would decrease the load of the Cuirass publish server
> that was flooded by worker connections. While the situation is better,
> there are still a lot of substitute timeout errors on Cuirass.
>
> I have also observed those timeouts using the publish server behind
> ci.guix.gnu.org. This makes me think that there's something else that is
> preventing the server from honoring every connection request in less
> than 5 seconds (the timeout duration).

I don=E2=80=99t see this when substituting from https://ci.guix.gnu.org the=
se
days.

You mentioned on IRC that nginx logs show that =E2=80=98guix publish=E2=80=
=99 times out.
Looking at /var/log/nginx/error.log, I see =E2=80=9CConnection reset by pee=
r=E2=80=9D
and =E2=80=9CBroken pipe=E2=80=9D, which could indicate that the client clo=
sed the
connection (which was open) prematurely, maybe due to an internal
timeout.

What I=E2=80=99d like to know is whether those timeouts you mention appear
during connection establishment (connect(2) on the client side doesn=E2=80=
=99t
complete within 5 seconds) or after connection establishment.

Connection establishment is handled by the kernel and =E2=80=98guix publish=
=E2=80=99 is
not involved.  However, the listen(2) call made in (web server http)
sets a maximum backlog of 128 connections; if at some point 128
connections are already queued, then the 129th client will have
connect(2) take some time.  Seems unlikely, but who knows.

Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 07 Dec 2022 09:45:01 +0000
Resent-Message-ID: <handler.48468.B48468.16704062596263 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16704062596263
          (code B ref 48468); Wed, 07 Dec 2022 09:45:01 +0000
Received: (at 48468) by debbugs.gnu.org; 7 Dec 2022 09:44:19 +0000
Received: from localhost ([127.0.0.1]:48909 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p2qyQ-0001cx-SL
	for submit <at> debbugs.gnu.org; Wed, 07 Dec 2022 04:44:19 -0500
Received: from eggs.gnu.org ([209.51.188.92]:60034)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1p2qyN-0001cr-Av
 for 48468 <at> debbugs.gnu.org; Wed, 07 Dec 2022 04:44:17 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1p2qyH-00045e-Do; Wed, 07 Dec 2022 04:44:09 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=9xYEt4HDXobSZZuvXN7OeOueQs4B+Q050cNQ+t23BMU=; b=J5g3m6XVh69+J0o9HdVS
 XyMdTuApKqnAkH2l6dVuJZNt2dxK8ACt99ixsNi02All0LRjr0o7ExMth6hPfTL0a470gmF8opUC6
 ge3ukhNWtnkFQ01kOWb1jLDAIV6uF4bvSOEzKUJm2bbpuh70FtFzR9F9CcoQFZl+V0q7SQR83Ev7E
 b9rFP/dzaSw7M5+f9TAhtSgifKlQDqJsZhhXkPHLhI3eUwYBpnn2J6HVk5Ze9y8YBKcjUDSK9RDal
 7V0uLyqyRNSM6YuiGxZyWT0Ttr17EG9nq47IHUfCjCetWAQgBVOEX162KMIKQ6b1Gpl7dDUK50xvL
 lP52ox9k7f2DBA==;
Received: from 2a02-8429-81d2-3d01-94c9-8097-ea5c-2774.rev.sfr.net
 ([2a02:8429:81d2:3d01:94c9:8097:ea5c:2774] helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1p2qyC-0005rX-DF; Wed, 07 Dec 2022 04:44:08 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN> <87lenmrnos.fsf@HIDDEN>
Date: Wed, 07 Dec 2022 10:44:01 +0100
In-Reply-To: <87lenmrnos.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 05 Dec 2022 14:21:23 +0100")
Message-ID: <87cz8v8s66.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello,

> You mentioned on IRC that nginx logs show that =E2=80=98guix publish=E2=
=80=99 times out.
> Looking at /var/log/nginx/error.log, I see =E2=80=9CConnection reset by p=
eer=E2=80=9D
> and =E2=80=9CBroken pipe=E2=80=9D, which could indicate that the client c=
losed the
> connection (which was open) prematurely, maybe due to an internal
> timeout.

Could it be that the client is receiving 404 because the baking of some
NAR was deferred to a worker, and then it closes the connection?

I think that's what I had in mind with the patch 2/2 of this patchset:
https://issues.guix.gnu.org/50040.

Thanks,

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 07 Dec 2022 13:39:02 +0000
Resent-Message-ID: <handler.48468.B48468.16704203183583 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16704203183583
          (code B ref 48468); Wed, 07 Dec 2022 13:39:02 +0000
Received: (at 48468) by debbugs.gnu.org; 7 Dec 2022 13:38:38 +0000
Received: from localhost ([127.0.0.1]:50421 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p2udC-0000vj-9k
	for submit <at> debbugs.gnu.org; Wed, 07 Dec 2022 08:38:38 -0500
Received: from eggs.gnu.org ([209.51.188.92]:45806)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1p2udA-0000vd-Bd
 for 48468 <at> debbugs.gnu.org; Wed, 07 Dec 2022 08:38:37 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1p2ud5-0006eW-1B
 for 48468 <at> debbugs.gnu.org; Wed, 07 Dec 2022 08:38:31 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=VMpl2g1kW8uavHNeZi6SkTR2Agv4PQwDjU2wki/AnD8=; b=XwMPTTe/8xc9QlNfP/b9
 e+awrw7HWTl7bp9yYqWy6gtXg+4D7v0TBkxRIHfZAW7LwnUk0D933R79koxRm3pmPlbhg4RJ/BPL/
 x0+x9Fb8dmYUOZqNHx1n0ez2u3PdZ1sg5ax7Fqi9BLpDsx1/3Rzq4XOyhYjx7TMh3IIZWAliQx7vr
 iIjUCfMddm0WgzJ9BtcU1/f0mLZnVdxDUojmdXCahyntERo05nV3nJO0NvBQnI6zvnvmKzfnplQ4S
 SFISVBIdVte8laVhNu+IbpOgikPxUpIFTpF911in6naeHirjeG4d8QjRLwB4KG0m1BYyP/8DXi5/a
 PGl1MRmPJtbAQg==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1p2ud1-0000l0-Na; Wed, 07 Dec 2022 08:38:27 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN> <87lenmrnos.fsf@HIDDEN>
 <87cz8v8s66.fsf@HIDDEN>
Date: Wed, 07 Dec 2022 14:38:25 +0100
In-Reply-To: <87cz8v8s66.fsf@HIDDEN> (Mathieu Othacehe's message of "Wed, 07
 Dec 2022 10:44:01 +0100")
Message-ID: <87cz8vjpv2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Mathieu Othacehe <othacehe@HIDDEN> skribis:

>> You mentioned on IRC that nginx logs show that =E2=80=98guix publish=E2=
=80=99 times out.
>> Looking at /var/log/nginx/error.log, I see =E2=80=9CConnection reset by =
peer=E2=80=9D
>> and =E2=80=9CBroken pipe=E2=80=9D, which could indicate that the client =
closed the
>> connection (which was open) prematurely, maybe due to an internal
>> timeout.
>
> Could it be that the client is receiving 404 because the baking of some
> NAR was deferred to a worker, and then it closes the connection?

Unlikely.  Take
/gnu/store/qmzr030rzgikdxv3g9msqv0l8qp5j6y2-btrfs-raid-root-os.drv,
which was marked as failed earlier today due to missing .drv.  It=E2=80=99s=
 a
4KiB file, and the cache-bypass-threshold is =E2=80=98guix publish=E2=80=99=
 is typically
set to something much higher than that.  So =E2=80=98guix publish=E2=80=99 =
won=E2=80=99t return
404 in that case.

If you find a =E2=80=9Cmissing derivation=E2=80=9D error in Cuirass, how wo=
uld you
search logs to find what happened?  I=E2=80=99m not sure where to look for
useful debugging info.

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Mathieu Othacehe <othacehe@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 07 Dec 2022 14:33:02 +0000
Resent-Message-ID: <handler.48468.B48468.16704235345733 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.16704235345733
          (code B ref 48468); Wed, 07 Dec 2022 14:33:02 +0000
Received: (at 48468) by debbugs.gnu.org; 7 Dec 2022 14:32:14 +0000
Received: from localhost ([127.0.0.1]:50524 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p2vT3-0001UP-Ki
	for submit <at> debbugs.gnu.org; Wed, 07 Dec 2022 09:32:14 -0500
Received: from eggs.gnu.org ([209.51.188.92]:59208)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1p2vT1-0001UJ-S1
 for 48468 <at> debbugs.gnu.org; Wed, 07 Dec 2022 09:32:12 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1p2vSv-0006eU-Fe; Wed, 07 Dec 2022 09:32:05 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=+QIVLCTr0wnscpqQ49rkieXQI15PIBR56NfsJYUJpWc=; b=CM3BAH5dr+8eOgJD7zhX
 vS5iPZtcvhuXiW6rJ4xeoeE32LixDkl6KPlcha2IbJv/hVOCGtN405dJeiWeUk8RQQw5JUN7oNEsB
 wuLEjM0o9UYiafKwIhMB1YZeT0Lc/nw6qPxfA30MQEe6YkjeovMmb3jxnbR+TpL8wWIcZb/sjqYWP
 my/3fwmI8bnyc+eAeNX5IozFXoP3Ukqw7lQYExEMpc+j3pfvceSSmzyNVARj7PfA9P5Lc1vX+X52c
 6zipxQWcIgGbkXd4jtWhvd/OwGtzH9PopfxcM+Qo6Bh3iglTLrRuPS9907ixBt+wzajrdvFNJUqfV
 aEmcFFPliIUB6w==;
Received: from 2a02-8429-81d2-3d01-94c9-8097-ea5c-2774.rev.sfr.net
 ([2a02:8429:81d2:3d01:94c9:8097:ea5c:2774] helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>)
 id 1p2vSf-0005IU-4F; Wed, 07 Dec 2022 09:32:03 -0500
From: Mathieu Othacehe <othacehe@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN> <87lenmrnos.fsf@HIDDEN>
 <87cz8v8s66.fsf@HIDDEN> <87cz8vjpv2.fsf@HIDDEN>
Date: Wed, 07 Dec 2022 15:31:44 +0100
In-Reply-To: <87cz8vjpv2.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Wed, 07 Dec 2022 14:38:25 +0100")
Message-ID: <878rjj8eun.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello,

> /gnu/store/qmzr030rzgikdxv3g9msqv0l8qp5j6y2-btrfs-raid-root-os.drv,
> which was marked as failed earlier today due to missing .drv.  It=E2=80=
=99s a
> 4KiB file, and the cache-bypass-threshold is =E2=80=98guix publish=E2=80=
=99 is typically
> set to something much higher than that.  So =E2=80=98guix publish=E2=80=
=99 won=E2=80=99t return
> 404 in that case.

Yes but that derivation also depends on other derivations, for instance
qemu-minimal and if I try:

--8<---------------cut here---------------start------------->8---
mathieu@berlin ~$ guix build qemu-minimal
...
/gnu/store/lwv2pl0m6dkf6bkzip755w5p71g5akq4-qemu-minimal-7.1.0
--8<---------------cut here---------------end--------------->8---

and then, from my machine.

--8<---------------cut here---------------start------------->8---
curl https://ci.guix.gnu.org/lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
We're baking it: /lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
--8<---------------cut here---------------end--------------->8---

wget exhibits the same behaviour and returns 404.

So any build that requires a heavy substitute, heavier than the cache
bypass threshold at least, will fail on the workers, as it would fail
locally.

That's not really a surprise as baking substitutes takes time and there
is a time window between the moment Cuirass triggers NAR baking and the
moment the NAR is baked, where every user will get a 404.

Mathieu

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Thu, 08 Dec 2022 10:27:02 +0000
Resent-Message-ID: <handler.48468.B48468.167049519932240 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.167049519932240
          (code B ref 48468); Thu, 08 Dec 2022 10:27:02 +0000
Received: (at 48468) by debbugs.gnu.org; 8 Dec 2022 10:26:39 +0000
Received: from localhost ([127.0.0.1]:55883 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p3E6x-0008Nw-3E
	for submit <at> debbugs.gnu.org; Thu, 08 Dec 2022 05:26:39 -0500
Received: from eggs.gnu.org ([209.51.188.92]:58260)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1p3E6v-0008Np-Cb
 for 48468 <at> debbugs.gnu.org; Thu, 08 Dec 2022 05:26:38 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1p3E6q-0008Pw-3l
 for 48468 <at> debbugs.gnu.org; Thu, 08 Dec 2022 05:26:32 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=1OwmOglqgnVtHkHRwkKwAXDWfPcjCygSAB5v1BC1iP4=; b=V9Q4lME8a3zVPYYVrWX/
 1qs32wxPu+w+D0j8tXPsS00VxuHKpF4Qt60HhOnckfpnH8p6zDiXjM55pS5CP7Oj/3i/gORGlVlpg
 dIAoo3BOFxGbhHW9o5E9q0wclFR6o5lZ1YIi8nYdoAhronnrdY5kXXxpsGoiG7vZSm1j5LrzAppEl
 lc1t8rMLXzqRnbW9MNP7KPDbJDTWxqzGHzLeHr+8IkaOQikBJLT79P1wO8kML12Np0lAXV2t4qMQl
 n7nvKVAtNLVFKgQbLInM6Qtm958LFKKPOauNC1kuiW7WYnvNY+j00GZv1rcpukgg8I4ZCt3hqx4Eu
 5SRu729RBVB0Lw==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1p3E6p-0003x0-Gz; Thu, 08 Dec 2022 05:26:31 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN> <87lenmrnos.fsf@HIDDEN>
 <87cz8v8s66.fsf@HIDDEN> <87cz8vjpv2.fsf@HIDDEN>
 <878rjj8eun.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Octidi 18 Frimaire an 231 de la =?UTF-8?Q?R=C3=A9volution,?= jour du Lierre
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Thu, 08 Dec 2022 11:26:28 +0100
In-Reply-To: <878rjj8eun.fsf@HIDDEN> (Mathieu Othacehe's message of "Wed, 07
 Dec 2022 15:31:44 +0100")
Message-ID: <87wn72fay3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Mathieu Othacehe <othacehe@HIDDEN> skribis:

>> /gnu/store/qmzr030rzgikdxv3g9msqv0l8qp5j6y2-btrfs-raid-root-os.drv,
>> which was marked as failed earlier today due to missing .drv.  It=E2=80=
=99s a
>> 4KiB file, and the cache-bypass-threshold is =E2=80=98guix publish=E2=80=
=99 is typically
>> set to something much higher than that.  So =E2=80=98guix publish=E2=80=
=99 won=E2=80=99t return
>> 404 in that case.
>
> Yes but that derivation also depends on other derivations, for instance
> qemu-minimal and if I try:
>
> mathieu@berlin ~$ guix build qemu-minimal
> ...
> /gnu/store/lwv2pl0m6dkf6bkzip755w5p71g5akq4-qemu-minimal-7.1.0
>
>
> and then, from my machine.
>
> curl https://ci.guix.gnu.org/lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
> We're baking it: /lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo

Yes, but derivations (.drv) are depend only on =E2=80=9Csources=E2=80=9D (l=
ike
*-guile-builder, *.patch, *.scm) and on other derivations, all of which
are typically less than 1=C2=A0MiB.

Derivations don=E2=80=99t depend on derivation outputs like that of qemu-mi=
nimal
above.

> So any build that requires a heavy substitute, heavier than the cache
> bypass threshold at least, will fail on the workers, as it would fail
> locally.

The main =E2=80=98guix publish=E2=80=99 instance on berlin has cache-bypass=
-threshold
set to 100=C2=A0MiB; I don=E2=80=99t think we have any source or .drv that =
is this
big, or even a tenth of it.  :-)

The =E2=80=98guix publish=E2=80=99 instance spawned in (cuirass remote) doe=
sn=E2=80=99t use
caching at all, so there=E2=80=99s no bypass threshold.

The only clue we have so far is a long chain of successful .drv
substitutions followed by a failing one:

--8<---------------cut here---------------start------------->8---
Downloading http://141.80.167.131/nar/zstd/9nm35401i8j42559iadi3iz3kmhmj7pr=
-guix-system-tests.drv...

[K guix-system-tests.drv  1KiB             0B/s 00:00 [                  ] =
  0.0%
[K guix-system-tests.drv  1KiB         1.3MiB/s 00:00 [##################] =
100.0%
[K guix-system-tests.drv  1KiB         408KiB/s 00:00 [##################] =
100.0%

@ substituter-succeeded /gnu/store/9nm35401i8j42559iadi3iz3kmhmj7pr-guix-sy=
stem-tests.drv
fetching path `/gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-test=
s-modules.drv'...
@ substituter-started /gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-syst=
em-tests-modules.drv substitute
Downloading http://141.80.167.131/nar/zstd/9zf6inizcb09m136c44dj35fmzf2g3hs=
-guix-system-tests-modules.drv...

[K guix-system-tests-modules.drv  543B     0B/s 00:00 [                  ] =
  0.0%
[K guix-system-tests-modules.drv  543B 851KiB/s 00:00 [##################] =
100.0%
[K guix-system-tests-modules.drv  543B 484KiB/s 00:00 [##################] =
100.0%

@ substituter-succeeded /gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-sy=
stem-tests-modules.drv
cannot build missing derivation ?/gnu/store/zq7idl0j51fdzqhhqm9ql90d0f2326k=
7-btrfs-root-on-subvolume-os.drv?
--8<---------------cut here---------------end--------------->8---

(From <https://ci.guix.gnu.org/build/224849/log/raw>.)

Plausible explanations that come to mind:

  1. =E2=80=98guix publish=E2=80=99 returning 404, but not due to baking.  =
Instead the
     .drv is simply not in store, hence 404.

  2. Client timeout (=E2=80=98guix publish=E2=80=99 fails to reply on time).

  3. Cached 404 in nginx (though we=E2=80=99re not supposed to cache those I
     think?), or timeout in nginx (again due to =E2=80=98guix publish=E2=80=
=99 being too
     slow and =E2=80=98proxy_read_timeout=E2=80=99 is reached, currently 10=
s).

We should see if we can get useful info from nginx/publish logs.

Thanks,
Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#48468: substitute server connection timeout
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 10 Dec 2022 10:57:02 +0000
Resent-Message-ID: <handler.48468.B48468.167066976423542 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 48468
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Mathieu Othacehe <othacehe@HIDDEN>
Cc: 48468 <at> debbugs.gnu.org
Received: via spool by 48468-submit <at> debbugs.gnu.org id=B48468.167066976423542
          (code B ref 48468); Sat, 10 Dec 2022 10:57:02 +0000
Received: (at 48468) by debbugs.gnu.org; 10 Dec 2022 10:56:04 +0000
Received: from localhost ([127.0.0.1]:42078 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1p3xWV-00067e-Us
	for submit <at> debbugs.gnu.org; Sat, 10 Dec 2022 05:56:04 -0500
Received: from eggs.gnu.org ([209.51.188.92]:48908)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1p3xWU-00067H-QX
 for 48468 <at> debbugs.gnu.org; Sat, 10 Dec 2022 05:56:03 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1p3xWP-0002VM-5W
 for 48468 <at> debbugs.gnu.org; Sat, 10 Dec 2022 05:55:57 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=YD/ILx6kPDmpu3G7BGvmOwwpa1gjGAnaH2rwXlq5sdM=; b=LYilsh4IhxJCfcKZreWs
 0KQ1aIAC6wv3DPn/lzIDFns7L03mwsN3Utgn341YHPfKpl/Mj9IxP3o1zsVlFiru4L52usD/dRpzC
 qoPKcKY8Wn+S4LzjUdKqfuLwevKkmXgn9f0bygC8n/QxrQGq5M32jE2lEaDVaz4KXthUapUjfXEyL
 oqTbW00IhxTGIsHI44dDtGmkaoAmpFdfc4Hi07oe0dq+SL98UNBt7zexZImcMLIvs0qyFKCT0rtEx
 GThPh85vw9IVR0rja8uwKqCSWLDtEo+/XRRL3jVCyNmkcKzZFE4siS0MF9IGjH2imeSBf98SwO8O4
 ERGw6somlMXViQ==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1p3xWM-0004Fc-S8; Sat, 10 Dec 2022 05:55:56 -0500
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
References: <87lf8e4l42.fsf@HIDDEN> <87im2bfj3i.fsf@HIDDEN>
 <87sg108vl5.fsf@HIDDEN> <87lenmrnos.fsf@HIDDEN>
 <87cz8v8s66.fsf@HIDDEN> <87cz8vjpv2.fsf@HIDDEN>
 <878rjj8eun.fsf@HIDDEN> <87wn72fay3.fsf@HIDDEN>
Date: Sat, 10 Dec 2022 11:55:52 +0100
In-Reply-To: <87wn72fay3.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 08 Dec 2022 11:26:28 +0100")
Message-ID: <87fsdn5xzb.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi,

Ludovic Court=C3=A8s <ludo@HIDDEN> skribis:

> Mathieu Othacehe <othacehe@HIDDEN> skribis:

[...]

>> curl https://ci.guix.gnu.org/lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
>> We're baking it: /lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
>
> Yes, but derivations (.drv) are depend only on =E2=80=9Csources=E2=80=9D =
(like
> *-guile-builder, *.patch, *.scm) and on other derivations, all of which
> are typically less than 1=C2=A0MiB.
>
> Derivations don=E2=80=99t depend on derivation outputs like that of qemu-=
minimal
> above.
>
>> So any build that requires a heavy substitute, heavier than the cache
>> bypass threshold at least, will fail on the workers, as it would fail
>> locally.
>
> The main =E2=80=98guix publish=E2=80=99 instance on berlin has cache-bypa=
ss-threshold
> set to 100=C2=A0MiB; I don=E2=80=99t think we have any source or .drv tha=
t is this
> big, or even a tenth of it.  :-)

On closer inspection, we do get bigger sources for those system test
derivations:

--8<---------------cut here---------------start------------->8---
$ du  $(guix gc -R /gnu/store/2laznragkfjf0jn6n7b8c53vgmb8maqp-installed-ex=
tlinux-os.drv ) |sort -k1 -n |tail
6304	/gnu/store/mvvf60w8p0lx8vlixryb1a1gfib401rf-guix-b129026/gnu/packages
6956	/gnu/store/9cs4i3hygldsan8ahf0p7x5kgzmqz9gr-guix-packages-base-source/=
gnu/packages/patches
7000	/gnu/store/mvvf60w8p0lx8vlixryb1a1gfib401rf-guix-b129026/gnu
7716	/gnu/store/z0gc56x1w83wn0pi4mz8nhyjdr0d10fs-guix
12352	/gnu/store/mvvf60w8p0lx8vlixryb1a1gfib401rf-guix-b129026
22548	/gnu/store/ky96vygv1hpa8iz6677qz96binbz201h-packages
22684	/gnu/store/rrg9ki4bdjffvmgapjcqwjwjwkhb0qhl-doc
39516	/gnu/store/9cs4i3hygldsan8ahf0p7x5kgzmqz9gr-guix-packages-base-source=
/gnu/packages
39524	/gnu/store/9cs4i3hygldsan8ahf0p7x5kgzmqz9gr-guix-packages-base-source=
/gnu
39528	/gnu/store/9cs4i3hygldsan8ahf0p7x5kgzmqz9gr-guix-packages-base-source
--8<---------------cut here---------------end--------------->8---

The items at the bottom weigh around 40=C2=A0MiB.

That=E2=80=99s still below the 100=C2=A0MiB cache bypass threshold of the m=
ain =E2=80=98guix
publish=E2=80=99 instance though.

Ludo=E2=80=99.

Last modified: Sat, 10 Dec 2022 11:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.