GNU bug report logs - #55848
[cuirass] workers stalled

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Severity: important; Reported by: Mathieu Othacehe <othacehe@HIDDEN>; dated Wed, 8 Jun 2022 15:32:02 UTC; Maintainer for guix is bug-guix@HIDDEN.
Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 55848 <at> debbugs.gnu.org:


Received: (at 55848) by debbugs.gnu.org; 11 Jun 2022 10:44:54 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 11 06:44:54 2022
Received: from localhost ([127.0.0.1]:51954 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nzybt-0007tb-RU
	for submit <at> debbugs.gnu.org; Sat, 11 Jun 2022 06:44:54 -0400
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:43587)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <tom@HIDDEN>) id 1nzybo-0007tE-UK
 for 55848 <at> debbugs.gnu.org; Sat, 11 Jun 2022 06:44:50 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id C93A85C00E4;
 Sat, 11 Jun 2022 06:44:43 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Sat, 11 Jun 2022 06:44:43 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 tom-fitzhenry.me.uk; h=cc:cc:content-type:date:date:from:from
 :in-reply-to:in-reply-to:message-id:mime-version:references
 :reply-to:sender:subject:subject:to:to; s=fm2; t=1654944283; x=
 1655030683; bh=Harm9+hQKY4Qxssnj+8WoVEWHLVCpyLHdK/iGYysuOQ=; b=I
 R45o8k/68LYSrCJVjlw5iiu5qaObpnbVxzdiew54gGIbY9wlAryCV1pc1RaiXwi+
 qNQ2ORHH1Xk3DA11ZVLx1A9GGmWvLJ/g705ZQCC09b3BLtM7n95FtL4Ou1Mqkgq0
 D7NwsQbhCngqvmyZZQ6lYWeUrQHBNZKXfxDX/p0+KxBQg+Nba8ayBk9CD4tjLb9q
 mzEnD0s57O363dUgFHJ0NHKt+SlyGQTw6+y+2H+/0nb5Coav8/ADxcfshUAncpET
 axez86H0g4WTimHZgWXshnUAEO108BMMeD44hsjVILoDIcDbbEBHJlsN4zGZkoN1
 zYNWkLHpt9Ts779yyR8TQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:date:date:feedback-id
 :feedback-id:from:from:in-reply-to:in-reply-to:message-id
 :mime-version:references:reply-to:sender:subject:subject:to:to
 :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
 fm2; t=1654944283; x=1655030683; bh=Harm9+hQKY4Qxssnj+8WoVEWHLVC
 pyLHdK/iGYysuOQ=; b=T7RkSnbvTO9ism7ynRV4G3/c8P0GD0//QWxYwZ5Mj4pD
 S+n66O+JBor+v4vrrRw374cGHszf1PfO9c0IysEvOyaqGe0LDh2SRXEeWteWlVtO
 3yJK0P2w21UhNSULj8/kXW33li77zqKythyIejq/rb9HLmh8t9ALX9EtTBTji25V
 Y9781TzLAwAsfy56mTpkoWmLoSF2IznZKILQGSfK9z2EjBZRuWWim8gUR2QEtjup
 Nb8Dimqu7jgjvNkzZaMnLdYMJH34VJ8HfX0H1h2eb3POSJRIqLDm9E//E4sBxjcN
 Kbcf2B4Od+JI1ltZd0hMr8zyg29D7eUX7SjlOww1Yg==
X-ME-Sender: <xms:G3KkYkpsm9XOLrxE4N_8sdxIXB1rLmqTxKF03qiicS4Do6PJX1zawA>
 <xme:G3KkYqqzpd4YoCZ82-vDwFHnkdBF8z1QWEULwVHi6Gz5EFGfvD54H2M3CGsSgxpqz
 womNmWZfZLEkfNXQw>
X-ME-Received: <xmr:G3KkYpOTELDyTV7oY7xRBXn_IeseCg72T2LjekQKmcsyIGz4pAEXf333AEGvMhsn0drc_o_ONuxPlsU939KjeKY>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddufedgfedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvvefufhffjgfkfgggtgesthdtredttdertdenucfhrhhomhepvfhomhcu
 hfhithiihhgvnhhrhicuoehtohhmsehtohhmqdhfihhtiihhvghnrhihrdhmvgdruhhkqe
 enucggtffrrghtthgvrhhnpeekfeelteduleekkeehfeejkeduueduhfeugfehtdelgeev
 veduhfefhfejleekieenucffohhmrghinhepghhnuhdrohhrghenucevlhhushhtvghruf
 hiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohhmsehtohhmqdhfihhtiihh
 vghnrhihrdhmvgdruhhk
X-ME-Proxy: <xmx:G3KkYr5TDKZJRyyLsMSrPAJIerQ38gHUfCGPxyq6ij1ec8Y-x3tY6Q>
 <xmx:G3KkYj4g6Pk3STHdWhVwIT1KBRjE_a518VF2nap9YictvOXJqHaVng>
 <xmx:G3KkYrgK3PIFinSP08r7aySt4NFmF1e-3855y9rw44J5Rb2njw30WA>
 <xmx:G3KkYpTYjae9lPKws82whcLRKiPGw-1RPnEfvOUHkiwh_B5RKCoYvQ>
Feedback-ID: iefc945ae:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat,
 11 Jun 2022 06:44:42 -0400 (EDT)
From: Tom Fitzhenry <tom@HIDDEN>
To: Greg Hogan <code@HIDDEN>
Subject: Re: bug#55848: [cuirass] workers stalled
References: <87h74v2mu7.fsf@HIDDEN>
 <CA+3U0ZmY4jcrZ6FPeQCgHHZKpvSh0BZ1ki5KTXfGemQRf-ZOkw@HIDDEN>
Date: Sat, 11 Jun 2022 20:44:16 +1000
In-Reply-To: <CA+3U0ZmY4jcrZ6FPeQCgHHZKpvSh0BZ1ki5KTXfGemQRf-ZOkw@HIDDEN>
 (Greg Hogan's message of "Wed, 8 Jun 2022 15:07:18 -0400")
Message-ID: <878rq3scn3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 55848
Cc: Mathieu Othacehe <othacehe@HIDDEN>, 55848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Greg Hogan <code@HIDDEN> writes:

> On Wed, Jun 8, 2022 at 11:32 AM Mathieu Othacehe <othacehe@HIDDEN> wrote:
>> The aarch64 workers were all idle whereas 70k builds were
>> available. Once restarted, they started building again.

From following the builds on http://ci.guix.gnu.org/workers , many
(all?) builds are failing on the following workers:

* grunewald
* kreuzberg
* pankow

The builds are failing with the same error:

"substitute: updating substitutes from 'https://ci.guix.gnu.org'...
0.0%guix substitute: error: TLS error in procedure 'handshake': Error in
the pull function."

Here's some examples:
* http://ci.guix.gnu.org/build/998403/details
* http://ci.guix.gnu.org/build/978678/details
* http://ci.guix.gnu.org/build/978243/details


On worker overdrive1, in the raw log of
http://ci.guix.gnu.org/build/875908/details we can see this
rust-async-mutex build managing to pull substitutes, but it 
seems to be compiling rust-1.57 itself.




Information forwarded to bug-guix@HIDDEN:
bug#55848; Package guix. Full text available.

Message received at 55848 <at> debbugs.gnu.org:


Received: (at 55848) by debbugs.gnu.org; 8 Jun 2022 19:07:40 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 08 15:07:40 2022
Received: from localhost ([127.0.0.1]:44318 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nz11n-0002Pt-Ru
	for submit <at> debbugs.gnu.org; Wed, 08 Jun 2022 15:07:40 -0400
Received: from mail-io1-f49.google.com ([209.85.166.49]:45686)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <code@HIDDEN>) id 1nz11j-0002Pe-98
 for 55848 <at> debbugs.gnu.org; Wed, 08 Jun 2022 15:07:38 -0400
Received: by mail-io1-f49.google.com with SMTP id 19so578439iou.12
 for <55848 <at> debbugs.gnu.org>; Wed, 08 Jun 2022 12:07:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=greghogan-com.20210112.gappssmtp.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=Hrvu0h90HESa5IQkHxDRFff9A2vmnpKmqgwOITkQlQA=;
 b=KHC48EHgnYGHwvvl9+aj5WmrgYmRY9ARUu1nwJkE9SLggYrJlu1FEp3R1K9oAuSV/b
 FxBjyDBfhpe+gEDDK7AeDqGM2qp6WTdwmKCdfrV4lT/+IxMzfh5JrmSNtipqekkxYE/Z
 5bggaBIdID5AT/d041nEAdYsF1G+TNF7GZ2i8EohihZhBK2v+Y/hIEi1eFKzGSmmT+QA
 cB3W2jaK9VZszU1uSkI32FG29XdudTbRzKhO8h4SsOApR+DzUZAxfZLTI4SxPL3d7QGl
 X9pu4N83mK3ip4pSQC6TkUiW3F5VUK5OZwjHlXqMIRoaRKSEQz51JHX7Q5vBiO8Zj/3W
 yQsA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=Hrvu0h90HESa5IQkHxDRFff9A2vmnpKmqgwOITkQlQA=;
 b=XDNl0mpYupuuJ9YQoB8r4d5nbJ53ks/Cd1QWoc9+iQgAQlQiaY6vm4+cUREy4IUvap
 12/4RXWmxPQH6Xsmh/p3AEpaTNW1HpGqqfRYyyghq0n8iIL++sPdh6g5SHRo43nekpdc
 mpWdmrhxY9WgPqKSdPuLElL/byYH2LN7M+Fr2bLqrpVn+jdIIQlNsA9sJ+Wf6XD1TOMy
 Xxk58nZFFyuVlennHxDhPisAWIpCKnHuOhwhCPeIe2IZ/krV2tNUko4T5I3CN5gkezG3
 FmXyTGHAgu39nhrn2wlp35gSvVgQ5xHg86Z5rMKg4DJhbmak4bJkxV+x98Oa8zzi89TT
 WriA==
X-Gm-Message-State: AOAM5338G02nvKLGC8fW/clMKuN47fzcxXm7cN9S2nywDJPkW7ABD2Mr
 Vb7VQ8IsDmdnop0BQnuSA9ub8Rs/qfCnmLT83w9Wb0YCeB4cjmCW
X-Google-Smtp-Source: ABdhPJyb6Tuama5tkipXoKX9aHDW+bKTfn8QTGGftIXwczFwk3O2O17Q2fd9Y8saBioxxyn2eCiZ8VgexXVYuED+Tqs=
X-Received: by 2002:a6b:ed0f:0:b0:657:b1ff:be52 with SMTP id
 n15-20020a6bed0f000000b00657b1ffbe52mr16152567iog.34.1654715249046; Wed, 08
 Jun 2022 12:07:29 -0700 (PDT)
MIME-Version: 1.0
References: <87h74v2mu7.fsf@HIDDEN>
In-Reply-To: <87h74v2mu7.fsf@HIDDEN>
From: Greg Hogan <code@HIDDEN>
Date: Wed, 8 Jun 2022 15:07:18 -0400
Message-ID: <CA+3U0ZmY4jcrZ6FPeQCgHHZKpvSh0BZ1ki5KTXfGemQRf-ZOkw@HIDDEN>
Subject: Re: bug#55848: [cuirass] workers stalled
To: Mathieu Othacehe <othacehe@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 55848
Cc: 55848 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Wed, Jun 8, 2022 at 11:32 AM Mathieu Othacehe <othacehe@HIDDEN> wrote:
>
>
> Hello,
>
> The aarch64 workers were all idle whereas 70k builds were
> available. Once restarted, they started building again.
>
> The problem might be that when the server is unavailable for a while the
> worker connections expire and cannot be resumed once the server is
> available again.
>
> Thanks,
>
> Mathieu

The recent aarch64 builds look to all be failing with the following message.

===== <cut> =====
substitute:
substitute:  [Kupdating substitutes from 'https://ci.guix.gnu.org'...
 0.0%guix substitute: error: TLS error in procedure 'handshake': Error
in the pull function.
===== </cut> =====




Information forwarded to bug-guix@HIDDEN:
bug#55848; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 8 Jun 2022 15:31:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 08 11:31:55 2022
Received: from localhost ([127.0.0.1]:44005 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nyxf0-0006w1-Sj
	for submit <at> debbugs.gnu.org; Wed, 08 Jun 2022 11:31:55 -0400
Received: from lists.gnu.org ([209.51.188.17]:60576)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <othacehe@HIDDEN>) id 1nyxew-0006vq-3l
 for submit <at> debbugs.gnu.org; Wed, 08 Jun 2022 11:31:53 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:44914)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1nyxet-00030V-Sv
 for bug-guix@HIDDEN; Wed, 08 Jun 2022 11:31:48 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36212)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1nyxet-00046K-JT
 for bug-guix@HIDDEN; Wed, 08 Jun 2022 11:31:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
 references; bh=mGUIV2OooEy919a16wiiIuGCd4pyQP9XtwqO2++BrJg=; b=mmZI4v0mCcsMJY
 VQoe4RUmdG4fAVmT0hI+CObZcOadETjUCSE1ABPb8+xkyYjzYDtYd3sJS4VqZ5RYHwl+XOQx7oloz
 3oy174yGLlwQ2fTC9eQvb2a7ez3SNiJoJQLxTO1eNJS/g2KIKwk6tpcKtsgKjXdJmDGLOuIdRsMjm
 T95EvbjgTWcTZZ05yxK8LQYV/MZ0F4EvYSObhiOJBmdbI1L8J6QdDFgnent2EJ9paT3fhcMWeB9mB
 hyDdBclqu1T99k9aJuodchrE0HBXfZV8MGhr6b8YHJDvFXrtC4B3rC1IjQSLSXCwl6qE0cvdNJvXU
 wMj17lHV4/0H4BFBapZw==;
Received: from 2a02-8429-81d2-3d01-94c9-8097-ea5c-2775.rev.sfr.net
 ([2a02:8429:81d2:3d01:94c9:8097:ea5c:2775]:58406 helo=meije)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <othacehe@HIDDEN>) id 1nyxet-00063d-53
 for bug-guix@HIDDEN; Wed, 08 Jun 2022 11:31:47 -0400
From: Mathieu Othacehe <othacehe@HIDDEN>
To: bug-guix@HIDDEN
Subject: [cuirass] workers stalled
Date: Wed, 08 Jun 2022 17:31:44 +0200
Message-ID: <87h74v2mu7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)


Hello,

The aarch64 workers were all idle whereas 70k builds were
available. Once restarted, they started building again.

The problem might be that when the server is unavailable for a while the
worker connections expire and cannot be resumed once the server is
available again.

Thanks,

Mathieu




Acknowledgement sent to Mathieu Othacehe <othacehe@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#55848; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sat, 11 Jun 2022 20:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.