GNU logs - #33410, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#33410: Offloaded builds can get stuck indefinitely due to network issues
Resent-From: Mark H Weaver <mhw@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 17 Nov 2018 04:10:01 +0000
Resent-Message-ID: <handler.33410.B.15424277842023 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 33410 <at> debbugs.gnu.org
From: Mark H Weaver <mhw@HIDDEN>
Date: Fri, 16 Nov 2018 23:08:50 -0500
Message-ID: <87a7m8xs42.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

I just discovered that 4 out of 5 armhf build slots on Hydra have been
stuck for 24 hours, apparently after the network connections to the
build slaves were lost, possibly due to a temporary network outage.

I've seen this kind of thing happen periodically since we switched to
using guile-ssh for offloaded builds.

On Hydra I can monitor the builds and investigate when a given build
seems to be taking far too long, and I can kill those jobs to free up
the build slots.  There's no way to kill the builds from Hydra's web
interface, but I can kill them manually by logging into Hydra.

This might become a more serious problem on Berlin, as we add ARM build
slaves that are not on the same local network as Berlin itself, until
the web interface allows for this kind of monitoring and intervention.

      Mark

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Mark H Weaver <mhw@HIDDEN>
Subject: bug#33410: Acknowledgement (Offloaded builds can get stuck
 indefinitely due to network issues)
Message-ID: <handler.33410.B.15424277842023.ack <at> debbugs.gnu.org>
References: <87a7m8xs42.fsf@HIDDEN>
X-Gnu-PR-Message: ack 33410
X-Gnu-PR-Package: guix
Reply-To: 33410 <at> debbugs.gnu.org
Date: Sat, 17 Nov 2018 04:10:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 33410 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
33410: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33410
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#33410: Offloaded builds can get stuck indefinitely due to network issues
Resent-From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sat, 17 Nov 2018 14:22:02 +0000
Resent-Message-ID: <handler.33410.B33410.15424645019089 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Mark H Weaver <mhw@HIDDEN>
Cc: 33410 <at> debbugs.gnu.org
From: ludo@HIDDEN (Ludovic =?UTF-8?Q?Court=C3=A8s?=)
References: <87a7m8xs42.fsf@HIDDEN>
Date: Sat, 17 Nov 2018 15:21:32 +0100
In-Reply-To: <87a7m8xs42.fsf@HIDDEN> (Mark H. Weaver's message of "Fri, 16
 Nov 2018 23:08:50 -0500")
Message-ID: <87efbjokcz.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello,

Mark H Weaver <mhw@HIDDEN> skribis:

> I just discovered that 4 out of 5 armhf build slots on Hydra have been
> stuck for 24 hours, apparently after the network connections to the
> build slaves were lost, possibly due to a temporary network outage.
>
> I've seen this kind of thing happen periodically since we switched to
> using guile-ssh for offloaded builds.

Which guix-daemon version is hydra running?

Commit a708de151c255712071e42e5c8284756b51768cd adds a safeguard to make
sure timeouts are honored, though there might be some cases where it
doesn=E2=80=99t quite work as expected (I suspect libssh handles EINTR
internally by looping, in which case our signal handling async doesn=E2=80=
=99t
get a chance to run.)

> On Hydra I can monitor the builds and investigate when a given build
> seems to be taking far too long, and I can kill those jobs to free up
> the build slots.  There's no way to kill the builds from Hydra's web
> interface, but I can kill them manually by logging into Hydra.
>
> This might become a more serious problem on Berlin, as we add ARM build
> slaves that are not on the same local network as Berlin itself, until
> the web interface allows for this kind of monitoring and intervention.

The current situation on berlin is suboptimal: I run =E2=80=98guix processe=
s=E2=80=99
when I suspect something is wrong, and that=E2=80=99s how I found about
<https://issues.guix.info/issue/33239>.

Thanks,
Ludo=E2=80=99.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.