GNU bug report logs - #59514
Stuck builds in Cuirass

Previous Next

To reply to this bug, email your comments to 59514 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <marius <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: Stuck builds in Cuirass
Date: Wed, 23 Nov 2022 13:50:35 +0100

[Message part 1 (text/plain, inline)]

Hi,

Cuirass has a tendency to not notice when a build is finished, leaving
it in a "running" state.

The phenomenon can be observed by going to
<https://ci.guix.gnu.org/status> and look at builds that are running for
a suspiciously long time.

Typically the build log will indicate that it has finished, yet Cuirass
is patiently waiting...and not scheduling further builds.

Restarting the builds typically get things going again.

I wrote a nasty script to automatically restart builds that are running
for >1 hour, but it's not a sustainable solution:

[restart-old-builds.py (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 59514 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Marius Bakke <marius <at> gnu.org>
Cc: 59514 <at> debbugs.gnu.org
Subject: Re: bug#59514: Stuck builds in Cuirass
Date: Wed, 23 Nov 2022 14:26:50 +0100

Hello Marius,

> Cuirass has a tendency to not notice when a build is finished, leaving
> it in a "running" state.
>
> The phenomenon can be observed by going to
> <https://ci.guix.gnu.org/status> and look at builds that are running for
> a suspiciously long time.

I suspect this is caused by https://issues.guix.gnu.org/59510 which
causes the worker threads to bail out.

We can probably merge those two issues. The
/var/log/cuirass-remote-server.log file on Berlin also indicates when
the build-succeeded or build-failed message is received by the server,
and how long the fetch from the worker took.

Thanks,

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.