GNU logs - #60803, boring messages

Message sent to othacehe@HIDDEN, bug-guix@HIDDEN:

Subject: bug#60803: Cuirass stopped processing jobs for aarch64-linux and x86_64-linux
Resent-From: Marius Bakke <marius@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: othacehe@HIDDEN, bug-guix@HIDDEN
Resent-Date: Sat, 14 Jan 2023 05:20:02 +0000
Resent-Message-ID: <handler.60803.B.16736735767893 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 60803 <at> debbugs.gnu.org
Cc: othacehe@HIDDEN
From: Marius Bakke <marius@HIDDEN>
Date: Sat, 14 Jan 2023 06:19:22 +0100
Message-ID: <87bkn1n15x.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hello Guix,

Cuirass has stopped processing (old) jobs for aarch64 and x86_64.  After
digging through the database it's because (db-get-pending-build ...)
returns a build that is missing from the Jobs table:

  WITH pending_dependencies AS
  (SELECT Builds.id, count(dep.id) as deps FROM Builds
  LEFT JOIN BuildDependencies as bd ON bd.source = Builds.id
  LEFT JOIN Builds AS dep ON bd.target = dep.id AND dep.status != 0
  WHERE Builds.status = -2 AND Builds.system = 'x86_64-linux'
  GROUP BY builds.id
  ORDER BY Builds.priority ASC, Builds.timestamp DESC)
  SELECT id FROM pending_dependencies where deps = 0 limit 1;

     id
  --------
   335212

However:

  select * from jobs  where  build = 335212;
   name | evaluation | build | status | system
  ------+------------+-------+--------+--------
  (0 rows)

For clarity:

  select id,derivation,evaluation,job_name,nix_name,status from builds where id = 335212;
     id   |                            derivation                             | evaluation |       job_name        |     nix_name      | status
  --------+-------------------------------------------------------------------+------------+-----------------------+-------------------+--------
   335212 | /gnu/store/yzgcza0nijnp79mzz878q9a61p6jykgh-perftest-4.5-0.20.drv |     103435 | perftest.x86_64-linux | perftest-4.5-0.20 |     -2

The derivation is also missing from the Outputs table, which causes the
monster query in (db-get-builds ...), which is what workers call to
fetch the next job, to return nothing.

335212 belongs to evaluation 103435 according to the above query, but
does not show up here:

  https://ci.guix.gnu.org/eval/103435?all=&paginate=0

The build id sequence appears to belong to this evaluation:

  https://ci.guix.gnu.org/eval/103436?all=&paginate=0

(notice how it has 335211 and 335213).

I'm not sure how to recover from this.  Either manually create the
entries in Jobs and Outputs, or delete the offending Builds entry?

The 335212 build is for x86_64-linux, we have the same problem with
335087 (also perftest) on aarch64.  i686-linux and powerpc64le-linux is
fine.

Ideas?

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Marius Bakke <marius@HIDDEN>
Subject: bug#60803: Acknowledgement (Cuirass stopped processing jobs for
 aarch64-linux and x86_64-linux)
Message-ID: <handler.60803.B.16736735767893.ack <at> debbugs.gnu.org>
References: <87bkn1n15x.fsf@HIDDEN>
X-Gnu-PR-Message: ack 60803
X-Gnu-PR-Package: guix
Reply-To: 60803 <at> debbugs.gnu.org
Date: Sat, 14 Jan 2023 05:20:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

As you requested using X-Debbugs-CC, your message was also forwarded to
  othacehe@HIDDEN
(after having been given a bug report number, if it did not have one).

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 60803 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
60803: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D60803
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#60803: Cuirass stopped processing jobs for aarch64-linux and x86_64-linux
Resent-From: Marius Bakke <marius@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 15 Jan 2023 04:27:02 +0000
Resent-Message-ID: <handler.60803.B60803.167375680214925 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 60803 <at> debbugs.gnu.org
Cc: othacehe@HIDDEN
From: Marius Bakke <marius@HIDDEN>
In-Reply-To: <87bkn1n15x.fsf@HIDDEN>
References: <87bkn1n15x.fsf@HIDDEN>
Date: Sun, 15 Jan 2023 05:26:27 +0100
Message-ID: <878ri4mnik.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="=-=-=";
 micalg=pgp-sha512; protocol="application/pgp-signature"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: text/plain

Marius Bakke <marius@HIDDEN> skriver:

> The 335212 build is for x86_64-linux, we have the same problem with
> 335087 (also perftest) on aarch64.  i686-linux and powerpc64le-linux is
> fine.

I deleted these two from the Builds and BuildDependencies tables which
allowed Cuirass to move forward (or backwards, really, as it was
processing new jobs just fine).

Not sure how to mitigate the problem (race when two evaluations create
different derivations with identical outputs at the same time?), but at
least we know how to deal with it.

Speaking of builds, I started debugging #60016 and accidentally deleted
build 175246!  Enough late night debugging for me...  I'll set up my own
Cuirass to experiment on "soon".

--=-=-=
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCY8OAcw8cbWFyaXVzQGdu
dS5vcmcACgkQ6HGLpZEUEHe+tgEAx8zHVH7VGv9EY/Xc3FKqHDeMY35FA4zCvcG2
YlE/Dr8BAPiTogxuXk3DdA7uDvOEFoBF/fGtmXH0cCf9Pwrd89QJ
=05EW
-----END PGP SIGNATURE-----
--=-=-=--

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 14 Jul 2024 21:43:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Jul 14 17:43:18 2024
Received: from localhost ([127.0.0.1]:58665 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1sT702-00054b-KY
	for submit <at> debbugs.gnu.org; Sun, 14 Jul 2024 17:43:18 -0400
Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:6986)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1sT700-00054K-86
 for control <at> debbugs.gnu.org; Sun, 14 Jul 2024 17:43:16 -0400
Authentication-Results: mail3-relais-sop.national.inria.fr;
 dkim=none (message not signed) header.i=none;
 spf=SoftFail smtp.mailfrom=ludo@HIDDEN;
 dmarc=fail (p=none dis=none) d=gnu.org
X-IronPort-AV: E=Sophos;i="6.09,208,1716242400"; d="scan'208";a="92131729"
Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201])
 by mail3-relais-sop.national.inria.fr with
 ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2024 23:43:07 +0200
Date: Sun, 14 Jul 2024 23:43:06 +0200
Message-Id: <87sewby7it.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Subject: control message for bug #60803
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

close 60803 
quit

Last modified: Sun, 14 Jul 2024 21:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.