GNU bug report logs - #32583
Cuirass: some builds are marked as failed, but they were never built

Previous Next

Package: guix;

Reported by: Clément Lassieur <clement <at> lassieur.org>

Date: Thu, 30 Aug 2018 09:51:01 UTC

Severity: normal

Done: ludo <at> gnu.org (Ludovic Courtès)

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32583 in the body.
You can then email your comments to 32583 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#32583; Package guix. (Thu, 30 Aug 2018 09:51:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to Clément Lassieur <clement <at> lassieur.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Thu, 30 Aug 2018 09:51:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: bug-guix <at> gnu.org
Subject: Cuirass: some builds are marked as failed, but they were never built
Date: Thu, 30 Aug 2018 11:44:54 +0200
Reported by Ludovic on IRC.

Some builds are marked as failed, but they were never built.  Plus,
starttime and stoptime don't make sense.

--8<---------------cut here---------------start------------->8---
<civodul> snape: i just noticed something fishy:
	  https://berlin.guixsd.org/build/467599 is marked as failed
								        [11:24]
<civodul> however,
	  /gnu/store/nr0w89qymmnfbcb9vw9q86p0a4v1h6w2-libsvgtiny-0.1.7.drv has
	  never been built
<civodul> so it's not even a dependency-failed situation
<civodul> i wonder if this could be a regression in Cuirass
<civodul> it has "starttime":0,"stoptime":1535613271  [11:25]
<civodul> which normally cannot happen (starttime shouldn't be 0 i stoptime is
	  not 0)
<snape> civodul: indeed it's weird  [11:27]
<snape> civodul: how do you know
	/gnu/store/nr0w89qymmnfbcb9vw9q86p0a4v1h6w2-libsvgtiny-0.1.7.drv has
	never been built?  [11:28]
<snape> well, I guess because there is no log  [11:32]
<civodul> snape: yes and i ssh'd into berlin
<snape> civodul: it builds if you spawn it manually?  [11:37]
<civodul> snape: i didn't let it run to completion because i wanted us to be
	  able to debug it  [11:39]
<civodul> but most likely it does  [11:40]
<civodul> (i tested with
	  /gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv, which
	  was another build marked as failed)
--8<---------------cut here---------------end--------------->8---

Clément




Information forwarded to bug-guix <at> gnu.org:
bug#32583; Package guix. (Sat, 01 Sep 2018 23:36:02 GMT) Full text and rfc822 format available.

Message #8 received at 32583 <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: Ludovic Courtès <ludo <at> gnu.org>, 32583 <at> debbugs.gnu.org
Subject: Re: bug#32583: Cuirass: some builds are marked as failed,
 but they were never built
Date: Sun, 02 Sep 2018 01:35:05 +0200
Clément Lassieur <clement <at> lassieur.org> writes:

> Reported by Ludovic on IRC.
>
> Some builds are marked as failed, but they were never built.  Plus,
> starttime and stoptime don't make sense.
>
> --8<---------------cut here---------------start------------->8---
> <civodul> snape: i just noticed something fishy:
> 	  https://berlin.guixsd.org/build/467599 is marked as failed
> 								        [11:24]
> <civodul> however,
> 	  /gnu/store/nr0w89qymmnfbcb9vw9q86p0a4v1h6w2-libsvgtiny-0.1.7.drv has
> 	  never been built
> <civodul> so it's not even a dependency-failed situation
> <civodul> i wonder if this could be a regression in Cuirass
> <civodul> it has "starttime":0,"stoptime":1535613271  [11:25]
> <civodul> which normally cannot happen (starttime shouldn't be 0 i stoptime is
> 	  not 0)
> <snape> civodul: indeed it's weird  [11:27]
> <snape> civodul: how do you know
> 	/gnu/store/nr0w89qymmnfbcb9vw9q86p0a4v1h6w2-libsvgtiny-0.1.7.drv has
> 	never been built?  [11:28]
> <snape> well, I guess because there is no log  [11:32]
> <civodul> snape: yes and i ssh'd into berlin
> <snape> civodul: it builds if you spawn it manually?  [11:37]
> <civodul> snape: i didn't let it run to completion because i wanted us to be
> 	  able to debug it  [11:39]
> <civodul> but most likely it does  [11:40]
> <civodul> (i tested with
> 	  /gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv, which
> 	  was another build marked as failed)
> --8<---------------cut here---------------end--------------->8---
>
> Clément

This is from the logs:

--8<---------------cut here---------------start------------->8---
2018-08-30T09:14:31 batch of builds (partially) failed:build of
`/gnu/store/06yy5cl59baa6iczvcrl7gzbs82x5j30-cleanup.drv',
`/gnu/store/0ilx18r8s4rw00kxynn6nida1grggx46-separate-store-os.drv',
`/gnu/store/0n887pl0igppd7mr1hh2zrc889sfbh5h-dovecot-test.drv',
`/gnu/store/0qad25im46dnqwfcd63k2lljgd7h9gy2-dicod.drv',
`/gnu/store/0y5yb81mx5m98ddmbvhivhyszd5mh1bi-encrypted-root-os.drv',
`/gnu/store/1af46bw6km0qd6mvm63xsx10f0ipyy2x-nfs.drv',
`/gnu/store/1b28w2c9bgsjnypmnwyl5bnq7yc8k3pv-btrfs-root-os.drv',
`/gnu/store/1vv5wp52l1wbfg45b0a44fs9icx31m8f-encrypted-root-os.drv',
`/gnu/store/2149hi6f7gccn5x8cbgdx8w7sgadp1h0-basic.drv',
`/gnu/store/21i0ylg5p3q1s6qg395vyr55n64hkbil-separate-home-os.drv',
`/gnu/store/228h2p10ysd7n4cw3fwg1n7aj0k4xmrr-raid-root-os.drv',
`/gnu/store/2a4gsnndyslw7cwg9rjc5yhqhkpfag6w-libparserutils-0.2.4.drv',
`/gnu/store/2i8xq16s6bcwnxfpn7l6yq9jyip3ayxc-installed-extlinux-os.drv',
`/gnu/store/2lp48f2jdjampri0g9137qcz6ra4v5n3-mongodb-test.drv',
`/gnu/store/2zvk8ml5ankw0mbq6m693p4a9ppx7x83-exim-test.drv',
`/gnu/store/3dzssjfjzy5nngfm0cbg747lw2sddfvv-mcron.drv',
`/gnu/store/43pdwcgm49vs4grfzkw1n4mg7hhc0ih5-libvirt-test.drv',
`/gnu/store/4li2cm5igrcrkis8b4ilipf9823zy2h1-iso-image-installer.drv',
`/gnu/store/4w8hbibw77vxr5iwf1avgf0y95l4fwwd-libsvgtiny-0.1.7.drv',
`/gnu/store/56cz6yfy5s8w95kqxhgrry4iymx9zdbq-hubbub-0.3.5.drv',
  [...]
`/gnu/store/w83xmggaj5qxzfafnv1pyhff24s2c40m-installed-os.drv',
`/gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv',
`/gnu/store/wqpy6m5kf4wmnjhdqqwfyyx6n0nis06s-libdom-0.3.3.drv',
`/gnu/store/x2k166lqcmwad0mli8qsrql9a4vsrq45-installed-extlinux-os.drv',
`/gnu/store/x6v7bdvcvfbny1kdgsq6jilbszrfksbc-libcss-0.8.0.drv',
`/gnu/store/xg4dzxz54qpf678syvf4379vbps1wqhs-tailon-test.drv',
`/gnu/store/y08j1qivdlam0kp1w636f4lycnmm866v-disk-image.drv',
`/gnu/store/ya89cfswkgv7m3az3rzmbggy433lg288-libvirt-test.drv',
`/gnu/store/ya9q9x208fif1ci4pbsfm7vyk39znzmh-memcached-test.drv',
`/gnu/store/yhc1liighrcsgl1smpwrdahy3xrj4zd6-libnsutils-0.0.5.drv',
`/gnu/store/ykhlibxssbg42r2cfyzm65gkkjyqky7h-opensmtpd-test.drv',
`/gnu/store/yxv7ny3v4ng41rvhd0r245lnr4nwwch4-raid-root-os.drv',
`/gnu/store/zdakqvwx1na7p8c8ivacz4wlcm6pbdh6-dicod.drv'
failed (status: 100)
--8<---------------cut here---------------end--------------->8---

So it seems that the daemon tried to build them, but they all failed.
Probably at that point the log file weren't created.

I don't really understand why you were able to build
/gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv manually
then.

'starttime == 0 && stoptime != 0' is explained by the fact the exception
(from 'build-derivations&'?) was thrown before 'handle-build-event' was
called (which should have set starttime), and later on
'(update-build-statuses! store batch)' was called, which only sets
stoptime (because the output is invalid).

Any idea?

Clément




Information forwarded to bug-guix <at> gnu.org:
bug#32583; Package guix. (Sun, 02 Sep 2018 14:40:02 GMT) Full text and rfc822 format available.

Message #11 received at 32583 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Clément Lassieur <clement <at> lassieur.org>
Cc: 32583 <at> debbugs.gnu.org
Subject: Re: bug#32583: Cuirass: some builds are marked as failed,
 but they were never built
Date: Sun, 02 Sep 2018 16:39:49 +0200
Hi Clément,

Clément Lassieur <clement <at> lassieur.org> skribis:

> This is from the logs:
>
> 2018-08-30T09:14:31 batch of builds (partially) failed:build of
> `/gnu/store/06yy5cl59baa6iczvcrl7gzbs82x5j30-cleanup.drv',
> `/gnu/store/0ilx18r8s4rw00kxynn6nida1grggx46-separate-store-os.drv',
> `/gnu/store/0n887pl0igppd7mr1hh2zrc889sfbh5h-dovecot-test.drv',
> `/gnu/store/0qad25im46dnqwfcd63k2lljgd7h9gy2-dicod.drv',
> `/gnu/store/0y5yb81mx5m98ddmbvhivhyszd5mh1bi-encrypted-root-os.drv',
> `/gnu/store/1af46bw6km0qd6mvm63xsx10f0ipyy2x-nfs.drv',
> `/gnu/store/1b28w2c9bgsjnypmnwyl5bnq7yc8k3pv-btrfs-root-os.drv',
> `/gnu/store/1vv5wp52l1wbfg45b0a44fs9icx31m8f-encrypted-root-os.drv',
> `/gnu/store/2149hi6f7gccn5x8cbgdx8w7sgadp1h0-basic.drv',
> `/gnu/store/21i0ylg5p3q1s6qg395vyr55n64hkbil-separate-home-os.drv',
> `/gnu/store/228h2p10ysd7n4cw3fwg1n7aj0k4xmrr-raid-root-os.drv',
> `/gnu/store/2a4gsnndyslw7cwg9rjc5yhqhkpfag6w-libparserutils-0.2.4.drv',
> `/gnu/store/2i8xq16s6bcwnxfpn7l6yq9jyip3ayxc-installed-extlinux-os.drv',
> `/gnu/store/2lp48f2jdjampri0g9137qcz6ra4v5n3-mongodb-test.drv',
> `/gnu/store/2zvk8ml5ankw0mbq6m693p4a9ppx7x83-exim-test.drv',
> `/gnu/store/3dzssjfjzy5nngfm0cbg747lw2sddfvv-mcron.drv',
> `/gnu/store/43pdwcgm49vs4grfzkw1n4mg7hhc0ih5-libvirt-test.drv',
> `/gnu/store/4li2cm5igrcrkis8b4ilipf9823zy2h1-iso-image-installer.drv',
> `/gnu/store/4w8hbibw77vxr5iwf1avgf0y95l4fwwd-libsvgtiny-0.1.7.drv',
> `/gnu/store/56cz6yfy5s8w95kqxhgrry4iymx9zdbq-hubbub-0.3.5.drv',
>   [...]
> `/gnu/store/w83xmggaj5qxzfafnv1pyhff24s2c40m-installed-os.drv',
> `/gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv',
> `/gnu/store/wqpy6m5kf4wmnjhdqqwfyyx6n0nis06s-libdom-0.3.3.drv',
> `/gnu/store/x2k166lqcmwad0mli8qsrql9a4vsrq45-installed-extlinux-os.drv',
> `/gnu/store/x6v7bdvcvfbny1kdgsq6jilbszrfksbc-libcss-0.8.0.drv',
> `/gnu/store/xg4dzxz54qpf678syvf4379vbps1wqhs-tailon-test.drv',
> `/gnu/store/y08j1qivdlam0kp1w636f4lycnmm866v-disk-image.drv',
> `/gnu/store/ya89cfswkgv7m3az3rzmbggy433lg288-libvirt-test.drv',
> `/gnu/store/ya9q9x208fif1ci4pbsfm7vyk39znzmh-memcached-test.drv',
> `/gnu/store/yhc1liighrcsgl1smpwrdahy3xrj4zd6-libnsutils-0.0.5.drv',
> `/gnu/store/ykhlibxssbg42r2cfyzm65gkkjyqky7h-opensmtpd-test.drv',
> `/gnu/store/yxv7ny3v4ng41rvhd0r245lnr4nwwch4-raid-root-os.drv',
> `/gnu/store/zdakqvwx1na7p8c8ivacz4wlcm6pbdh6-dicod.drv'
> failed (status: 100)
>
> So it seems that the daemon tried to build them, but they all failed.
> Probably at that point the log file weren't created.
>
> I don't really understand why you were able to build
> /gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv manually
> then.

It could be that the failure above is due to an offloading timeout (I’ve
seen ‘guix offload’ hang occasionally before on berlin, and
a708de151c255712071e42e5c8284756b51768cd turned those into timeouts
reported as transient failures.)  So we may have a bug where things get
“stuck” for some reason.

> 'starttime == 0 && stoptime != 0' is explained by the fact the exception
> (from 'build-derivations&'?) was thrown before 'handle-build-event' was
> called (which should have set starttime), and later on
> '(update-build-statuses! store batch)' was called, which only sets
> stoptime (because the output is invalid).

I wonder if we should get rid of ‘update-build-statuses!’: if a build
fails, then we get a ‘build-failure’ event and we already update the
build status at that point, right?

WDYT?

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#32583; Package guix. (Tue, 20 Nov 2018 09:57:01 GMT) Full text and rfc822 format available.

Message #14 received at 32583 <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 32583 <at> debbugs.gnu.org
Subject: Re: bug#32583: Cuirass: some builds are marked as failed,
 but they were never built
Date: Tue, 20 Nov 2018 10:56:11 +0100
Sorry to reply that late.  I think
be489a26c0e6a5f23a48142a87728a0ec8bc3c9c fixed this.  Can we close it?

Thank you,
Clément




Reply sent to ludo <at> gnu.org (Ludovic Courtès):
You have taken responsibility. (Tue, 20 Nov 2018 12:45:01 GMT) Full text and rfc822 format available.

Notification sent to Clément Lassieur <clement <at> lassieur.org>:
bug acknowledged by developer. (Tue, 20 Nov 2018 12:45:01 GMT) Full text and rfc822 format available.

Message #19 received at 32583-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Clément Lassieur <clement <at> lassieur.org>
Cc: 32583-done <at> debbugs.gnu.org
Subject: Re: bug#32583: Cuirass: some builds are marked as failed,
 but they were never built
Date: Tue, 20 Nov 2018 13:44:12 +0100
Clément Lassieur <clement <at> lassieur.org> skribis:

> Sorry to reply that late.  I think
> be489a26c0e6a5f23a48142a87728a0ec8bc3c9c fixed this.  Can we close it?

Yep, done!

Ludo'.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Wed, 19 Dec 2018 12:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 5 years and 101 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.