GNU bug report logs - #38062
Offloading + timeout + --keep-going leads to assertion failure

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 38062 in the body.
You can then email your comments to 38062 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-Guix <at> gnu.org
Subject: Offloading + timeout + --keep-going leads to assertion failure
Date: Mon, 04 Nov 2019 21:45:47 +0100

Hello,

An offloaded build that times out in the presence of ‘--keep-going’
leads to an assertion failure.  To reproduce, set up offloading and run
something like this:

--8<---------------cut here---------------start------------->8---
$ guix build vim --no-substitutes --timeout=5 --keep-going
The following derivation will be built:
   /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv
process 4277 acquired build slot '/var/guix/offload/localhost:2222/0'
load on machine 'localhost' is 0.04 (normalized: 0.01)
building /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv...
sending 1 store item (48 MiB) to 'localhost'...
exporting path `/gnu/store/mlwyk5vcja0gqm20xxj8mwgf0fbqv8cz-vim-8.1.0644-checkout'
building of `/gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv' timed out after 5 seconds
build of /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv failed
View build log at '/var/log/guix/drvs/5m/nnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv.bz2'.
guix build: error: corrupt input while restoring archive from #<closed: file 7ff2769cdc40>
--8<---------------cut here---------------end--------------->8---

The last error is due to premature EOF on the client socket: the child
‘guix-daemon’ process crashed, and thus ‘guix build’ gets EPIPE on its
client socket.  Here’s how the ‘guix-daemon’ process crashed:

--8<---------------cut here---------------start------------->8---
$ tail -3 /var/log/guix-daemon.log
accepted connection from pid 4270, user ludo
accepted connection from pid 4277, user root
guix-daemon: nix/libstore/build.cc:3448: void nix::Worker::run(const Goals&): Assertion `!settings.keepGoing || children.empty()' failed.
--8<---------------cut here---------------end--------------->8---

In Cuirass, this manifests itself with EPIPE while writing to the client
socket:

--8<---------------cut here---------------start------------->8---
2019-11-04T20:11:26 fatal: uncaught exception 'system-error' in 'restart-builds' fiber!
2019-11-04T20:11:26 exception arguments: ("fport_write" "~A" ("Broken pipe") (32))
In ice-9/boot-9.scm:
    829:9  5 (catch _ _ #<procedure 7f79d663c320 at cuirass/utils.s?> ?)
   751:25  4 (dispatch-exception 0 system-error ("fport_write" "~A" ?))
In cuirass/utils.scm:
    183:8  3 (_ _ "fport_write" "~A" ("Broken pipe") (32))
In ice-9/boot-9.scm:
    829:9  2 (catch #t #<procedure 7f79cc7e41c0 at cuirass/utils.sc?> ?)
In cuirass/utils.scm:
   184:22  1 (_)
In unknown file:
           0 (make-stack #t)
ERROR: In procedure make-stack:
In procedure fport_write: Broken pipe
--8<---------------cut here---------------end--------------->8---

This bug is almost certainly a consequence of commit
ada9a19a2dca74feafcf24df1152abd685d4142f.

To be continued…

Ludo’.

Message #10 received at 38062-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 38062-done <at> debbugs.gnu.org
Subject: Re: bug#38062: Offloading + timeout + --keep-going leads to assertion
 failure
Date: Mon, 04 Nov 2019 23:38:12 +0100

Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> An offloaded build that times out in the presence of ‘--keep-going’
> leads to an assertion failure.  To reproduce, set up offloading and run
> something like this:
>
> $ guix build vim --no-substitutes --timeout=5 --keep-going
> The following derivation will be built:
>    /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv
> process 4277 acquired build slot '/var/guix/offload/localhost:2222/0'
> load on machine 'localhost' is 0.04 (normalized: 0.01)
> building /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv...
> sending 1 store item (48 MiB) to 'localhost'...
> exporting path `/gnu/store/mlwyk5vcja0gqm20xxj8mwgf0fbqv8cz-vim-8.1.0644-checkout'
> building of `/gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv' timed out after 5 seconds
> build of /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv failed
> View build log at '/var/log/guix/drvs/5m/nnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv.bz2'.
> guix build: error: corrupt input while restoring archive from #<closed: file 7ff2769cdc40>
>
>
> The last error is due to premature EOF on the client socket: the child
> ‘guix-daemon’ process crashed, and thus ‘guix build’ gets EPIPE on its
> client socket.  Here’s how the ‘guix-daemon’ process crashed:
>
> $ tail -3 /var/log/guix-daemon.log
> accepted connection from pid 4270, user ludo
> accepted connection from pid 4277, user root
> guix-daemon: nix/libstore/build.cc:3448: void nix::Worker::run(const Goals&): Assertion `!settings.keepGoing || children.empty()' failed.

Fixed by af73beeba1fc9effab60b11aea1d7ed8c24e7367.

I’ll update the ‘guix’ package soonish.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.