GNU bug report logs - #47759
python-minimal tests hang

Previous Next

Package: guix;

Reported by: Danny Milosavljevic <dannym <at> scratchpost.org>

Date: Tue, 13 Apr 2021 20:39:01 UTC

Severity: normal

Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 47759 in the body.
You can then email your comments to 47759 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox

Report forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Tue, 13 Apr 2021 20:39:01 GMT) Full text and rfc822 format available.
Acknowledgement sent to Danny Milosavljevic <dannym <at> scratchpost.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Tue, 13 Apr 2021 20:39:01 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: <bug-guix <at> gnu.org>
Subject: python-minimal tests hang
Date: Tue, 13 Apr 2021 22:38:17 +0200

[Message part 1 (text/plain, inline)]
On x86_64, python-minimal build hangs when running the tests:

$ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
[...]
1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)

It's reproducible every time.

$ guix describe
Generation 230  Apr 13 2021 12:15:27    (current)
  guix 822eacc
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 822eacc6bb0878323e6687d4460a7c53066545e1
$ uname -a
Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux

[Message part 2 (application/pgp-signature, inline)]
Information forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Tue, 13 Apr 2021 21:31:02 GMT) Full text and rfc822 format available.

Message #8 received at 47759 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 47759 <at> debbugs.gnu.org
Subject: Re: bug#47759: python-minimal tests hang
Date: Tue, 13 Apr 2021 17:30:34 -0400

[Message part 1 (text/plain, inline)]
On Tue, Apr 13, 2021 at 10:38:17PM +0200, Danny Milosavljevic wrote:
> On x86_64, python-minimal build hangs when running the tests:
> 
> $ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
> [...]
> 1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)

Where does this derivation come from? I can't seem to find it on the
machines I have access to.

> $ guix describe
> Generation 230  Apr 13 2021 12:15:27    (current)
>   guix 822eacc
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ uname -a
> Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux

------
$ guix describe
Generation 15   Apr 13 2021 23:26:10    (current)
  guix 822eacc
    repository URL: https://git.savannah.gnu.org/git/guix.git
    commit: 822eacc6bb0878323e6687d4460a7c53066545e1
$ guix build --derivations --no-grafts python-minimal 
/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
$ guix build --derivations --no-grafts python-minimal-wrapper
/gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
------

[signature.asc (application/pgp-signature, inline)]
Information forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Tue, 20 Apr 2021 12:07:01 GMT) Full text and rfc822 format available.

Message #11 received at 47759 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Leo Famulari <leo <at> famulari.name>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>,
 GNU Debbugs <control <at> debbugs.gnu.org>, 47759 <at> debbugs.gnu.org
Subject: Re: bug#47759: python-minimal tests hang
Date: Tue, 20 Apr 2021 08:05:53 -0400

block 47297 by 47759
thanks

Hi!

Leo Famulari <leo <at> famulari.name> writes:

> On Tue, Apr 13, 2021 at 10:38:17PM +0200, Danny Milosavljevic wrote:
>> On x86_64, python-minimal build hangs when running the tests:
>> 
>> $ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
>> [...]
>> 1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)
>
> Where does this derivation come from? I can't seem to find it on the
> machines I have access to.
>
>> $ guix describe
>> Generation 230  Apr 13 2021 12:15:27    (current)
>>   guix 822eacc
>>     repository URL: https://git.savannah.gnu.org/git/guix.git
>>     branch: master
>>     commit: 822eacc6bb0878323e6687d4460a7c53066545e1
>> $ uname -a
>> Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux
>
> ------
> $ guix describe
> Generation 15   Apr 13 2021 23:26:10    (current)
>   guix 822eacc
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ guix build --derivations --no-grafts python-minimal 
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix build --derivations --no-grafts python-minimal-wrapper
> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
> ------

I can also reproduce this on the version-1.3.0, building for
armhf-linux.  Another release blocker :-/.

Thanks,

Maxim
Added indication that bug 47759 blocks47297 Request was from Maxim Cournoyer <maxim.cournoyer <at> gmail.com> to control <at> debbugs.gnu.org. (Tue, 20 Apr 2021 12:07:02 GMT) Full text and rfc822 format available.
Information forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Thu, 22 Apr 2021 20:30:02 GMT) Full text and rfc822 format available.

Message #16 received at 47759 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 47759 <at> debbugs.gnu.org
Subject: Re: bug#47759: python-minimal tests hang
Date: Thu, 22 Apr 2021 22:29:08 +0200

Hi!

Leo Famulari <leo <at> famulari.name> skribis:

> $ guix describe
> Generation 15   Apr 13 2021 23:26:10    (current)
>   guix 822eacc
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ guix build --derivations --no-grafts python-minimal 
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix build --derivations --no-grafts python-minimal-wrapper
> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv

FWIW, berlin has binaries for x86_64 and armhf (the two you mentioned):

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux -d
/gnu/store/y9g688q1ffk6gbyiwda3pw5wb29q0a2x-python-minimal-3.8.2.drv
/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux 
/gnu/store/c7s4wslhi6idzprnnigabw8bz4lp7gn8-python-minimal-3.8.2
/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- weather python-minimal -s x86_64-linux -s armhf-linux 
computing 1 package derivations for armhf-linux...
computing 1 package derivations for x86_64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  100.0% substitutes available (2 out of 2)
  at least 67.2 MiB of nars (compressed)
  153.0 MiB on disk (uncompressed)
  0.085 seconds per request (0.2 seconds in total)
  11.8 requests per second

  at least 1,000 queued builds
      aarch64-linux: 1000 (100.0%)
  build rate: 36.16 builds per hour
      x86_64-linux: 8.38 builds per hour
      i686-linux: 20.25 builds per hour
      aarch64-linux: 8.93 builds per hour
--8<---------------cut here---------------end--------------->8---

I also rebuilt it locally on my x86_64 laptop and it ran to completion
(although there’s a bitwise reproducibility issue):

--8<---------------cut here---------------start------------->8---
$ time guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --check --no-grafts -v1
La jena derivo estos konstruata:
   /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv

building /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv...
- 'compress-documentation' phaseguix build: error: derivation `/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv' may not be deterministic: output `/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2' differs

real    9m45.336s
user    0m3.490s
sys     0m0.289s
--8<---------------cut here---------------end--------------->8---

So… no problem?  Or perhaps it’s a function of the number of cores or
something?  (Here that’s with 2 cores/4 threads.)

Ludo’.
Information forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Thu, 22 Apr 2021 21:02:02 GMT) Full text and rfc822 format available.

Message #19 received at 47759 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>, 47759 <at> debbugs.gnu.org,
 Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#47759: python-minimal tests hang
Date: Thu, 22 Apr 2021 17:01:04 -0400

Hi,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi!
>
> Leo Famulari <leo <at> famulari.name> skribis:
>
>> $ guix describe
>> Generation 15   Apr 13 2021 23:26:10    (current)
>>   guix 822eacc
>>     repository URL: https://git.savannah.gnu.org/git/guix.git
>>     commit: 822eacc6bb0878323e6687d4460a7c53066545e1
>> $ guix build --derivations --no-grafts python-minimal 
>> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
>> $ guix build --derivations --no-grafts python-minimal-wrapper
>> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
>
> FWIW, berlin has binaries for x86_64 and armhf (the two you mentioned):
>
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux -d
> /gnu/store/y9g688q1ffk6gbyiwda3pw5wb29q0a2x-python-minimal-3.8.2.drv
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux 
> /gnu/store/c7s4wslhi6idzprnnigabw8bz4lp7gn8-python-minimal-3.8.2
> /gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- weather python-minimal -s x86_64-linux -s armhf-linux 
> computing 1 package derivations for armhf-linux...
> computing 1 package derivations for x86_64-linux...
> looking for 2 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   100.0% substitutes available (2 out of 2)
>   at least 67.2 MiB of nars (compressed)
>   153.0 MiB on disk (uncompressed)
>   0.085 seconds per request (0.2 seconds in total)
>   11.8 requests per second
>
>   at least 1,000 queued builds
>       aarch64-linux: 1000 (100.0%)
>   build rate: 36.16 builds per hour
>       x86_64-linux: 8.38 builds per hour
>       i686-linux: 20.25 builds per hour
>       aarch64-linux: 8.93 builds per hour
>
>
> I also rebuilt it locally on my x86_64 laptop and it ran to completion
> (although there’s a bitwise reproducibility issue):
>
> $ time guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --check --no-grafts -v1
> La jena derivo estos konstruata:
>    /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
>
> building /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv...
> - 'compress-documentation' phaseguix build: error: derivation `/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv' may not be deterministic: output `/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2' differs
>
> real    9m45.336s
> user    0m3.490s
> sys     0m0.289s
>
> So… no problem?  Or perhaps it’s a function of the number of cores or
> something?  (Here that’s with 2 cores/4 threads.)

For me it seems to happen only when using QEMU transparent emulation,
e.g. to build it for armhf-linux on an amd64 machine.

Danny, are you still able to reproduce this problem from the master
branch, without using QEMU emulation?

Thank you,

Maxim
Information forwarded to bug-guix <at> gnu.org:
bug#47759; Package guix. (Sat, 01 May 2021 03:46:01 GMT) Full text and rfc822 format available.

Message #22 received at 47759 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>,
 GNU Debbugs <control <at> debbugs.gnu.org>, 47759 <at> debbugs.gnu.org,
 Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#47759: python-minimal tests hang
Date: Fri, 30 Apr 2021 23:45:30 -0400

unblock 47297 by 47759
thanks

A first RC1 release candidate with the 5 supported architectures could
be produced without this bug being a blocker.  The key was in using real
hardware rather than QEMU emulation.

Unblocking.

Maxim
Removed indication that bug 47759 blocks Request was from Maxim Cournoyer <maxim.cournoyer <at> gmail.com> to control <at> debbugs.gnu.org. (Sat, 01 May 2021 03:46:02 GMT) Full text and rfc822 format available.
Reply sent to Maxim Cournoyer <maxim.cournoyer <at> gmail.com>:
You have taken responsibility. (Thu, 14 Jul 2022 04:09:03 GMT) Full text and rfc822 format available.
Notification sent to Danny Milosavljevic <dannym <at> scratchpost.org>:
bug acknowledged by developer. (Thu, 14 Jul 2022 04:09:03 GMT) Full text and rfc822 format available.

Message #29 received at 47759-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>,
 GNU Debbugs <control <at> debbugs.gnu.org>, 47759-done <at> debbugs.gnu.org,
 Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#47759: python-minimal tests hang
Date: Thu, 14 Jul 2022 00:08:01 -0400

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> unblock 47297 by 47759
> thanks
>
> A first RC1 release candidate with the 5 supported architectures could
> be produced without this bug being a blocker.  The key was in using real
> hardware rather than QEMU emulation.
>
> Unblocking.

Closing, as I don't think tracking QEMU-induced build problems is in
scope for our bug tracker.

Thanks,

Maxim
bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 11 Aug 2022 11:24:11 GMT) Full text and rfc822 format available.
This bug report was last modified 1 year and 255 days ago.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.