GNU bug report logs - #24503
unstable test results (on hydra)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 24503 in the body.
You can then email your comments to 24503 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #3 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: submit <at> debbugs.gnu.org
Subject: unstable test results (on hydra)
Date: Wed, 21 Sep 2016 16:22:24 -0400

Package: emacs
Version: 25.1

On hydra (at least), some tests fail in one iteration and succeed in the
next, for no clear reason. This cause noise on the emacs-buildstatus
mailing list and makes it harder to spot real problems.

Someone (TM) should look into this and try to figure out if the tests
really have a problem, or if the problematic ones should just be
disabled on hydra.

eieio-test-37-obsolete-name-in-constructor is one I have noticed several
times.

Failed: http://hydra.nixos.org/build/40880158
Passed: http://hydra.nixos.org/build/40890883

There were no relevant changes between those builds.

Message #6 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 11 Jan 2017 14:41:34 -0500

Hi Stefan,

Do you have any idea why eieio-test-37-obsolete-name-in-constructor
sometimes (randomly?) fails with:

(eieio--testing "toto") -> [eieio-class-tag--eieio--testing]

rather than ("toto" 2) ?

Eg http://hydra.nixos.org/build/46213612/log/raw

Message #9 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 11 Jan 2017 16:45:35 -0500

> Do you have any idea why eieio-test-37-obsolete-name-in-constructor
> sometimes (randomly?) fails with:
> (eieio--testing "toto") -> [eieio-class-tag--eieio--testing]
> rather than ("toto" 2) ?

Hmm... not off the top of my head no.  I'll have to dig deeper to give
you a better answer,


        Stefan

Message #12 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 11 Jan 2017 17:09:53 -0500

I've failed to reproduce this issue myself in 1000+ trials. So maybe
it is some hydra-specific weirdness, though I can't see how.
If all else fails, the test can just be skipped there.

Message #15 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 11 Jan 2017 18:06:46 -0500

> I've failed to reproduce this issue myself in 1000+ trials. So maybe
> it is some hydra-specific weirdness, though I can't see how.
> If all else fails, the test can just be skipped there.

It could be due to a parallel-make issue where sometimes we load a .el
file and sometimes we load a .elc file depending on the relative order
of execution of the various parallel commands.


        Stefan "who still haven't looked at the actual code, so is just
                talking about very general ideas"

Message #18 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 11 Jan 2017 18:16:39 -0500

Stefan Monnier wrote:

> It could be due to a parallel-make issue where sometimes we load a .el
> file and sometimes we load a .elc file depending on the relative order
> of execution of the various parallel commands.

I think hydra builds are single-core.
But obviously we need to wait till you can have a proper look. :)

Message #21 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 19 Apr 2017 17:38:48 -0400

Another one I've seen intermittently fail:

Test eieio-test-method-order-list-6 condition:
    (error "eieio-test-methodinvoke.el: Test Failed:
    ((:STATIC C) (:STATIC C-base1) (:STATIC C-base2)) != ((:STATIC C))")

Eg http://hydra.nixos.org/build/51808781

Message #24 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Tue, 28 Nov 2017 12:57:37 -0500

Tests that "randomly" fail, like the ones mentioned in this report, are
a problem for downstream distributions that run "make check" and pay
attention to the results (I sometimes think they pay more attention than
Emacs does). They can waste their time investigating things that seem to
be platform-specific just because they happen to fail once on a
particular architecture.

Eg
https://bugs.debian.org/879020

So we do a disservice to the wider community with these tests that
sometimes fail. If they aren't good enough to be enabled on Emacs's own
CI system, they probably aren't good enough full-stop, and should be
disabled by default for everyone.

Message #27 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Wed, 29 Nov 2017 08:35:58 -0500

Glenn Morris <rgm <at> gnu.org> writes:

> Tests that "randomly" fail, like the ones mentioned in this report, are
> a problem for downstream distributions that run "make check" and pay
> attention to the results (I sometimes think they pay more attention than
> Emacs does). They can waste their time investigating things that seem to
> be platform-specific just because they happen to fail once on a
> particular architecture.
>
> Eg
> https://bugs.debian.org/879020
>
> So we do a disservice to the wider community with these tests that
> sometimes fail. If they aren't good enough to be enabled on Emacs's own
> CI system, they probably aren't good enough full-stop, and should be
> disabled by default for everyone.

Hmm, I can't see any reason for these tests to fail in a
non-deterministic way.  This sounds like it could be a serious bug.  And
if happens on machines other than the Hydra ones, the bug is almost
certainly in Emacs.  I think we need to add some logging statements to
figure out what is going.  I will try to figure how to do that.

Message #34 received at 24503-done <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: 24503-done <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Tue, 09 Jan 2018 23:02:05 -0500

Version: 26.0.91

I've tagged the relevant tests as unstable and skipped them by default.
I will tag more tests as unstable if and when they come up.

Message #42 received at 24503 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 24503 <at> debbugs.gnu.org
Subject: Re: bug#24503: unstable test results (on hydra)
Date: Tue, 09 Jan 2018 23:21:02 -0500

Noam Postavsky <npostavs <at> users.sourceforge.net> writes:

> Hmm, I can't see any reason for these tests to fail in a
> non-deterministic way.  This sounds like it could be a serious bug.  And
> if happens on machines other than the Hydra ones, the bug is almost
> certainly in Emacs.  I think we need to add some logging statements to
> figure out what is going.  I will try to figure how to do that.

I didn't see any failures since I added tracing code [1: cfa50d30f2],
probably the tracing itself changed something which somehow prevents the
bug from triggering.

[1: cfa50d30f2]: 2017-12-03 15:39:02 -0500
  ; Tracing for eieio-test random failure (Bug#24503)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=cfa50d30f2876ca30158082e9a91d19e804a7e09

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.