GNU bug report logs - #79056
tab-bar-tests-quit-restore-window sometimes fails in *compilation* buffer

Previous Next

To reply to this bug, email your comments to 79056 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Emacs bug reports and feature requests <bug-gnu-emacs <at> gnu.org>
Subject: tab-bar-tests-quit-restore-window sometimes fails in *compilation*
 buffer
Date: Sat, 19 Jul 2025 15:48:44 -0700

This is a timing related bug. To try to reproduce it on Ubuntu 25.04 
x86-64, I start up Emacs, then build Emacs from master (commit 
291cd2d3419c7be20200db3f0ee5936661ab79f2), then cd to the test 
subdirectory, then run:

M-x compile RET make check LOGFILES=lisp/tab-bar-tests.log RET

Sometimes it works, although it outputs a stray 
"^[[H^[[J^[[H^[[J^[[H^[[J^[[H^[[J" string (sans trailing newline) to 
stdout, where "^[" stands for ESC. This string is a minor bug in the 
test: it shouldn't output junk to stdout.

Sometimes it fails, resulting in the following *compilation* buffer, 
where "^[" again represents an ESC character.

The test always seems to work if I run it in a *shell* window, so 
apparently it has something with whether Emacs has a controlling tty, or 
something like that.

–-*- mode: compilation; default-directory: 
"~/src/gnu/emacs/static-checking/test/" -*-
Compilation started at Sat Jul 19 15:36:46

make check LOGFILES=lisp/tab-bar-tests.log
rm -f ./*.tmp
make[1]: Entering directory 
'/home/eggert/src/gnu/emacs/static-checking/test'
make[2]: Entering directory 
'/home/eggert/src/gnu/emacs/static-checking/test'
  GEN      lisp/tab-bar-tests.log
^[[H^[[JRunning 3 tests (2025-07-19 15:36:46-0700, selector ‘(not (or 
(tag :expensive-test) (tag :unstable) (tag :nativecomp)))’)
Renamed tab to ’1’
No more closed tabs to undo
   passed  1/3  tab-bar-tests-close-other-tabs-default (0.001130 sec)
No more closed tabs to undo
No more closed tabs to undo
No more closed tabs to undo
No more closed tabs to undo
No more closed tabs to undo
   passed  2/3  tab-bar-tests-close-other-tabs-with-arg (0.001069 sec)
Display next command buffer in a new frame...
Composing main Info directory...
Composing main Info directory...done
Test tab-bar-tests-quit-restore-window backtrace:
  signal(ert-test-failed (((should (eq (length (window-list)) 2)) :for
  ert-fail(((should (eq (length (window-list)) 2)) :form (eq 1 2) :val
  #f(compiled-function () #<bytecode -0x57ca7793bcf5d97>)()
  #f(compiled-function () #<bytecode 0xf90bd418f219aee>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0xf90bd418f219aee>
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name tab-bar-tests-quit-restore-window :do
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ... ...))
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable) (tag :n
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable) (
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "--eval" "(setq treesit-extra-l
  command-line()
  normal-top-level()
Test tab-bar-tests-quit-restore-window condition:
    (ert-test-failed
     ((should (eq (length ...) 2)) :form (eq 1 2) :value nil))
   FAILED  3/3  tab-bar-tests-quit-restore-window (0.017622 sec) at 
lisp/tab-bar-tests.el:54

Ran 3 tests, 2 results as expected, 1 unexpected (2025-07-19 
15:36:46-0700, 0.062801 sec)

1 unexpected results:
   FAILED  tab-bar-tests-quit-restore-window

make[2]: *** [Makefile:185: lisp/tab-bar-tests.log] Error 1
make[2]: Leaving directory '/home/eggert/src/gnu/emacs/static-checking/test'
make[1]: [Makefile:348: check-doit] Error 2 (ignored)

SUMMARY OF TEST RESULTS
-----------------------
Files examined: 1
Ran 3 tests, 2 results as expected, 1 unexpected, 0 skipped
1 files contained unexpected results:
  lisp/tab-bar-tests.log
make[1]: *** [Makefile:349: check-doit] Error 1
make[1]: Leaving directory '/home/eggert/src/gnu/emacs/static-checking/test'
make: *** [Makefile:315: check] Error 2

Compilation exited abnormally with code 2 at Sat Jul 19 15:36:47, 
duration 1.37 s

Message #8 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 79056 <at> debbugs.gnu.org
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Sun, 20 Jul 2025 09:56:41 +0300

> This is a timing related bug. To try to reproduce it on Ubuntu 25.04
> x86-64, I start up Emacs, then build Emacs from master (commit
> 291cd2d3419c7be20200db3f0ee5936661ab79f2), then cd to the test
> subdirectory, then run:
>
> M-x compile RET make check LOGFILES=lisp/tab-bar-tests.log RET
>
> Sometimes it works, although it outputs a stray
> "^[[H^[[J^[[H^[[J^[[H^[[J^[[H^[[J" string (sans trailing newline) to
> stdout, where "^[" stands for ESC. This string is a minor bug in the test:
> it shouldn't output junk to stdout.
>
> Sometimes it fails, resulting in the following *compilation* buffer, where
> "^[" again represents an ESC character.
>
> The test always seems to work if I run it in a *shell* window, so
> apparently it has something with whether Emacs has a controlling tty, or
> something like that.

Maybe tuning frame parameters would help here:

  (let* ((frame-params (when noninteractive
                         '((window-system . nil)
                           (tty-type . "linux"))))

I tried different tty-type values such as "ansi", "dumb", "vt100", "xterm"
with different effects.

Message #11 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79056 <at> debbugs.gnu.org
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Sun, 20 Jul 2025 17:07:08 -0700

On 2025-07-19 23:56, Juri Linkov wrote:
> Maybe tuning frame parameters would help here:
> 
>    (let* ((frame-params (when noninteractive
>                           '((window-system . nil)
>                             (tty-type . "linux"))))
> 
> I tried different tty-type values such as "ansi", "dumb", "vt100", "xterm"
> with different effects.

I don't know what the test is trying to do, but when it's noninteractive 
why does the test access /dev/tty or need TERM? /dev/tty and TERM 
ordinarily ought to be irrelevant and unused in a noninteractive test.

How about something like the following patch?

diff --git a/test/lisp/tab-bar-tests.el b/test/lisp/tab-bar-tests.el
index 257f6bd37d3..9529f43853b 100644
--- a/test/lisp/tab-bar-tests.el
+++ b/test/lisp/tab-bar-tests.el
@@ -52,15 +52,7 @@ tab-bar-tests-close-other-tabs-with-arg
   (tab-bar-tabs-set nil))

 (ert-deftest tab-bar-tests-quit-restore-window ()
-  (skip-when (pcase system-type
-               ;; Skip test on MS-Windows in batch mode, since terminal
-               ;; frames cannot be created in that case.
-               ('windows-nt noninteractive)
-               ;; Emba runs the container without "--tty"
-               ;; (the environment variable "TERM" is nil), and this
-               ;; test fails with '(error "Could not open file: 
/dev/tty")'.
-               ;; Therefore skip it unless it can use '(tty-type . 
"linux")'.
-               ('gnu/linux (null (getenv "TERM")))))
+  (skip-when (or noninteractive (null (getenv "TERM"))))

   (let* ((frame-params (when noninteractive
                          '((window-system . nil)

Message #14 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 79056 <at> debbugs.gnu.org
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Tue, 22 Jul 2025 09:38:35 +0300

> I don't know what the test is trying to do, but when it's noninteractive 
> why does the test access /dev/tty or need TERM? /dev/tty and TERM 
> ordinarily ought to be irrelevant and unused in a noninteractive test.
>
> How about something like the following patch?
>
> @@ -52,15 +52,7 @@ tab-bar-tests-close-other-tabs-with-arg
> -  (skip-when (pcase system-type
> -               ;; Skip test on MS-Windows in batch mode, since terminal
> -               ;; frames cannot be created in that case.
> -               ('windows-nt noninteractive)
> -               ;; Emba runs the container without "--tty"
> -               ;; (the environment variable "TERM" is nil), and this
> -               ;; test fails with '(error "Could not open file: /dev/tty")'.
> -               ;; Therefore skip it unless it can use '(tty-type . "linux")'.
> -               ('gnu/linux (null (getenv "TERM")))))
> +  (skip-when (or noninteractive (null (getenv "TERM"))))

If there is no reliable way to create terminal frames in noninteractive
tests, then probably such tests should be skipped by default to be used
only manually when someone will remember to run them periodically.

PS: Many web browsers support headless mode that is used in test automation.
I wonder how it would be possible to implement headless mode in Emacs?

Message #17 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79056 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Tue, 22 Jul 2025 16:07:50 +0300

> Cc: 79056 <at> debbugs.gnu.org
> From: Juri Linkov <juri <at> linkov.net>
> Date: Tue, 22 Jul 2025 09:38:35 +0300
> 
> > +  (skip-when (or noninteractive (null (getenv "TERM"))))
> 
> If there is no reliable way to create terminal frames in noninteractive
> tests, then probably such tests should be skipped by default to be used
> only manually when someone will remember to run them periodically.

Why not allow them in interactive runs, as the patch above does?

> PS: Many web browsers support headless mode that is used in test automation.
> I wonder how it would be possible to implement headless mode in Emacs?

It should be possible, if someone writes the headless back-end for the
display engine, akin to xterm.c+xfns.c for X, just much simpler.
(Probably starting with term.c should be easier.)

Patches welcome.

Message #20 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79056 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Wed, 23 Jul 2025 09:51:22 +0300

>> > +  (skip-when (or noninteractive (null (getenv "TERM"))))
>> 
>> If there is no reliable way to create terminal frames in noninteractive
>> tests, then probably such tests should be skipped by default to be used
>> only manually when someone will remember to run them periodically.
>
> Why not allow them in interactive runs, as the patch above does?

What are interactive runs?

Maybe I'm missing something, but I see two distinctions:

1. automatic tests that run on EMBA
2. manual tests that are invoked locally by e.g. `make -C test tab-bar-tests`

1. noninteractive tests that run without term or a connection to the X display
2. interactive tests that can create X frames, etc.

Automatic tests on EMBA can't be interactive without headless mode.
So what remains to do is to manually run interactive tests locally.
But IIUC `make -C test tab-bar-tests` is noninteractive.
And I don't know what command line could run them interactively,
so that tests could pop up X frames, etc.

>> PS: Many web browsers support headless mode that is used in test automation.
>> I wonder how it would be possible to implement headless mode in Emacs?
>
> It should be possible, if someone writes the headless back-end for the
> display engine, akin to xterm.c+xfns.c for X, just much simpler.

Probably in such headless mode X calls can be replaced with stubs or mocks
unless they are stateful.

Message #23 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: 79056 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Wed, 23 Jul 2025 14:44:05 +0300

> From: Juri Linkov <juri <at> linkov.net>
> Cc: eggert <at> cs.ucla.edu,  79056 <at> debbugs.gnu.org
> Date: Wed, 23 Jul 2025 09:51:22 +0300
> 
> >> > +  (skip-when (or noninteractive (null (getenv "TERM"))))
> >> 
> >> If there is no reliable way to create terminal frames in noninteractive
> >> tests, then probably such tests should be skipped by default to be used
> >> only manually when someone will remember to run them periodically.
> >
> > Why not allow them in interactive runs, as the patch above does?
> 
> What are interactive runs?

When you start Emacs, load the test file, then say

  M-x ert-run-tests-interactively RET

> >> I wonder how it would be possible to implement headless mode in Emacs?
> >
> > It should be possible, if someone writes the headless back-end for the
> > display engine, akin to xterm.c+xfns.c for X, just much simpler.
> 
> Probably in such headless mode X calls can be replaced with stubs or mocks
> unless they are stateful.

There should be no X calls at all.  Emacs only calls X functions when
the frame is an X frame.  In other case, the display code calls a
suitable method that corresponds to the frame kind.  See the
initialization of these methods in x_create_terminal and a similar
initialization for TTY frames in set_tty_hooks.  We need something
like that for the so-called "initial" frames created by the
noninteractive session.

Message #26 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Juri Linkov <juri <at> linkov.net>
Cc: Eli Zaretskii <eliz <at> gnu.org>, eggert <at> cs.ucla.edu, 79056 <at> debbugs.gnu.org
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Wed, 23 Jul 2025 16:46:05 +0200

Juri Linkov <juri <at> linkov.net> writes:

Hi Juri,

> Automatic tests on EMBA can't be interactive without headless mode.
> So what remains to do is to manually run interactive tests locally.
> But IIUC `make -C test tab-bar-tests` is noninteractive.
> And I don't know what command line could run them interactively,
> so that tests could pop up X frames, etc.

See test/README:

--8<---------------cut here---------------start------------->8---
The tests are run in batch mode by default; sometimes it's useful to
get precisely the same environment but run in interactive mode for
debugging.  To do that, use

    make TEST_INTERACTIVE=yes ...
--8<---------------cut here---------------end--------------->8---

(I haven't used this for years, so I don't know whether it still works)

Best regards, Michael.

Message #29 received at 79056 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, eggert <at> cs.ucla.edu, 79056 <at> debbugs.gnu.org
Subject: Re: bug#79056: tab-bar-tests-quit-restore-window sometimes fails in
 *compilation* buffer
Date: Wed, 23 Jul 2025 19:58:50 +0300

>> Automatic tests on EMBA can't be interactive without headless mode.
>> So what remains to do is to manually run interactive tests locally.
>> But IIUC `make -C test tab-bar-tests` is noninteractive.
>> And I don't know what command line could run them interactively,
>> so that tests could pop up X frames, etc.
>
> See test/README:
>
> The tests are run in batch mode by default; sometimes it's useful to
> get precisely the same environment but run in interactive mode for
> debugging.  To do that, use
>
>     make TEST_INTERACTIVE=yes ...
>
> (I haven't used this for years, so I don't know whether it still works)

Thanks for the reference.  I expected something like this should exist,
but didn't remember if it really existed.

Now I tried to run

  make TEST_INTERACTIVE=yes -C test tab-bar-tests

after applying the patch provided by Paul.

And interactive tests are passed successfully,
the output buffer displays:

  Passed:  3
  Failed:  0
  Skipped: 0
  Total:   3/3

Probably this can't be used on EMBA.  So for automatic tests
we could still try to add headless mode by implementing
a set of hooks like described by Eli.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.