GNU bug report logs - #76447
29.4; ert-run-tests-batch now aborts if a test signals an undefined error

Previous Next

To reply to this bug, email your comments to 76447 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Philipp <p.stephani2 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.4; ert-run-tests-batch now aborts if a test signals an undefined
 error
Date: Thu, 20 Feb 2025 21:26:41 +0100

With Emacs 29.4:

$ emacs -Q -batch -eval '(ert-deftest test-1 () (signal (quote foo) nil))' -eval '(ert-deftest test-2 ())' -f ert-run-tests-batch-and-exit
Running 2 tests (2025-02-20 21:13:09+0100, selector ‘t’)
Test test-1 backtrace:
  signal(foo nil)
  (closure (t) nil (signal 'foo nil))()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name test-1 :documentation nil :body (clos
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test 
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  command-line-1(("-eval" "(ert-deftest test-1 () (signal (quote foo) 
  command-line()
  normal-top-level()
Test test-1 condition:
    (foo)
   FAILED  1/2  test-1 (0.000060 sec) 
   passed  2/2  test-2 (0.000030 sec)

Ran 2 tests, 1 results as expected, 1 unexpected (2025-02-20 21:13:09+0100, 0.019570 sec)

1 unexpected results:
   FAILED  test-1


With Emacs 30.1 release candidate 1:

$ emacs -Q -batch -eval '(ert-deftest test-1 () (signal (quote foo) nil))' -eval '(ert-deftest test-2 ())' -f ert-run-tests-batch-and-exit
Running 2 tests (2025-02-20 21:13:03+0100, selector ‘t’)
Test test-1 aborted with non-local exit
  ABORTED  1/2  test-1 (0.000073 sec) 

Aborted: Ran 2 tests, 0 results as expected, 0 unexpected (2025-02-20 21:13:03+0100, 0.000140 sec)

Error running tests
  backtrace()
  #f(compiled-function () #<bytecode -0x174c28bd26a373a>)()
  ert-run-tests-batch-and-exit()
  command-line-1(("-eval" "(ert-deftest test-1 () (signal (quote foo) nil))" "-eval" "(ert-deftest test-2 ())" "-f" "ert-run-tests-batch-and-exit"))
  command-line()
  normal-top-level()


test-2 isn't executed.  I think the Emacs 29.4 behavior is more
"user-friendly", though this (signaling an error symbol that doesn't
derive from 'error') is arguably a corner case.

Probably the behavior change is fallout from switching to handler-bind.
To restore the Emacs 29.4 behavior, probably a construct such as

(condition-case err
    (handler-bind ((t ...)) ...)
  (t ...))

could be used.  The pseudo-condition t should be able to catch all
signals.


In GNU Emacs 29.4 (build 1, aarch64-apple-darwin24.3.0, NS
 appkit-2575.40 Version 15.3.1 (Build 24D70)) of 2025-02-20
Repository revision: 6a299b3caceb2c73b932ba73849738faa8c5d975
Repository branch: HEAD
Windowing system distributor 'Apple', version 10.3.2575
System Description:  macOS 15.3.1

Configured using:
 'configure --with-modules --with-xml2 --without-pop --with-mailutils
 'CFLAGS=-g -O3' LDFLAGS=-O3'

Configured features:
ACL GNUTLS JSON LCMS2 LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG
SQLITE3 THREADS TOOLKIT_SCROLL_BARS TREE_SITTER WEBP ZLIB

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug phst skeleton pcase ffap thingatpt url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util url-parse auth-source cl-seq
eieio eieio-core cl-macs json map byte-opt gv bytecomp byte-compile
url-vars rx message sendmail mailcap yank-media dired dired-loaddefs
rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config
gnus-util time-date mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader gnutls puny elp dbus xml subr-x compile
text-property-search comint ansi-osc ansi-color ring cl-loaddefs cl-lib
rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads kqueue cocoa ns lcms2
multi-tty make-network-process emacs)

Memory information:
((conses 16 69214 10345)
 (symbols 48 8174 0)
 (strings 32 23097 2385)
 (string-bytes 1 704838)
 (vectors 16 16338)
 (vector-slots 8 223298 13523)
 (floats 8 29 54)
 (intervals 56 229 0)
 (buffers 984 10))

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.