GNU bug report logs - #22471
ert batch should print compile parsable error messages

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22471 in the body.
You can then email your comments to 22471 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: phillip.lord <at> russet.org.uk (Phillip Lord)
To: bug-gnu-emacs <at> gnu.org
Subject: ert batch should print compile parsable error messages
Date: Wed, 27 Jan 2016 09:15:44 +0000

Currently, when running in batch, ert prints out messages like so:


Running 24 tests (2016-01-27 09:05:17+0000)
   passed   1/24  buffer-string=
   passed   2/24  buffer=
...
   passed  22/24  sisyphus-test-with-find-file
   passed  23/24  to-string
   passed  24/24  with-temp-buffers

Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)

1 unexpected results:
   FAILED  crash-out

It would be nice to add some compilation-mode parsable data to this. So,
something like


   passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

It would also be nice to add colourisation: red and green for passed and
failed. In the idea world, these could be added as ansi colour codes to
the batch output of ERT. They should work on the command line then also.

More of an RFE than a bug, but it is odd that compilation mode works
with lots and lots of test frameworks, but not Emacs'

Phil

Message #8 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: phillip.lord <at> russet.org.uk (Phillip Lord)
Cc: 22471 <at> debbugs.gnu.org
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Wed, 27 Jan 2016 18:06:04 +0200

> From: phillip.lord <at> russet.org.uk (Phillip Lord)
> Date: Wed, 27 Jan 2016 09:15:44 +0000
> 
> 
> Currently, when running in batch, ert prints out messages like so:
> 
> 
> Running 24 tests (2016-01-27 09:05:17+0000)
>    passed   1/24  buffer-string=
>    passed   2/24  buffer=
> ...
>    passed  22/24  sisyphus-test-with-find-file
>    passed  23/24  to-string
>    passed  24/24  with-temp-buffers
> 
> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
> 
> 1 unexpected results:
>    FAILED  crash-out
> 
> It would be nice to add some compilation-mode parsable data to this. So,
> something like
> 
> 
>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

FWIW, I have never seen a test suite that displayed in
compilation-mode compatible format.  If we do that (and I'm
necessarily opposed to that), we'd be the odd one out, I think.

Message #11 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: phillip.lord <at> russet.org.uk (Phillip Lord)
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 22471 <at> debbugs.gnu.org
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Wed, 27 Jan 2016 21:53:58 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: phillip.lord <at> russet.org.uk (Phillip Lord)
>> Date: Wed, 27 Jan 2016 09:15:44 +0000
>> 
>> 
>> Currently, when running in batch, ert prints out messages like so:
>> 
>> 
>> Running 24 tests (2016-01-27 09:05:17+0000)
>>    passed   1/24  buffer-string=
>>    passed   2/24  buffer=
>> ...
>>    passed  22/24  sisyphus-test-with-find-file
>>    passed  23/24  to-string
>>    passed  24/24  with-temp-buffers
>> 
>> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>> 
>> 1 unexpected results:
>>    FAILED  crash-out
>> 
>> It would be nice to add some compilation-mode parsable data to this. So,
>> something like
>> 
>> 
>>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> FWIW, I have never seen a test suite that displayed in
> compilation-mode compatible format.  If we do that (and I'm
> necessarily opposed to that), we'd be the odd one out, I think.

There's a couple in the compilation mode regexp -- cucumber, ruby and
perl test harnesses and so forth.

I put a quick attempt at this up on feature/parsable-ert-output.

   passed  22/24  sisyphus-test-with-find-file at test/sisyphus-test.el line 277.
   passed  23/24  to-string at test/sisyphus-test.el line 96.
   passed  24/24  with-temp-buffers at test/sisyphus-test.el line 232.

Works, but makes for rather a cluttered output.

Thoughts welcome.

Phil

Message #14 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: phillip.lord <at> russet.org.uk (Phillip Lord)
Cc: Eli Zaretskii <eliz <at> gnu.org>, 22471 <at> debbugs.gnu.org
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Wed, 27 Jan 2016 20:28:22 -0500

Why would you call the batch version of ert from within Emacs,
when M-x ert exists and does colour, hyperlinking to source, etc?

Message #17 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: phillip.lord <at> russet.org.uk (Phillip Lord)
To: Glenn Morris <rgm <at> gnu.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 22471 <at> debbugs.gnu.org
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Thu, 28 Jan 2016 17:14:57 +0000

Glenn Morris <rgm <at> gnu.org> writes:

> Why would you call the batch version of ert from within Emacs,
> when M-x ert exists and does colour, hyperlinking to source, etc?

M-x ert is nice, but has one main failure. It's not running in a clean,
controllable Emacs; it's running in my Emacs. I use M-x ert only when I
am debugging failing tests.

The rest of the time, I run tests in a clean, newly launched Emacs with
a sandboxed .emacs.d/elpa. As a secondary advantage, it can be easier to
test two or three Emacs versions at the same time.

I've been writing a library to help get around this problem, but
fundamentally, the only clean way to test code is not to run it in the
same Emacs as is launching it. At some point, I'd like to add the
ability to launch an emacs for every test.

Phil

Message #20 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: phillip.lord <at> russet.org.uk (Phillip Lord)
Cc: 22471 <at> debbugs.gnu.org
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Thu, 20 Aug 2020 18:39:45 +0200

phillip.lord <at> russet.org.uk (Phillip Lord) writes:

> Currently, when running in batch, ert prints out messages like so:
>
> Running 24 tests (2016-01-27 09:05:17+0000)
>    passed   1/24  buffer-string=
>    passed   2/24  buffer=
> ...
>    passed  22/24  sisyphus-test-with-find-file
>    passed  23/24  to-string
>    passed  24/24  with-temp-buffers
>
> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>
> 1 unexpected results:
>    FAILED  crash-out
>
> It would be nice to add some compilation-mode parsable data to this. So,
> something like
>
>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)

I think that would look rather cluttered for the "passed" lines, but it
would indeed be helpful on the FAILED lines.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #23 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Thu, 20 Aug 2020 18:51:32 +0200

Am Do., 20. Aug. 2020 um 18:41 Uhr schrieb Lars Ingebrigtsen <larsi <at> gnus.org>:
>
> phillip.lord <at> russet.org.uk (Phillip Lord) writes:
>
> > Currently, when running in batch, ert prints out messages like so:
> >
> > Running 24 tests (2016-01-27 09:05:17+0000)
> >    passed   1/24  buffer-string=
> >    passed   2/24  buffer=
> > ...
> >    passed  22/24  sisyphus-test-with-find-file
> >    passed  23/24  to-string
> >    passed  24/24  with-temp-buffers
> >
> > Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
> >
> > 1 unexpected results:
> >    FAILED  crash-out
> >
> > It would be nice to add some compilation-mode parsable data to this. So,
> > something like
> >
> >    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> I think that would look rather cluttered for the "passed" lines, but it
> would indeed be helpful on the FAILED lines.
>

I'm doing something similar in my Bazel ruleset for Elisp
(https://github.com/phst/rules_elisp/blob/0b24aa1660af2f6c668899bdd78aaba383d7ac18/elisp/ert/runner.el#L446).
It would be nice to have an API for this as my code needs a few hacks
and subtleties.

Message #26 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Fri, 22 Oct 2021 17:15:35 -0700

[Message part 1 (text/plain, inline)]

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> phillip.lord <at> russet.org.uk (Phillip Lord) writes:
>
>> Currently, when running in batch, ert prints out messages like so:
>>
>> Running 24 tests (2016-01-27 09:05:17+0000)
>>    passed   1/24  buffer-string=
>>    passed   2/24  buffer=
>> ...
>>    passed  22/24  sisyphus-test-with-find-file
>>    passed  23/24  to-string
>>    passed  24/24  with-temp-buffers
>>
>> Ran 24 tests, 23 results as expected, 1 unexpected (2016-01-27 09:05:18+0000)
>>
>> 1 unexpected results:
>>    FAILED  crash-out
>>
>> It would be nice to add some compilation-mode parsable data to this. So,
>> something like
>>
>>    passed   1/24  buffer-string=  (in ./test.sisyphus-test.el:22:)
>
> I think that would look rather cluttered for the "passed" lines, but it
> would indeed be helpful on the FAILED lines.

This would in my opinion be very useful and a great timesaver.  But yes,
for FAILED results only, much like how the compiler only gives us line
numbers if there is a problem.  (AFAIR, that's how the test frameworks
I've used do it.)

So I updated the patch from the feature/parsable-ert-output against
current master (patch attached), made sure that one of our tests failed,
and then tested it with "make check".

I got the following backtrace, but gave up after looking at it for a
while:

Debugger entered--Lisp error: (error "Don’t know where
‘fns-tests-string-bytes’ is defin...")
  signal(error ("Don’t know where ‘fns-tests-string-bytes’ is defin..."))
  error("Don't know where `%s' is defined" fns-tests-string-bytes)
  find-function-search-for-symbol(fns-tests-string-bytes ert-deftest nil)
  find-definition-noselect(fns-tests-string-bytes ert-deftest)
  ert-test-location([...])
[...]

[0001-Add-source-information-to-ERT-batch-mode.patch (text/x-diff, attachment)]

Message #29 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Sun, 24 Oct 2021 14:57:40 +0200

Stefan Kangas <stefan <at> marxist.se> writes:

> I got the following backtrace, but gave up after looking at it for a
> while:
>
> Debugger entered--Lisp error: (error "Don’t know where
> ‘fns-tests-string-bytes’ is defin...")
>   signal(error ("Don’t know where ‘fns-tests-string-bytes’ is defin..."))

Well -- ert-deftest doesn't record what file it's from, I think?
Looking at the definition, it just adds the test to the symbol-plist.
So that macro would also have to stash the file name somewhere.
load-history would be possible (but sounds unclean), so stashing it in
the symbol-plist would be better.

Then find-definition-noselect could be extended to use that for this
type.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #32 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Sun, 24 Oct 2021 07:27:31 -0700

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Well -- ert-deftest doesn't record what file it's from, I think?
> Looking at the definition, it just adds the test to the symbol-plist.
> So that macro would also have to stash the file name somewhere.
> load-history would be possible (but sounds unclean), so stashing it in
> the symbol-plist would be better.
>
> Then find-definition-noselect could be extended to use that for this
> type.

Right, I'm just wondering what has changed as presumably the patch
worked when Phillip Lord first wrote it?

Message #35 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Sun, 24 Oct 2021 22:01:17 +0200

Stefan Kangas <stefan <at> marxist.se> writes:

> Right, I'm just wondering what has changed as presumably the patch
> worked when Phillip Lord first wrote it?

Oh, I interpreted the branch as a sketch towards a solution but I see
that he posted apparent output from it, too:

> I put a quick attempt at this up on feature/parsable-ert-output.
> 
>    passed 22/24 sisyphus-test-with-find-file at test/sisyphus-test.el
> line 277.
>    passed  23/24  to-string at test/sisyphus-test.el line 96.
>    passed  24/24  with-temp-buffers at test/sisyphus-test.el line 232.

So it worked at one point?  (I'm not able to compile that branch at
all -- it segfaults while bootstrapping.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #38 received at 22471 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: 22471 <at> debbugs.gnu.org, Phillip Lord <phillip.lord <at> russet.org.uk>
Subject: Re: bug#22471: ert batch should print compile parsable error messages
Date: Mon, 24 Jan 2022 13:54:17 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

>> Right, I'm just wondering what has changed as presumably the patch
>> worked when Phillip Lord first wrote it?
>
> Oh, I interpreted the branch as a sketch towards a solution but I see
> that he posted apparent output from it, too:

I've now fixed up the patch and am pushing to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.