GNU bug report logs - #62471
29.0.60; `erts-run-test' results inconsistent with batch mode

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 26 Mar 2023 22:28:46 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 26 18:28:46 2023
Received: from localhost ([127.0.0.1]:46078 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pgYr0-00010C-2B
	for submit <at> debbugs.gnu.org; Sun, 26 Mar 2023 18:28:46 -0400
Received: from lists.gnu.org ([209.51.188.17]:54678)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <troy.s.brown@HIDDEN>) id 1pgYqy-000104-8B
 for submit <at> debbugs.gnu.org; Sun, 26 Mar 2023 18:28:44 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <troy.s.brown@HIDDEN>)
 id 1pgYqx-00031n-V9
 for bug-gnu-emacs@HIDDEN; Sun, 26 Mar 2023 18:28:43 -0400
Received: from mail-ed1-f42.google.com ([209.85.208.42])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <troy.s.brown@HIDDEN>)
 id 1pgYqw-00087n-9p
 for bug-gnu-emacs@HIDDEN; Sun, 26 Mar 2023 18:28:43 -0400
Received: by mail-ed1-f42.google.com with SMTP id b20so28650405edd.1
 for <bug-gnu-emacs@HIDDEN>; Sun, 26 Mar 2023 15:28:41 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112; t=1679869720;
 h=content-transfer-encoding:to:subject:message-id:date:from
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=ZYE8J3HhQYRBHRziNrCJ1NxXZV9FYDO7un7NJQn5APg=;
 b=aL/fILSdRkF+I4rIgPYyFI/SxpilTL9y5ioZMjQoZJIsqXut3YJTN06V38IFeCGYDl
 WoWg6+9dq62Wk9ya+MF7jOqLKYTnUdI2lTYgMRHeTEUQhTQhIz3SPpvrLUg98TYZlv9g
 Qik3Sr3QKyYCq+mZgqT92+rQHZLuT0aZYqLgQomlHsOv4UqXfKyHwrJvCorUok9cOywC
 tZ0ukMAKR+bKbKG4ivvuaEnwjFV6DNNdcVRnzwRwe4NZeuRc/uTc1Are7vSo2v7JEeYC
 +/uXf+U+TD138DitKNR1f3E7nRJBfBG0HW3b7BuvTMQSCkKl29PLJw8a+fKVhUmhyMTS
 Ggbw==
X-Gm-Message-State: AAQBX9cUmPzj6pU5cax7qAIspzNEr4rGGpyS2upAPI0/GcM8sQ2CyHun
 E1nM9o4Qn4cL0WTfUx+ui3ZHQXVdh5/d2g==
X-Google-Smtp-Source: AKy350ZqfKqPcp5YQQWkqFs8558KzIzjYdyjLK7QtUmKugfIgqw6o3iD4t0woulbH0cRjhG8TgFHsg==
X-Received: by 2002:a17:906:eda3:b0:931:7350:a4f3 with SMTP id
 sa3-20020a170906eda300b009317350a4f3mr11683171ejb.10.1679869719534; 
 Sun, 26 Mar 2023 15:28:39 -0700 (PDT)
Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com.
 [209.85.208.41]) by smtp.gmail.com with ESMTPSA id
 g5-20020a170906394500b0092fb818127dsm13484992eje.94.2023.03.26.15.28.39
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Sun, 26 Mar 2023 15:28:39 -0700 (PDT)
Received: by mail-ed1-f41.google.com with SMTP id cn12so28612239edb.4
 for <bug-gnu-emacs@HIDDEN>; Sun, 26 Mar 2023 15:28:39 -0700 (PDT)
X-Received: by 2002:a50:d7d9:0:b0:502:227a:d0da with SMTP id
 m25-20020a50d7d9000000b00502227ad0damr3863891edj.2.1679869719141; Sun, 26 Mar
 2023 15:28:39 -0700 (PDT)
MIME-Version: 1.0
From: Troy Brown <brownts@HIDDEN>
Date: Sun, 26 Mar 2023 18:28:27 -0400
X-Gmail-Original-Message-ID: <CABvCZ41xUozoZGWAW88HMD0HFfBWW=0DG2H=CVqaVNAaScB0Bw@HIDDEN>
Message-ID: <CABvCZ41xUozoZGWAW88HMD0HFfBWW=0DG2H=CVqaVNAaScB0Bw@HIDDEN>
Subject: 29.0.60; `erts-run-test' results inconsistent with batch mode
To: bug-gnu-emacs@HIDDEN
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=209.85.208.42;
 envelope-from=troy.s.brown@HIDDEN; helo=mail-ed1-f42.google.com
X-Spam_score_int: -13
X-Spam_score: -1.4
X-Spam_bar: -
X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.25,
 FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.1 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

Interactively running `erts-run-test' provides different (i.e.,
incorrect) results as when the test is run in batch mode.  I noticed
this happens when a new "code" block is introduced.  In the following
situation, there are two ERTS tests, each one supplies its own "code"
block associated with the test case.  When I interactively attempt to
execute the second test case (i.e., C-c C-c from within `erts-mode'),
the test fails and the expected results match the output from the first
case.

I used the following two test files.  `hi-test.el' is used to run the
test in batch mode (emacs -batch -l ert -l hi-test.el -f
ert-run-tests-batch-and-exit).  This just runs the ERTS test cases.
The second file (located relatively from `hi-test.el' in
resources/hi.erts) contains the two test cases.

hi-test.el:

(require 'ert)
(require 'ert-x)

(ert-deftest hi-test-run ()
  (ert-test-erts-file (ert-resource-file "hi.erts")))


resources/hi.erts:

Code:
  (lambda ()
    (fundamental-mode)
    (goto-char (point-max))
    (insert "there\n"))

Name: Hi there

=3D-=3D
Hi
=3D-=3D
Hi
there
=3D-=3D-=3D

Code:
  (lambda ()
    (fundamental-mode)
    (goto-char (point-max))
    (insert "you\n"))

Name: Hi you

=3D-=3D
Hi
=3D-=3D
Hi
you
=3D-=3D-=3D

Running in batch mode, I get the following result:

Running 1 tests (2023-03-26 18:10:33-0400, selector =E2=80=98t=E2=80=99)
   passed  1/1  hi-test-run (0.000384 sec)

Ran 1 tests, 1 results as expected, 0 unexpected (2023-03-26
18:10:33-0400, 0.000493 sec)

Running interactively in the `erts-mode' buffer, I get the following
result:

Test failure; result:
Hi
there