GNU bug report logs - #59477
python-tests fail without tree-sitter

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 59477 in the body.
You can then email your comments to 59477 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Emacs Bug Report <bug-gnu-emacs <at> gnu.org>
Cc: Yuan Fu <casouri <at> gmail.com>
Subject: python-tests fail without tree-sitter
Date: Tue, 22 Nov 2022 11:13:17 +0100

[Message part 1 (text/plain, inline)]

The python-tests fail when Emacs is built without tree-sitter:

[python-tests.log (application/octet-stream, attachment)]

Message #8 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: mattias.engdegard <at> gmail.com
Cc: 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Tue, 22 Nov 2022 18:40:39 -0800

Mattias Engdegård <mattias.engdegard <at> gmail.com> writes:

> The python-tests fail when Emacs is built without tree-sitter:

Thanks. Seems to be some problem with expanding the font-lock region.
I’m working on it.

Yuan

Message #11 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: mattias.engdegard <at> gmail.com
Cc: 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Wed, 23 Nov 2022 14:00:51 -0800

Yuan Fu <casouri <at> gmail.com> writes:

> Mattias Engdegård <mattias.engdegard <at> gmail.com> writes:
>
>> The python-tests fail when Emacs is built without tree-sitter:
>
> Thanks. Seems to be some problem with expanding the font-lock region.
> I’m working on it.
>
> Yuan

The python-font-lock-assignment-statement-multiline-xxx tests failing
are due to me accidentally dropping code when copying around:

(setq-local font-lock-defaults
              `(,python-font-lock-keywords
                nil nil nil nil
                (font-lock-syntactic-face-function
                 . python-font-lock-syntactic-face-function)
                (font-lock-extend-after-change-region-function
                 . python-font-lock-extend-region)))

became

(setq-local font-lock-defaults
              `(,python-font-lock-keywords
                nil nil nil nil
                (font-lock-syntactic-face-function
                 . python-font-lock-syntactic-face-function)))

The python-tests--fill-long-first-line test fails because the retunred
buffer-string doesn’t have the newline at the end.

Specifically, the return of this form

(with-temp-buffer
      (insert "def asdf():
    \"\"\"123 123 123 123 123 123 123 123 123 123 123 123 123 SHOULDBEWRAPPED 123 123 123 123

    \"\"\"
    a = 1
")
      (python-mode)
      (goto-char (point-min))
      (forward-line 1)
      (end-of-line)
      (fill-paragraph)
      (buffer-substring-no-properties (point-min) (point-max)))

Doesn’t have the newline at the end.

But if I run this in a buffer manually I get the ending newline. I’m not
sure what’s the cause of that.  Bisecting give
7c5d4348330b206aff1f8e5bc4fd241d6a6dc0b5, but that commit doesn’t change
anything filling-related.

Yuan

Message #14 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Yuan Fu <casouri <at> gmail.com>
Cc: mattias.engdegard <at> gmail.com, 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Thu, 24 Nov 2022 08:45:42 +0200

> Cc: 59477 <at> debbugs.gnu.org
> From: Yuan Fu <casouri <at> gmail.com>
> Date: Wed, 23 Nov 2022 14:00:51 -0800
> 
> But if I run this in a buffer manually I get the ending newline. I’m not
> sure what’s the cause of that.

Perhaps mode-require-final-newline?

Message #19 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Yuan Fu <casouri <at> gmail.com>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 59477 <at> debbugs.gnu.org
Subject: bug#59477: python-tests fail without tree-sitter
Date: Fri, 25 Nov 2022 17:50:10 +0100

> But if I run this in a buffer manually I get the ending newline. I’m not sure what’s the cause of that. Bisecting give 7c5d4348330b206aff1f8e5bc4fd241d6a6dc0b5, but that commit doesn’t change anything filling-related. 

No idea really, but it might have something to do with the fact that the changes move the assignments

  (setq-local font-lock-defaults
              `(,python-font-lock-keywords
                nil nil nil nil
                (font-lock-syntactic-face-function
                 . python-font-lock-syntactic-face-function)))
  (setq-local syntax-propertize-function
              python-syntax-propertize-function)

so that they are executed after

  (when python-indent-guess-indent-offset
    (python-indent-guess-indent-offset))

instead of before. `python-indent-guess-indent-offset` has the side-effect of setting syntax properties, in particular for the string terminator (triple-quote in the test).

This is important, because python-fill-string (called as part of fill-paragraph in the test) assumes this having already been done and if not, str-end-pos isn't computed correctly and things take a turn for the worse after that.

Stefan probably knows better how this is supposed to work, but presumably python-fill-string should take measures to ensure accurate syntax properties before doing things like

  (re-search-forward (rx (syntax string-delimiter)) nil t)

and so on. Sorry about not being of much help here.

Message #22 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Sat, 26 Nov 2022 14:18:37 -0800

> On Nov 25, 2022, at 8:50 AM, Mattias Engdegård <mattias.engdegard <at> gmail.com> wrote:
> 
>> But if I run this in a buffer manually I get the ending newline. I’m not sure what’s the cause of that. Bisecting give 7c5d4348330b206aff1f8e5bc4fd241d6a6dc0b5, but that commit doesn’t change anything filling-related. 
> 
> No idea really, but it might have something to do with the fact that the changes move the assignments
> 
>  (setq-local font-lock-defaults
>              `(,python-font-lock-keywords
>                nil nil nil nil
>                (font-lock-syntactic-face-function
>                 . python-font-lock-syntactic-face-function)))
>  (setq-local syntax-propertize-function
>              python-syntax-propertize-function)
> 
> so that they are executed after
> 
>  (when python-indent-guess-indent-offset
>    (python-indent-guess-indent-offset))
> 
> instead of before. `python-indent-guess-indent-offset` has the side-effect of setting syntax properties, in particular for the string terminator (triple-quote in the test).
> 
> This is important, because python-fill-string (called as part of fill-paragraph in the test) assumes this having already been done and if not, str-end-pos isn't computed correctly and things take a turn for the worse after that.
> 
> Stefan probably knows better how this is supposed to work, but presumably python-fill-string should take measures to ensure accurate syntax properties before doing things like
> 
>  (re-search-forward (rx (syntax string-delimiter)) nil t)
> 
> and so on. Sorry about not being of much help here.
> 

Thanks, that’s a very useful information. And I can only blame myself for breaking the tests :-)

While still unable to find the culprit. I have the following observations:

1. Setting require-final-newline to t doesn’t work
2. If I change with-temp-buffer to with-current-buffer (get-buffer-create "*test*”), the problem disappears, the newlines is not dropped
3. I edebugged fill-paragraph, the newlines in the temp buffer disappears at line 865 in fill.el, where the recursive call returns. Before (funcall function justify) returns (`function` is fill-paragraph itself), the newline still exists, but after we return to the caller at line 865, the newline disappears.

I checked for newline by hitting e and evaluating (char-before (point-max))

Yuan

Message #25 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Yuan Fu <casouri <at> gmail.com>
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Sun, 27 Nov 2022 14:07:13 +0100

[Message part 1 (text/plain, inline)]

26 nov. 2022 kl. 23.18 skrev Yuan Fu <casouri <at> gmail.com>:

> 1. Setting require-final-newline to t doesn’t work

As expected; it's not relevant here.

> 2. If I change with-temp-buffer to with-current-buffer (get-buffer-create "*test*”), the problem disappears, the newlines is not dropped

Yes, because you then reuse an old buffer that already has local variables set.

I'm tempted to push this patch that sinks the call to python-indent-guess-indent-offset so that it comes last in mode initialisation, as discussed in my previous message. Any objection?

As mentioned, I'm not sure it really addresses the root problem but at least it should be no worse than before the changes that broke the test.

[python-mode.diff (application/octet-stream, attachment)]

Message #28 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Yuan Fu <casouri <at> gmail.com>
Cc: kobarity <kobarity <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Sun, 27 Nov 2022 14:16:14 +0100

[Message part 1 (text/plain, inline)]

Oh, there is one more test in python-tests that fails (on my Mac at least): `python-ffap-module-path-1`. It was added fairly recently (see bug#58713) so I'm not sure it was already broken at the time or if that is a recent change. CC:ing the author and attaching the test log.

[python-ffap-module-path-1.log (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

This looks like an error when parsing output from Python, mistaking input for output.

(How we ended up parsing a buffer containing both input, output and prompts intermingled is anyone's guess -- it doesn't look like a very reliable way of doing it, and evidently isn't.)

Message #31 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: Yuan Fu <casouri <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Sun, 27 Nov 2022 23:48:22 +0900

Mattias Engdegård wrote:
> Oh, there is one more test in python-tests that fails (on my Mac at least): `python-ffap-module-path-1`. It was added fairly recently (see bug#58713) so I'm not sure it was already broken at the time or if that is a recent change. CC:ing the author and attaching the test log.

`python-ffap-module-path-1' does not fail on my Ubuntu 22.04
environment.  It seems to me that `python-ffap-module-path' or
`python-shell-send-string-no-output' is not working as expected on
your environment.

Could you tell me the results of doing the following?

1. emacs -Q
2. Open Python file. (Empty file is OK)
3. C-c C-p (M-x run-python)
4. C-x o (Back to the Python mode buffer)
5. M-: (python-shell-send-string-no-output "print('a')")

My expectation is "a".

Message #34 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Yuan Fu <casouri <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Mon, 28 Nov 2022 11:01:52 +0100

[Message part 1 (text/plain, inline)]

27 nov. 2022 kl. 15.48 skrev kobarity <kobarity <at> gmail.com>:

> It seems to me that `python-ffap-module-path' or
> `python-shell-send-string-no-output' is not working as expected on
> your environment.

Indeed, something in Emacs is making incorrect assumptions.

> 1. emacs -Q
> 2. Open Python file. (Empty file is OK)
> 3. C-c C-p (M-x run-python)
> 4. C-x o (Back to the Python mode buffer)
> 5. M-: (python-shell-send-string-no-output "print('a')")
> 
> My expectation is "a".

The result here is

 "__PYTHON_EL_eval(\"print('a')\", \"/Users/mattias/emacs/asdf.py\")\r\na"

which is consistent with the test failure -- something in python-mode is having trouble parsing the reply. I traced `python-shell-output-filter`; see attached file.

This might be bug#25753 again (via a comment in etc/PROBLEMS). Even if no effort is made to fix the bug properly, we should make sure the test doesn't fail.

[python-shell-output-filter-trace.log (application/octet-stream, attachment)]

Message #37 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Yuan Fu <casouri <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Mon, 28 Nov 2022 11:18:32 +0100

[Message part 1 (text/plain, inline)]

Here is a trace of python-shell-output-filter when running the failing test (python-ffap-module-path-1):

[python-test-trace.log (application/octet-stream, attachment)]

Message #40 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: Yuan Fu <casouri <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Tue, 29 Nov 2022 00:18:10 +0900

[Message part 1 (text/plain, inline)]

Mattias Engdegård wrote:
> > 1. emacs -Q
> > 2. Open Python file. (Empty file is OK)
> > 3. C-c C-p (M-x run-python)
> > 4. C-x o (Back to the Python mode buffer)
> > 5. M-: (python-shell-send-string-no-output "print('a')")
> > 
> > My expectation is "a".
> 
> The result here is
> 
>  "__PYTHON_EL_eval(\"print('a')\", \"/Users/mattias/emacs/asdf.py\")\r\na"
> 
> which is consistent with the test failure -- something in python-mode is having trouble parsing the reply. I traced `python-shell-output-filter`; see attached file.
> 
> This might be bug#25753 again (via a comment in etc/PROBLEMS). Even if no effort is made to fix the bug properly, we should make sure the test doesn't fail.

Thank you for the logs and letting me know about bug#25753.  It seems
to be a bug that has existed for some time on Mac.  Here is the trace
on my Ubuntu.

======================================================================
1 -> (python-shell-output-filter ">>> ")
1 <- python-shell-output-filter: ""
======================================================================
1 -> (python-shell-output-filter ">>> ")
1 <- python-shell-output-filter: ""
======================================================================
1 -> (python-shell-output-filter "python.el: native completion setup loaded\n")
1 <- python-shell-output-filter: ""
======================================================================
1 -> (python-shell-output-filter ">>> ")
1 <- python-shell-output-filter: ""
======================================================================
1 -> (python-shell-output-filter "a\n")
1 <- python-shell-output-filter: ""
======================================================================
1 -> (python-shell-output-filter ">>> ")
1 <- python-shell-output-filter: ""

As for `python-ffap-module-path-1', it assumes that
`python-ffap-module-path' and its underlying
`python-shell-send-string-no-output' operate correctly.  So I think
the first option is to mark as an expected failure on Mac as the
attached patch.  However, I'm not sure if it fails on all Macs.
Another option may be to add :unstable tag.  Which do you think is
better?

[0001-Mark-python-ffap-module-path-1-as-an-expected-failur.patch (application/octet-stream, attachment)]

Message #43 received at 59477 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: Yuan Fu <casouri <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 59477 <at> debbugs.gnu.org
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Mon, 28 Nov 2022 20:22:53 +0100

28 nov. 2022 kl. 16.18 skrev kobarity <kobarity <at> gmail.com>:

> It seems
> to be a bug that has existed for some time on Mac.  Here is the trace
> on my Ubuntu.

Right, it looks like the input lines aren't echoed. (Conversing through a TTY is really a terrible mode for issuing commands and getting replies reliably.)

> As for `python-ffap-module-path-1', it assumes that
> `python-ffap-module-path' and its underlying
> `python-shell-send-string-no-output' operate correctly.  So I think
> the first option is to mark as an expected failure on Mac as the
> attached patch.  However, I'm not sure if it fails on all Macs.
> Another option may be to add :unstable tag.  Which do you think is
> better?

It's probably best to always skip the test on macOS, since there is no telling whether the Python used is the system-supplied one with libedit or one where the user installed readline. That's the change that I ended up pushing.

Thanks for your kind help!

Message #48 received at 59477-done <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Yuan Fu <casouri <at> gmail.com>
Cc: 59477-done <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Wed, 30 Nov 2022 14:41:29 +0100

27 nov. 2022 kl. 14.07 skrev Mattias Engdegård <mattias.engdegard <at> gmail.com>:

> I'm tempted to push this patch that sinks the call to python-indent-guess-indent-offset

and have now succumbed to the temptation: pushed to emacs-29.

This should fix the last observed failure in python-tests and the bug can be closed.

Message #51 received at 59477-done <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: 59477-done <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#59477: python-tests fail without tree-sitter
Date: Sat, 3 Dec 2022 23:51:52 -0800

> On Nov 30, 2022, at 5:41 AM, Mattias Engdegård <mattias.engdegard <at> gmail.com> wrote:
> 
> 27 nov. 2022 kl. 14.07 skrev Mattias Engdegård <mattias.engdegard <at> gmail.com>:
> 
>> I'm tempted to push this patch that sinks the call to python-indent-guess-indent-offset
> 
> and have now succumbed to the temptation: pushed to emacs-29.
> 
> This should fix the last observed failure in python-tests and the bug can be closed.
> 

Sorry for the delay, I just got around and came back to this bug. It didn’t fix the test for me when I tried you suggestion. But it’s probably some problem with my execution, since your patch clearly fixes the test. Anyway, thanks a ton for fixing this! One more stone off my chest :-)

Yuan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.