GNU bug report logs - #47657
python-shell font-lock with multi-line input: runaway fontification buffer length

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 47657 in the body.
You can then email your comments to 47657 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: python-shell font-lock with multi-line input: runaway fontification
 buffer length
Date: Thu, 8 Apr 2021 11:48:25 -0400

[Message part 1 (text/plain, inline)]

	
`python-shell-font-lock-post-command-hook’ makes the presumption that only a single line of input can appear at the prompt.  It includes this preparatory “clean-up” of its font lock buffer after first having copied all input lines after the prompt:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))
           ...
             (replacement
              (python-shell-font-lock-with-font-lock-buffer
                (delete-region (line-beginning-position)
                               (point-max))
                (setq font-lock-buffer-pos (point))
                (insert input)


This means that using, e.g., C-c SPC (`comint-accumulate’) to insert a newline at the shell prompt leads to runaway growth of the font-lock buffer (“ *Python-font-lock”, by default).  Any command, like an insert, produces a new copy of all but the final line of input in this buffer.  Since this entire buffer is font-locked from scratch after every single command, this becomes very inefficient.  

Find an example, below.

The simple fix would be to delete the entire font-lock buffer file contents:

	(delete-region (point-min) (point-max))

A perhaps smarter and more efficient fix would be to enable font locking in the inferior Python buffer itself, but constrain it to the text after the process mark.  This would also prevent front-lock from constantly having to re-fontify the entire input text after each command. 

++++

(In *Python* shell buffer):

In [2]: a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I’ 

(In " *Python-font-lock"; this can become arbitrarily long):

a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
a= 'now is the time'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
a= 'now is the time'
for i in range(12):
a = 12; print(i+4-4) 
b='one two one two and through and through'
d='and I' 

[Message part 2 (text/html, inline)]

Message #8 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input:
 runaway fontification buffer length
Date: Mon, 12 Apr 2021 11:32:54 +0200

JD Smith <jdtsmith <at> gmail.com> writes:

> `python-shell-font-lock-post-command-hook’ makes the presumption that only a
> single line of input can appear at the prompt.  It includes this preparatory
> “clean-up” of its font lock buffer after first having copied all input lines after the
> prompt:
>
>        (let* ((input (buffer-substring-no-properties
>                       prompt-end (point-max)))
>             ...
>               (replacement
>                (python-shell-font-lock-with-font-lock-buffer
>                  (delete-region (line-beginning-position)
>                                 (point-max))
>                  (setq font-lock-buffer-pos (point))
>                  (insert input)

I think the point here is that if we're in a multi-line expression, then
the font locking may be dependent on the previous lines, too, so we
can't just delete the contents of the buffer.

> Find an example, below.

[...]

> (In *Python* shell buffer):
>
> In [2]: a= 'now is the time'
> for i in range(12):
> a = 12; print(i+4-4) 
> b='one two one two and through and through'
> d='and I’ 
>
> (In " *Python-font-lock"; this can become arbitrarily long):
>
> a= 'now is the time'
> a= 'now is the time'

I was unable to reproduce this (in Emacs 28).  I pasted this into the
*Python* buffer:

a= 'now is the time'
for i in range(12):
    a = 12; print(i+4-4) 
    b='one two one two and through and through'
    d='and I'

and the font-lock buffer never became very big.

Do you have a step-by-step recipe, starting from "emacs -Q", to
reproduce this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #13 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input: runaway
 fontification buffer length
Date: Mon, 12 Apr 2021 09:18:13 -0400

[Message part 1 (text/plain, inline)]

> On Apr 12, 2021, at 5:32 AM, Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
> 
> I was unable to reproduce this (in Emacs 28).  I pasted this into the
> *Python* buffer:

Pasting multiple lines constitutes a single command for post-command-hook. In that case you would not expect to see any repeated lines.

> Do you have a step-by-step recipe, starting from "emacs -Q", to
> reproduce this problem?

emacs -Q
M-x run-python
C-x 5 b “ *Python-font-lock*”
In inferior python shell: type any line.
C-c SPC (`comint-accumulate’) to continue.
Type another line.  Notice the first line is repeated. 
Repeat steps 5 & 6 several times. 

The issue is how input is gathered from the prompt to the end of shell buffer in the post-command-hook:

      (let* ((input (buffer-substring-no-properties
                     prompt-end (point-max)))

Line-at-a-time also won’t be able to handle editing lines before the last.  On emacs-devel Stefan has been helpful with some ideas to perform font lock in-place in the shell buffer, which is working well in my tests.  I’m using that as part of a package to implement true multi-line editing (ala iPython).

[Message part 2 (text/html, inline)]

Message #16 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input:
 runaway fontification buffer length
Date: Tue, 13 Apr 2021 10:05:40 +0200

JD Smith <jdtsmith <at> gmail.com> writes:

>  Do you have a step-by-step recipe, starting from "emacs -Q", to
>  reproduce this problem?
>
> 1 emacs -Q
> 2 M-x run-python
> 3 C-x 5 b “ *Python-font-lock*”
> 4 In inferior python shell: type any line.
> 5 C-c SPC (`comint-accumulate’) to continue.
> 6 Type another line.  Notice the first line is repeated. 
> 7 Repeat steps 5 & 6 several times. 

You didn't say what Emacs version you're using -- I tested this in Emacs
28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
never contains more than a single non-blank line using this recipe.
(The "any line" I used to test was "5".)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #19 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input: runaway
 fontification buffer length
Date: Tue, 13 Apr 2021 09:14:20 -0400

[Message part 1 (text/plain, inline)]

> On Apr 13, 2021, at 4:05 AM, Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
> 
> JD Smith <jdtsmith <at> gmail.com> writes:
> 
>> Do you have a step-by-step recipe, starting from "emacs -Q", to
>> reproduce this problem?
>> 
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line.  Notice the first line is repeated. 
>> 7 Repeat steps 5 & 6 several times. 
> 
> You didn't say what Emacs version you're using -- I tested this in Emacs
> 28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
> never contains more than a single non-blank line using this recipe.
> (The "any line" I used to test was "5”.)

Mac port, v27.2, python-mode v0.26.1.  I have a hard time understanding how this would not reproduce, as (buffer-substring-no-properties  prompt-end (point-max)) clearly takes all of the text (multi-lines included) from the prompt onward.  I took a look here <https://github.com/emacs-mirror/emacs/blob/b064ddd3f600ed28e62b09d556ecced5f80d9883/lisp/progmodes/python.el#L2710> (python-mode v0.27.1) and the input is gathered in the same manner in the post-command-hook.  The “repeated” lines in #6 are in the *Python-font-lock* buffer, btw. 

[Message part 2 (text/html, inline)]

Message #22 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input:
 runaway fontification buffer length
Date: Sun, 25 Apr 2021 19:34:50 +0200

[Message part 1 (text/plain, inline)]

JD Smith <jdtsmith <at> gmail.com> writes:

>  1 emacs -Q
>  2 M-x run-python
>  3 C-x 5 b “ *Python-font-lock*”
>  4 In inferior python shell: type any line.
>  5 C-c SPC (`comint-accumulate’) to continue.
>  6 Type another line.  Notice the first line is repeated. 
>  7 Repeat steps 5 & 6 several times. 
>
>  You didn't say what Emacs version you're using -- I tested this in Emacs
>  28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
>  never contains more than a single non-blank line using this recipe.
>  (The "any line" I used to test was "5”.)
>
> Mac port, v27.2, python-mode v0.26.1.  I have a hard time
> understanding how this would not reproduce, as
> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
> all of the text (multi-lines included) from the prompt onward.  I took
> a look here (python-mode v0.27.1) and the input is gathered in the
> same manner in the post-command-hook.  The “repeated” lines in #6 are
> in the *Python-font-lock* buffer, btw.

After typing "1" in 4) and "7" in 6), the buffer looks like:

[Message part 2 (image/png, inline)]

[Message part 3 (text/plain, inline)]

So just the "7", and not the first line, which was "1".

Perhaps there's some missing element in your recipe?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #25 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input: runaway
 fontification buffer length
Date: Sun, 25 Apr 2021 13:53:35 -0400

[Message part 1 (text/plain, inline)]

Very odd, I get :

1
1
7

in *Python-font-lock* after precisely these same steps.  What if you use C-j in step #5 instead of C-c SPC?


> On Apr 25, 2021, at 1:34 PM, Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
> 
> JD Smith <jdtsmith <at> gmail.com> writes:
> 
>> 1 emacs -Q
>> 2 M-x run-python
>> 3 C-x 5 b “ *Python-font-lock*”
>> 4 In inferior python shell: type any line.
>> 5 C-c SPC (`comint-accumulate’) to continue.
>> 6 Type another line.  Notice the first line is repeated. 
>> 7 Repeat steps 5 & 6 several times. 
>> 
>> You didn't say what Emacs version you're using -- I tested this in Emacs
>> 28, and was unable to reproduce this.  The " *Python-font-lock*" buffer
>> never contains more than a single non-blank line using this recipe.
>> (The "any line" I used to test was "5”.)
>> 
>> Mac port, v27.2, python-mode v0.26.1.  I have a hard time
>> understanding how this would not reproduce, as
>> (buffer-substring-no-properties prompt-end (point-max)) clearly takes
>> all of the text (multi-lines included) from the prompt onward.  I took
>> a look here (python-mode v0.27.1) and the input is gathered in the
>> same manner in the post-command-hook.  The “repeated” lines in #6 are
>> in the *Python-font-lock* buffer, btw.
> 
> After typing "1" in 4) and "7" in 6), the buffer looks like:
> 
> <Mail Attachment.png>
> So just the "7", and not the first line, which was "1".
> 
> Perhaps there's some missing element in your recipe?
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no

[Message part 2 (text/html, inline)]

Message #28 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input:
 runaway fontification buffer length
Date: Sun, 25 Apr 2021 21:19:22 +0200

JD Smith <jdtsmith <at> gmail.com> writes:

> Very odd, I get :
>
>  1
>  1
>  7
>
> in *Python-font-lock* after precisely these same steps.  What if you use C-j in step
> #5 instead of C-c SPC?

I don't see any difference.

(This is with Emacs 27.2 on Debian/Bullseye.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #33 received at 47657 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 47657 <at> debbugs.gnu.org
Subject: Re: bug#47657: python-shell font-lock with multi-line input:
 runaway fontification buffer length
Date: Wed, 13 Oct 2021 18:45:34 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> JD Smith <jdtsmith <at> gmail.com> writes:
>
>> Very odd, I get :
>>
>>  1
>>  1
>>  7
>>
>> in *Python-font-lock* after precisely these same steps.  What if you
>> use C-j in step
>> #5 instead of C-c SPC?
>
> I don't see any difference.
>
> (This is with Emacs 27.2 on Debian/Bullseye.)

I was finally able to reproduce this -- I'm not sure what I was doing
differently before.  This is now fixed in Emacs 29 (I'm not putting the
fix into emacs-28, because I'm not 100% sure about it).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.