GNU bug report logs - #67262
python-ts-mode cannot identify triple-quoted-strings

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 67262 in the body.
You can then email your comments to 67262 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 18 Nov 2023 10:52:05 -0500

[Message part 1 (text/plain, inline)]

Inside this triple-quoted string, in a python buffer:

a = """This is a test"""

python-mode yields (python-info-triple-quoted-string-p)=t, whereas python-ts-mode gives nil, defeating the fancy doc string folding both modes implement.

The reason seems to be that (syntax-ppss) returns something different in position 3 (which is "non-nil if inside a string”) between these modes: 

t for python-mode (which signals a triple quote)
 ?34=" in python-ts-mode 

If you first load python-mode, then load python-ts-mode, the syntax parse becomes equal between the modes, and this bug vanishes.  

python.el v0.28, Emacs v29.1

[Message part 2 (text/html, inline)]

Message #8 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 18 Nov 2023 18:29:28 +0200

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Sat, 18 Nov 2023 10:52:05 -0500
> 
> Inside this triple-quoted string, in a python buffer:
> 
> a = """This is a test"""
> 
> python-mode yields (python-info-triple-quoted-string-p)=t, whereas python-ts-mode gives nil,
> defeating the fancy doc string folding both modes implement.
> 
> The reason seems to be that (syntax-ppss) returns something different in position 3 (which is "non-nil
> if inside a string”) between these modes: 
> 
> * t for python-mode (which signals a triple quote)
> *  ?34=" in python-ts-mode 
> 
> If you first load python-mode, then load python-ts-mode, the syntax parse becomes equal between the
> modes, and this bug vanishes.  

Can you figure out which part of python-mode's initialization makes
the above work correctly, and why?  Then we could discuss whether
moving that part into python-base-mode is TRT.

Thanks.

Message #11 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 18 Nov 2023 12:18:02 -0500

[Message part 1 (text/plain, inline)]

> On Nov 18, 2023, at 11:29 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: JD Smith <jdtsmith <at> gmail.com>
>> Date: Sat, 18 Nov 2023 10:52:05 -0500
>> 
>> Inside this triple-quoted string, in a python buffer:
>> 
>> a = """This is a test"""
>> 
>> python-mode yields (python-info-triple-quoted-string-p)=t, whereas python-ts-mode gives nil,
> 
> Can you figure out which part of python-mode's initialization makes
> the above work correctly, and why?  Then we could discuss whether
> moving that part into python-base-mode is TRT.

I took a look. It’s `syntax-propertize-function' that is not being setup.  From a buffer in `python-ts-mode', this reenables triple-quote recognition:

(progn 
  (setq-local syntax-propertize-function python-syntax-propertize-function) 
  (syntax-ppss-flush-cache (point-min)))

Note that `python-syntax-propertize-function' mentions `python-syntax-stringify', which scans the syntax for triple quotes and marks their 'syntax-table.  I am not sure whether this was an oversight, or was omitted purposefully from the body of python-ts-mode.

[Message part 2 (text/html, inline)]

Message #14 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: JD Smith <jdtsmith <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 19 Nov 2023 00:52:06 +0200

On 18/11/2023 19:18, JD Smith wrote:
>    (setq-local syntax-propertize-function 
> python-syntax-propertize-function)

This is not a bad choice, but even better would be to write a s-p-f 
based on the tree-sitter parse tree.

There are examples in ruby-ts-mode and js-ts-mode.

Message #17 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 25 Nov 2023 12:01:56 +0200

> Date: Sun, 19 Nov 2023 00:52:06 +0200
> Cc: 67262 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry <at> gutov.dev>
> 
> On 18/11/2023 19:18, JD Smith wrote:
> >    (setq-local syntax-propertize-function 
> > python-syntax-propertize-function)
> 
> This is not a bad choice, but even better would be to write a s-p-f 
> based on the tree-sitter parse tree.
> 
> There are examples in ruby-ts-mode and js-ts-mode.

JD, would you like to try writing such a syntax-propertize-function?

Message #20 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 25 Nov 2023 09:42:23 -0500

Bridging emacs syntax to treesitter in a robust way seems like it could be a subtle enterprise, so I’d prefer to leave that to one of the experts.  Right now the syntax-propertize-function in python-mode does one simple thing: ensure triple quotes are properly marked as strings.  Since the treesitter grammar doesn’t distinguish between different flavors of strings, something similar would still be needed, if we want to continue to treat various string flavors distinctly using syntax.  

Is moving all syntactification (beyond just font-lock) over to TS an explicit goal for all the *-ts-mode’s?

> On Nov 25, 2023, at 5:01 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> Date: Sun, 19 Nov 2023 00:52:06 +0200
>> Cc: 67262 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry <at> gutov.dev>
>> 
>> On 18/11/2023 19:18, JD Smith wrote:
>>>   (setq-local syntax-propertize-function 
>>> python-syntax-propertize-function)
>> 
>> This is not a bad choice, but even better would be to write a s-p-f 
>> based on the tree-sitter parse tree.
>> 
>> There are examples in ruby-ts-mode and js-ts-mode.
> 
> JD, would you like to try writing such a syntax-propertize-function?

Message #23 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: JD Smith <jdtsmith <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 Yuan Fu <casouri <at> gmail.com>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 26 Nov 2023 04:04:07 +0200

[Message part 1 (text/plain, inline)]

On 25/11/2023 16:42, JD Smith wrote:
> Bridging emacs syntax to treesitter in a robust way seems like it could be a subtle enterprise, so I’d prefer to leave that to one of the experts.  Right now the syntax-propertize-function in python-mode does one simple thing: ensure triple quotes are properly marked as strings.  Since the treesitter grammar doesn’t distinguish between different flavors of strings, something similar would still be needed, if we want to continue to treat various string flavors distinctly using syntax.
> 
> Is moving all syntactification (beyond just font-lock) over to TS an explicit goal for all the *-ts-mode’s?

It would make sense - since this way we would only have one source of 
syntax-recognition bugs, rather than two (both the grammar and the 
definition in Elisp).

Attached is a patch you can try (that uses treesit for s-p-f).

Unfortunately, it's not quite perfect (nor is python-syntax-stringify, 
according to its FIXME inside): after certain modifications, the 
syntax-table property is not applied.

I've done some print-debugging in python--treesit-parser-after-change, 
and it looks like the problem is this: in certain cases (e.g. when 
electric-pair-post-self-insert-function fires) the parser notifier fires 
only after syntax-propertize has been called -- and it fires inside of 
it. Meaning it's too late to flush the syntax-propertize cache at that 
point.

The reason for it is, overall, the fast that we're trigger parser's 
after-change notifiers lazily: only after some other feature has to 
initialize the parser, calling treesit_ensure_parsed from 
treesit-parser-root-node.

I think bug#66732 might also be a variation of this problem.

As for what to do about this one -- probably something involving 
syntax-propertize-extend-region-functions, adding an entry which would 
initialize the parser, but not call syntax-ppss-flush-cache directly (or 
at least not just that). It would signal the earlier position to extend 
to through some dynamic variable. This is getting tricky enough to move 
from the individual major modes into treesit.el proper, I think.

Yuan and others, thoughts welcome.

JD, I do believe the attached patch is TRT (or close to it), but 
depending on how it works for you, and how quickly we deal with the 
above problem, it might make sense to enact your original suggestion first.

And finally, here's the backtrace that led me to the above conclusions:

  backtrace()
  (message "in progress, backtrace %s" (backtrace))
  (progn (message "in progress, backtrace %s" (backtrace)))
  (if (syntax-propertize--in-process-p) (progn (message "in progress, 
backtrace %s" (backtrace))))
  (save-current-buffer (set-buffer (treesit-parser-buffer parser)) 
(message "flushing %s up to %s" ranges (let* ((--cl-var-- ranges) (r 
nil) (--cl-var-- nil)) (while (consp --cl-var--) (setq r (car 
--cl-var--)) (let* ((temp (car r))) (setq --cl-var-- (if --cl-var-- (min 
--cl-var-- temp) temp))) (setq --cl-var-- (cdr --cl-var--))) 
--cl-var--)) (syntax-ppss-flush-cache (let* ((--cl-var-- ranges) (r nil) 
(--cl-var-- nil)) (while (consp --cl-var--) (setq r (car --cl-var--)) 
(let* ((temp (car r))) (setq --cl-var-- (if --cl-var-- (min --cl-var-- 
temp) temp))) (setq --cl-var-- (cdr --cl-var--))) --cl-var--)) (if 
(syntax-propertize--in-process-p) (progn (message "in progress, 
backtrace %s" (backtrace)))) (message "flushed up to %d, %s" 
syntax-propertize--done syntax-ppss-wide))
  (progn (save-current-buffer (set-buffer (treesit-parser-buffer 
parser)) (message "flushing %s up to %s" ranges (let* ((--cl-var-- 
ranges) (r nil) (--cl-var-- nil)) (while (consp --cl-var--) (setq r (car 
--cl-var--)) (let* ((temp ...)) (setq --cl-var-- (if --cl-var-- ... 
temp))) (setq --cl-var-- (cdr --cl-var--))) --cl-var--)) 
(syntax-ppss-flush-cache (let* ((--cl-var-- ranges) (r nil) (--cl-var-- 
nil)) (while (consp --cl-var--) (setq r (car --cl-var--)) (let* ((temp 
...)) (setq --cl-var-- (if --cl-var-- ... temp))) (setq --cl-var-- (cdr 
--cl-var--))) --cl-var--)) (if (syntax-propertize--in-process-p) (progn 
(message "in progress, backtrace %s" (backtrace)))) (message "flushed up 
to %d, %s" syntax-propertize--done syntax-ppss-wide)))
  (if ranges (progn (save-current-buffer (set-buffer 
(treesit-parser-buffer parser)) (message "flushing %s up to %s" ranges 
(let* ((--cl-var-- ranges) (r nil) (--cl-var-- nil)) (while (consp 
--cl-var--) (setq r (car --cl-var--)) (let* (...) (setq --cl-var-- ...)) 
(setq --cl-var-- (cdr --cl-var--))) --cl-var--)) 
(syntax-ppss-flush-cache (let* ((--cl-var-- ranges) (r nil) (--cl-var-- 
nil)) (while (consp --cl-var--) (setq r (car --cl-var--)) (let* (...) 
(setq --cl-var-- ...)) (setq --cl-var-- (cdr --cl-var--))) --cl-var--)) 
(if (syntax-propertize--in-process-p) (progn (message "in progress, 
backtrace %s" (backtrace)))) (message "flushed up to %d, %s" 
syntax-propertize--done syntax-ppss-wide))))
  python--treesit-parser-after-change(((27 . 50)) #<treesit-parser for 
python>)
  treesit-buffer-root-node(python)
  treesit-node-at(42)
  (let ((node (treesit-node-at (point)))) (cond ((equal 
(treesit-node-type node) "string_content") (put-text-property (- (point) 
3) (- (point) 2) 'syntax-table (string-to-syntax "|"))) ((and (equal 
(treesit-node-type node) "\"") (= (treesit-node-start node) (- (point) 
3))) (put-text-property (1- (point)) (point) 'syntax-table 
(string-to-syntax "|")))))
  (cond (t (message "pt %s" (point)) (let ((node (treesit-node-at 
(point)))) (cond ((equal (treesit-node-type node) "string_content") 
(put-text-property (- (point) 3) (- (point) 2) 'syntax-table 
(string-to-syntax "|"))) ((and (equal (treesit-node-type node) "\"") (= 
(treesit-node-start node) (- ... 3))) (put-text-property (1- (point)) 
(point) 'syntax-table (string-to-syntax "|")))))))
  (while (and (< (point) end) (re-search-forward "\\(?:\"\"\"\\|'''\\)" 
end t)) (cond (t (message "pt %s" (point)) (let ((node (treesit-node-at 
(point)))) (cond ((equal (treesit-node-type node) "string_content") 
(put-text-property (- ... 3) (- ... 2) 'syntax-table (string-to-syntax 
"|"))) ((and (equal ... "\"") (= ... ...)) (put-text-property (1- ...) 
(point) 'syntax-table (string-to-syntax "|"))))))))
  (closure (t) (start end) (goto-char start) (while (and (< (point) 
end) (re-search-forward "\\(?:\"\"\"\\|'''\\)" end t)) (cond (t (message 
"pt %s" (point)) (let ((node ...)) (cond (... ...) (... ...)))))))(39 50)
  funcall((closure (t) (start end) (goto-char start) (while (and (< 
(point) end) (re-search-forward "\\(?:\"\"\"\\|'''\\)" end t)) (cond (t 
(message "pt %s" (point)) (let ((node ...)) (cond (... ...) (... 
...))))))) 39 50)
  python--treesit-syntax-propertize-function-1(39 50)
  syntax-propertize(42)
  syntax-ppss(42)
  electric-pair-syntax-info(39)
  electric-pair-post-self-insert-function()
  self-insert-command(1 39)
  funcall-interactively(self-insert-command 1 39)
  #<subr call-interactively>(self-insert-command nil nil)
  call-interactively <at> ido-cr+-record-current-command(#<subr 
call-interactively> self-insert-command nil nil)
  apply(call-interactively <at> ido-cr+-record-current-command #<subr 
call-interactively> (self-insert-command nil nil))
  call-interactively(self-insert-command nil nil)
  command-execute(self-insert-command)

[python--treesit-syntax-propertize-function.diff (text/x-patch, attachment)]

Message #26 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: JD Smith <jdtsmith <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>,
 Yuan Fu <casouri <at> gmail.com>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 26 Nov 2023 16:58:41 +0200

On 26/11/2023 04:04, Dmitry Gutov wrote:
> As for what to do about this one -- probably something involving 
> syntax-propertize-extend-region-functions, adding an entry which would 
> initialize the parser, but not call syntax-ppss-flush-cache directly (or 
> at least not just that). It would signal the earlier position to extend 
> to through some dynamic variable. This is getting tricky enough to move 
> from the individual major modes into treesit.el proper, I think.

Alternatively, we'd trigger updates eagerly from within 
treesit_record_change -- that would make it slower, invalidating the 
comment above it. Not sure by how much, though.

Message #29 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Dmitry Gutov <dmitry <at> gutov.dev>, JD Smith <jdtsmith <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 26 Nov 2023 15:43:22 -0800

On 11/26/23 6:58 AM, Dmitry Gutov wrote:
> On 26/11/2023 04:04, Dmitry Gutov wrote:
>> As for what to do about this one -- probably something involving 
>> syntax-propertize-extend-region-functions, adding an entry which 
>> would initialize the parser, but not call syntax-ppss-flush-cache 
>> directly (or at least not just that). It would signal the earlier 
>> position to extend to through some dynamic variable. This is getting 
>> tricky enough to move from the individual major modes into treesit.el 
>> proper, I think.
>
> Alternatively, we'd trigger updates eagerly from within 
> treesit_record_change -- that would make it slower, invalidating the 
> comment above it. Not sure by how much, though.

It seems to me that what we need is to force a re-parse at the beginning 
of syntax-propertize or in syntax-ppss-flush-cache; the re-parse would 
cause the notifier to run, which runs python--treesit-parser-after-change.

I'm not quite sure about how do we cause this re-parse. The 
straightforward approach would be calling treesit-force-reparse[1] in 
syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to keep 
tree-sitter transparent for syntax.el. Maybe we can add a hook in 
syntax-propertize/syntax-ppss-flush-cache.

[1] This function doesn't exist yet, but it's easy to define in lisp.

Yuan

Message #32 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>, JD Smith <jdtsmith <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Mon, 27 Nov 2023 02:05:10 +0200

On 27/11/2023 01:43, Yuan Fu wrote:
> 
> On 11/26/23 6:58 AM, Dmitry Gutov wrote:
>> On 26/11/2023 04:04, Dmitry Gutov wrote:
>>> As for what to do about this one -- probably something involving 
>>> syntax-propertize-extend-region-functions, adding an entry which 
>>> would initialize the parser, but not call syntax-ppss-flush-cache 
>>> directly (or at least not just that). It would signal the earlier 
>>> position to extend to through some dynamic variable. This is getting 
>>> tricky enough to move from the individual major modes into treesit.el 
>>> proper, I think.
>>
>> Alternatively, we'd trigger updates eagerly from within 
>> treesit_record_change -- that would make it slower, invalidating the 
>> comment above it. Not sure by how much, though.
> 
> It seems to me that what we need is to force a re-parse at the beginning 
> of syntax-propertize or in syntax-ppss-flush-cache; the re-parse would 
> cause the notifier to run, which runs python--treesit-parser-after-change.

syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
seems like it will be odd to have execution the other way around and/or 
add some hook into it which would call the re-parse and extend the 
region to be invalidated.

syntax-propertize could have another hook added, yes. Or an advice.

But it seems better to reuse some of the existing hooks, such as 
syntax-propertize-extend-region-functions. It treesit.c provided a way 
to fetch the newly-invalidated region, the treesit-major-mode-setup 
could add a new function to syntax-propertize-extend-region-functions 
which would invoke that feature. But even now it can instantiate the 
parse, which would call treesit-force-reparse internally, and then 
collect the info from the callbacks.

And yet another way - is to extend the region to be propertized from 
inside the major mode's syntax-propertize-function, invalidating some 
earlier entries too. The main problem with that, I think, is that every 
ts mode will have to repeat that trick. And that authors would have to 
know to do that. How to make that easier and more obvious, is a question.

Finally, if I'm right that bug#66732 has a similar cause, then a shared 
solution that can be reused by syntax and font-lock (or preferably just 
fix both in the same place) would be ideal.

> I'm not quite sure about how do we cause this re-parse. The 
> straightforward approach would be calling treesit-force-reparse[1] in 
> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to keep 
> tree-sitter transparent for syntax.el. Maybe we can add a hook in 
> syntax-propertize/syntax-ppss-flush-cache.
> 
> [1] This function doesn't exist yet, but it's easy to define in lisp.

treesit-parser-root-node calls it anyway and does little else, so we 
could get by with just using it.

Message #37 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Dmitry Gutov <dmitry <at> gutov.dev>, JD Smith <jdtsmith <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Tue, 12 Dec 2023 00:32:09 -0800

On 11/26/23 4:05 PM, Dmitry Gutov wrote:
> On 27/11/2023 01:43, Yuan Fu wrote:
>>
>> On 11/26/23 6:58 AM, Dmitry Gutov wrote:
>>> On 26/11/2023 04:04, Dmitry Gutov wrote:
>>>> As for what to do about this one -- probably something involving 
>>>> syntax-propertize-extend-region-functions, adding an entry which 
>>>> would initialize the parser, but not call syntax-ppss-flush-cache 
>>>> directly (or at least not just that). It would signal the earlier 
>>>> position to extend to through some dynamic variable. This is 
>>>> getting tricky enough to move from the individual major modes into 
>>>> treesit.el proper, I think.
>>>
>>> Alternatively, we'd trigger updates eagerly from within 
>>> treesit_record_change -- that would make it slower, invalidating the 
>>> comment above it. Not sure by how much, though.
>>
>> It seems to me that what we need is to force a re-parse at the 
>> beginning of syntax-propertize or in syntax-ppss-flush-cache; the 
>> re-parse would cause the notifier to run, which runs 
>> python--treesit-parser-after-change.
>
> syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
> seems like it will be odd to have execution the other way around 
> and/or add some hook into it which would call the re-parse and extend 
> the region to be invalidated.
>
> syntax-propertize could have another hook added, yes. Or an advice.
>
> But it seems better to reuse some of the existing hooks, such as 
> syntax-propertize-extend-region-functions. It treesit.c provided a way 
> to fetch the newly-invalidated region, the treesit-major-mode-setup 
> could add a new function to syntax-propertize-extend-region-functions 
> which would invoke that feature. But even now it can instantiate the 
> parse, which would call treesit-force-reparse internally, and then 
> collect the info from the callbacks.
syntax-propertize-extend-region-functions looks perfect. We just need to 
force a reparse in it and the notifier will do the rest.
>
> And yet another way - is to extend the region to be propertized from 
> inside the major mode's syntax-propertize-function, invalidating some 
> earlier entries too. The main problem with that, I think, is that 
> every ts mode will have to repeat that trick. And that authors would 
> have to know to do that. How to make that easier and more obvious, is 
> a question.
>
> Finally, if I'm right that bug#66732 has a similar cause, then a 
> shared solution that can be reused by syntax and font-lock (or 
> preferably just fix both in the same place) would be ideal.
>
>> I'm not quite sure about how do we cause this re-parse. The 
>> straightforward approach would be calling treesit-force-reparse[1] in 
>> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to 
>> keep tree-sitter transparent for syntax.el. Maybe we can add a hook 
>> in syntax-propertize/syntax-ppss-flush-cache.
>>
>> [1] This function doesn't exist yet, but it's easy to define in lisp.
>
> treesit-parser-root-node calls it anyway and does little else, so we 
> could get by with just using it.
>
Yep.

Yuan

Message #40 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>, JD Smith <jdtsmith <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Tue, 12 Dec 2023 23:55:43 +0200

On 12/12/2023 10:32, Yuan Fu wrote:

>> syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
>> seems like it will be odd to have execution the other way around 
>> and/or add some hook into it which would call the re-parse and extend 
>> the region to be invalidated.
>>
>> syntax-propertize could have another hook added, yes. Or an advice.
>>
>> But it seems better to reuse some of the existing hooks, such as 
>> syntax-propertize-extend-region-functions. It treesit.c provided a way 
>> to fetch the newly-invalidated region, the treesit-major-mode-setup 
>> could add a new function to syntax-propertize-extend-region-functions 
>> which would invoke that feature. But even now it can instantiate the 
>> parse, which would call treesit-force-reparse internally, and then 
>> collect the info from the callbacks.
> syntax-propertize-extend-region-functions looks perfect. We just need to 
> force a reparse in it and the notifier will do the rest.
>>
>> And yet another way - is to extend the region to be propertized from 
>> inside the major mode's syntax-propertize-function, invalidating some 
>> earlier entries too. The main problem with that, I think, is that 
>> every ts mode will have to repeat that trick. And that authors would 
>> have to know to do that. How to make that easier and more obvious, is 
>> a question.
>>
>> Finally, if I'm right that bug#66732 has a similar cause, then a 
>> shared solution that can be reused by syntax and font-lock (or 
>> preferably just fix both in the same place) would be ideal.
>>
>>> I'm not quite sure about how do we cause this re-parse. The 
>>> straightforward approach would be calling treesit-force-reparse[1] in 
>>> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to 
>>> keep tree-sitter transparent for syntax.el. Maybe we can add a hook 
>>> in syntax-propertize/syntax-ppss-flush-cache.
>>>
>>> [1] This function doesn't exist yet, but it's easy to define in lisp.
>>
>> treesit-parser-root-node calls it anyway and does little else, so we 
>> could get by with just using it.
>>
> Yep.

For the casual observer: I posted a solution using the above method in 
comments to bug#66732 yesterday.

Message #43 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: casouri <at> gmail.com, Dmitry Gutov <dmitry <at> gutov.dev>
Cc: 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 14:47:46 +0200

How should we make progress with this bug?  I'd like to fix it before
Emacs 29.2 is released.

> Date: Tue, 12 Dec 2023 23:55:43 +0200
> Cc: 67262 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry <at> gutov.dev>
> 
> On 12/12/2023 10:32, Yuan Fu wrote:
> 
> >> syntax-ppss-flush-cache is called by edits (and by the re-parse). It 
> >> seems like it will be odd to have execution the other way around 
> >> and/or add some hook into it which would call the re-parse and extend 
> >> the region to be invalidated.
> >>
> >> syntax-propertize could have another hook added, yes. Or an advice.
> >>
> >> But it seems better to reuse some of the existing hooks, such as 
> >> syntax-propertize-extend-region-functions. It treesit.c provided a way 
> >> to fetch the newly-invalidated region, the treesit-major-mode-setup 
> >> could add a new function to syntax-propertize-extend-region-functions 
> >> which would invoke that feature. But even now it can instantiate the 
> >> parse, which would call treesit-force-reparse internally, and then 
> >> collect the info from the callbacks.
> > syntax-propertize-extend-region-functions looks perfect. We just need to 
> > force a reparse in it and the notifier will do the rest.
> >>
> >> And yet another way - is to extend the region to be propertized from 
> >> inside the major mode's syntax-propertize-function, invalidating some 
> >> earlier entries too. The main problem with that, I think, is that 
> >> every ts mode will have to repeat that trick. And that authors would 
> >> have to know to do that. How to make that easier and more obvious, is 
> >> a question.
> >>
> >> Finally, if I'm right that bug#66732 has a similar cause, then a 
> >> shared solution that can be reused by syntax and font-lock (or 
> >> preferably just fix both in the same place) would be ideal.
> >>
> >>> I'm not quite sure about how do we cause this re-parse. The 
> >>> straightforward approach would be calling treesit-force-reparse[1] in 
> >>> syntax-propertize/syntax-ppss-flush-cache. But ideally I'd like to 
> >>> keep tree-sitter transparent for syntax.el. Maybe we can add a hook 
> >>> in syntax-propertize/syntax-ppss-flush-cache.
> >>>
> >>> [1] This function doesn't exist yet, but it's easy to define in lisp.
> >>
> >> treesit-parser-root-node calls it anyway and does little else, so we 
> >> could get by with just using it.
> >>
> > Yep.
> 
> For the casual observer: I posted a solution using the above method in 
> comments to bug#66732 yesterday.
> 

Message #46 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Eli Zaretskii <eliz <at> gnu.org>, casouri <at> gmail.com
Cc: 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 15:37:40 +0200

On 16/12/2023 14:47, Eli Zaretskii wrote:
> How should we make progress with this bug?  I'd like to fix it before
> Emacs 29.2 is released.

We're currently working on the general issue in bug#66732.

Message #49 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: casouri <at> gmail.com, 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 15:59:57 +0200

> Date: Sat, 16 Dec 2023 15:37:40 +0200
> Cc: jdtsmith <at> gmail.com, 67262 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry <at> gutov.dev>
> 
> On 16/12/2023 14:47, Eli Zaretskii wrote:
> > How should we make progress with this bug?  I'd like to fix it before
> > Emacs 29.2 is released.
> 
> We're currently working on the general issue in bug#66732.

Ok, so please close this bug as well, when that general issue is
resolved.

And I hope bug#66732 will be solved soon?  Because I'm waiting for
this to be solved for Emacs 29.2.  If you think I shouldn't wait,
please tell.

Thanks.

Message #52 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, casouri <at> gmail.com, 67262 <at> debbugs.gnu.org
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 10:11:12 -0500

[Message part 1 (text/plain, inline)]

> On Dec 16, 2023, at 8:59 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> Date: Sat, 16 Dec 2023 15:37:40 +0200
>> Cc: jdtsmith <at> gmail.com, 67262 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry <at> gutov.dev>
>> 
>> On 16/12/2023 14:47, Eli Zaretskii wrote:
>>> How should we make progress with this bug?  I'd like to fix it before
>>> Emacs 29.2 is released.
>> 
>> We're currently working on the general issue in bug#66732.
> 
> Ok, so please close this bug as well, when that general issue is
> resolved.
> 
> And I hope bug#66732 will be solved soon?  Because I'm waiting for
> this to be solved for Emacs 29.2.  If you think I shouldn't wait,
> please tell.

If #66732 is not ready in time for 29.2, I’d suggest at least implementing the “simple” fix of returning `python-syntax-propertize-function’ into service, mentioned above.

[Message part 2 (text/html, inline)]

Message #55 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: casouri <at> gmail.com, 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 17:21:55 +0200

On 16/12/2023 15:59, Eli Zaretskii wrote:
>> Date: Sat, 16 Dec 2023 15:37:40 +0200
>> Cc:jdtsmith <at> gmail.com,67262 <at> debbugs.gnu.org
>> From: Dmitry Gutov<dmitry <at> gutov.dev>
>>
>> On 16/12/2023 14:47, Eli Zaretskii wrote:
>>> How should we make progress with this bug?  I'd like to fix it before
>>> Emacs 29.2 is released.
>> We're currently working on the general issue in bug#66732.
> Ok, so please close this bug as well, when that general issue is
> resolved.
> 
> And I hope bug#66732 will be solved soon?  Because I'm waiting for
> this to be solved for Emacs 29.2.  If you think I shouldn't wait,
> please tell.

We have a few more days, right?

Message #58 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: casouri <at> gmail.com, 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 17:57:01 +0200

> Date: Sat, 16 Dec 2023 17:21:55 +0200
> Cc: casouri <at> gmail.com, jdtsmith <at> gmail.com, 67262 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry <at> gutov.dev>
> 
> On 16/12/2023 15:59, Eli Zaretskii wrote:
> >> Date: Sat, 16 Dec 2023 15:37:40 +0200
> >> Cc:jdtsmith <at> gmail.com,67262 <at> debbugs.gnu.org
> >> From: Dmitry Gutov<dmitry <at> gutov.dev>
> >>
> >> On 16/12/2023 14:47, Eli Zaretskii wrote:
> >>> How should we make progress with this bug?  I'd like to fix it before
> >>> Emacs 29.2 is released.
> >> We're currently working on the general issue in bug#66732.
> > Ok, so please close this bug as well, when that general issue is
> > resolved.
> > 
> > And I hope bug#66732 will be solved soon?  Because I'm waiting for
> > this to be solved for Emacs 29.2.  If you think I shouldn't wait,
> > please tell.
> 
> We have a few more days, right?

Yes.  Likely more than just "few days".

Message #61 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 67262 <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 16 Dec 2023 15:36:49 -0800

[Message part 1 (text/plain, inline)]

> On Dec 16, 2023, at 5:37 AM, Dmitry Gutov <dmitry <at> gutov.dev> wrote:
> 
> On 16/12/2023 14:47, Eli Zaretskii wrote:
>> How should we make progress with this bug?  I'd like to fix it before
>> Emacs 29.2 is released.
> 
> We're currently working on the general issue in bug#66732.

How do you think of this patch? This extends the patch in bug#66732, and adds correct text property to the quotes (I think).

It doesn’t handle the case of “””””” (no content inside the quotes), but I think that’s fine?

[python-triple-quote.patch (application/octet-stream, attachment)]

Message #64 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 67262 <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 17 Dec 2023 15:46:29 +0200

On 17/12/2023 01:36, Yuan Fu wrote:
> How do you think of this patch? This extends the patch in bug#66732, and adds correct text property to the quotes (I think).

I do believe the common pattern should be followed here: a 
syntax-propertize-function and a call to syntax-ppss-flush-cache to 
invalidate when appropriate. See my patch in this bug for reference.

The reasons are performance (deferring work) and keeping state 
management logic in one place.

With your patch in particular, try this:

1. Type """foo -- no text properties.
2. Add """. Check the 3rd and 4th quote characters with 'C-u C-x ='. 
Both should have the (15) 'syntax-table' text property applied.
4. Backspace. Syntax highlighting changes -- the string is not 
highlighted now (that's good, I guess). But both 3rd and 4th quote chars 
retain the text property. (python-info-triple-quoted-string-p) still 
returns the same value, and, more importantly, (syntax-ppss) does too.
5. Backspace 2 more times (the text in the buffer is now '"""foo'). The 
4th quote is gone now, but the 3rd still has the text property.

(nth 3 (syntax-ppss)) still returns t, and 
(python-info-triple-quoted-string-p) still returns the same value.

On steps 1 and 5 the buffer contents are the same, but the syntax-table 
properties are different.

> It doesn’t handle the case of “””””” (no content inside the quotes), but I think that’s fine?

It might be important for some callers of 
(python-info-triple-quoted-string-p). But it should be pretty easy to 
fix, now or later.

Message #67 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: casouri <at> gmail.com, 67262 <at> debbugs.gnu.org, jdtsmith <at> gmail.com
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 23 Dec 2023 11:52:11 +0200

Ping!  Can we please make further progress here to resolve this issue?

> Date: Sun, 17 Dec 2023 15:46:29 +0200
> Cc: Eli Zaretskii <eliz <at> gnu.org>, JD Smith <jdtsmith <at> gmail.com>,
>  67262 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry <at> gutov.dev>
> 
> On 17/12/2023 01:36, Yuan Fu wrote:
> > How do you think of this patch? This extends the patch in bug#66732, and adds correct text property to the quotes (I think).
> 
> I do believe the common pattern should be followed here: a 
> syntax-propertize-function and a call to syntax-ppss-flush-cache to 
> invalidate when appropriate. See my patch in this bug for reference.
> 
> The reasons are performance (deferring work) and keeping state 
> management logic in one place.
> 
> With your patch in particular, try this:
> 
> 1. Type """foo -- no text properties.
> 2. Add """. Check the 3rd and 4th quote characters with 'C-u C-x ='. 
> Both should have the (15) 'syntax-table' text property applied.
> 4. Backspace. Syntax highlighting changes -- the string is not 
> highlighted now (that's good, I guess). But both 3rd and 4th quote chars 
> retain the text property. (python-info-triple-quoted-string-p) still 
> returns the same value, and, more importantly, (syntax-ppss) does too.
> 5. Backspace 2 more times (the text in the buffer is now '"""foo'). The 
> 4th quote is gone now, but the 3rd still has the text property.
> 
> (nth 3 (syntax-ppss)) still returns t, and 
> (python-info-triple-quoted-string-p) still returns the same value.
> 
> On steps 1 and 5 the buffer contents are the same, but the syntax-table 
> properties are different.
> 
> > It doesn’t handle the case of “””””” (no content inside the quotes), but I think that’s fine?
> 
> It might be important for some callers of 
> (python-info-triple-quoted-string-p). But it should be pretty easy to 
> fix, now or later.
> 

Message #70 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, 67262 <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 23 Dec 2023 10:03:42 -0800

> On Dec 23, 2023, at 1:52 AM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> Ping!  Can we please make further progress here to resolve this issue?

Once we merge the fix for bug#66732, we can proceed on this. I have a patch for this bug too.

Yuan

Message #73 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: 67262 <at> debbugs.gnu.org, JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 23 Dec 2023 23:01:35 +0200

On 23/12/2023 20:03, Yuan Fu wrote:
>> On Dec 23, 2023, at 1:52 AM, Eli Zaretskii<eliz <at> gnu.org>  wrote:
>>
>> Ping!  Can we please make further progress here to resolve this issue?
> Once we merge the fix for bug#66732, we can proceed on this. I have a patch for this bug too.

FWIW, I suggest using the first half of the patch attached to 
https://debbugs.gnu.org/67262#23

Only adjusting the positions to add the property to the "inner" pair of 
quotes rather than the outer one (to match python-mode).

Message #76 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 67262 <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 23 Dec 2023 15:51:24 -0800

[Message part 1 (text/plain, inline)]

> On Dec 23, 2023, at 1:01 PM, Dmitry Gutov <dmitry <at> gutov.dev> wrote:
> 
> On 23/12/2023 20:03, Yuan Fu wrote:
>>> On Dec 23, 2023, at 1:52 AM, Eli Zaretskii<eliz <at> gnu.org>  wrote:
>>> 
>>> Ping!  Can we please make further progress here to resolve this issue?
>> Once we merge the fix for bug#66732, we can proceed on this. I have a patch for this bug too.
> 
> FWIW, I suggest using the first half of the patch attached to https://debbugs.gnu.org/67262#23
> 
> Only adjusting the positions to add the property to the "inner" pair of quotes rather than the outer one (to match python-mode).

Something like this? IMHO the code is easier to understand without the syntax-propertize-rules.

Yuan

[python-syntax.patch (application/octet-stream, attachment)]

Message #79 received at 67262 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Yuan Fu <casouri <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 67262 <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sun, 24 Dec 2023 02:45:25 +0200

On 24/12/2023 01:51, Yuan Fu wrote:
> Something like this? IMHO the code is easier to understand without the syntax-propertize-rules.

Sure, fine with me.

> +    (while (search-forward "\"\"\"" end t)+            (put-text-property (1- start) start

I think we also should support the triple-single-quotes, like 
python-syntax-propertize-function does.

> +                               'syntax-table (string-to-syntax "| "))
> +            (put-text-property end (min (1+ end) (point-max))
> +                               'syntax-table (string-to-syntax "| "))))))))

The spaces after "|" seem unnecessary.

Message #84 received at 67262-done <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 67262-done <at> debbugs.gnu.org,
 JD Smith <jdtsmith <at> gmail.com>
Subject: Re: bug#67262: python-ts-mode cannot identify triple-quoted-strings
Date: Sat, 23 Dec 2023 19:10:05 -0800

> On Dec 23, 2023, at 4:45 PM, Dmitry Gutov <dmitry <at> gutov.dev> wrote:
> 
> On 24/12/2023 01:51, Yuan Fu wrote:
>> Something like this? IMHO the code is easier to understand without the syntax-propertize-rules.
> 
> Sure, fine with me.
> 
>> +    (while (search-forward "\"\"\"" end t)+            (put-text-property (1- start) start
> 
> I think we also should support the triple-single-quotes, like python-syntax-propertize-function does.
> 
>> +                               'syntax-table (string-to-syntax "| "))
>> +            (put-text-property end (min (1+ end) (point-max))
>> +                               'syntax-table (string-to-syntax "| "))))))))
> 
> The spaces after "|" seem unnecessary.

Thanks, fixed those things and pushed to emacs-29.

Yuan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.