GNU bug report logs - #45897
27.1; python mode font-lock confused by string concatenation

Previous Next

Package: emacs;

Reported by: Tom Tromey <tom <at> tromey.com>

Date: Fri, 15 Jan 2021 18:05:02 UTC

Severity: normal

Found in version 27.1

Done: Eli Zaretskii <eliz <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 45897 in the body.
You can then email your comments to 45897 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Fri, 15 Jan 2021 18:05:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tom Tromey <tom <at> tromey.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Fri, 15 Jan 2021 18:05:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tom Tromey <tom <at> tromey.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.1; python mode font-lock confused by string concatenation
Date: Fri, 15 Jan 2021 11:03:52 -0700
Consider this 2-line Python file:

x = "hello"""
y = "confused"

If you put this into a .py file, Emacs will font-lock the second line
incorrectly.  It appears that Emacs thinks the `"""' on the first line
is an opening triple quote.  However, it is actually string
concatenation, with the second string being empty.

I tripped over a case like this in some real code.

In GNU Emacs 27.1 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.21, cairo version 1.16.0)
 of 2020-08-20 built on buildvm-x86-24.iad2.fedoraproject.org
Windowing system distributor 'Fedora Project', version 11.0.12010000
System Description: Fedora 32 (Workstation Edition)

Recent messages:
Wrote /home/tromey/.newsrc.eld
Saving /home/tromey/.newsrc.eld...done
(No changes need to be saved)
(New file)
Can’t guess python-indent-offset, using defaults: 4
EOL while scanning string literal [3 times]
Saving file /tmp/q.py...
Wrote /tmp/q.py
Quit
Undo

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules --with-harfbuzz
 --with-cairo --with-json build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS
LIBSYSTEMD JSON PDUMPER GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-pcomplete-mode: t
  erc-netsplit-mode: t
  erc-spelling-mode: t
  erc-truncate-mode: t
  display-fill-column-indicator-mode: t
  flyspell-mode: t
  shell-dirtrack-mode: t
  which-function-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-notify-mode: t
  erc-notifications-mode: t
  erc-match-mode: t
  erc-services-mode: t
  erc-networks-mode: t
  erc-hl-nicks-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  flycheck-mode: t
  savehist-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: python-do-auto-fill
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug tramp-cmds yaml-mode gnus-draft vc-annotate tramp-cache
thai-util thai-word goto-addr log-edit sh-script smie executable cc-mode
cc-fonts cc-guess cc-menus cc-cmds tabify man find-dired rst ffap grep
webjump log-view pcvs-util dabbrev mule-util ggtags etags fileloop
generator bug-reference tcl jka-compr gnus-html url-queue help-fns
radix-tree url-cache mm-url flow-fill misearch multi-isearch shr-color
supercite regi smerge-mode diff python tramp-sh ada-mode ada-imenu align
ada-skel wisi-skel skeleton ada-process wisi-process-parse
ada-indent-user-options ada-core wisi-prj wisi xref wisi-fringe
wisi-parse-common semantic/lex semantic/fw mode-local uniquify-files
project find-file compile erc-list erc-menu erc-join erc-ring
erc-pcomplete erc-netsplit erc-spelling erc-truncate cl-extra smiley
mm-archive gnus-bcklg qp gnus-async gnus-ml disp-table sort gnus-cite
vc-mtn vc-hg mailalias mail-hist nnir gnus-topic nndraft nnmh nnfolder
utf-7 bbdb-gnus bbdb-mua bbdb-com crm gnutls network-stream nsm
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum url
url-proxy url-privacy url-expand url-methods url-history mailcap shr
url-cookie url-domsuf svg dom gnus-group gnus-undo smtpmail gnus-start
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range gnus-win gnus nnheader bbdb-message sendmail mail-extr
message rmc puny rfc822 mml mml-sec epa derived epg epg-config gnus-util
rmail rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader rfc2368 copyright add-log
make-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc-git diff-mode
easy-mmode term/xterm xterm display-fill-column-indicator flyspell
ispell diminish appt diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat shell pcomplete parse-time iso8601 time-date ls-lisp
which-func imenu autorevert filenotify desktop frameset cus-start
cus-load git-link url-util erc-track erc-notify
erc-desktop-notifications erc-match erc-services erc-networks
notifications dbus erc-hl-nicks color erc-button erc-fill erc-stamp
wid-edit erc-goodies erc erc-backend erc-compat format-spec thingatpt pp
erc-loaddefs dired-aux dired-x dired dired-loaddefs warnings advice
vc-dir ewoc vc vc-dispatcher flycheck find-func help-mode rx dash
cc-styles cc-align cc-engine cc-vars cc-defs bbdb bbdb-site timezone
ange-ftp comint ansi-color ring server savehist finder-inf clang-rename
clang-include-fixer let-alist clang-format xml info package easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting xwidget-internal cairo move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1294698 143374)
 (symbols 48 40600 90)
 (strings 32 416876 28749)
 (string-bytes 1 9986293)
 (vectors 16 135630)
 (vector-slots 8 2470110 287678)
 (floats 8 541 727)
 (intervals 56 112191 2695)
 (buffers 1000 395))




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Fri, 19 Jan 2024 16:02:02 GMT) Full text and rfc822 format available.

Message #8 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: Jakub Ječmínek <jecminek.k <at> gmail.com>
To: 45897 <at> debbugs.gnu.org
Cc: Tom Tromey <tom <at> tromey.com>
Subject: Re: bug#45897: 27.1; python mode font-lock confused by string
 concatenation
Date: Fri, 19 Jan 2024 17:01:48 +0100
Tom Tromey <tom <at> tromey.com> writes:

> Consider this 2-line Python file:
>
> x = "hello"""
> y = "confused"
>
> If you put this into a .py file, Emacs will font-lock the second line
> incorrectly.  It appears that Emacs thinks the `"""' on the first line
> is an opening triple quote.  However, it is actually string
> concatenation, with the second string being empty.
>
> I tripped over a case like this in some real code.

Hi, thanks for the bug report and sorry for the late response! I confirm
that this is an issue.

I've prepared a patch which handles this special case:

From f34c84b8b4c629c51d547dd41d20222b32f838bb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k <at> gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
literal concatenation.
---
 lisp/progmodes/python.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..d6534a7ddae 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -908,7 +908,11 @@ is used to limit the scan."
 (defun python-syntax-stringify ()
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
+        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
+                                                      (line-beginning-position) (point))))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+        (string-literal-concat (and (null string-start)
+                           (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +925,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+         (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
--
2.39.3 (Apple Git-145)


If the patch is good enough, I'll sign the paperwork (I've most likely
crossed the line limit).

Best,

Jakub Ječmínek




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Fri, 19 Jan 2024 17:27:02 GMT) Full text and rfc822 format available.

Message #11 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: Jakub Ječmínek <jecminek.k <at> gmail.com>
To: 45897 <at> debbugs.gnu.org
Subject: bug#45897: [PATCH] 27.1; python mode font-lock confused by string
 concatenation
Date: Fri, 19 Jan 2024 18:25:57 +0100
Tags: patch

I'm sending the patch once again with correct indentation and patch tag
(I'm still figuring out how the bug tracker works).

From 93d60a342c3d12b46d9fece443d63cdb4101a745 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k <at> gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
literal concatenation.
---
 lisp/progmodes/python.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..ca014469952 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -908,7 +908,11 @@ is used to limit the scan."
 (defun python-syntax-stringify ()
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
+        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
+                                                      (line-beginning-position) (point))))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+        (string-literal-concat (and (null string-start)
+                                    (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +925,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+         (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
--
2.39.3 (Apple Git-145)

I'm more than happy to make further changes if need be.

Best,

Jakub Ječmínek




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Sat, 20 Jan 2024 10:11:02 GMT) Full text and rfc822 format available.

Message #14 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jakub Ječmínek <jecminek.k <at> gmail.com>,
 kobarity <kobarity <at> gmail.com>
Cc: 45897 <at> debbugs.gnu.org
Subject: Re: bug#45897: [PATCH] 27.1;
 python mode font-lock confused by string concatenation
Date: Sat, 20 Jan 2024 12:09:50 +0200
> From: Jakub Ječmínek <jecminek.k <at> gmail.com>
> Date: Fri, 19 Jan 2024 18:25:57 +0100
> 
> Tags: patch
> 
> I'm sending the patch once again with correct indentation and patch tag
> (I'm still figuring out how the bug tracker works).

Thanks.

Kobarity, any comments?

> >From 93d60a342c3d12b46d9fece443d63cdb4101a745 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k <at> gmail.com>
> Date: Fri, 19 Jan 2024 16:38:21 +0100
> Subject: [PATCH] Fix syntax highlighting after string literal concatenation in
>  python-mode
> 
> * lisp/progmodes/python.el (python-syntax-stringify): Fix incorrect font-lock after string
> literal concatenation.
> ---
>  lisp/progmodes/python.el | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index e2f614f52c2..ca014469952 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -908,7 +908,11 @@ is used to limit the scan."
>  (defun python-syntax-stringify ()
>    "Put `syntax-table' property correctly on single/triple quotes."
>    (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
> +        (line-ppss (save-excursion (backward-char 3) (parse-partial-sexp
> +                                                      (line-beginning-position) (point))))
>           (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
> +        (string-literal-concat (and (null string-start)
> +                                    (and (not (null (nth 3 line-ppss))) (nth 8 line-ppss))))
>           (quote-starting-pos (- (point) 3))
>           (quote-ending-pos (point)))
>      (cond ((or (nth 4 ppss)             ;Inside a comment
> @@ -921,6 +925,8 @@ is used to limit the scan."
>            ((nth 5 ppss)
>             ;; The first quote is escaped, so it's not part of a triple quote!
>             (goto-char (1+ quote-starting-pos)))
> +          ;; Handle string literal concatenation (bug#45897)
> +         (string-literal-concat nil)
>            ((null string-start)
>             ;; This set of quotes delimit the start of a string.  Put
>             ;; string fence syntax on last quote. (bug#49518)
> --
> 2.39.3 (Apple Git-145)
> 
> I'm more than happy to make further changes if need be.
> 
> Best,
> 
> Jakub Ječmínek
> 
> 
> 
> 




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Sat, 20 Jan 2024 15:42:01 GMT) Full text and rfc822 format available.

Message #17 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Jakub Ječmínek <jecminek.k <at> gmail.com>
Cc: 45897 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#45897: [PATCH] 27.1;
 python mode font-lock confused by string concatenation
Date: Sun, 21 Jan 2024 00:41:22 +0900
[Message part 1 (text/plain, inline)]
On Sat, 20 Jan 2024 19:09:50 +0900,
Eli Zaretskii wrote:
> 
> > From: Jakub Ječmínek <jecminek.k <at> gmail.com>
> > Date: Fri, 19 Jan 2024 18:25:57 +0100
> > 
> > Tags: patch
> > 
> > I'm sending the patch once again with correct indentation and patch tag
> > (I'm still figuring out how the bug tracker works).
> 
> Thanks.
> 
> Kobarity, any comments?

Thank you for the patch.  I tried it and confirmed that the example
shown by Tom Tromey is fixed.

I think ppss and line-ppss would be the same in this example.  Can you
give an example of a case where ppss and line-ppss are different?

I noticed an error in the commit-msg hook when applying the patch.
CONTRIBUTE states the following:

- Lines in ChangeLog entries should preferably be not longer than 63
  characters, and must not exceed 78 characters, unless they consist
  of a single word of at most 140 characters; this 78/140 limit is
  enforced by a commit hook.

Also, the indentation still looks wrong to me.  It might be better to
attach the patch instead of pasting it into the body of the mail.

Attached is a patch to add an ERT that identifies this issue.  Please
add it to your patch if you like.
[0001-Add-an-ERT-for-bug-45897.patch (text/plain, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Sat, 20 Jan 2024 23:35:02 GMT) Full text and rfc822 format available.

Message #20 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: Jakub Ječmínek <jecminek.k <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: 45897 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#45897: [PATCH] 27.1; python mode font-lock confused by
 string concatenation
Date: Sun, 21 Jan 2024 00:33:55 +0100
[Message part 1 (text/plain, inline)]
kobarity <kobarity <at> gmail.com> writes:

> Thank you for the patch.  I tried it and confirmed that the example
> shown by Tom Tromey is fixed.
>
> I think ppss and line-ppss would be the same in this example.  Can you
> give an example of a case where ppss and line-ppss are different?

Well, it would be different in case like this:

#+BEGIN_SRC python
" # <- forgotten quote
a = "abc"""
#+END_SRC

`ppss' would in the context of `python-syntax-stringify' evaluate to
'(0 nil 8 nil nil nil 0 nil nil nil nil), while `line-ppss' to
'(0 nil 5 34 nil nil 0 nil 7 nil nil). Note the 3rd and 8th
element. Here's the snippet to test it:

#+BEGIN_SRC emacs-lisp
(with-temp-buffer
  (insert "\"
a = \"abc\"\"\"")
  (backward-char 3)
  (syntax-ppss)
  (parse-partial-sexp (line-beginning-position) (point)))
#+END_SRC

I was thinking that we want to consider only cases where the quote is on
the same line but I guess I wanted to solve a problem we don't
have. I've revised the patch to be more simple, please see attached.

> I noticed an error in the commit-msg hook when applying the patch.
> CONTRIBUTE states the following:
>
> - Lines in ChangeLog entries should preferably be not longer than 63
>   characters, and must not exceed 78 characters, unless they consist
>   of a single word of at most 140 characters; this 78/140 limit is
>   enforced by a commit hook.

Thanks, fixed.

> Also, the indentation still looks wrong to me.  It might be better to
> attach the patch instead of pasting it into the body of the mail.

Please see attached.

> Attached is a patch to add an ERT that identifies this issue.  Please
> add it to your patch if you like.

Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
description.

Best,

Jakub Ječmínek

[0001-Fix-syntax-highlighting-after-string-literal-concat-.patch (text/plain, inline)]
From af1f029bde36456b359e9768a8e525fdac6db3e3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <jecminek.k <at> gmail.com>
Date: Fri, 19 Jan 2024 16:38:21 +0100
Subject: [PATCH] Fix syntax highlighting after string literal concat in
 python-mode

* lisp/progmodes/python.el (python-syntax-stringify): Fix
incorrect font-lock after string literal concatenation.
(Bug#45897)

* test/lisp/progmodes/python-tests.el
(python-font-lock-string-literal-concatenation): New test.

Co-authored-by: kobarity <kobarity <at> gmail.com>
---
 lisp/progmodes/python.el            |  3 +++
 test/lisp/progmodes/python-tests.el | 12 ++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e2f614f52c2..bee9c1d8cda 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -909,6 +909,7 @@ is used to limit the scan."
   "Put `syntax-table' property correctly on single/triple quotes."
   (let* ((ppss (save-excursion (backward-char 3) (syntax-ppss)))
          (string-start (and (eq t (nth 3 ppss)) (nth 8 ppss)))
+         (string-literal-concat (numberp (nth 3 ppss)))
          (quote-starting-pos (- (point) 3))
          (quote-ending-pos (point)))
     (cond ((or (nth 4 ppss)             ;Inside a comment
@@ -921,6 +922,8 @@ is used to limit the scan."
           ((nth 5 ppss)
            ;; The first quote is escaped, so it's not part of a triple quote!
            (goto-char (1+ quote-starting-pos)))
+          ;; Handle string literal concatenation (bug#45897)
+          (string-literal-concat nil)
           ((null string-start)
            ;; This set of quotes delimit the start of a string.  Put
            ;; string fence syntax on last quote. (bug#49518)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..59957ff0712 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -660,6 +660,18 @@ r'\\x12 \123 \\n \\u1234 \\U00010348 \\N{Plus-Minus Sign}'"
      (3 . font-lock-string-face) (14)
      (16 . font-lock-string-face))))
 
+(ert-deftest python-font-lock-string-literal-concatenation ()
+  "Test for bug#45897."
+  (python-tests-assert-faces
+   "x = \"hello\"\"\"
+y = \"confused\""
+   '((1 . font-lock-variable-name-face) (2)
+     (3 . font-lock-operator-face) (4)
+     (5 . font-lock-string-face) (14)
+     (15 . font-lock-variable-name-face) (16)
+     (17 . font-lock-operator-face) (18)
+     (19 . font-lock-string-face))))
+
 
 ;;; Indentation
 
-- 
2.39.3 (Apple Git-145)


Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#45897; Package emacs. (Sun, 21 Jan 2024 07:40:02 GMT) Full text and rfc822 format available.

Message #23 received at 45897 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: Jakub Ječmínek <jecminek.k <at> gmail.com>
Cc: 45897 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#45897: [PATCH] 27.1;
 python mode font-lock confused by string concatenation
Date: Sun, 21 Jan 2024 16:39:43 +0900
Jakub Ječmínek wrote:
> kobarity <kobarity <at> gmail.com> writes:
> 
> > Thank you for the patch.  I tried it and confirmed that the example
> > shown by Tom Tromey is fixed.
> >
> > I think ppss and line-ppss would be the same in this example.  Can you
> > give an example of a case where ppss and line-ppss are different?
> 
> Well, it would be different in case like this:
> 
> #+BEGIN_SRC python
> " # <- forgotten quote
> a = "abc"""
> #+END_SRC
> 
> `ppss' would in the context of `python-syntax-stringify' evaluate to
> '(0 nil 8 nil nil nil 0 nil nil nil nil), while `line-ppss' to
> '(0 nil 5 34 nil nil 0 nil 7 nil nil). Note the 3rd and 8th
> element. Here's the snippet to test it:
> 
> #+BEGIN_SRC emacs-lisp
> (with-temp-buffer
>   (insert "\"
> a = \"abc\"\"\"")
>   (backward-char 3)
>   (syntax-ppss)
>   (parse-partial-sexp (line-beginning-position) (point)))
> #+END_SRC
> 
> I was thinking that we want to consider only cases where the quote is on
> the same line but I guess I wanted to solve a problem we don't
> have. I've revised the patch to be more simple, please see attached.
> 
> > I noticed an error in the commit-msg hook when applying the patch.
> > CONTRIBUTE states the following:
> >
> > - Lines in ChangeLog entries should preferably be not longer than 63
> >   characters, and must not exceed 78 characters, unless they consist
> >   of a single word of at most 140 characters; this 78/140 limit is
> >   enforced by a commit hook.
> 
> Thanks, fixed.
> 
> > Also, the indentation still looks wrong to me.  It might be better to
> > attach the patch instead of pasting it into the body of the mail.
> 
> Please see attached.
> 
> > Attached is a patch to add an ERT that identifies this issue.  Please
> > add it to your patch if you like.
> 
> Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
> description.

Thank you.  The revised patch looks good to me.




Reply sent to Eli Zaretskii <eliz <at> gnu.org>:
You have taken responsibility. (Sat, 27 Jan 2024 10:08:02 GMT) Full text and rfc822 format available.

Notification sent to Tom Tromey <tom <at> tromey.com>:
bug acknowledged by developer. (Sat, 27 Jan 2024 10:08:02 GMT) Full text and rfc822 format available.

Message #28 received at 45897-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: kobarity <kobarity <at> gmail.com>
Cc: 45897-done <at> debbugs.gnu.org, jecminek.k <at> gmail.com
Subject: Re: bug#45897: [PATCH] 27.1;
 python mode font-lock confused by string concatenation
Date: Sat, 27 Jan 2024 12:06:59 +0200
> Date: Sun, 21 Jan 2024 16:39:43 +0900
> From: kobarity <kobarity <at> gmail.com>
> Cc: 45897 <at> debbugs.gnu.org,
> 	Eli Zaretskii <eliz <at> gnu.org>
> 
> Jakub Ječmínek wrote:
> > kobarity <kobarity <at> gmail.com> writes:
> > 
> > 
> > > Attached is a patch to add an ERT that identifies this issue.  Please
> > > add it to your patch if you like.
> > 
> > Thanks, I've added it to my patch and added 'Co-authored-by:' to commit
> > description.
> 
> Thank you.  The revised patch looks good to me.

Thanks, installed on master, and closing the bug.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sat, 24 Feb 2024 12:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 33 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.