GNU bug report logs - #62142
30.0.50; python-fill-paragraph problems on filling strings

Previous Next

Package: emacs;

Reported by: kobarity <kobarity <at> gmail.com>

Date: Sun, 12 Mar 2023 08:40:02 UTC

Severity: normal

Found in version 30.0.50

Done: Eli Zaretskii <eliz <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 62142 in the body.
You can then email your comments to 62142 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Sun, 12 Mar 2023 08:40:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to kobarity <kobarity <at> gmail.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Sun, 12 Mar 2023 08:40:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>,
 João Távora <joaotavora <at> gmail.com>
Subject: 30.0.50; python-fill-paragraph problems on filling strings
Date: Sun, 12 Mar 2023 17:39:37 +0900
[Message part 1 (text/plain, inline)]
Hi,

There are some problems in filling Python's strings, which are caused
by the same portion of the code.

Here are the steps to reproduce the first problem:

1. emacs -Q
2. C-x C-f somefile1.py
3. Enter the following program:

#+begin_src python
s1 = 'abc'
s2 = 'def'
#+end_src

4. Place point within the first string (abc).
6. M-q

The lines will be merged into a single line.

#+begin_src python
s1 = 'abc' s2 = 'def'
#+end_src

This problem occurs in Emacs 27 and later.

Second problem can be reproduced with the following steps:

1. emacs -Q
2. C-x C-f somefile2.py
3. Enter the following program:

#+begin_src python
s = '''abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz'''
#+end_src

4. Place point at the first or third quote of the starting triple quotes.
5. M-q

The line is not filled.  However, if the point is placed at the second
quote or within the string, the line will be filled as follows:

#+begin_src python
s = '''abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyz'''
#+end_src

When the point is placed at the ending triple quotes, the line will be
filled if the point is at the first or third quote.

This problem occurs in Emacs 28 and later.

Third problem can be reproduced with the following steps:

1. emacs -Q
2. C-x C-f somefile3.py
3. Enter the following program:

#+begin_src python
s = 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz'
#+end_src

4. Place point within the string.
5. M-q

The line will be split into 2 lines.

#+begin_src python
s = 'abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyz'
#+end_src

This is not a correct Python program anymore because a single quoted
string can not span multiple lines unless the end of line is escaped
using a backslash.

The docstring of `python-fill-paragraph' says:
`fill-paragraph-function' handling multi-line strings and possibly comments.

Although it is possible to make a multi-line single quoted string by
escaping the end of line using backslash, triple quoted strings are
usually used for multi-line strings.  So I think
`python-fill-paragraph' should not do anything with single quoted
strings.


The cause of the first problem is similar to Bug#58780.  In
`python-fill-string', the string is searched for with the following
code:

           (or (python-syntax-context 'string)
               (and (equal (string-to-syntax "|")
                           (syntax-after (point)))
                    (point)))))

and the end of the string is searched for with the following code:

            (or (re-search-forward (rx (syntax string-delimiter)) nil t)
                (goto-char (point-max)))

These codes used to be appropriate, but it is not correct after the
following commit which is included in Emacs 27 and later:

commit 4b39b741f1949ebad1dfccc5032dfce521bedc2a
Author: Stefan Monnier <monnier <at> iro.umontreal.ca>
Date:   Tue Apr 9 15:08:21 2019 -0400

    python.el: don't syntax-propertize single/double quoted strings

    * lisp/progmodes/python.el (python-syntax-propertize-function):
    Only mark triple-quoted strings, let the normal syntax-table handle
    the rest.
    (python-syntax-stringify): Adjust accordingly.

From this commit, single quoted strings can not be identified by
string-delimiter and can only be identified by string-quote.

In the fix of Bug#58780, I added `forward-sexp' to search for the end
of single quoted string.  However, considering the third problem, I
would prefer to have `python-fill-paragraph' and `python-fill-string'
search only for triple quoted strings in this case.


The second problem occurs after the following commit included in Emacs
28 and later:

commit 0646c6817139aa905a2f6079fdc82eb4be944de0
Author: João Távora <joaotavora <at> gmail.com>
Date:   Sun Sep 19 11:42:20 2021 +0100

    Make syntax-ppss more accurate for Python triple quotes (bug#49518)

    By putting delimiter syntax on the last character of Python
    triple-quoted strings, this makes syntax-ppss be more accurate.

This commit narrows the range of string-delimiter from outermost
quotes to innermost quotes, and results in `python-fill-paragraph' not
recognizing the string when point is at some quotes.


Attached is a patch to fix these problems.  Main changes are:

1. single-quoted-string and triple-quoted-string are added as a TYPE
   argument of `python-syntax-context'.
2. New helper function `python-info-triple-quoted-string-p' is added
   to detect triple quoted strings including quotes.
3. `python-fill-paragraph' and `python-fill-string' use
   `python-info-triple-quoted-string-p' to detect triple quoted
   strings only.

I look forward to your comments.

--
In GNU Emacs 30.0.50 (build 3, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.16.0, Xaw scroll bars) of 2023-03-10 built on ubuntu
Repository revision: 8ee205d232574e12921b052c7e93b7e16d6f1187
Repository branch: master
System Description: Ubuntu 22.04.2 LTS

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XPM
LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils python json
subr-x map rx project pcase treesit cl-seq comint ansi-osc ring
cl-loaddefs cl-lib ansi-color term/screen term/xterm xterm byte-opt gv
bytecomp byte-compile rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads inotify dynamic-setting
system-font-setting font-render-setting cairo x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 60054 7620)
 (symbols 48 7362 0)
 (strings 32 20143 1278)
 (string-bytes 1 593570)
 (vectors 16 11583)
 (vector-slots 8 139242 9455)
 (floats 8 30 11543)
 (intervals 56 239 0)
 (buffers 984 12))
[0001-Fix-python-fill-paragraph-problems-on-filling-string.patch (application/octet-stream, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Sun, 12 Mar 2023 18:04:01 GMT) Full text and rfc822 format available.

Message #8 received at 62142 <at> debbugs.gnu.org (full text, mbox):

From: João Távora <joaotavora <at> gmail.com>
To: kobarity <kobarity <at> gmail.com>
Cc: 62142 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 30.0.50; python-fill-paragraph problems on filling strings
Date: Sun, 12 Mar 2023 18:05:48 +0000
kobarity <kobarity <at> gmail.com> writes:

> Attached is a patch to fix these problems.  Main changes are:
>
> 1. single-quoted-string and triple-quoted-string are added as a TYPE
>    argument of `python-syntax-context'.
> 2. New helper function `python-info-triple-quoted-string-p' is added
>    to detect triple quoted strings including quotes.
> 3. `python-fill-paragraph' and `python-fill-string' use
>    `python-info-triple-quoted-string-p' to detect triple quoted
>    strings only.
>
> I look forward to your comments.

Hi kobarity, 

I suppose the test suite still passes.  Did also enable
electric-pair-mode and play around and find acceptable behaviour?  Does
bug#49518 stay solved?

If so, this is fine with me, and let's let Stefan chime in.

João






Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Mon, 13 Mar 2023 14:18:01 GMT) Full text and rfc822 format available.

Message #11 received at 62142 <at> debbugs.gnu.org (full text, mbox):

From: kobarity <kobarity <at> gmail.com>
To: João Távora <joaotavora <at> gmail.com>
Cc: 62142 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 30.0.50; python-fill-paragraph problems on filling strings
Date: Mon, 13 Mar 2023 23:17:48 +0900
João Távora wrote:
> I suppose the test suite still passes.  Did also enable
> electric-pair-mode and play around and find acceptable behaviour?  Does
> bug#49518 stay solved?

Hi João,

As far as running some of the basic checks described in Bug#49518, I
don't see any problems.  Except for adding new argument values and a
new helper, my patch only modifies `python-fill-paragraph' and
`python-fill-string', which are not supposed to be called by
electric-pair-mode, so it should not affect electric-pair-mode.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Thu, 16 Mar 2023 07:21:01 GMT) Full text and rfc822 format available.

Message #14 received at 62142 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: kobarity <kobarity <at> gmail.com>
Cc: 62142 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca, joaotavora <at> gmail.com
Subject: Re: bug#62142: 30.0.50;
 python-fill-paragraph problems on filling strings
Date: Thu, 16 Mar 2023 09:20:04 +0200
> Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>,
>  João Távora <joaotavora <at> gmail.com>
> Date: Sun, 12 Mar 2023 17:39:37 +0900
> From: kobarity <kobarity <at> gmail.com>
> 
> The cause of the first problem is similar to Bug#58780.  In
> `python-fill-string', the string is searched for with the following
> code:
> 
>            (or (python-syntax-context 'string)
>                (and (equal (string-to-syntax "|")
>                            (syntax-after (point)))
>                     (point)))))
> 
> and the end of the string is searched for with the following code:
> 
>             (or (re-search-forward (rx (syntax string-delimiter)) nil t)
>                 (goto-char (point-max)))
> 
> These codes used to be appropriate, but it is not correct after the
> following commit which is included in Emacs 27 and later:
> 
> commit 4b39b741f1949ebad1dfccc5032dfce521bedc2a
> Author: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Date:   Tue Apr 9 15:08:21 2019 -0400
> 
>     python.el: don't syntax-propertize single/double quoted strings
> 
>     * lisp/progmodes/python.el (python-syntax-propertize-function):
>     Only mark triple-quoted strings, let the normal syntax-table handle
>     the rest.
>     (python-syntax-stringify): Adjust accordingly.
> 
> From this commit, single quoted strings can not be identified by
> string-delimiter and can only be identified by string-quote.
> 
> In the fix of Bug#58780, I added `forward-sexp' to search for the end
> of single quoted string.  However, considering the third problem, I
> would prefer to have `python-fill-paragraph' and `python-fill-string'
> search only for triple quoted strings in this case.
> 
> 
> The second problem occurs after the following commit included in Emacs
> 28 and later:
> 
> commit 0646c6817139aa905a2f6079fdc82eb4be944de0
> Author: João Távora <joaotavora <at> gmail.com>
> Date:   Sun Sep 19 11:42:20 2021 +0100
> 
>     Make syntax-ppss more accurate for Python triple quotes (bug#49518)
> 
>     By putting delimiter syntax on the last character of Python
>     triple-quoted strings, this makes syntax-ppss be more accurate.
> 
> This commit narrows the range of string-delimiter from outermost
> quotes to innermost quotes, and results in `python-fill-paragraph' not
> recognizing the string when point is at some quotes.
> 
> 
> Attached is a patch to fix these problems.  Main changes are:
> 
> 1. single-quoted-string and triple-quoted-string are added as a TYPE
>    argument of `python-syntax-context'.
> 2. New helper function `python-info-triple-quoted-string-p' is added
>    to detect triple quoted strings including quotes.
> 3. `python-fill-paragraph' and `python-fill-string' use
>    `python-info-triple-quoted-string-p' to detect triple quoted
>    strings only.
> 
> I look forward to your comments.

Stefan and João, any comments?  If not, I'd like to install this on
emacs-29 soon.




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Thu, 16 Mar 2023 08:56:02 GMT) Full text and rfc822 format available.

Message #17 received at 62142 <at> debbugs.gnu.org (full text, mbox):

From: João Távora <joaotavora <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 62142 <at> debbugs.gnu.org, kobarity <kobarity <at> gmail.com>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#62142: 30.0.50;
 python-fill-paragraph problems on filling strings
Date: Thu, 16 Mar 2023 08:55:16 +0000
On Thu, Mar 16, 2023, 07:20 Eli Zaretskii <eliz <at> gnu.org> wrote:
>
>
> > 3. `python-fill-paragraph' and `python-fill-string' use
> >    `python-info-triple-quoted-string-p' to detect triple quoted
> >    strings only.
> >
> > I look forward to your comments.
>
> Stefan and João, any comments?  If not, I'd like to install this on
> emacs-29 soon.


If electric-pair-mode integration is unaffected (which
kobarity reports it is), LGTM (but I didn't really "look").

João




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#62142; Package emacs. (Thu, 16 Mar 2023 13:28:01 GMT) Full text and rfc822 format available.

Message #20 received at 62142 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 62142 <at> debbugs.gnu.org, kobarity <kobarity <at> gmail.com>, joaotavora <at> gmail.com
Subject: Re: bug#62142: 30.0.50; python-fill-paragraph problems on filling
 strings
Date: Thu, 16 Mar 2023 09:27:45 -0400
> Stefan and João, any comments?  If not, I'd like to install this on
> emacs-29 soon.

I haven't had time to look deeply, but it looks sane,


        Stefan





Reply sent to Eli Zaretskii <eliz <at> gnu.org>:
You have taken responsibility. (Thu, 16 Mar 2023 15:01:02 GMT) Full text and rfc822 format available.

Notification sent to kobarity <kobarity <at> gmail.com>:
bug acknowledged by developer. (Thu, 16 Mar 2023 15:01:02 GMT) Full text and rfc822 format available.

Message #25 received at 62142-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 62142-done <at> debbugs.gnu.org, kobarity <at> gmail.com, joaotavora <at> gmail.com
Subject: Re: bug#62142: 30.0.50; python-fill-paragraph problems on filling
 strings
Date: Thu, 16 Mar 2023 17:00:07 +0200
> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: kobarity <kobarity <at> gmail.com>,  62142 <at> debbugs.gnu.org,
>   joaotavora <at> gmail.com
> Date: Thu, 16 Mar 2023 09:27:45 -0400
> 
> > Stefan and João, any comments?  If not, I'd like to install this on
> > emacs-29 soon.
> 
> I haven't had time to look deeply, but it looks sane,

Thanks, installed on the emacs-29 branch, and closing the bug.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Fri, 14 Apr 2023 11:24:08 GMT) Full text and rfc822 format available.

This bug report was last modified 348 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.