GNU bug report logs - #58687
29.0.50; Enabling pp-use-max-width dramatically slows down formatting of large sexps like org-persist--index

Previous Next

To reply to this bug, email your comments to 58687 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.50; Enabling pp-use-max-width dramatically slows down
 formatting of large sexps like org-persist--index
Date: Fri, 21 Oct 2022 12:59:17 +0000

[Message part 1 (text/plain, inline)]

I set the new setting `pp-use-max-width' to t to fold the output of interactive commands  like `pp-eval-last-sexp'. However, I found out that this increased the time that kill-emacs took to run by 30 seconds. By profiling I found that this is because I use the latest org-version (9.5.5-gcb1359) on the main branch with persistent caching enabled and in `kill-emacs-hook' it then saves the `org-persist--index' to a file via `org-persist-write:index'. The index sexp can be quite large when one has many org files and when org uses pp on it that takes a long time.

What I wasn't aware when I customized pp-use-max-width is that it's used by other packages to format lisp code and this might slow down these operations quiet significantly, as I just wanted to set only for my custom pretty-printing purposes of usually small sexps.

My suggestion is to put a note into the emacs-news and the variable docstring that it can have significant performance penalties on large sexps. If the performance can be improved that would be also nice, but not sure if that's possible. I assume the authors are aware of the downsides, but it then should be documented well at least.

I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.

I had reported this first on the org-mode mailing list at https://lists.gnu.org/archive/html/emacs-orgmode/2022-10/msg00734.html.

Best regards,
Michael Eliachevitch

--
In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.34, cairo version 1.17.6) of 2022-10-20 built on e490
Repository revision: f61db42fc580fb671016c77be942506d9081ac2c
Repository branch: master
System Description: Arch Linux

Configured using:
'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
--localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
--with-modules --without-libotf --without-m17n-flt --without-gconf
--enable-link-time-optimization --with-native-compilation
--with-xinput2 --with-pgtk --without-xaw3d --with-sound=alsa
--with-xwidgets --without-gpm --without-compress-install
'--program-transform-name=s/\([ec]tags\)/\1.emacs/'
'CFLAGS=-march=native -mtune=generic -O3 -pipe -fno-plt -fexceptions
-Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
-fstack-clash-protection -fcf-protection'
LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM
XWIDGETS GTK3 ZLIB

Important settings:
 value of $XMODIFIERS: @im=fcitx
 locale-coding-system: nil

Major mode: ELisp/d

Minor modes in effect:
 tooltip-mode: t
 global-eldoc-mode: t
 eldoc-mode: t
 show-paren-mode: t
 electric-indent-mode: t
 mouse-wheel-mode: t
 tool-bar-mode: t
 menu-bar-mode: t
 file-name-shadow-mode: t
 global-font-lock-mode: t
 font-lock-mode: t
 blink-cursor-mode: t
 line-number-mode: t
 indent-tabs-mode: t
 transient-mark-mode: t
 auto-composition-mode: t
 auto-encryption-mode: t
 auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date edebug debug
backtrace find-func benchmark pp vc-git diff-mode easy-mmode
vc-dispatcher cl-loaddefs comp comp-cstr warnings icons subr-x rx cl-seq
cl-macs gv cl-extra help-mode bytecomp byte-compile cconv cl-lib rmc
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
pgtk-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads xwidget-internal dbusbind inotify dynamic-setting
system-font-setting font-render-setting cairo gtk pgtk lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 92875 7823)
(symbols 48 8353 1)
(strings 32 23567 2442)
(string-bytes 1 712274)
(vectors 16 17653)
(vector-slots 8 356365 12834)
(floats 8 30 53)
(intervals 56 434 3)
(buffers 1000 15))

[org-persist-index.el (text/plain, attachment)]

[benchmark-pp-on-org-persist-index.el (text/plain, attachment)]

Message #8 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
Cc: 58687 <at> debbugs.gnu.org
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Thu, 12 Jan 2023 11:27:35 +0000

Michael Eliachevitch <m.eliachevitch <at> posteo.de> writes:

> I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.

I'd like to bump this bug.

If it helps, here are the steps to reproduce:

1. Save the attached files to same folder
2. Open benchmark-pp-on-org-persist-index.el
3. M-x eval-buffer

Observed: (19.617422592 2 1.0353073050000035)
Expected: `pp' not taking ~20sec to write 15k of elisp data.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message #11 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 58687 <at> debbugs.gnu.org, m.eliachevitch <at> posteo.de
Subject: Re: bug#58687: 29.0.50;
 Enabling pp-use-max-width dramatically slows down formatting of large
 sexps like org-persist--index
Date: Thu, 12 Jan 2023 15:36:25 +0200

> Cc: 58687 <at> debbugs.gnu.org
> From: Ihor Radchenko <yantar92 <at> posteo.net>
> Date: Thu, 12 Jan 2023 11:27:35 +0000
> 
> Michael Eliachevitch <m.eliachevitch <at> posteo.de> writes:
> 
> > I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.
> 
> I'd like to bump this bug.

Why?

> 1. Save the attached files to same folder
> 2. Open benchmark-pp-on-org-persist-index.el
> 3. M-x eval-buffer
> 
> Observed: (19.617422592 2 1.0353073050000035)
> Expected: `pp' not taking ~20sec to write 15k of elisp data.

Did you look at what pp.el does when pp-use-max-width is non-nil?  I
show a profile below, to make that clear.

The "regular" pp (when pp-use-max-width is nil) finishes almost
instantaneously in this case.

       40048  85% - command-execute
       40048  85%  - call-interactively
       40046  85%   - funcall-interactively
       40046  85%    - execute-extended-command
       40044  85%     - command-execute
       40044  85%      - call-interactively
       40044  85%       - funcall-interactively
       40044  85%        - eval-buffer
       40043  85%         - let
       40042  85%          - benchmark-call
       40041  85%           - #<lambda 0x5461c05f>
       40041  85%            - pp
       40041  85%             - pp-to-string
       40041  85%              - pp-emacs-lisp-code
       40028  85%               - pp--insert-lisp
       40028  85%                - pp--format-list
       40028  85%                 - pp--insert
       38710  82%                  - pp--insert-lisp
       38710  82%                   - pp--format-list
       38708  82%                    - pp--insert
       20781  44%                     - pp--insert-lisp
       20780  44%                      - pp--format-list
       20780  44%                       - pp--insert
       13619  28%                        - pp--indent-buffer
       13603  28%                         - lisp-indent-line
       12595  26%                          - lisp-ppss
        9763  20%                           - syntax-ppss
           6   0%                              #<compiled -0x163d6f700e816ab6>
           1   0%                              syntax-propertize

Message #14 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 58687 <at> debbugs.gnu.org, m.eliachevitch <at> posteo.de
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Thu, 12 Jan 2023 16:19:37 +0000

Eli Zaretskii <eliz <at> gnu.org> writes:

>> > I attached a file with the value of my `org-persist--index` expression and a file with a benchmark where I run pp on it, which took me 25s when running it with emacs -Q. Until recently my org persist index was much longer, but I pruned it a bit back when I wasn't aware what exactly caused the slowdown.
>> 
>> I'd like to bump this bug.
>
> Why?

Because, I'd like to know if there are chances that it could be fixed.
The original report demonstrates how this issue slows down Emacs quit
time dramatically. I should either let-bind `pp-use-max-with' to nil in
org-persist.el to work around the problem or wait for the fix.

>> Observed: (19.617422592 2 1.0353073050000035)
>> Expected: `pp' not taking ~20sec to write 15k of elisp data.
>
> Did you look at what pp.el does when pp-use-max-width is non-nil?  I
> show a profile below, to make that clear.
>
> The "regular" pp (when pp-use-max-width is nil) finishes almost
> instantaneously in this case.
> ...
>        40041  85%            - pp
>        40041  85%             - pp-to-string
>        40041  85%              - pp-emacs-lisp-code
>        40028  85%               - pp--insert-lisp
>        40028  85%                - pp--format-list
>        40028  85%                 - pp--insert
>        38710  82%                  - pp--insert-lisp
>        38710  82%                   - pp--format-list

So, the current `pp' implementation is re-parsing from bob for every
nested list inside sexp. This is quadratic scaling, and, as the repro
demonstrates, the time goes up very quickly. Is the new option usable
at all in practice?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message #17 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Visuwesh <visuweshm <at> gmail.com>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 58687 <at> debbugs.gnu.org,
 m.eliachevitch <at> posteo.de
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Thu, 12 Jan 2023 22:03:50 +0530

[வியாழன் ஜனவரி 12, 2023] Ihor Radchenko wrote:

> [...]
> So, the current `pp' implementation is re-parsing from bob for every
> nested list inside sexp. This is quadratic scaling, and, as the repro
> demonstrates, the time goes up very quickly. Is the new option usable
> at all in practice?

IME, unfortunately not.  Like the OP, I saw the announcement of the new
user option, got excited and enabled it right away... only to be hit by
bugs more often than not, add to it the slowness, it wasn't a fun
experience.
Personally, I always thought it would be best if the user facing
commands like pp-eval-sexp and friends alone respected the user option.

Message #20 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ihor Radchenko <yantar92 <at> posteo.net>
Cc: 58687 <at> debbugs.gnu.org, m.eliachevitch <at> posteo.de
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Thu, 12 Jan 2023 18:39:35 +0200

> From: Ihor Radchenko <yantar92 <at> posteo.net>
> Cc: m.eliachevitch <at> posteo.de, 58687 <at> debbugs.gnu.org
> Date: Thu, 12 Jan 2023 16:19:37 +0000
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >        40041  85%            - pp
> >        40041  85%             - pp-to-string
> >        40041  85%              - pp-emacs-lisp-code
> >        40028  85%               - pp--insert-lisp
> >        40028  85%                - pp--format-list
> >        40028  85%                 - pp--insert
> >        38710  82%                  - pp--insert-lisp
> >        38710  82%                   - pp--format-list
> 
> So, the current `pp' implementation is re-parsing from bob for every
> nested list inside sexp.

Not "the current 'pp'", but the implementation for this optional
behavior.

> This is quadratic scaling, and, as the repro demonstrates, the time
> goes up very quickly. Is the new option usable at all in practice?

Maybe only for small code fragments.

Message #23 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Ihor Radchenko <yantar92 <at> posteo.net>, 58687 <at> debbugs.gnu.org
Subject: Fwd: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Thu, 12 Jan 2023 23:02:42 +0000

Sorry for the noise, but I need to resend (i.e. re-cc) this email to 58687 <at> debbugs.gnu.org. I had sent it already to that list earlier, but my e-mail provider had blocked it because I recently enabled the "TLS-sending guarantee" it had blocked 58687 <at> debbugs.gnu.org, therefore I disabled that setting again. Sorry for the noise.

To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Ihor Radchenko <yantar92 <at> posteo.net>, 58687 <at> debbugs.gnu.org

On 2023-01-12 at 23:22 +01, Michael Eliachevitch <m.eliachevitch <at> posteo.de> wrote:

> On 2023-01-12 at 18:39 +02, Eli Zaretskii <eliz <at> gnu.org> wrote:
>
>> Not "the current 'pp'", but the implementation for this optional
>> behavior.
>
> This is optional behavior, but I would prefer if the performance impact of
> enabling this optional behavior would be documented, e.g. in the variable
> docstring and NEWS.29. It's disabled by default and will only be used by those
> like me and Visuwesh who the documentation and news, so I'm fine with it being
> almost unusably slow if that's documented. Though better performance if possible
> would be nice of course.
>
> On 2023-01-12 at 22:03 +0530, Visuwesh <visuweshm <at> gmail.com> wrote:
>
>> Personally, I always thought it would be best if the user facing
>> commands like pp-eval-sexp and friends alone respected the user option.
>
> Sounds like a good idea. I definetly didn't expect that me personally setting
> this option for myself would affect how lisp objects are serialized to disk in
> external packages. Not sure if I would only enable it for interactive commands
> and the like, in the emacs-world we are all hackers and hard to say what is
> user-facing. People might write their own functions using pp on small s-exps,
> and wonder why this setting isn't doing anything. Not sure what's the best
> approach there
>
> Maybe pp isn't meant to be used for doing anything that's not meant primarily
> for human eyes, like serialization of lisp objects, maybe it's an error on
> package maintainers that use it that way, but at least that could also somehow
> be communicated more clearly to them.


--
Michael Eliachevitch
Public PGP Key: https://keyoxide.org/hkp/546908c782383ad0e7d894ec1b8f95c8125dce31

Message #26 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
Cc: yantar92 <at> posteo.net, 58687 <at> debbugs.gnu.org
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Fri, 13 Jan 2023 10:42:48 +0200

> From: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
> Cc: Ihor Radchenko <yantar92 <at> posteo.net>, 58687 <at> debbugs.gnu.org
> Date: Thu, 12 Jan 2023 22:22:20 +0000
> 
> On 2023-01-12 at 18:39 +02, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> > Not "the current 'pp'", but the implementation for this optional
> > behavior.
> 
> This is optional behavior, but I would prefer if the performance impact of enabling this optional behavior would be documented, e.g. in the variable docstring and NEWS.29.

Done.

> On 2023-01-12 at 22:03 +0530, Visuwesh <visuweshm <at> gmail.com> wrote:
> 
> > Personally, I always thought it would be best if the user facing
> > commands like pp-eval-sexp and friends alone respected the user option.
> 
> Sounds like a good idea.

Also done.  (Most of the functions already honored this option, but 2
of them didn't.)

Message #29 received at 58687 <at> debbugs.gnu.org (full text, mbox):

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Michael Eliachevitch <m.eliachevitch <at> posteo.de>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 58687 <at> debbugs.gnu.org
Subject: Re: bug#58687: 29.0.50; Enabling pp-use-max-width dramatically
 slows down formatting of large sexps like org-persist--index
Date: Fri, 13 Jan 2023 09:28:37 +0000

Michael Eliachevitch <m.eliachevitch <at> posteo.de> writes:

>> Personally, I always thought it would be best if the user facing
>> commands like pp-eval-sexp and friends alone respected the user option.
>
> Sounds like a good idea. I definetly didn't expect that me personally setting this option for myself would affect how lisp objects are serialized to disk in external packages. Not sure if I would only enable it for interactive commands and the like, in the emacs-world we are all hackers and hard to say what is user-facing. People might write their own functions using pp on small s-exps, and wonder why this setting isn't doing anything. Not sure what's the best approach there
>
> Maybe pp isn't meant to be used for doing anything that's not meant primarily for human eyes, like serialization of lisp objects, maybe it's an error on package maintainers that use it that way, but at least that could also somehow be communicated more clearly to them.

In org-persist, `pp' is used because "index" file might be something
users may want to check manually. Following the notion that Elisp data
should be something consumable by humans, if possible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.