GNU logs - #77588, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#77588: Catastrophic slowdown in eglot via `flymake-diag-region'
Resent-From: JD Smith <jdtsmith@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 06 Apr 2025 22:51:02 +0000
Resent-Message-ID: <handler.77588.B.17439798538217 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 77588 <at> debbugs.gnu.org
From: JD Smith <jdtsmith@HIDDEN>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_B8DC673C-8435-40E0-93A5-DE6AD84F4615"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.400.131.1.6\))
Message-Id: <CEA8A47E-2C5F-44A3-9909-DD342A7598A7@HIDDEN>
Date: Sun, 6 Apr 2025 18:50:26 -0400
Received-SPF: pass client-ip=2607:f8b0:4864:20::131;
 envelope-from=jdtsmith@HIDDEN; helo=mail-il1-x131.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)


--Apple-Mail=_B8DC673C-8435-40E0-93A5-DE6AD84F4615
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


I was diagnosing a painful eglot pause of about 10 seconds that would =
reliably occur every few edits when completing candidates and working in =
a long python file (~8k lines).  I am using the basedpyright LSP server. =
 I narrowed this down to eglot's `textDocument/publishDiagnostics' =
notification handler.

The file I was working on has several hundred warnings and errors, so =
that represents a substantial message from the LSP server.  After a fair =
bit of timing and sleuthing, I narrowed it down further.  A few =
interrelated misbehaviors combined to produce the huge intermittent =
pauses:

When completing text directly after entering/removing a newline, the =
Diagnostics ranges received by eglot are out of date and "one line off".
Some of these off-by-one-line diagnostic message ranges now land on =
blank lines in the file, so the effective range there is empty.
Seeing an empty range, eglot declares the server has "botched" it, so it =
tries to make a by-hand recalculation of the range using =
`flymake-diag-region'.
`flymake-diag-region' for some reason calls (end-of-thing 'sexp) and =
(end-of-thing 'symbol). =20
In some positions in a long python file these commands are VERY SLOW.  =
Rather than the typical call time of ~1ms, at these certain positions =
`flymake-diag-region' takes ~400ms.

It doesn't take too many "botched eglot ranges" interacting with slow =
`thingatpt' misbehavior to add up to a 10s delay.

I'm not certain of the best solution.  A few ideas, from hardest to =
easiest:

Teach eglot textDocument/diagnostic
++++++++++++++++++++++++++++

 textDocument/publishDiagnostics messages arrive way too frequently IMO, =
after every buffer change of any kind.  They are pushed to eglot from =
the LSP server, and if they contain hundreds of errors, this becomes =
very inefficient (re-painting with flymake the same hundreds of regions =
over and over after each keystroke).

The best solution here would probably be to adopt "pull" diagnostics =
using textDocument/diagnostic, perhaps in an idle-timer whose duration =
the user can configure.  I don't believe EGLOT can do diagnostic pulls =
at the moment.

Don't use thingatpt in `flymake-diag-region'
+++++++++++++++++++++++++++++++++

`flymake-diag-region' should perhaps not use thingapt, which is subject =
to the performance vagaries of the major-mode and underlying file.  I am =
uncertain why it relies on that.  Perhaps the performance of those will =
be improved with treesitter variants.

Eglot could detect off-by-one diagnostics
+++++++++++++++++++++++++++++++

Hard to know the best heuristic, but lots of null effective ranges is a =
good one.

Eglot can simply ignore null range diagnostics
+++++++++++++++++++++++++++++++++++

Eglot doesn't need to use `flymake-diag-region' to try to calculate an =
update range if it encounters a null diagnostic range.  It could simply =
drop those, as they are probably wrong anyway, and will shortly be =
updated.


--Apple-Mail=_B8DC673C-8435-40E0-93A5-DE6AD84F4615
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dus-ascii"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: =
after-white-space;"><div><br></div>I was diagnosing a painful eglot =
pause of about 10 seconds that would reliably occur every few edits when =
completing candidates and working in a long python file (~8k lines). =
&nbsp;I am using the basedpyright LSP server. &nbsp;I narrowed this down =
to eglot's `textDocument/publishDiagnostics' notification =
handler.<div><br></div><div>The file I was working on has several =
hundred warnings and errors, so that represents a substantial message =
from the LSP server. &nbsp;After a fair bit of timing and sleuthing, I =
narrowed it down further. &nbsp;A few interrelated misbehaviors combined =
to produce the<span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, =
0);">&nbsp;</span><span style=3D"caret-color: rgb(0, 0, 0); color: =
rgb(0, 0, 0);">huge&nbsp;</span>intermittent =
pauses:<div><br></div><div><ol class=3D"MailOutline"><li>When completing =
text directly after entering/removing a newline, the Diagnostics ranges =
received by eglot are out of date and "one line off".</li><li>Some of =
these off-by-one-line diagnostic message ranges now land on blank lines =
in the file, so the effective range there is empty.</li><li>Seeing an =
empty range, eglot declares the server has "botched" it, so it tries to =
make a by-hand recalculation of the range using =
`flymake-diag-region'.</li><li>`flymake-diag-region' for some reason =
calls&nbsp;(end-of-thing 'sexp) and&nbsp;(end-of-thing 'symbol). =
&nbsp;</li><li>In some positions in a long python file these commands =
are VERY SLOW. &nbsp;Rather than the typical call time of ~1ms, at these =
certain positions `flymake-diag-region' takes =
~400ms.</li></ol><div><br></div><div>It doesn't take too many "botched =
eglot ranges" interacting with slow `thingatpt' misbehavior to add up to =
a 10s delay.</div><div><br></div><div>I'm not certain of the best =
solution. &nbsp;A few ideas, from hardest to =
easiest:</div><div><br></div><div><span style=3D"caret-color: rgb(0, 0, =
0); color: rgb(0, 0, 0);">Teach eglot =
textDocument/diagnostic</span></div><div>++++++++++++++++++++++++++++</div=
><div><br></div><div>&nbsp;<font =
color=3D"#000000">textDocument/publishDiagnostics messages arrive way =
too frequently IMO, after every buffer change of any kind. &nbsp;They =
are pushed to eglot from the LSP server, and if they contain hundreds of =
errors, this becomes very inefficient (re-painting with flymake the same =
hundreds of regions over and over after each =
keystroke).</font></div><div><font =
color=3D"#000000"><br></font></div><div><font color=3D"#000000">The best =
solution here would probably be to adopt "pull" diagnostics =
using&nbsp;textDocument/diagnostic, perhaps in an idle-timer whose =
duration the user can configure. &nbsp;I don't believe EGLOT can do =
diagnostic pulls at the moment.</font></div><div><br></div><div>Don't =
use thingatpt in =
`flymake-diag-region'</div><div>+++++++++++++++++++++++++++++++++</div><di=
v><br></div><div><span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, =
0, 0);">`</span><font color=3D"#000000"><span style=3D"caret-color: =
rgb(0, 0, 0);">flymake-diag-region' should perhaps not use thingapt, =
which is subject to the performance vagaries of the major-mode and =
underlying file. &nbsp;I am uncertain why it relies on that. =
&nbsp;Perhaps the performance of those will be improved with treesitter =
variants.</span></font></div><div><br></div><div>Eglot could detect =
off-by-one =
diagnostics</div><div>+++++++++++++++++++++++++++++++</div><div><br></div>=
<div>Hard to know the best heuristic, but lots of null effective ranges =
is a good one.</div><div><br></div><div>Eglot can simply ignore null =
range =
diagnostics</div><div>+++++++++++++++++++++++++++++++++++</div><div><br></=
div><div><div><font color=3D"#000000">Eglot doesn't need to use `<span =
style=3D"caret-color: rgb(0, 0, 0);">flymake-diag-region' to try to =
calculate an update range if it encounters a null diagnostic range. =
&nbsp;It could simply drop those, as they are probably wrong anyway, and =
will shortly be =
updated.</span></font></div></div><div><br></div></div></div></body></html=
>=

--Apple-Mail=_B8DC673C-8435-40E0-93A5-DE6AD84F4615--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: JD Smith <jdtsmith@HIDDEN>
Subject: bug#77588: Acknowledgement (Catastrophic slowdown in eglot via
 `flymake-diag-region')
Message-ID: <handler.77588.B.17439798538217.ack <at> debbugs.gnu.org>
References: <CEA8A47E-2C5F-44A3-9909-DD342A7598A7@HIDDEN>
X-Gnu-PR-Message: ack 77588
X-Gnu-PR-Package: emacs
Reply-To: 77588 <at> debbugs.gnu.org
Date: Sun, 06 Apr 2025 22:51:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 77588 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
77588: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D77588
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#77588: Catastrophic slowdown in eglot via `flymake-diag-region'
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 07 Apr 2025 11:22:02 +0000
Resent-Message-ID: <handler.77588.B77588.174402491325943 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: JD Smith <jdtsmith@HIDDEN>, =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= <joaotavora@HIDDEN>, Spencer Baugh <sbaugh@HIDDEN>
Cc: 77588 <at> debbugs.gnu.org
Date: Mon, 07 Apr 2025 14:21:38 +0300
Message-Id: <86v7rgtdx9.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <CEA8A47E-2C5F-44A3-9909-DD342A7598A7@HIDDEN> (message from JD
 Smith on Sun, 6 Apr 2025 18:50:26 -0400)
References: <CEA8A47E-2C5F-44A3-9909-DD342A7598A7@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: JD Smith <jdtsmith@HIDDEN>
> Date: Sun, 6 Apr 2025 18:50:26 -0400
> 
> I was diagnosing a painful eglot pause of about 10 seconds that would reliably occur every few edits when
> completing candidates and working in a long python file (~8k lines).  I am using the basedpyright LSP server. 
> I narrowed this down to eglot's `textDocument/publishDiagnostics' notification handler.
> 
> The file I was working on has several hundred warnings and errors, so that represents a substantial message
> from the LSP server.  After a fair bit of timing and sleuthing, I narrowed it down further.  A few interrelated
> misbehaviors combined to produce the huge intermittent pauses:
> 
> 1 When completing text directly after entering/removing a newline, the Diagnostics ranges received by eglot
>  are out of date and "one line off".
> 2 Some of these off-by-one-line diagnostic message ranges now land on blank lines in the file, so the
>  effective range there is empty.
> 3 Seeing an empty range, eglot declares the server has "botched" it, so it tries to make a by-hand
>  recalculation of the range using `flymake-diag-region'.
> 4 `flymake-diag-region' for some reason calls (end-of-thing 'sexp) and (end-of-thing 'symbol).  
> 5 In some positions in a long python file these commands are VERY SLOW.  Rather than the typical call time
>  of ~1ms, at these certain positions `flymake-diag-region' takes ~400ms.
> 
> It doesn't take too many "botched eglot ranges" interacting with slow `thingatpt' misbehavior to add up to a
> 10s delay.
> 
> I'm not certain of the best solution.  A few ideas, from hardest to easiest:
> 
> Teach eglot textDocument/diagnostic
> ++++++++++++++++++++++++++++
> 
>  textDocument/publishDiagnostics messages arrive way too frequently IMO, after every buffer change of any
> kind.  They are pushed to eglot from the LSP server, and if they contain hundreds of errors, this becomes
> very inefficient (re-painting with flymake the same hundreds of regions over and over after each keystroke).
> 
> The best solution here would probably be to adopt "pull" diagnostics using textDocument/diagnostic, perhaps
> in an idle-timer whose duration the user can configure.  I don't believe EGLOT can do diagnostic pulls at the
> moment.
> 
> Don't use thingatpt in `flymake-diag-region'
> +++++++++++++++++++++++++++++++++
> 
> `flymake-diag-region' should perhaps not use thingapt, which is subject to the performance vagaries of the
> major-mode and underlying file.  I am uncertain why it relies on that.  Perhaps the performance of those will be
> improved with treesitter variants.
> 
> Eglot could detect off-by-one diagnostics
> +++++++++++++++++++++++++++++++
> 
> Hard to know the best heuristic, but lots of null effective ranges is a good one.
> 
> Eglot can simply ignore null range diagnostics
> +++++++++++++++++++++++++++++++++++
> 
> Eglot doesn't need to use `flymake-diag-region' to try to calculate an update range if it encounters a null
> diagnostic range.  It could simply drop those, as they are probably wrong anyway, and will shortly be updated.

Adding João and Spencer to the discussion.

Last modified: Mon, 7 Apr 2025 11:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.