GNU logs - #40897, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#40897: Negation of pixel-width :width expressions does not work
Resent-From: Pip Cet <pipcet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 27 Apr 2020 08:49:01 +0000
Resent-Message-ID: <handler.40897.B.15879773182610 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 40897 <at> debbugs.gnu.org
MIME-Version: 1.0
From: Pip Cet <pipcet@HIDDEN>
Date: Mon, 27 Apr 2020 08:47:50 +0000
Message-ID: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
Content-Type: multipart/mixed; boundary="0000000000004ad4da05a441c77d"
Received-SPF: pass client-ip=2607:f8b0:4864:20::32a;
 envelope-from=pipcet@HIDDEN; helo=mail-ot1-x32a.google.com
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--0000000000004ad4da05a441c77d
Content-Type: text/plain; charset="UTF-8"

In looking over the xdisp.c code, I noticed that
calc_pixel_width_or_height has at least three problems:

1. negation of width specifiers doesn't work
2. a circular list passed to (+) will hang Emacs
3. a circular list passed directly will crash Emacs

(1) is a significant bug and should be fixed. (2) is easy to fix, and
we should probably do so on the master branch. (3) is difficult to
fix, but I'd like to fix all these, and undiscovered similar issues,
by converting calc_pixel_width_or_height to using the Lisp
interpreter.

The underlying problem is we're writing yet another mini-interpreter
for a limited Lisp-like language, in C. That means no quitting, no
protection against stack overflows, and getting tricky semantics (such
as the different behavior of (- A) and (- A B ...)) wrong.

My proposal is to move the somewhat complex logic of
calc_pixel_width_or_height to Lisp code, run when the display spec is
actually evaluated (i.e. when "(when ...)" display specs run their
Lisp code). This could be achieved by a method similar to the patch
for bug#40845 in
https://lists.gnu.org/archive/html/bug-gnu-emacs/2020-04/msg01096.html.

This would also resolve bug#40856 by allowing more complicated space
calculations at a time when the font metrics are known.

I'm attaching a patch, against master, for (1) only. I haven't tried
to restore the previous "optimization" of avoiding (very cheap) calls
to EQ for all but the first argument.

Recipes:

(1):

(insert (propertize " " 'display '(space :width (+ 50 (- 50)))))

Expected result: a zero-width space
Actual result: a space of width 100

(2):
(let ((l (cons 0 nil)))
  (setcdr l l)
  (insert (propertize " " 'display `(space :width (+ . ,l)))))

Expected result: a quittable infinite loop
Actual result: a non-quittable infinite loop

(3):
(let ((l (cons 0 nil)))
  (setcdr l l)
  (insert (propertize " " 'display `(space :width ,l))))

Expected result: a Lisp stack overflow error
Actual result: a C stack overflow, Emacs crashes

--0000000000004ad4da05a441c77d
Content-Type: text/x-patch; charset="US-ASCII"; 
	name="0001-Fix-pixel-width-evaluation-of-negated-specs.patch"
Content-Disposition: attachment; 
	filename="0001-Fix-pixel-width-evaluation-of-negated-specs.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_k9i8n4k40>
X-Attachment-Id: f_k9i8n4k40

RnJvbSBiNzQ5M2ZmOTM0NTUzYTJkMDA4Y2NlNGZiODg0NGQyNWZkMmRmOTk4IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBQaXAgQ2V0IDxwaXBjZXRAZ21haWwuY29tPgpEYXRlOiBNb24s
IDI3IEFwciAyMDIwIDA4OjQxOjUyICswMDAwClN1YmplY3Q6IFtQQVRDSF0gRml4IHBpeGVsLXdp
ZHRoIGV2YWx1YXRpb24gb2YgbmVnYXRlZCBzcGVjcwoKKiBzcmMveGRpc3AuYyAoY2FsY19waXhl
bF93aWR0aF9vcl9oZWlnaHQpOiBGaXggZXZhbHVhdGlvbiBvZiAoLSBYKQpTUEVDLgotLS0KIHNy
Yy94ZGlzcC5jIHwgMTAgKysrKystLS0tLQogMSBmaWxlIGNoYW5nZWQsIDUgaW5zZXJ0aW9ucygr
KSwgNSBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9zcmMveGRpc3AuYyBiL3NyYy94ZGlzcC5j
CmluZGV4IDE0MGQxMzQ1NzIuLjFhYmE0Yzg5MDEgMTAwNjQ0Ci0tLSBhL3NyYy94ZGlzcC5jCisr
KyBiL3NyYy94ZGlzcC5jCkBAIC0yNzI3OCw3ICsyNzI3OCw3IEBAIGNhbGNfcGl4ZWxfd2lkdGhf
b3JfaGVpZ2h0IChkb3VibGUgKnJlcywgc3RydWN0IGl0ICppdCwgTGlzcF9PYmplY3QgcHJvcCwK
IAkgICAgIHJlY3Vyc2l2ZWx5IGNhbGN1bGF0ZWQgdmFsdWVzLiAgKi8KIAkgIGlmIChFUSAoY2Fy
LCBRcGx1cykgfHwgRVEgKGNhciwgUW1pbnVzKSkKIAkgICAgewotCSAgICAgIGJvb2wgZmlyc3Qg
PSB0cnVlOworCSAgICAgIHB0cmRpZmZfdCBjb3VudCA9IDA7CiAJICAgICAgZG91YmxlIHB4Owog
CiAJICAgICAgcGl4ZWxzID0gMDsKQEAgLTI3Mjg3LDEzICsyNzI4NywxMyBAQCBjYWxjX3BpeGVs
X3dpZHRoX29yX2hlaWdodCAoZG91YmxlICpyZXMsIHN0cnVjdCBpdCAqaXQsIExpc3BfT2JqZWN0
IHByb3AsCiAJCSAgaWYgKCFjYWxjX3BpeGVsX3dpZHRoX29yX2hlaWdodCAoJnB4LCBpdCwgWENB
UiAoY2RyKSwKIAkJCQkJCSAgIGZvbnQsIHdpZHRoX3AsIGFsaWduX3RvKSkKIAkJICAgIHJldHVy
biBmYWxzZTsKLQkJICBpZiAoZmlyc3QpCi0JCSAgICBwaXhlbHMgPSAoRVEgKGNhciwgUXBsdXMp
ID8gcHggOiAtcHgpLCBmaXJzdCA9IGZhbHNlOwotCQkgIGVsc2UKKwkJICBpZiAoY291bnQrKyA9
PSAwKQogCQkgICAgcGl4ZWxzICs9IHB4OworCQkgIGVsc2UKKwkJICAgIHBpeGVscyArPSAoRVEg
KGNhciwgUXBsdXMpID8gcHggOiAtcHgpOwogCQkgIGNkciA9IFhDRFIgKGNkcik7CiAJCX0KLQkg
ICAgICBpZiAoRVEgKGNhciwgUW1pbnVzKSkKKwkgICAgICBpZiAoRVEgKGNhciwgUW1pbnVzKSAm
JiBjb3VudCA9PSAxKQogCQlwaXhlbHMgPSAtcGl4ZWxzOwogCSAgICAgIHJldHVybiBPS19QSVhF
TFMgKHBpeGVscyk7CiAJICAgIH0KLS0gCjIuMjYuMgoK
--0000000000004ad4da05a441c77d--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Pip Cet <pipcet@HIDDEN>
Subject: bug#40897: Acknowledgement (Negation of pixel-width :width
 expressions does not work)
Message-ID: <handler.40897.B.15879773182610.ack <at> debbugs.gnu.org>
References: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
X-Gnu-PR-Message: ack 40897
X-Gnu-PR-Package: emacs
Reply-To: 40897 <at> debbugs.gnu.org
Date: Mon, 27 Apr 2020 08:49:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 40897 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
40897: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D40897
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#40897: Negation of pixel-width :width expressions does not work
Resent-From: Pip Cet <pipcet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 27 Apr 2020 12:20:02 +0000
Resent-Message-ID: <handler.40897.B40897.158798996732018 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 40897 <at> debbugs.gnu.org
MIME-Version: 1.0
References: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
In-Reply-To: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
From: Pip Cet <pipcet@HIDDEN>
Date: Mon, 27 Apr 2020 12:18:41 +0000
Message-ID: <CAOqdjBfZZG38fOu4icmB+r3c55tw6_7HMo1hAZBpn=hVtBAAXA@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Mon, Apr 27, 2020 at 8:49 AM Pip Cet <pipcet@HIDDEN> wrote:
> 3. a circular list passed directly will crash Emacs

I'm not saying this is a security issue, but you can create a
text/enriched file, in the obvious way, which will crash emacs -Q when
you open it. That's very inconvenient.

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#40897: Negation of pixel-width :width expressions does not work
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 27 Apr 2020 14:33:01 +0000
Resent-Message-ID: <handler.40897.B40897.158799794031271 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Pip Cet <pipcet@HIDDEN>
Cc: 40897 <at> debbugs.gnu.org
Date: Mon, 27 Apr 2020 17:32:05 +0300
Message-Id: <83y2qh3spm.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
 (message from Pip Cet on Mon, 27 Apr 2020 08:47:50 +0000)
References: <CAOqdjBcDg2t_kRFmcEftJ_hpC0a70c5HJwFYqa9o-r0bJttRzQ@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: Pip Cet <pipcet@HIDDEN>
> Date: Mon, 27 Apr 2020 08:47:50 +0000
> 
> In looking over the xdisp.c code, I noticed that
> calc_pixel_width_or_height has at least three problems:
> 
> 1. negation of width specifiers doesn't work

Thanks.  That should tell us how many people tried to use this feature
since it was introduced, 16 years ago ;-)

> 2. a circular list passed to (+) will hang Emacs
> 3. a circular list passed directly will crash Emacs
> 
> (1) is a significant bug and should be fixed. (2) is easy to fix, and
> we should probably do so on the master branch. (3) is difficult to
> fix, but I'd like to fix all these, and undiscovered similar issues,
> by converting calc_pixel_width_or_height to using the Lisp
> interpreter.

I don't think converting this to Lisp is a good idea, and I try to
explain below why.  It doesn't seem to be needed for fixing these
particular problems (see the proposed patch below), and even more so
given the evidence of this feature's popularity.

> The underlying problem is we're writing yet another mini-interpreter
> for a limited Lisp-like language, in C.

That is true, but there's a reason to this.

> That means no quitting, no protection against stack overflows

You cannot usefully quit or protect against stack overflow inside
redisplay anyway.  Quitting or signaling an error during redisplay
just gives you an endless redisplay loop, because signaling an error
immediately enters another redisplay cycle.  And even if you catch the
error and quietly return, in many/most cases you have another
redisplay either immediately or soon enough, and that presents a
locked-up or almost a locked-up Emacs ("almost" means that typing M-<
or M-> enough time might eventually get you out of the vicious circle,
if the problematic place is not at BOB/EOB).

The only reasonable way of avoiding these is to prevent the need to
error out.

You say (3) is difficult to fix, but the patch below does succeed in
preventing the stack overflow, so I don't think I understand why you
thought this to be difficult.  What am I missing?

> getting tricky semantics (such as the different behavior of (- A)
> and (- A B ...)) wrong.

AFAIU, the semantics is the same as in Lisp, isn't it?  The
implementation has a bug, but the semantics doesn't, I think.

> My proposal is to move the somewhat complex logic of
> calc_pixel_width_or_height to Lisp code, run when the display spec is
> actually evaluated (i.e. when "(when ...)" display specs run their
> Lisp code). This could be achieved by a method similar to the patch
> for bug#40845 in
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2020-04/msg01096.html.
> 
> This would also resolve bug#40856 by allowing more complicated space
> calculations at a time when the font metrics are known.

You have promoted similar views elsewhere, on at least 2 other
occasions, so let me now try to explain why I very much object to
moving display functionality to Lisp, as a general tendency (as
opposed to some selected circumstances where this could be justified).

My reasons are as follows:

  . Lisp slows down redisplay.  First it does that because it's Lisp,
    and second because it causes GC.  We already have enough
    complaints against redisplay slowness (even on today's fast
    desktop machines!), so we should be very careful about this.

  . Handling Lisp errors inside redisplay is tricky, for the reasons
    explained above.  It is even more tricky when Lisp computes
    something that the rest of redisplay needs for its operation (as
    opposed to calling various hooks, such as window-scroll-functions,
    which are basically a one-way street -- if the hook errors out, it
    doesn't disrupt the display itself).

  . Last, but not least -- and people tend to forget about this
    important factor -- using Lisp to calculate display elements makes
    it nigh impossible to know when certain redisplay optimizations
    can be validly applied and when not.  That's because A Lisp
    program can potentially do anything and depend on any numbers of
    global and local state variables, and users will expect the
    display to change when these variables change their values.
    Redisplay optimizations are based on tracking some key state
    variables to deduce when a more thorough redisplay is needed and
    when it is safe to use shortcuts.  This is already a hard task,
    especially since we consistently made Emacs redisplay more and
    more lazily starting from Emacs 25 -- we still get bug reports
    about some situations where some state change fails to trigger
    redisplay, even though the number of possible variables to track
    is not very large.  Using Lisp much more than we do now will make
    tracking the relevant variables impossible.  The result will be
    only one: we will have to disable more and more optimizations and
    shortcuts to keep what's on the glass accurate.  And Emacs's
    redisplay can be *really* slow without the optimizations,
    especially in simple cases, like when you just move the cursor, or
    delete or insert a character.

So my suggestion is to use Lisp as part of redisplay only where no
other solution is reasonable or practical, and when the feature we
want is really important to have.

Once again, there could be certain situations where calling Lisp from
the display code might be the only practical solution for some serious
and important problem.  The above just tries to explain why I don't
think that idea is good _in_general_, and why we should try to avoid
it if at all possible.

I hope I made my position on this clear enough.

Here's the patch I came up with to handle the problems you reported.
WDYT?

diff --git a/src/xdisp.c b/src/xdisp.c
index 140d134572..bc27fb15e0 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -27284,11 +27284,16 @@ calc_pixel_width_or_height (double *res, struct it *it, Lisp_Object prop,
 	      pixels = 0;
 	      while (CONSP (cdr))
 		{
+		  if (EQ (cdr, XCDR (cdr)))
+		    return false;
 		  if (!calc_pixel_width_or_height (&px, it, XCAR (cdr),
 						   font, width_p, align_to))
 		    return false;
 		  if (first)
-		    pixels = (EQ (car, Qplus) ? px : -px), first = false;
+		    pixels =
+		      (EQ (car, Qminus) && CONSP (XCDR (cdr))
+		       ? -px
+		       : px), first = false;
 		  else
 		    pixels += px;
 		  cdr = XCDR (cdr);
@@ -27305,13 +27310,15 @@ calc_pixel_width_or_height (double *res, struct it *it, Lisp_Object prop,
 
       /* '(NUM)': absolute number of pixels.  */
       if (NUMBERP (car))
-{
+	{
 	  double fact;
 	  int offset =
 	    width_p && align_to && *align_to < 0 ? it->lnum_pixel_width : 0;
 	  pixels = XFLOATINT (car);
 	  if (NILP (cdr))
 	    return OK_PIXELS (pixels + offset);
+	  if (EQ (cdr, XCDR (cdr)))
+	    return false;
 	  if (calc_pixel_width_or_height (&fact, it, cdr,
 					  font, width_p, align_to))
 	    return OK_PIXELS (pixels * fact + offset);

Last modified: Mon, 27 Apr 2020 14:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.