GNU bug report logs - #57684
locked narrowing breaks existing code without an apparent way to repair

Previous Next

To reply to this bug, email your comments to 57684 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Pogonyshev <pogonyshev <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: locked narrowing breaks existing code without an apparent way to
 repair
Date: Thu, 8 Sep 2022 21:37:02 +0200

[Message part 1 (text/plain, inline)]

"Locked narrowing" added in Emacs 29 cannot be (temporarily) canceled by
code called inside. This in particular breaks Logview (
https://github.com/doublep/logview), because this narrowing is in effect
during fontification and causes Logview to fall down into an infinite loop.
Moreover, variable `restrictions-locked' appears to be not exposed to
Elisp, meaning that this cannot be cleared even with an explicit let-bind.

The mode simply cannot work without an ability to temporarily widen the
text. Macro `logview--std-temporarily-widening' is used 35 times in its
code:

(defmacro logview--std-temporarily-widening (&rest body)
  (declare (indent 0) (debug t))
  `(save-restriction
     (let ((logview--point-min (logview--point-min))
           (logview--point-max (logview--point-max)))
       (widen)
       ,@body)))

To make it even harder to debug, Emacs sometimes hangs completely with even
C-g not aborting faulty code (in this case "faulty" because of incompatible
changes in Emacs itself).

Paul

[Message part 2 (text/html, inline)]

Message #8 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Thu, 08 Sep 2022 19:57:13 +0000

>
> "Locked narrowing" added in Emacs 29 cannot be (temporarily) canceled by 
> code called inside.
>

This issue is being worked on, see the (not yet finished) 
feature/improved-locked-narrowing branch.

Message #11 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Paul Pogonyshev <pogonyshev <at> gmail.com>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Thu, 8 Sep 2022 22:30:45 +0200

[Message part 1 (text/plain, inline)]

Good to hear. I think in all cases it should be left to the coder to
temporarily lift restrictions, even if those are "locked", maybe
with strong warnings in documentation. Speed should always
come a second priority to functionality. Besides, Logview (one
example of what would suffer if widening is impossible) is very
fast.

As an example of why widening might be needed: fontification
may depend on preceding text in the buffer. Maybe standard
Emacs fontification code handles that separately, but e.g.
Logview (almost) completely replaces the standard code here
with a home-grown solution aimed specifically at log files. And
actually this was done exactly for speed.

Anyway, I hope this is finished soon.

Paul

On Thu, 8 Sept 2022 at 21:57, Gregory Heytings <gregory <at> heytings.org> wrote:

>
> >
> > "Locked narrowing" added in Emacs 29 cannot be (temporarily) canceled by
> > code called inside.
> >
>
> This issue is being worked on, see the (not yet finished)
> feature/improved-locked-narrowing branch.
>

[Message part 2 (text/html, inline)]

Message #14 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Po Lu <luangruo <at> yahoo.com>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: Gregory Heytings <gregory <at> heytings.org>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Fri, 09 Sep 2022 09:46:44 +0800

Paul Pogonyshev <pogonyshev <at> gmail.com> writes:

> Good to hear. I think in all cases it should be left to the coder to
> temporarily lift restrictions, even if those are "locked", maybe
> with strong warnings in documentation. Speed should always
> come a second priority to functionality. Besides, Logview (one
> example of what would suffer if widening is impossible) is very
> fast.
>
> As an example of why widening might be needed: fontification
> may depend on preceding text in the buffer. Maybe standard
> Emacs fontification code handles that separately, but e.g.
> Logview (almost) completely replaces the standard code here
> with a home-grown solution aimed specifically at log files. And
> actually this was done exactly for speed.
>
> Anyway, I hope this is finished soon.

For the time being, you can either use an indirect buffer, or run the
fontification code in a timer.

The former already exists in the wild.

Message #17 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Paul Pogonyshev <pogonyshev <at> gmail.com>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Fri, 09 Sep 2022 18:16:55 +0200

Gregory Heytings <gregory <at> heytings.org> writes:

> This issue is being worked on, see the (not yet finished)
> feature/improved-locked-narrowing branch.

Is there much to be done on that branch before merging?

Message #20 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Paul Pogonyshev <pogonyshev <at> gmail.com>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Fri, 09 Sep 2022 16:20:23 +0000

>> This issue is being worked on, see the (not yet finished) 
>> feature/improved-locked-narrowing branch.
>
> Is there much to be done on that branch before merging?
>

I don't know yet.  Not "that much" I think, but there are a number of 
things that need to be fixed/improved.

Message #23 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Paul Pogonyshev <pogonyshev <at> gmail.com>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Tue, 13 Sep 2022 22:54:07 +0200

[Message part 1 (text/plain, inline)]

By the way, today Emacs 29 hung in Magit blaming for me, with C-g doing
nothing. Grepping Magit sources suggest it uses the "save-restriction -
temporarily widen" more than ten times in various places, 3 of them when
blaming. Cannot say for sure that was it, but all the outer symptoms are
identical with the hangs in Logview.

I really think there must be a way to "widen no matter which locks are
installed" - a lot of code seems to depend on that.

Paul

On Fri, 9 Sept 2022 at 18:20, Gregory Heytings <gregory <at> heytings.org> wrote:

>
> >> This issue is being worked on, see the (not yet finished)
> >> feature/improved-locked-narrowing branch.
> >
> > Is there much to be done on that branch before merging?
> >
>
> I don't know yet.  Not "that much" I think, but there are a number of
> things that need to be fixed/improved.
>

[Message part 2 (text/html, inline)]

Message #26 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Tue, 13 Sep 2022 21:06:20 +0000

>
> By the way, today Emacs 29 hung in Magit blaming for me, with C-g doing 
> nothing. Grepping Magit sources suggest it uses the "save-restriction - 
> temporarily widen" more than ten times in various places, 3 of them when 
> blaming. Cannot say for sure that was it, but all the outer symptoms are 
> identical with the hangs in Logview. I really think there must be a way 
> to "widen no matter which locks are installed" - a lot of code seems to 
> depend on that.
>

Yes, we know that, and as I said earlier it will be possible to unlock a 
locked narrowing.  That being said, your description is too vague to draw 
a conclusion, but given that you tell that Emacs hung in Magit, I'm not 
quite sure it's locked narrowing that is the culprit.  Locked narrowing is 
currently used only in buffers with very long lines, and only when those 
buffers are on display.  Files with very long lines are typically 
machine-generated, and not under version control.

Message #29 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Paul Pogonyshev <pogonyshev <at> gmail.com>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Tue, 13 Sep 2022 23:13:52 +0200

[Message part 1 (text/plain, inline)]

Well, probably not it then.

> Files with very long lines are typically machine-generated

Typically, but not always. The Logview hangs for me likely were because
the logs I worked with contained long command lines for Java process
starting (when you use a huge application with over 100 library JARs,
classpath specification can get _really_ long). Actually, in a sense, those
are machine-generated too, but the files are meant for human consumption,
not e.g. as generated sources that are fed to a compiler.

Anyway, I really hope something is done about this soon so that I don't
have to switch between Emacs 29 and 28 all the time.

Paul

On Tue, 13 Sept 2022 at 23:06, Gregory Heytings <gregory <at> heytings.org>
wrote:

>
> >
> > By the way, today Emacs 29 hung in Magit blaming for me, with C-g doing
> > nothing. Grepping Magit sources suggest it uses the "save-restriction -
> > temporarily widen" more than ten times in various places, 3 of them when
> > blaming. Cannot say for sure that was it, but all the outer symptoms are
> > identical with the hangs in Logview. I really think there must be a way
> > to "widen no matter which locks are installed" - a lot of code seems to
> > depend on that.
> >
>
> Yes, we know that, and as I said earlier it will be possible to unlock a
> locked narrowing.  That being said, your description is too vague to draw
> a conclusion, but given that you tell that Emacs hung in Magit, I'm not
> quite sure it's locked narrowing that is the culprit.  Locked narrowing is
> currently used only in buffers with very long lines, and only when those
> buffers are on display.  Files with very long lines are typically
> machine-generated, and not under version control.
>

[Message part 2 (text/html, inline)]

Message #32 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Tue, 13 Sep 2022 21:35:58 +0000

> 
> Anyway, I really hope something is done about this soon so that I don't 
> have to switch between Emacs 29 and 28 all the time.
>

Note that you can safely change long-line-threshold to a somewhat larger 
value, say 25000, or even 50000.

Message #35 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: gregory <at> heytings.org, larsi <at> gnus.org, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Wed, 14 Sep 2022 05:34:02 +0300

> Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
> From: Paul Pogonyshev <pogonyshev <at> gmail.com>
> Date: Tue, 13 Sep 2022 22:54:07 +0200
> 
> By the way, today Emacs 29 hung in Magit blaming for me, with C-g doing nothing. Grepping Magit sources
> suggest it uses the "save-restriction - temporarily widen" more than ten times in various places, 3 of them
> when blaming. Cannot say for sure that was it, but all the outer symptoms are identical with the hangs in
> Logview.
> 
> I really think there must be a way to "widen no matter which locks are installed" - a lot of code seems to
> depend on that.

What does long-line-optimizations-p produce in Magit buffers?  If it
returns nil, what you see there has nothing to do with locked
narrowing, which is only used when buffer text has very long lines.

(Magit hides buffer text almost completely, presenting you what is
largely overlays and 'display' properties, so you may not be aware of
what the actual buffer text looks like.)

Message #38 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Paul Pogonyshev <pogonyshev <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Gregory Heytings <gregory <at> heytings.org>, Lars Ingebrigtsen <larsi <at> gnus.org>,
 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Wed, 14 Sep 2022 11:45:01 +0200

[Message part 1 (text/plain, inline)]

No, that's not it, I have checked now. Also, blaming is done in normal file
buffers, not in the status buffer.

By the way, it would really be nice if Emacs could do something about hangs
irrespective of what causes that. Even if Elisp code is buggy, Emacs itself
should never allow it to fall into an infinite loop and stop responding to
C-g, leaving full restart as the only way out.

Paul

On Wed, 14 Sept 2022 at 04:34, Eli Zaretskii <eliz <at> gnu.org> wrote:

> > Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
> > From: Paul Pogonyshev <pogonyshev <at> gmail.com>
> > Date: Tue, 13 Sep 2022 22:54:07 +0200
> >
> > By the way, today Emacs 29 hung in Magit blaming for me, with C-g doing
> nothing. Grepping Magit sources
> > suggest it uses the "save-restriction - temporarily widen" more than ten
> times in various places, 3 of them
> > when blaming. Cannot say for sure that was it, but all the outer
> symptoms are identical with the hangs in
> > Logview.
> >
> > I really think there must be a way to "widen no matter which locks are
> installed" - a lot of code seems to
> > depend on that.
>
> What does long-line-optimizations-p produce in Magit buffers?  If it
> returns nil, what you see there has nothing to do with locked
> narrowing, which is only used when buffer text has very long lines.
>
> (Magit hides buffer text almost completely, presenting you what is
> largely overlays and 'display' properties, so you may not be aware of
> what the actual buffer text looks like.)
>

[Message part 2 (text/html, inline)]

Message #41 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Pogonyshev <pogonyshev <at> gmail.com>
Cc: gregory <at> heytings.org, larsi <at> gnus.org, 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Wed, 14 Sep 2022 14:57:11 +0300

> From: Paul Pogonyshev <pogonyshev <at> gmail.com>
> Date: Wed, 14 Sep 2022 11:45:01 +0200
> Cc: Gregory Heytings <gregory <at> heytings.org>, Lars Ingebrigtsen <larsi <at> gnus.org>, 57684 <at> debbugs.gnu.org
> 
> By the way, it would really be nice if Emacs could do something about hangs irrespective of what causes
> that. Even if Elisp code is buggy, Emacs itself should never allow it to fall into an infinite loop and stop
> responding to C-g, leaving full restart as the only way out.

I think that's impossible in general, unless we restrict what Lisp
programs can do.  Every programming language can be used to write a
buggy program.

However, it should be possible to prevent some cases of such
problematic behavior, certainly so when the infloop is caused by our
bug.  But for that we need to know the details of the specific case in
order to investigate.

Message #44 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Thu, 15 Sep 2022 23:38:15 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Even if Elisp code is buggy, Emacs itself should never allow it to fall into an infinite loop and stop
  > > responding to C-g, leaving full restart as the only way out.

It used to be impossible to have an infinite loop in Emacs Lisp that
you could not quit out of.  The Lisp interpreter and tye bytecode
interpreter both had calls to QUIT in all the loops of Lisp execution.
Likewise, all the loops in C code that corresponded to Lisp loops, and
might fail to terminate if given circular lists, such as Fmemq,
had calls to QUIT.

If that is now no longertrue, what made it cease to be true?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #47 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rms <at> gnu.org
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Fri, 16 Sep 2022 09:04:28 +0300

> From: Richard Stallman <rms <at> gnu.org>
> Cc: 57684 <at> debbugs.gnu.org
> Date: Thu, 15 Sep 2022 23:38:15 -0400
> 
>   > Even if Elisp code is buggy, Emacs itself should never allow it to fall into an infinite loop and stop
>   > > responding to C-g, leaving full restart as the only way out.
> 
> It used to be impossible to have an infinite loop in Emacs Lisp that
> you could not quit out of.  The Lisp interpreter and tye bytecode
> interpreter both had calls to QUIT in all the loops of Lisp execution.
> Likewise, all the loops in C code that corresponded to Lisp loops, and
> might fail to terminate if given circular lists, such as Fmemq,
> had calls to QUIT.
> 
> If that is now no longertrue, what made it cease to be true?

It is still true.  The problem exists only with Lisp code that is run
when QUITs are either intentionally disabled, or when Lisp is run from
the display engine, where QUIT is caught and basically ignored.  The
particular case which started this thread is of the latter variety.

Message #50 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Fri, 16 Sep 2022 23:40:19 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is still true.  The problem exists only with Lisp code that is run
  > when QUITs are either intentionally disabled, or when Lisp is run from
  > the display engine, where QUIT is caught and basically ignored.  The
  > particular case which started this thread is of the latter variety.

I never allowed redisplay to run Lisp code -- it seemed very dangerous.
An error there could leave redisplay data structures inconsistent.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #53 received at 57684 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rms <at> gnu.org
Cc: 57684 <at> debbugs.gnu.org
Subject: Re: bug#57684: locked narrowing breaks existing code without an
 apparent way to repair
Date: Sat, 17 Sep 2022 09:33:06 +0300

> From: Richard Stallman <rms <at> gnu.org>
> Cc: 57684 <at> debbugs.gnu.org
> Date: Fri, 16 Sep 2022 23:40:19 -0400
> 
>   > It is still true.  The problem exists only with Lisp code that is run
>   > when QUITs are either intentionally disabled, or when Lisp is run from
>   > the display engine, where QUIT is caught and basically ignored.  The
>   > particular case which started this thread is of the latter variety.
> 
> I never allowed redisplay to run Lisp code -- it seemed very dangerous.
> An error there could leave redisplay data structures inconsistent.

That ship sailed a long time ago: Emacs 21 introduced jit-lock.el,
which is a way of running font-lock as part of redisplay, on the
portion of the buffer which is about to be displayed.  This is on by
default since Emacs 21.

In addition, we have several hooks, notably window-scroll-functions,
which redisplay runs when appropriate.

Search xdisp.c for "safe_call" to see where and why we call Lisp from
redisplay.  Yes, it's somewhat dangerous, and caused us nontrivial
issues over the years, but I don't think it's possible to go back to
not calling Lisp from the display code.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.