GNU bug report logs - #15841
Display bugs with cache-long-lines non-nil

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 15841 in the body.
You can then email your comments to 15841 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Nathan Trapuzzano <nbtrap <at> nbtrap.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 10:18:28 +0200

Redirected to a new bug report, please use this one in the future.

> From: Nathan Trapuzzano <nbtrap <at> nbtrap.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  Michael Heerdegen <michael_heerdegen <at> web.de>,  15797 <at> debbugs.gnu.org,  kjambunathan <at> gmail.com
> Date: Fri, 08 Nov 2013 14:07:44 -0500
> 
> Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
> 
> >> I wonder if we should simply turn it on by default, and leave it
> >> there.  That way, any bugs that it exposes will be flushed out very
> >> quickly.  Stefan?
> >
> > Let's try it.
> 
> Sorry I'm late to the party, but this broke linum and nlinum for me,
> using the defaults.  Any time I insert a new line, the line numbers get
> totally messed up--most of them don't even display any more.  And then
> using motion commands sometimes results in really bizarre behavior
> (apparently only with linum/nlinum so far), such as when I do
> forwad-sexp and point goes to the end of some sexp other than the one at
> point.

Message #8 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 10:31:45 +0200

> From: Michael Heerdegen <michael_heerdegen <at> web.de>
> Date: Sat, 09 Nov 2013 03:37:49 +0100
> Cc: 15797 <at> debbugs.gnu.org, kjambunathan <at> gmail.com
> 
> I see something probably related.  Sometimes, evaluating
> 
>   (line-number-at-pos)
> 
> at the end of my .emacs (10000 lines) needs over a second.  Doing
> 
>   (setq cache-long-scans nil)
> 
> immediately fixes this.  Dunno yet how to provoke the problem.

Please do try to find a recipe.  I just did that with xdisp.c (almost
30000 lines), and couldn't reproduce this.  line-number-at-pos is
instantaneous, whether I try it at the beginning, the end, or the
middle of the file.

Message #11 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: nbtrap <at> nbtrap.com
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 10:51:29 +0200

[Message part 1 (text/plain, inline)]

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap <at> nbtrap.com>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  Michael Heerdegen <michael_heerdegen <at> web.de>,  15797 <at> debbugs.gnu.org,  kjambunathan <at> gmail.com
> Date: Fri, 08 Nov 2013 16:36:50 -0500
> 
> Stefan Monnier <monnier <at> IRO.UMontreal.CA> writes:
> 
> > I tried:
> >
> >   src/emacs -Q src/xdisp.c -l .../nlinum.el -f nlinum-mode
> >
> > and then moved about in the buffer, but I didn't notice any problem.
> 
> Try this with the attached file:
> 
> src/emacs -nw -Q foo.el
> 
> M-x linum-mode    (you may notice the problem already here)
> M-x goto-line 10
> C-o
> 
> The numbers in the margin get messed up.  I haven't figured out a
> consistent way to screw up the motion commands yet.

[foo.el (application/emacs-lisp, attachment)]

Message #14 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: nbtrap <at> nbtrap.com
Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 10:52:33 +0200

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap <at> nbtrap.com>
> Date: Fri, 08 Nov 2013 18:11:44 -0500
> Cc: Michael Heerdegen <michael_heerdegen <at> web.de>, 15797 <at> debbugs.gnu.org,
> 	kjambunathan <at> gmail.com
> 
> Nathan Trapuzzano <nbtrap <at> nbtrap.com> writes:
> 
> > I haven't figured out a consistent way to screw up the motion commands
> > yet.
> 
> Here we go.  Visit the same file and do:
> 
> M-x linum-mode
> C-o
> C-n
> C-e
> 
> Cursor moves almost to end of entire file.

Message #17 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: nbtrap <at> nbtrap.com
Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 13:18:29 +0200

Redirecting to a new bug report.

> From: Nathan Trapuzzano <nbtrap <at> nbtrap.com>
> Date: Fri, 08 Nov 2013 18:11:44 -0500
> Cc: Michael Heerdegen <michael_heerdegen <at> web.de>, 15797 <at> debbugs.gnu.org,
> 	kjambunathan <at> gmail.com
> 
> Nathan Trapuzzano <nbtrap <at> nbtrap.com> writes:
> 
> > I haven't figured out a consistent way to screw up the motion commands
> > yet.
> 
> Here we go.  Visit the same file and do:
> 
> M-x linum-mode
> C-o
> C-n
> C-e
> 
> Cursor moves almost to end of entire file.

Should be fixed in trunk revision 115050.

However, there are still problems which can be seen with this recipe:

 emacs -Q --eval "(setq-default cache-long-scans nil)"
 C-x C-f src/xdisp.c
 M-: (setq cache-long-scans t) RET
 M-x linum-mode RET
 M->

Many line numbers near the end of the buffer are not shown in the
margin.

Message #20 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: nbtrap <at> nbtrap.com
Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 16:02:21 +0200

> Date: Sat, 09 Nov 2013 13:18:29 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
> 
>  emacs -Q --eval "(setq-default cache-long-scans nil)"
>  C-x C-f src/xdisp.c
>  M-: (setq cache-long-scans t) RET
>  M-x linum-mode RET
>  M->
> 
> Many line numbers near the end of the buffer are not shown in the
> margin.

This seems to be related to font-lock: turning off
global-font-lock-mode makes the problems go away.  Manually invoking
font-lock-fontify-buffer before turning on linum-mode also eliminates
the problems, so it looks like JIT Font Lock is the prime suspect.

Message #23 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: nbtrap <at> nbtrap.com
Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 09 Nov 2013 23:27:01 +0200

> Date: Sat, 09 Nov 2013 16:02:21 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
> 
> > Date: Sat, 09 Nov 2013 13:18:29 +0200
> > From: Eli Zaretskii <eliz <at> gnu.org>
> > Cc: michael_heerdegen <at> web.de, 15841 <at> debbugs.gnu.org
> > 
> >  emacs -Q --eval "(setq-default cache-long-scans nil)"
> >  C-x C-f src/xdisp.c
> >  M-: (setq cache-long-scans t) RET
> >  M-x linum-mode RET
> >  M->
> > 
> > Many line numbers near the end of the buffer are not shown in the
> > margin.
> 
> This seems to be related to font-lock: turning off
> global-font-lock-mode makes the problems go away.  Manually invoking
> font-lock-fontify-buffer before turning on linum-mode also eliminates
> the problems, so it looks like JIT Font Lock is the prime suspect.

The problem was that while the "dumb loop" in find_newline was running,
memory allocation in know_region_cache could have caused relocation of
buffer text, so C pointers needed to be adjusted, but weren't.

Fixed in trunk revision 115052.

Message #26 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sun, 10 Nov 2013 19:20:13 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> > I see something probably related.  Sometimes, evaluating
> > 
> >   (line-number-at-pos)
> > 
> > at the end of my .emacs (10000 lines) needs over a second.  Doing
> > 
> >   (setq cache-long-scans nil)
> > 
> > immediately fixes this.  Dunno yet how to provoke the problem.
>
> Please do try to find a recipe.  I just did that with xdisp.c (almost
> 30000 lines), and couldn't reproduce this.  line-number-at-pos is
> instantaneous, whether I try it at the beginning, the end, or the
> middle of the file.

Sorry, I was misguided.  What was slow wasn't `line-number-at-pos' but
redisplay (so only the result of evaluation showed up delayed).  This
long time for redisplay does only happen with some code I'm currently
developing, but also only when cache-long-scans is non-nil.  Anyway,
I'll tell you when I have a recipe.

Thanks,

Michael.

Message #29 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sun, 10 Nov 2013 20:31:10 +0200

> From: Michael Heerdegen <michael_heerdegen <at> web.de>
> Cc: 15841 <at> debbugs.gnu.org,  nbtrap <at> nbtrap.com
> Date: Sun, 10 Nov 2013 19:20:13 +0100
> 
> Sorry, I was misguided.  What was slow wasn't `line-number-at-pos' but
> redisplay (so only the result of evaluation showed up delayed).  This
> long time for redisplay does only happen with some code I'm currently
> developing, but also only when cache-long-scans is non-nil.

Thanks for clarifying.

> Anyway, I'll tell you when I have a recipe.

Please do, and thanks.

Message #32 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 11 Nov 2013 04:39:56 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> > Anyway, I'll tell you when I have a recipe.

Some elaborations, feel free to ignore.

The culprit was my own code: it placed myriads of invisible overlays
with no properties into the buffer.  Under these extreme circumstances,
`line-number-at-pos' indeed gets extremely slow at the end of my 10000
lines buffer: one invocation needs over a second.  I saw that with elp
as well as with profiler.  Setting `cache-long-scans' to nil (or
removing the overlays) cures this.

Although this is a corner case, I wonder why overlays slow down
`line-number-at-pos' so much for `cache-long-scans' non-nil - is that
expected?  Or can the profiler times I saw span redisplay times?

Because, when I use this:

(defmacro my-measure-time (expr)
  "Eval EXPR, display how much time it took."
  (with-gensyms (time)
    `(let ((,time (current-time)))
       ,expr
       (message "%s secs"
          (float-time (time-subtract (current-time) ,time))))))

and evaluate (my-measure-time (line-number-at-pos)) manually with M-:
(in the same situation), it shows a very tiny value.  But I'm sure that
from code, `line-number-at-pos' really lasts over a second.  Strange, I
don't understand it.



Regards,

Michael.

Message #35 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 11 Nov 2013 15:12:15 +0100

This change broke dired-maybe-insert-subdir.  Perhaps this should be a
new bug number, but since part of the breakage is display-related, I
decided to report it here.  To reproduce:

0. emacs -Q (built from trunk bzr 115033 or later)
1. Open a directory in Dired, e.g. the Emacs source tree root.
2. Put point on an entry listing a directory, e.g. admin.
3. Type `i'.
=> Emacs hangs.

If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
nil)', then at step 3, `i' works as expected.

Here are more details of the breakage:

Typing `C-g' releases the hang and shows the admin subdirectory, but its
display is corrupted: (i) the "available" blocks information, which
should appear at the end of the "total" line, instead appears to the
right of the first entry `.' and is fontified with dired-directory face;
(ii) while all the entries of admin are listed below the `..' entry, the
entire line of each entry is fontified with dired-directory face, and
these lines cannot be accessed by vertical motion commands like `C-n' or
`C-p', though they can be with horizontal motion commands like `C-f' and
`C-p'.  In fact, all of these entries appear to Dired to be part of the
line of the `..' entry, and AFACT this is what causes the hang: Emacs
infloops in dired-move-to-filename because, when point appears to be at
eof, beginning-of-line moves it back to the start of the `..' entry.

Steve Berman

Message #38 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 11 Nov 2013 18:23:10 +0200

> From: Michael Heerdegen <michael_heerdegen <at> web.de>
> Cc: 15841 <at> debbugs.gnu.org,  nbtrap <at> nbtrap.com
> Date: Mon, 11 Nov 2013 04:39:56 +0100
> 
> The culprit was my own code: it placed myriads of invisible overlays
> with no properties into the buffer.  Under these extreme circumstances,
> `line-number-at-pos' indeed gets extremely slow at the end of my 10000
> lines buffer: one invocation needs over a second.  I saw that with elp
> as well as with profiler.  Setting `cache-long-scans' to nil (or
> removing the overlays) cures this.

Can you show some simple enough code that puts so many overlays, and
has this effect?  It sounds like some optimization is in order.

> Although this is a corner case, I wonder why overlays slow down
> `line-number-at-pos' so much for `cache-long-scans' non-nil - is that
> expected?

I don't know.  Armed with a simple test case, I will look into this
and see what I find.

> Because, when I use this:
> 
> (defmacro my-measure-time (expr)
>   "Eval EXPR, display how much time it took."
>   (with-gensyms (time)
>     `(let ((,time (current-time)))
>        ,expr
>        (message "%s secs"
>           (float-time (time-subtract (current-time) ,time))))))
> 
> and evaluate (my-measure-time (line-number-at-pos)) manually with M-:
> (in the same situation), it shows a very tiny value.  But I'm sure that
> from code, `line-number-at-pos' really lasts over a second.  Strange, I
> don't understand it.

Redisplay is indeed a likely suspect, but still cache-long-scans
somehow comes into play.

Thanks.

Message #41 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 11 Nov 2013 22:13:36 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 15841 <at> debbugs.gnu.org
> Date: Mon, 11 Nov 2013 15:12:15 +0100
> 
> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
> new bug number, but since part of the breakage is display-related, I
> decided to report it here.  To reproduce:
> 
> 0. emacs -Q (built from trunk bzr 115033 or later)
> 1. Open a directory in Dired, e.g. the Emacs source tree root.
> 2. Put point on an entry listing a directory, e.g. admin.
> 3. Type `i'.
> => Emacs hangs.

Actually, if you rebuild with assertions (--enable-checking), Emacs
hits an assertion violation before it starts inflooping, and aborts.

Message #44 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 11 Nov 2013 21:38:58 +0100

On Mon, 11 Nov 2013 22:13:36 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: 15841 <at> debbugs.gnu.org
>> Date: Mon, 11 Nov 2013 15:12:15 +0100
>> 
>> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
>> new bug number, but since part of the breakage is display-related, I
>> decided to report it here.  To reproduce:
>> 
>> 0. emacs -Q (built from trunk bzr 115033 or later)
>> 1. Open a directory in Dired, e.g. the Emacs source tree root.
>> 2. Put point on an entry listing a directory, e.g. admin.
>> 3. Type `i'.
>> => Emacs hangs.
>
> Actually, if you rebuild with assertions (--enable-checking), Emacs
> hits an assertion violation before it starts inflooping, and aborts.

As a matter of fact, I've also gotten an abort a couple of times while
trying to track down this bug, though I did not build with
--enable-checking; unfortunately, I don't remember exactly what I what
doing, nor was I running under gdb, so no backtrace.  I also have
another observation, which seems to be reproducible, though I haven't
tested it extensively: if I type `i' on any directory entry in Dired,
which makes Emacs hang, then type `C-g' to unhang Emacs, then type `i'
on a *higher* directory entry in the same Dired buffer, whose name
begins with `.' (e.g. `.bzr'), then `i' works as expected.  This appears
to hold for all higher dotted (and only dotted) directory entries; but
if I type `i' on a *lower* dotted directory entry, then Emacs again
hangs.

Steve Berman

Message #47 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Tue, 12 Nov 2013 01:38:34 +0100

On Mon, 11 Nov 2013 15:12:15 +0100 Stephen Berman <stephen.berman <at> gmx.net> wrote:

> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
> new bug number, but since part of the breakage is display-related, I
> decided to report it here.  To reproduce:
>
> 0. emacs -Q (built from trunk bzr 115033 or later)
> 1. Open a directory in Dired, e.g. the Emacs source tree root.
> 2. Put point on an entry listing a directory, e.g. admin.
> 3. Type `i'.
> => Emacs hangs.
>
> If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
> nil)', then at step 3, `i' works as expected.
>
> Here are more details of the breakage:
>
> Typing `C-g' releases the hang and shows the admin subdirectory, but its
> display is corrupted: (i) the "available" blocks information, which
> should appear at the end of the "total" line, instead appears to the
> right of the first entry `.' and is fontified with dired-directory face;
> (ii) while all the entries of admin are listed below the `..' entry, the
> entire line of each entry is fontified with dired-directory face, and
> these lines cannot be accessed by vertical motion commands like `C-n' or
> `C-p', though they can be with horizontal motion commands like `C-f' and
> `C-p'.  In fact, all of these entries appear to Dired to be part of the
> line of the `..' entry, and AFACT this is what causes the hang: Emacs
> infloops in dired-move-to-filename because, when point appears to be at
> eof, beginning-of-line moves it back to the start of the `..' entry.

I tracked the problematic fontification and motion behavior to
insert-directory in files.el: it happens during the loop when
decode-coding-region is called on the file names of the subdirectory
entries.  I stepped through the code with Edebug but could not tell why
it goes wrong here, and I'm too tired to pursue it further now.  Also,
when I step through this code, the "available" information is added at
the end of the entire subdirectory listing, unlike what I observed above
when just invoking `i' and then `C-g'.  I guess this is due to the
interaction of redisplay with stepping through the code; it's still
clear that the subdirectory listing is being treated as part of the line
containing the `..' entry.

Steve Berman

Message #50 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Tue, 12 Nov 2013 11:03:47 +0100

On Tue, 12 Nov 2013 01:38:34 +0100 Stephen Berman <stephen.berman <at> gmx.net> wrote:

> On Mon, 11 Nov 2013 15:12:15 +0100 Stephen Berman <stephen.berman <at> gmx.net> wrote:
>
>> This change broke dired-maybe-insert-subdir.  Perhaps this should be a
>> new bug number, but since part of the breakage is display-related, I
>> decided to report it here.  To reproduce:
>>
>> 0. emacs -Q (built from trunk bzr 115033 or later)
>> 1. Open a directory in Dired, e.g. the Emacs source tree root.
>> 2. Put point on an entry listing a directory, e.g. admin.
>> 3. Type `i'.
>> => Emacs hangs.
>>
>> If I repeat the recipe but after step 1 type `M-: (setq cache-long-lines
>> nil)', then at step 3, `i' works as expected.
>>
>> Here are more details of the breakage:
>>
>> Typing `C-g' releases the hang and shows the admin subdirectory, but its
>> display is corrupted: (i) the "available" blocks information, which
>> should appear at the end of the "total" line, instead appears to the
>> right of the first entry `.' and is fontified with dired-directory face;
>> (ii) while all the entries of admin are listed below the `..' entry, the
>> entire line of each entry is fontified with dired-directory face, and
>> these lines cannot be accessed by vertical motion commands like `C-n' or
>> `C-p', though they can be with horizontal motion commands like `C-f' and
>> `C-p'.  In fact, all of these entries appear to Dired to be part of the
>> line of the `..' entry, and AFACT this is what causes the hang: Emacs
>> infloops in dired-move-to-filename because, when point appears to be at
>> eof, beginning-of-line moves it back to the start of the `..' entry.
>
> I tracked the problematic fontification and motion behavior to
> insert-directory in files.el: it happens during the loop when
> decode-coding-region is called on the file names of the subdirectory
> entries.  I stepped through the code with Edebug but could not tell why
> it goes wrong here, and I'm too tired to pursue it further now.  Also,
> when I step through this code, the "available" information is added at
> the end of the entire subdirectory listing, unlike what I observed above
> when just invoking `i' and then `C-g'.  I guess this is due to the
> interaction of redisplay with stepping through the code; it's still
> clear that the subdirectory listing is being treated as part of the line
> containing the `..' entry.

I set a breakpoint on decode_coding_object and stepped through it after
invoking `i', but the Dired display changed to show the problematic
fontification only upon exiting decode_coding_object.  I don't know how
to find out when cache_long_scans is checked other than manually tracing
the call chains of each subroutine of decode_coding_object, which is too
laborious.  Is it possible to have execution halt when cache_long_scans
is checked, and if so, how?  Or is there a better way to try to track
down this bug?

Steve Berman

Message #53 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Tue, 12 Nov 2013 18:31:32 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 15841 <at> debbugs.gnu.org
> Date: Tue, 12 Nov 2013 11:03:47 +0100
> 
> > I tracked the problematic fontification and motion behavior to
> > insert-directory in files.el: it happens during the loop when
> > decode-coding-region is called on the file names of the subdirectory
> > entries.  I stepped through the code with Edebug but could not tell why
> > it goes wrong here, and I'm too tired to pursue it further now.  Also,
> > when I step through this code, the "available" information is added at
> > the end of the entire subdirectory listing, unlike what I observed above
> > when just invoking `i' and then `C-g'.  I guess this is due to the
> > interaction of redisplay with stepping through the code; it's still
> > clear that the subdirectory listing is being treated as part of the line
> > containing the `..' entry.
> 
> I set a breakpoint on decode_coding_object and stepped through it after
> invoking `i', but the Dired display changed to show the problematic
> fontification only upon exiting decode_coding_object.  I don't know how
> to find out when cache_long_scans is checked other than manually tracing
> the call chains of each subroutine of decode_coding_object, which is too
> laborious.

There's no need: I already established that, as you point out, the
changes to the buffer that cause the problem are indeed made by
decode-coding-region.  The problem becomes visible when redisplay,
entered after decode-coding-region finishes its job, re-fontifies
portions of the buffer that were affected by the changes.  The way
this affects redisplay is through forward-line and
line-beginning-position, of which JIT Font Lock is a heavy user.
That's why you only see the effect after decode-coding-region returns.

> Is it possible to have execution halt when cache_long_scans is
> checked, and if so, how?

Watchpoints are the answer.  But in this case, there's only one place
in the whole Emacs where this variable is consulted: in search.c,
around line 610, so you could just put a breakpoint there.

In any case, I already traced through the code that is involved, and
the immediate reason for the assertion violation is that the cache
isn't being updated wrt changes in buffer size (which are caused by
decoding the stuff brought in by 'ls').  However, a naive attempt to
force such updates didn't solve the whole problem: the aborts are
gone, but the infloop is still there, and also other minor display
issues.  So I guess there's another factor at work there...

I also need to figure out how to keep the cache up to date without
penalizing performance, which would render the cache worthless.

> Or is there a better way to try to track down this bug?

The cache has only 3 public interfaces (see region-cache.c), so it is
easy to put breakpoints in all of them and see what happens.  That's
what I did.

Thanks.

Message #56 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Tue, 12 Nov 2013 22:34:43 +0100

On Tue, 12 Nov 2013 18:31:32 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

> I already established that, as you point out, the
> changes to the buffer that cause the problem are indeed made by
> decode-coding-region.  The problem becomes visible when redisplay,
> entered after decode-coding-region finishes its job, re-fontifies
> portions of the buffer that were affected by the changes.  The way
> this affects redisplay is through forward-line and
> line-beginning-position, of which JIT Font Lock is a heavy user.
> That's why you only see the effect after decode-coding-region returns.

Thanks for the explanation.

>> Is it possible to have execution halt when cache_long_scans is
>> checked, and if so, how?
>
> Watchpoints are the answer.  But in this case, there's only one place
> in the whole Emacs where this variable is consulted: in search.c,
> around line 610, so you could just put a breakpoint there.
>
> In any case, I already traced through the code that is involved, and
> the immediate reason for the assertion violation is that the cache
> isn't being updated wrt changes in buffer size (which are caused by
> decoding the stuff brought in by 'ls').  However, a naive attempt to
> force such updates didn't solve the whole problem: the aborts are
> gone, but the infloop is still there, and also other minor display
> issues.  So I guess there's another factor at work there...

Fascinating, what unsuspected and apparently unrelated effects can be
brought to the surface by toggling the value of a variable!

> I also need to figure out how to keep the cache up to date without
> penalizing performance, which would render the cache worthless.
>
>> Or is there a better way to try to track down this bug?
>
> The cache has only 3 public interfaces (see region-cache.c), so it is
> easy to put breakpoints in all of them and see what happens.  That's
> what I did.

Thanks for the advice and for working on this bug.

Steve Berman

Message #59 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Thu, 14 Nov 2013 00:45:17 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> > From: Michael Heerdegen <michael_heerdegen <at> web.de>
> > Cc: 15841 <at> debbugs.gnu.org,  nbtrap <at> nbtrap.com
> > Date: Mon, 11 Nov 2013 04:39:56 +0100
> > 
> > The culprit was my own code: it placed myriads of invisible overlays
> > with no properties into the buffer.  Under these extreme circumstances,
> > `line-number-at-pos' indeed gets extremely slow at the end of my 10000
> > lines buffer: one invocation needs over a second.  I saw that with elp
> > as well as with profiler.  Setting `cache-long-scans' to nil (or
> > removing the overlays) cures this.
>
> Can you show some simple enough code that puts so many overlays, and
> has this effect?  It sounds like some optimization is in order.

Sorry, I did not find an easy test case.  I tried to simplify the
essence of what my code does, but the effect didn't occur.  Seems that
lots of different things must come together to provoke this symptom.
Given the fact that my code was really broken, I don't think it's worth
the time to follow this up.  It would cost me many hours to compile some
test code for emacs -Q.  Or do you think it would be worth it?  If you
think it could be very important, I would do it, but it would be very
time intensive.  My code works well now with `cache-long-scans' t after
the right fixes, btw.


Regards,

Michael.

Message #62 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 15841 <at> debbugs.gnu.org, nbtrap <at> nbtrap.com
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Thu, 14 Nov 2013 05:51:26 +0200

> From: Michael Heerdegen <michael_heerdegen <at> web.de>
> Cc: 15841 <at> debbugs.gnu.org,  nbtrap <at> nbtrap.com
> Date: Thu, 14 Nov 2013 00:45:17 +0100
> 
> > Can you show some simple enough code that puts so many overlays, and
> > has this effect?  It sounds like some optimization is in order.
> 
> Sorry, I did not find an easy test case.  I tried to simplify the
> essence of what my code does, but the effect didn't occur.  Seems that
> lots of different things must come together to provoke this symptom.
> Given the fact that my code was really broken, I don't think it's worth
> the time to follow this up.  It would cost me many hours to compile some
> test code for emacs -Q.  Or do you think it would be worth it?  If you
> think it could be very important, I would do it, but it would be very
> time intensive.  My code works well now with `cache-long-scans' t after
> the right fixes, btw.

If the bug isn't reproducible, it isn't worth working on.

Thanks.

Message #67 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: stephen.berman <at> gmx.net
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Fri, 15 Nov 2013 18:34:05 +0200

> Date: Tue, 12 Nov 2013 18:31:32 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 15841 <at> debbugs.gnu.org
> 
> In any case, I already traced through the code that is involved, and
> the immediate reason for the assertion violation is that the cache
> isn't being updated wrt changes in buffer size (which are caused by
> decoding the stuff brought in by 'ls').  However, a naive attempt to
> force such updates didn't solve the whole problem: the aborts are
> gone, but the infloop is still there, and also other minor display
> issues.  So I guess there's another factor at work there...

I think I might have found a solution for this.  Could you please run
with the patch below for a while, and see if it gives good results?

=== modified file 'src/coding.c'
--- src/coding.c	2013-10-08 06:40:09 +0000
+++ src/coding.c	2013-11-15 16:27:56 +0000
@@ -9358,6 +9358,14 @@ code_convert_region (Lisp_Object start, 
   setup_coding_system (coding_system, &coding);
   coding.mode |= CODING_MODE_LAST_BLOCK;
 
+  if (BUFFERP (dst_object) && !EQ (dst_object, src_object))
+    {
+      struct buffer *buf = XBUFFER (dst_object);
+      ptrdiff_t buf_pt = BUF_PT (buf);
+
+      invalidate_buffer_caches (buf, buf_pt, buf_pt);
+    }
+
   if (encodep)
     encode_coding_object (&coding, src_object, from, from_byte, to, to_byte,
 			  dst_object);
@@ -9447,6 +9455,15 @@ code_convert_string (Lisp_Object string,
   coding.mode |= CODING_MODE_LAST_BLOCK;
   chars = SCHARS (string);
   bytes = SBYTES (string);
+
+  if (BUFFERP (dst_object))
+    {
+      struct buffer *buf = XBUFFER (dst_object);
+      ptrdiff_t buf_pt = BUF_PT (buf);
+
+      invalidate_buffer_caches (buf, buf_pt, buf_pt);
+    }
+
   if (encodep)
     encode_coding_object (&coding, string, 0, 0, chars, bytes, dst_object);
   else

=== modified file 'src/insdel.c'
--- src/insdel.c	2013-11-11 05:18:53 +0000
+++ src/insdel.c	2013-11-15 16:27:56 +0000
@@ -993,6 +993,11 @@ insert_from_gap (ptrdiff_t nchars, ptrdi
   if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
     nchars = nbytes;
 
+  /* No need to call prepare_to_modify_buffer, since this is called
+     from places that replace some region with a different text, so
+     prepare_to_modify_buffer was already called by the deletion part
+     of this dance.  */
+  invalidate_buffer_caches (current_buffer, GPT, GPT);
   record_insert (GPT, nchars);
   MODIFF++;
 
@@ -1869,19 +1874,26 @@ prepare_to_modify_buffer (ptrdiff_t star
 			  ptrdiff_t *preserve_ptr)
 {
   prepare_to_modify_buffer_1 (start, end, preserve_ptr);
+  invalidate_buffer_caches (current_buffer, start, end);
+}
 
-  if (current_buffer->newline_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->newline_cache,
-                             start - BEG, Z - end);
-  if (current_buffer->width_run_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->width_run_cache,
-                             start - BEG, Z - end);
-  if (current_buffer->bidi_paragraph_cache)
-    invalidate_region_cache (current_buffer,
-                             current_buffer->bidi_paragraph_cache,
-                             start - BEG, Z - end);
+/* Invalidate the caches maintained by the buffer BUF, if any, for the
+   region between buffer positions START and END.  */
+void
+invalidate_buffer_caches (struct buffer *buf, ptrdiff_t start, ptrdiff_t end)
+{
+  if (buf->newline_cache)
+    invalidate_region_cache (buf,
+                             buf->newline_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
+  if (buf->width_run_cache)
+    invalidate_region_cache (buf,
+                             buf->width_run_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
+  if (buf->bidi_paragraph_cache)
+    invalidate_region_cache (buf,
+                             buf->bidi_paragraph_cache,
+                             start - BUF_BEG (buf), BUF_Z (buf) - end);
 }
 
 /* These macros work with an argument named `preserve_ptr'

=== modified file 'src/lisp.h'
--- src/lisp.h	2013-11-15 08:18:37 +0000
+++ src/lisp.h	2013-11-15 16:27:56 +0000
@@ -3486,6 +3486,7 @@ extern Lisp_Object del_range_2 (ptrdiff_
 extern void modify_text (ptrdiff_t, ptrdiff_t);
 extern void prepare_to_modify_buffer (ptrdiff_t, ptrdiff_t, ptrdiff_t *);
 extern void prepare_to_modify_buffer_1 (ptrdiff_t, ptrdiff_t, ptrdiff_t *);
+extern void invalidate_buffer_caches (struct buffer *, ptrdiff_t, ptrdiff_t);
 extern void signal_after_change (ptrdiff_t, ptrdiff_t, ptrdiff_t);
 extern void adjust_after_insert (ptrdiff_t, ptrdiff_t, ptrdiff_t,
 				 ptrdiff_t, ptrdiff_t);

Message #72 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Fri, 15 Nov 2013 19:05:42 +0100

On Fri, 15 Nov 2013 18:34:05 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> Date: Tue, 12 Nov 2013 18:31:32 +0200
>> From: Eli Zaretskii <eliz <at> gnu.org>
>> Cc: 15841 <at> debbugs.gnu.org
>> 
>> In any case, I already traced through the code that is involved, and
>> the immediate reason for the assertion violation is that the cache
>> isn't being updated wrt changes in buffer size (which are caused by
>> decoding the stuff brought in by 'ls').  However, a naive attempt to
>> force such updates didn't solve the whole problem: the aborts are
>> gone, but the infloop is still there, and also other minor display
>> issues.  So I guess there's another factor at work there...
>
> I think I might have found a solution for this.  Could you please run
> with the patch below for a while, and see if it gives good results?

Initial tests succeeded: `i' in Dired works as expected and there is no
infloop or display oddities.  I'll report back if any problems crop up
on further use.  Otherwise, thanks for fixing this!

Steve Berman

Message #75 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 16 Nov 2013 18:53:31 +0000

On Fri 15 Nov 2013, Eli Zaretskii wrote:

>> Date: Tue, 12 Nov 2013 18:31:32 +0200
>> From: Eli Zaretskii <eliz <at> gnu.org>
>> Cc: 15841 <at> debbugs.gnu.org
>> 
>> In any case, I already traced through the code that is involved, and
>> the immediate reason for the assertion violation is that the cache
>> isn't being updated wrt changes in buffer size (which are caused by
>> decoding the stuff brought in by 'ls').  However, a naive attempt to
>> force such updates didn't solve the whole problem: the aborts are
>> gone, but the infloop is still there, and also other minor display
>> issues.  So I guess there's another factor at work there...
>
> I think I might have found a solution for this.  Could you please run
> with the patch below for a while, and see if it gives good results?

Thanks Eli - I've seen the same problem with inserting a subdirectory in
a dired buffer. Applying your patch to r115122 fixed dired for me.

    AndyM

Message #78 received at 15841 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 15841 <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Sat, 16 Nov 2013 21:02:38 +0200

> From: Andy Moreton <andrewjmoreton <at> gmail.com>
> Date: Sat, 16 Nov 2013 18:53:31 +0000
> 
> Thanks Eli - I've seen the same problem with inserting a subdirectory in
> a dired buffer. Applying your patch to r115122 fixed dired for me.

Thanks for trying.  I will wait till Monday, to see if no adverse
effects pop up, then commit those changes.

Message #85 received at 15841-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andy Moreton <andrewjmoreton <at> gmail.com>,
 Stephen Berman <stephen.berman <at> gmx.net>
Cc: 15841-done <at> debbugs.gnu.org
Subject: Re: bug#15841: Display bugs with cache-long-lines non-nil
Date: Mon, 18 Nov 2013 18:32:05 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: 15841 <at> debbugs.gnu.org
> Date: Fri, 15 Nov 2013 19:05:42 +0100
> 
> Initial tests succeeded: `i' in Dired works as expected and there is no
> infloop or display oddities.  I'll report back if any problems crop up
> on further use.  Otherwise, thanks for fixing this!

No further problems, so I committed the changes as trunk revision
115138, and I'm closing this bug.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.