GNU bug report logs - #78545
31.0.50; project-mode-line is slow because it tries to read files on each update

Previous Next

To reply to this bug, email your comments to 78545 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Yikai Zhao <yikai <at> z1k.dev>
To: bug-gnu-emacs <at> gnu.org
Subject: 31.0.50; project-mode-line is slow because it tries to read files on
 each update
Date: Thu, 22 May 2025 14:15:49 +0800

After enabling `project-mode-line`, I find my emacs slightly less
responsive. I think it's because project-mode-line is slow due to
it trying to read files on each update. Here's how I reproduce it:

- Run `emacs -Q`
- Evaluate `(setopt project-mode-line t)`
- Open some random file in a non-project directory, e.g. `/tmp/test`,
start typing

Here's part of the profiling result:

         114  62% - redisplay_internal (C function)
          42  23%  - eval
          41  22%   - project-mode-line-format
          41  22%    - project-current
          41  22%     - project--find-in-directory
          41  22%      - run-hook-with-args-until-success
          41  22%       - project-try-vc
          41  22%        - project-try-vc--search
          25  13%         - project--value-in-dir
          22  12%          - hack-dir-local-variables
          22  12%           - #<byte-code-function 24E>
          22  12%            - hack-dir-local--get-variables
          22  12%             - dir-locals-find-file
          22  12%              - locate-dominating-file
          12   6%               + abbreviate-file-name
           3   1%          + #<byte-code-function 98F>
          13   7%         - locate-dominating-file
           6   3%          + #<byte-code-function 4D9>
           3   1%            abbreviate-file-name
           3   1%

It shows that `project-mode-line-format` can take a significant portion
of the redisplay time. It increases even more when the directory is deep
or the disk is slow.

IMO the performance of this function should be improved.

I have two questions:

1. Is it possible to simply cache the result as buffer-local?
2. IIUC some of the file reading comes from `project--value-in-dir`,
which opens a temporary buffer to load dir local variables. Why doesn't
it simply read the variable value in the current buffer?


Thanks
Yikai

Message #8 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Yikai Zhao <yikai <at> z1k.dev>
Cc: 78545 <at> debbugs.gnu.org
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to
 read files on each update
Date: Thu, 22 May 2025 07:47:18 -0400

[Message part 1 (text/plain, inline)]

On Thu, May 22, 2025 at 2:17 AM Yikai Zhao <yikai <at> z1k.dev> wrote:

> After enabling `project-mode-line`, I find my emacs slightly less
> responsive. I think it's because project-mode-line is slow due to
> it trying to read files on each update. Here's how I reproduce it:
>
> - Run `emacs -Q`
> - Evaluate `(setopt project-mode-line t)`
> - Open some random file in a non-project directory, e.g. `/tmp/test`,
> start typing
>
> Here's part of the profiling result:
>
>          114  62% - redisplay_internal (C function)
>           42  23%  - eval
>           41  22%   - project-mode-line-format
>           41  22%    - project-current
>           41  22%     - project--find-in-directory
>           41  22%      - run-hook-with-args-until-success
>           41  22%       - project-try-vc
>           41  22%        - project-try-vc--search
>           25  13%         - project--value-in-dir
>           22  12%          - hack-dir-local-variables
>           22  12%           - #<byte-code-function 24E>
>           22  12%            - hack-dir-local--get-variables
>           22  12%             - dir-locals-find-file
>           22  12%              - locate-dominating-file
>           12   6%               + abbreviate-file-name
>            3   1%          + #<byte-code-function 98F>
>           13   7%         - locate-dominating-file
>            6   3%          + #<byte-code-function 4D9>
>            3   1%            abbreviate-file-name
>            3   1%
>
> It shows that `project-mode-line-format` can take a significant portion
> of the redisplay time. It increases even more when the directory is deep
> or the disk is slow.
>
> IMO the performance of this function should be improved.
>
> I have two questions:
>
> 1. Is it possible to simply cache the result as buffer-local?
> 2. IIUC some of the file reading comes from `project--value-in-dir`,
> which opens a temporary buffer to load dir local variables. Why doesn't
> it simply read the variable value in the current buffer?
>

I've advised project functions to cache the "invariants" current project
and project root to incur these costs once per buffer.  I also cache the
nuance of a "non project" to avoid repeated project probing on buffers that
don't have a project.

I thought maybe I was the only person silly enough to invoke costly project
functions for every tab in `tab-bar` and every mode-line update.

I could contribute optional caching to project.el if there is sufficient
demand for it.  Unless someone else wants to.

[Message part 2 (text/html, inline)]

Message #11 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: bug-gnu-emacs <at> gnu.org, Ship Mints <shipmints <at> gmail.com>,
 Yikai Zhao <yikai <at> z1k.dev>
Cc: 78545 <at> debbugs.gnu.org
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to read files on each update
Date: Thu, 22 May 2025 11:53:57 -0400

On May 22, 2025 7:47:18 AM EDT, Ship Mints <shipmints <at> gmail.com> wrote:
>On Thu, May 22, 2025 at 2:17 AM Yikai Zhao <yikai <at> z1k.dev> wrote:
>
>> After enabling `project-mode-line`, I find my emacs slightly less
>> responsive. I think it's because project-mode-line is slow due to
>> it trying to read files on each update. Here's how I reproduce it:
>>
>> - Run `emacs -Q`
>> - Evaluate `(setopt project-mode-line t)`
>> - Open some random file in a non-project directory, e.g. `/tmp/test`,
>> start typing
>>
>> Here's part of the profiling result:
>>
>>          114  62% - redisplay_internal (C function)
>>           42  23%  - eval
>>           41  22%   - project-mode-line-format
>>           41  22%    - project-current
>>           41  22%     - project--find-in-directory
>>           41  22%      - run-hook-with-args-until-success
>>           41  22%       - project-try-vc
>>           41  22%        - project-try-vc--search
>>           25  13%         - project--value-in-dir
>>           22  12%          - hack-dir-local-variables
>>           22  12%           - #<byte-code-function 24E>
>>           22  12%            - hack-dir-local--get-variables
>>           22  12%             - dir-locals-find-file
>>           22  12%              - locate-dominating-file
>>           12   6%               + abbreviate-file-name
>>            3   1%          + #<byte-code-function 98F>
>>           13   7%         - locate-dominating-file
>>            6   3%          + #<byte-code-function 4D9>
>>            3   1%            abbreviate-file-name
>>            3   1%
>>
>> It shows that `project-mode-line-format` can take a significant portion
>> of the redisplay time. It increases even more when the directory is deep
>> or the disk is slow.
>>
>> IMO the performance of this function should be improved.
>>
>> I have two questions:
>>
>> 1. Is it possible to simply cache the result as buffer-local?
>> 2. IIUC some of the file reading comes from `project--value-in-dir`,
>> which opens a temporary buffer to load dir local variables. Why doesn't
>> it simply read the variable value in the current buffer?
>>
>
>I've advised project functions to cache the "invariants" current project
>and project root to incur these costs once per buffer.  I also cache the
>nuance of a "non project" to avoid repeated project probing on buffers that
>don't have a project.
>
>I thought maybe I was the only person silly enough to invoke costly project
>functions for every tab in `tab-bar` and every mode-line update.
>
>I could contribute optional caching to project.el if there is sufficient
>demand for it.  Unless someone else wants to.

We could also just have the IO functions signal error if called during a mode line update.

Message #17 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Kristoffer Balintona <krisbalintona <at> gmail.com>
To: Yikai Zhao <yikai <at> z1k.dev>, 78545 <at> debbugs.gnu.org
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to
 read files on each update
Date: Thu, 22 May 2025 08:58:15 -0700

On Thu, May 22 2025, Yikai Zhao wrote:

> After enabling `project-mode-line`, I find my emacs slightly less
> responsive. I think it's because project-mode-line is slow due to
> it trying to read files on each update. Here's how I reproduce it:
>
> - Run `emacs -Q`
> - Evaluate `(setopt project-mode-line t)`
> - Open some random file in a non-project directory, e.g. `/tmp/test`,
> start typing
>
> Here's part of the profiling result:
>
>          114  62% - redisplay_internal (C function)
>           42  23%  - eval
>           41  22%   - project-mode-line-format
>           41  22%    - project-current
>           41  22%     - project--find-in-directory
>           41  22%      - run-hook-with-args-until-success
>           41  22%       - project-try-vc
>           41  22%        - project-try-vc--search
>           25  13%         - project--value-in-dir
>           22  12%          - hack-dir-local-variables
>           22  12%           - #<byte-code-function 24E>
>           22  12%            - hack-dir-local--get-variables
>           22  12%             - dir-locals-find-file
>           22  12%              - locate-dominating-file
>           12   6%               + abbreviate-file-name
>            3   1%          + #<byte-code-function 98F>
>           13   7%         - locate-dominating-file
>            6   3%          + #<byte-code-function 4D9>
>            3   1%            abbreviate-file-name
>            3   1%
>
> It shows that `project-mode-line-format` can take a significant portion
> of the redisplay time. It increases even more when the directory is deep
> or the disk is slow.
>
> IMO the performance of this function should be improved.
>
> I have two questions:
>
> 1. Is it possible to simply cache the result as buffer-local?
> 2. IIUC some of the file reading comes from `project--value-in-dir`,
> which opens a temporary buffer to load dir local variables. Why doesn't
> it simply read the variable value in the current buffer?
>

I experienced similar, actually, on Nix where the Emacs source files,
having been built in the special Nix store, are somewhere deep under
/nix/store/. I suspected it was project.el trying to find a root. I
never got around to confirming it nor finding a fix.

-- 
In gratitude,
Kristoffer

Message #20 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Kristoffer Balintona <krisbalintona <at> gmail.com>
Cc: Yikai Zhao <yikai <at> z1k.dev>, 78545 <at> debbugs.gnu.org
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to
 read files on each update
Date: Thu, 22 May 2025 12:23:01 -0400

[Message part 1 (text/plain, inline)]

On Thu, May 22, 2025 at 11:59 AM Kristoffer Balintona <
krisbalintona <at> gmail.com> wrote:

> On Thu, May 22 2025, Yikai Zhao wrote:
>
> > After enabling `project-mode-line`, I find my emacs slightly less
> > responsive. I think it's because project-mode-line is slow due to
> > it trying to read files on each update. Here's how I reproduce it:
> >
> > - Run `emacs -Q`
> > - Evaluate `(setopt project-mode-line t)`
> > - Open some random file in a non-project directory, e.g. `/tmp/test`,
> > start typing
> >
> > Here's part of the profiling result:
> >
> >          114  62% - redisplay_internal (C function)
> >           42  23%  - eval
> >           41  22%   - project-mode-line-format
> >           41  22%    - project-current
> >           41  22%     - project--find-in-directory
> >           41  22%      - run-hook-with-args-until-success
> >           41  22%       - project-try-vc
> >           41  22%        - project-try-vc--search
> >           25  13%         - project--value-in-dir
> >           22  12%          - hack-dir-local-variables
> >           22  12%           - #<byte-code-function 24E>
> >           22  12%            - hack-dir-local--get-variables
> >           22  12%             - dir-locals-find-file
> >           22  12%              - locate-dominating-file
> >           12   6%               + abbreviate-file-name
> >            3   1%          + #<byte-code-function 98F>
> >           13   7%         - locate-dominating-file
> >            6   3%          + #<byte-code-function 4D9>
> >            3   1%            abbreviate-file-name
> >            3   1%
> >
> > It shows that `project-mode-line-format` can take a significant portion
> > of the redisplay time. It increases even more when the directory is deep
> > or the disk is slow.
> >
> > IMO the performance of this function should be improved.
> >
> > I have two questions:
> >
> > 1. Is it possible to simply cache the result as buffer-local?
> > 2. IIUC some of the file reading comes from `project--value-in-dir`,
> > which opens a temporary buffer to load dir local variables. Why doesn't
> > it simply read the variable value in the current buffer?
> >
>
> I experienced similar, actually, on Nix where the Emacs source files,
> having been built in the special Nix store, are somewhere deep under
> /nix/store/. I suspected it was project.el trying to find a root. I
> never got around to confirming it nor finding a fix.
>

I'm taking some time today to rework my approach rather than use advice.
That was hacky, and good enough for personal use to scratch an immediate
itch.  If this comes out well, I'll propose a project.el patch and see what
people think.

Daniel, I'm not so sure about blocking all IO in mode-line updates.  That
sounds like it will surprise a lot of people.

-Stephane

[Message part 2 (text/html, inline)]

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Yikai Zhao <yikai <at> z1k.dev>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: bug-gnu-emacs <at> gnu.org, 78545 <at> debbugs.gnu.org,
 Ship Mints <shipmints <at> gmail.com>
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to
 read files on each update
Date: Fri, 23 May 2025 01:02:02 +0800

[Message part 1 (text/plain, inline)]

On Thu, May 22, 2025 at 23:54 Daniel Colascione <dancol <at> dancol.org> wrote:

>
>
> On May 22, 2025 7:47:18 AM EDT, Ship Mints <shipmints <at> gmail.com> wrote:
> >On Thu, May 22, 2025 at 2:17 AM Yikai Zhao <yikai <at> z1k.dev> wrote:
> >
> >> After enabling `project-mode-line`, I find my emacs slightly less
> >> responsive. I think it's because project-mode-line is slow due to
> >> it trying to read files on each update. Here's how I reproduce it:
> >>
> >> - Run `emacs -Q`
> >> - Evaluate `(setopt project-mode-line t)`
> >> - Open some random file in a non-project directory, e.g. `/tmp/test`,
> >> start typing
> >>
> >> Here's part of the profiling result:
> >>
> >>          114  62% - redisplay_internal (C function)
> >>           42  23%  - eval
> >>           41  22%   - project-mode-line-format
> >>           41  22%    - project-current
> >>           41  22%     - project--find-in-directory
> >>           41  22%      - run-hook-with-args-until-success
> >>           41  22%       - project-try-vc
> >>           41  22%        - project-try-vc--search
> >>           25  13%         - project--value-in-dir
> >>           22  12%          - hack-dir-local-variables
> >>           22  12%           - #<byte-code-function 24E>
> >>           22  12%            - hack-dir-local--get-variables
> >>           22  12%             - dir-locals-find-file
> >>           22  12%              - locate-dominating-file
> >>           12   6%               + abbreviate-file-name
> >>            3   1%          + #<byte-code-function 98F>
> >>           13   7%         - locate-dominating-file
> >>            6   3%          + #<byte-code-function 4D9>
> >>            3   1%            abbreviate-file-name
> >>            3   1%
> >>
> >> It shows that `project-mode-line-format` can take a significant portion
> >> of the redisplay time. It increases even more when the directory is deep
> >> or the disk is slow.
> >>
> >> IMO the performance of this function should be improved.
> >>
> >> I have two questions:
> >>
> >> 1. Is it possible to simply cache the result as buffer-local?
> >> 2. IIUC some of the file reading comes from `project--value-in-dir`,
> >> which opens a temporary buffer to load dir local variables. Why doesn't
> >> it simply read the variable value in the current buffer?
> >>
> >
> >I've advised project functions to cache the "invariants" current project
> >and project root to incur these costs once per buffer.  I also cache the
> >nuance of a "non project" to avoid repeated project probing on buffers
> that
> >don't have a project.
> >
> >I thought maybe I was the only person silly enough to invoke costly
> project
> >functions for every tab in `tab-bar` and every mode-line update.
> >
> >I could contribute optional caching to project.el if there is sufficient
> >demand for it.  Unless someone else wants to.
>
> We could also just have the IO functions signal error if called during a
> mode line update.


Personally I think this is a great idea. Aside from mode line updates, IO
functions can cause undesired delay in many other functions (like tab bar
or maybe post command hook). If we have a general way to mark certain hooks
as “should not block by IO” and be able to capture those violations, it
would be very helpful.


Yikai


>

[Message part 2 (text/html, inline)]

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: Yikai Zhao <yikai <at> z1k.dev>
Cc: bug-gnu-emacs <at> gnu.org, 78545 <at> debbugs.gnu.org,
 Ship Mints <shipmints <at> gmail.com>
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to read files on each update
Date: Thu, 22 May 2025 13:21:18 -0400

On May 22, 2025 1:02:02 PM EDT, Yikai Zhao <yikai <at> z1k.dev> wrote:
>On Thu, May 22, 2025 at 23:54 Daniel Colascione <dancol <at> dancol.org> wrote:
>
>>
>>
>> On May 22, 2025 7:47:18 AM EDT, Ship Mints <shipmints <at> gmail.com> wrote:
>> >On Thu, May 22, 2025 at 2:17 AM Yikai Zhao <yikai <at> z1k.dev> wrote:
>> >
>> >> After enabling `project-mode-line`, I find my emacs slightly less
>> >> responsive. I think it's because project-mode-line is slow due to
>> >> it trying to read files on each update. Here's how I reproduce it:
>> >>
>> >> - Run `emacs -Q`
>> >> - Evaluate `(setopt project-mode-line t)`
>> >> - Open some random file in a non-project directory, e.g. `/tmp/test`,
>> >> start typing
>> >>
>> >> Here's part of the profiling result:
>> >>
>> >>          114  62% - redisplay_internal (C function)
>> >>           42  23%  - eval
>> >>           41  22%   - project-mode-line-format
>> >>           41  22%    - project-current
>> >>           41  22%     - project--find-in-directory
>> >>           41  22%      - run-hook-with-args-until-success
>> >>           41  22%       - project-try-vc
>> >>           41  22%        - project-try-vc--search
>> >>           25  13%         - project--value-in-dir
>> >>           22  12%          - hack-dir-local-variables
>> >>           22  12%           - #<byte-code-function 24E>
>> >>           22  12%            - hack-dir-local--get-variables
>> >>           22  12%             - dir-locals-find-file
>> >>           22  12%              - locate-dominating-file
>> >>           12   6%               + abbreviate-file-name
>> >>            3   1%          + #<byte-code-function 98F>
>> >>           13   7%         - locate-dominating-file
>> >>            6   3%          + #<byte-code-function 4D9>
>> >>            3   1%            abbreviate-file-name
>> >>            3   1%
>> >>
>> >> It shows that `project-mode-line-format` can take a significant portion
>> >> of the redisplay time. It increases even more when the directory is deep
>> >> or the disk is slow.
>> >>
>> >> IMO the performance of this function should be improved.
>> >>
>> >> I have two questions:
>> >>
>> >> 1. Is it possible to simply cache the result as buffer-local?
>> >> 2. IIUC some of the file reading comes from `project--value-in-dir`,
>> >> which opens a temporary buffer to load dir local variables. Why doesn't
>> >> it simply read the variable value in the current buffer?
>> >>
>> >
>> >I've advised project functions to cache the "invariants" current project
>> >and project root to incur these costs once per buffer.  I also cache the
>> >nuance of a "non project" to avoid repeated project probing on buffers
>> that
>> >don't have a project.
>> >
>> >I thought maybe I was the only person silly enough to invoke costly
>> project
>> >functions for every tab in `tab-bar` and every mode-line update.
>> >
>> >I could contribute optional caching to project.el if there is sufficient
>> >demand for it.  Unless someone else wants to.
>>
>> We could also just have the IO functions signal error if called during a
>> mode line update.
>
>
>Personally I think this is a great idea. Aside from mode line updates, IO
>functions can cause undesired delay in many other functions (like tab bar
>or maybe post command hook). If we have a general way to mark certain hooks
>as “should not block by IO” and be able to capture those violations, it
>would be very helpful.

I've been meaning to write a slow hook detector for a few weeks now. Feel free to beat me to it. I figure we can start by just calling each entry in the major hooks with a new run hooks (with args) variant that records times of each and reports anything over a threshold as a warning.


>
>
>Yikai
>
>
>>

Message #35 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Yikai Zhao <yikai <at> z1k.dev>
Cc: dancol <at> dancol.org, 78545 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#78545: 31.0.50;
 project-mode-line is slow because it tries to read files on each
 update
Date: Thu, 22 May 2025 22:25:05 +0300

> Cc: 78545 <at> debbugs.gnu.org, shipmints <at> gmail.com
> From: Yikai Zhao <yikai <at> z1k.dev>
> Date: Fri, 23 May 2025 01:02:02 +0800
> 
>  We could also just have the IO functions signal error if called during a mode line update.
> 
> Personally I think this is a great idea. Aside from mode line updates, IO functions can cause undesired delay
> in many other functions (like tab bar or maybe post command hook). If we have a general way to mark
> certain hooks as “should not block by IO” and be able to capture those violations, it would be very helpful. 

Signaling an error inside redisplay is not useful.  It just writes a
message into *Messages* and that's it.  Unless you are very vigilant,
it will take you a long time to even look in *Messages* and realize
Emacs signaled that error.

If a function doesn't want to do its job in some situation, it should
return without doing anything, rather than signaling an error.

Besides, I see nothing wrong with calling from redisplay functions
that access the filesystem, not in general, anyway.  Sure, calling
locate-dominating-file from a deep directory is not a very wise thing
to do as part of updating the mode line, but Emacs always gives us
enough rope to hang ourselves, trusting us that we won't.  We should
live up to that trust by not doing silly things.

Message #38 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: yikai <at> z1k.dev, 78545 <at> debbugs.gnu.org, shipmints <at> gmail.com
Subject: Re: bug#78545: 31.0.50;
 project-mode-line is slow because it tries to read files on each
 update
Date: Thu, 22 May 2025 22:28:07 +0300

> Cc: 78545 <at> debbugs.gnu.org, shipmints <at> gmail.com
> Date: Thu, 22 May 2025 13:21:18 -0400
> From: Daniel Colascione <dancol <at> dancol.org>
> 
> I've been meaning to write a slow hook detector for a few weeks now. Feel free to beat me to it. I figure we can start by just calling each entry in the major hooks with a new run hooks (with args) variant that records times of each and reports anything over a threshold as a warning.

I think that would be a very useful feature.  But it won't find this
particular problem, because project.el doesn't use a hook to update
the mode line in this case.

Message #41 received at 78545 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Daniel Colascione <dancol <at> dancol.org>, 78545 <at> debbugs.gnu.org,
 shipmints <at> gmail.com, yikai <at> z1k.dev
Subject: Re: bug#78545: 31.0.50; project-mode-line is slow because it tries to
 read files on each update
Date: Fri, 23 May 2025 03:37:21 +0300

On 22/05/2025 18:53, Daniel Colascione wrote:
> We could also just have the IO functions signal error if called during a mode line update.

The first 'project-current' call might as well happen during the 
mode-line refresh, when the project root is not cached. What's 
problematic is having these calls do I/O every time.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.