GNU bug report logs -
#41646
Startup in Windows is very slow when load-path contains many entries.
Previous Next
Reported by: Nicolas Bértolo <nicolasbertolo <at> gmail.com>
Date: Mon, 1 Jun 2020 14:27:02 UTC
Severity: minor
Tags: fixed, patch
Fixed in version 28.1
Done: Lars Ingebrigtsen <larsi <at> gnus.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 41646 in the body.
You can then email your comments to 41646 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 01 Jun 2020 14:27:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Nicolas Bértolo <nicolasbertolo <at> gmail.com>:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org.
(Mon, 01 Jun 2020 14:27:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
I have an issue regarding startup times in Windows. My configuration is
Spacemacs with many layers enabled. My load-path contains 380 entries.
I have profiled Emacs in Windows and found that it spends most of the startup
time calling wopen(). This is because when calling (load "foo") it checks all
directories in load-path for ("foo.el" "foo.elc" "foo.el.gz" "foo.elc.gz"
"foo.dll"). It gets worse when load-prefer-newer is t.
In my case `load-path` contains 380 entries, so every call to load will perform
380 * 5 = 1900 calls to wopen. This is very slow in Windows because its
filesystem is not optimized for so many accesses to small files.
I thought that a caching mechanism would help.
This "cache" would consist of a mapping of files that would get probed when
(load "foo") runs. It would be implemented as a hash table. The contents of this
hash table could be loaded from load-cache.el files in each package directory.
The directory foo-pkg could have a file load-cache.el with:
foo -> ("foo-pkg/foo.el"
"foo-pkg/foo.elc")
[...]
The directory bar-pkg could have a file load-cache.el with:
bar -> ("bar-pkg/bar.el"
"bar-pkg/bar.elc")
[...]
When a package is activated we could update the in-memory hash table by loading
its load-cache.el file. Then, when (require 'foo) runs, the loading code could
look at the hash table and only fopen() the files associated with the feature we
are loading. This would reduce the number of calls to fopen() from thousands to
~2 in the worst case.
Or we could have a big load-cache.el in `package-user-dir'. I prefer many small
files because maintaining the big file when many Emacs instances could be
installing or removing packages is synchronization nightmare.
Of course, this feature would be disabled by default.
What do you think?
Nicolas
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 01 Jun 2020 16:07:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Nicolas Bértolo <nicolasbertolo <at> gmail.com>
> Date: Mon, 1 Jun 2020 11:26:35 -0300
>
> I have an issue regarding startup times in Windows. My configuration is
> Spacemacs with many layers enabled. My load-path contains 380 entries.
>
> I have profiled Emacs in Windows and found that it spends most of the startup
> time calling wopen(). This is because when calling (load "foo") it checks all
> directories in load-path for ("foo.el" "foo.elc" "foo.el.gz" "foo.elc.gz"
> "foo.dll"). It gets worse when load-prefer-newer is t.
>
> In my case `load-path` contains 380 entries, so every call to load will perform
> 380 * 5 = 1900 calls to wopen. This is very slow in Windows because its
> filesystem is not optimized for so many accesses to small files.
So this is not specific to Windows, it's just that Windows has slower
file access. IOW, if load-path becomes significantly larger, the slow
startup will show on Posix systems as well, right?
Next question: are the 'wopen' calls coming from 'openp'? if so,
perhaps we could first try a cheaper call, like 'chmod' (or its Win32
API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
try that?
> I thought that a caching mechanism would help.
My main concern with a cache is how to make sure it reflects what's on
the disk, when files are added or removed.
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 01 Jun 2020 16:47:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> So this is not specific to Windows, it's just that Windows has slower
> file access. IOW, if load-path becomes significantly larger, the slow
> startup will show on Posix systems as well, right?
Exactly, but not nearly as bad. I profiled the same Spacemacs configuration in
Ubuntu 20.04 and startup takes 10 seconds, with 5 seconds spent inside openp().
Compare this to 40 seconds in total and 32 seconds inside openp() in Windows.
> Next question: are the 'wopen' calls coming from 'openp'? if so,
> perhaps we could first try a cheaper call, like 'chmod' (or its Win32
> API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
> try that?
They come from openp(), yes. I haven't tried it. I think the issue is related to
the IO architecture in Windows, so I don't expect major speedups, but it could
certainly help.
> My main concern with a cache is how to make sure it reflects what's on
> the disk, when files are added or removed.
That is my main concern as well. A good option would be to rely on package.el to
generate the load-cache.el file for a package when it generates autoloads. If we
can't find any of the files mentioned in the cache then we fallback to the
current load-path code. That is a simple solution that would take care of
addition and removal of files, I think.
Nico.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 01 Jun 2020 17:18:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Nicolas Bértolo <nicolasbertolo <at> gmail.com>
> Date: Mon, 1 Jun 2020 13:46:01 -0300
> Cc: 41646 <at> debbugs.gnu.org
>
> > So this is not specific to Windows, it's just that Windows has slower
> > file access. IOW, if load-path becomes significantly larger, the slow
> > startup will show on Posix systems as well, right?
>
> Exactly, but not nearly as bad. I profiled the same Spacemacs configuration in
> Ubuntu 20.04 and startup takes 10 seconds, with 5 seconds spent inside openp().
>
> Compare this to 40 seconds in total and 32 seconds inside openp() in Windows.
You may be surprised how many users of GNU/Linux consider 5 sec during
startup an unacceptable slowdown.
> > Next question: are the 'wopen' calls coming from 'openp'? if so,
> > perhaps we could first try a cheaper call, like 'chmod' (or its Win32
> > API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
> > try that?
>
> They come from openp(), yes. I haven't tried it. I think the issue is related to
> the IO architecture in Windows, so I don't expect major speedups, but it could
> certainly help.
'chmod' doesn't access the file, it only accesses the directory. So
it could be much faster.
> > My main concern with a cache is how to make sure it reflects what's on
> > the disk, when files are added or removed.
>
> That is my main concern as well. A good option would be to rely on package.el to
> generate the load-cache.el file for a package when it generates autoloads. If we
> can't find any of the files mentioned in the cache then we fallback to the
> current load-path code. That is a simple solution that would take care of
> addition and removal of files, I think.
Is this supposed to work only when packages are installed by
package.el? What about manual installation?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 01 Jun 2020 19:52:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> You may be surprised how many users of GNU/Linux consider 5 sec during
> startup an unacceptable slowdown.
Me too. But I had given up on it until I read how it was implemented.
> 'chmod' doesn't access the file, it only accesses the directory. So
> it could be much faster.
As far as I can see there is no way to issue a call to 'chmod' that does not
try to change the permissions of the file passed. I will try to do it
with access().
> Is this supposed to work only when packages are installed by
> package.el? What about manual installation?
Everything should work OK as long as you provide a load-cache.el file in the
directory you add to load-path. We could provide a function to generate it
automatically.
Thanks, Nico
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Tue, 02 Jun 2020 02:19:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Nicolas Bértolo <nicolasbertolo <at> gmail.com>
> Date: Mon, 1 Jun 2020 16:51:11 -0300
> Cc: 41646 <at> debbugs.gnu.org
>
> > 'chmod' doesn't access the file, it only accesses the directory. So
> > it could be much faster.
>
> As far as I can see there is no way to issue a call to 'chmod' that does not
> try to change the permissions of the file passed. I will try to do it
> with access().
Sorry, I meant 'access', of course. More accurately,
GetFileAttributes.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Wed, 03 Jun 2020 01:09:01 GMT)
Full text and
rfc822 format available.
Message #23 received at 41646 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
> Sorry, I meant 'access', of course. More accurately,
> GetFileAttributes.
That was a very good idea. The attached patch reduces time spent
in openp() during startup by 5 seconds, or 15%.
Thanks, Nico.
[0001-In-Windows-check-if-file-exists-before-opening-it.patch (application/octet-stream, attachment)]
Added tag(s) patch.
Request was from
Stefan Kangas <stefan <at> marxist.se>
to
control <at> debbugs.gnu.org.
(Thu, 13 Aug 2020 01:13:03 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 14 Aug 2020 17:23:01 GMT)
Full text and
rfc822 format available.
Message #28 received at 41646 <at> debbugs.gnu.org (full text, mbox):
Nicolas Bértolo <nicolasbertolo <at> gmail.com> writes:
>> Sorry, I meant 'access', of course. More accurately,
>> GetFileAttributes.
>
> That was a very good idea. The attached patch reduces time spent
> in openp() during startup by 5 seconds, or 15%.
That's a huge speed-up indeed.
[...]
> - fd = emacs_open (pfn, O_RDONLY, 0);
> + /* In some systems (like Windows) finding out if a
> + file exists is cheaper to do than actually opening
> + it. Only open the file when we are sure that it
> + exists. */
> +#ifdef WINDOWSNT
> + if (faccessat (AT_FDCWD, pfn, R_OK, AT_EACCESS))
> + fd = -1;
> + else
> +#endif
> + fd = emacs_open (pfn, O_RDONLY, 0);
> +
This is a Windows-specific patch, so I have no opinion on it. Eli?
Does this make sense to you?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 14 Aug 2020 19:08:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 41646 <at> debbugs.gnu.org
> Date: Fri, 14 Aug 2020 19:22:26 +0200
>
> > That was a very good idea. The attached patch reduces time spent
> > in openp() during startup by 5 seconds, or 15%.
>
> That's a huge speed-up indeed.
>
> [...]
>
> > - fd = emacs_open (pfn, O_RDONLY, 0);
> > + /* In some systems (like Windows) finding out if a
> > + file exists is cheaper to do than actually opening
> > + it. Only open the file when we are sure that it
> > + exists. */
> > +#ifdef WINDOWSNT
> > + if (faccessat (AT_FDCWD, pfn, R_OK, AT_EACCESS))
> > + fd = -1;
> > + else
> > +#endif
> > + fd = emacs_open (pfn, O_RDONLY, 0);
> > +
>
> This is a Windows-specific patch, so I have no opinion on it. Eli?
> Does this make sense to you?
Not sure. 15% sounds too little for having OS-specific code in that
place. I hoped it will be much more.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Thu, 13 May 2021 09:18:02 GMT)
Full text and
rfc822 format available.
Message #34 received at 41646 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> This is a Windows-specific patch, so I have no opinion on it. Eli?
>> Does this make sense to you?
>
> Not sure. 15% sounds too little for having OS-specific code in that
> place. I hoped it will be much more.
So would your preference be to not apply the patch?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Thu, 13 May 2021 10:21:01 GMT)
Full text and
rfc822 format available.
Message #37 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: nicolasbertolo <at> gmail.com, 41646 <at> debbugs.gnu.org
> Date: Thu, 13 May 2021 11:17:07 +0200
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
> >> This is a Windows-specific patch, so I have no opinion on it. Eli?
> >> Does this make sense to you?
> >
> > Not sure. 15% sounds too little for having OS-specific code in that
> > place. I hoped it will be much more.
>
> So would your preference be to not apply the patch?
No, I guess it's okay after all, so let's install it.
TIA
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Thu, 13 May 2021 11:32:02 GMT)
Full text and
rfc822 format available.
Message #40 received at 41646 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
> No, I guess it's okay after all, so let's install it.
OK; I tested the patch on a Windows VM, and Emacs seemed to start
normally, as far as I could tell (but I didn't test it further than
that).
Pushed to Emacs 28 now.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Added tag(s) fixed.
Request was from
Lars Ingebrigtsen <larsi <at> gnus.org>
to
control <at> debbugs.gnu.org.
(Thu, 13 May 2021 11:32:02 GMT)
Full text and
rfc822 format available.
bug marked as fixed in version 28.1, send any further explanations to
41646 <at> debbugs.gnu.org and Nicolas Bértolo <nicolasbertolo <at> gmail.com>
Request was from
Lars Ingebrigtsen <larsi <at> gnus.org>
to
control <at> debbugs.gnu.org.
(Thu, 13 May 2021 11:32:03 GMT)
Full text and
rfc822 format available.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Fri, 11 Jun 2021 11:24:05 GMT)
Full text and
rfc822 format available.
bug unarchived.
Request was from
Stefan Kangas <stefankangas <at> gmail.com>
to
control <at> debbugs.gnu.org.
(Sun, 13 Oct 2024 09:31:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 09:53:02 GMT)
Full text and
rfc822 format available.
Message #51 received at 41646 <at> debbugs.gnu.org (full text, mbox):
[I have unarchived Bug#41646 so that this discussion is archived in the
bug tracker.]
Lin Sun <sunlin7.mail <at> gmail.com> writes:
> On Sun, Oct 13, 2024 at 5:45 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>> To summarize: my point is that I think we prefer extending existing
>> libraries instead of introducing new ones with partially-overlapping
>> functionalities. I wonder what Stefan and Andrea (CC'ed) think about
>> this.
>
> Totally understand, and thanks for inviting Stefan and Andrea in the
> conversation, to let new participants know the context quickly, I had
> post a patch to speed up Emacs for the scenario that Emacs will very
> slow on startup with hundreds packages (>300) installed, the keypoint
> is hundreds packages will add hundreds paths into the `load-path',
> then a simple `(require X)' may trigger hundreds searching according
> to the `load-path', so the patch build a map for feature --> filepath
> from the variable `load-history' and store to disk, after that loading
> the cache and get the filepath from the cache for requiring a feature
> will significantly speed up Emacs on startup or requiring heavy
> features even there were hundreds packages. I also give the examples
> based on Spacemacs, whose startup time can be reduced from 9.703 to
> 4.175 seconds (341packages) on a Windows system, with simply add two
> lines to enable the `loadhints' from patch: (require 'loadhints)
> (loadhints-init 'startup).
Thanks, it sounds like a useful feature.
I agree with Eli that it would be better if it was not implemented as a
separate library, but integrated into the existing functionality. For
example, I see that you use `define-advice`. We generally frown upon
using that in core, since we could adapt our code to make it work with
the new use case.
Here's a suggestion:
Since this is about speeding up load time during startup, how about
adding a new defvar that enables a cache for `require`, and then set
that to t during startup and nil after? That would make the feature
work as expected without any user customization. In general, this is
preferable, as this reduces the overall complexity of Emacs, both for
users and developers.
Why should this be MS-Windows specific, BTW? Is slow startup time with
many packages much less of an issue on other operating systems? Is disk
access somehow slower on MS-Windows?
> From 52f79eedb1944169b6c8ac4cfde101f59345d815 Mon Sep 17 00:00:00 2001
> From: Lin Sun <sunlin7 <at> hotmail.com>
> Date: Sat, 19 Aug 2023 06:55:59 +0000
> Subject: [PATCH] * lisp/loadhints.el: new file to cache hints for `require'
> function
>
> ---
> lisp/loadhints.el | 114 ++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 114 insertions(+)
> create mode 100644 lisp/loadhints.el
>
> diff --git a/lisp/loadhints.el b/lisp/loadhints.el
> new file mode 100644
> index 00000000000..9befd885f0b
> --- /dev/null
> +++ b/lisp/loadhints.el
> @@ -0,0 +1,114 @@
> +;;; loadhints.el --- Give hints for `require' -*- lexical-binding:t -*-
> +
> +;; Copyright (C) 2023-2024 Free Software Foundation, Inc.
> +
> +;; Author: Lin Sun <sunlin7 <at> hotmail.com>
> +;; Keywords: utility
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; loadhints will save the feature pathes to a cache file, and uses the
> +;; cache to speed up the `require' function, it will rapidly speed up the
> +;; `require' function when hundreds directories in `load-path' (especially
> +;; for windows). Just call `(loadhints-init 'startup)' in emacs user init
> +;; file.
> +
> +;;; Code:
> +
> +(defcustom loadhints-type (if (eq system-type 'windows-nt) 'startup)
> + "The loadhints-type behavior.
> +A nil value means to disable this feature.
> +`startup' means to work on startup.
> +`manual' means depending on user manually update the cache.
> +`aggressive' means update the cache at emacs quit."
> + :type '(choice (const :tag "Disable" nil)
> + (const :tag "Startup" startup)
> + (const :tag "Manual" manual)
> + (const :tag "Aggressive" aggressive)))
> +
> +(defcustom loadhints-cache-file
> + (expand-file-name "loadhints-cache" user-emacs-directory)
> + "File to save the recent list into."
> + :version "31.0"
> + :type 'file
> + :initialize 'custom-initialize-default
> + :set (lambda (symbol value)
> + (let ((oldvalue (symbol-value symbol)))
> + (custom-set-default symbol value)
> + (and loadhints-type
> + (not (equal value oldvalue))
> + (load oldvalue t)))))
> +
> +(defvar loadhints--cache nil)
> +(defvar loadhints--modified nil)
> +
> +;;;###autoload
> +(defun loadhints-refresh-maybe (&optional force async)
> + "(Re)generate the loadhints cache file.
> +When call with prefix, will FORCE refresh the loadhints cache."
> + (interactive "P")
> + (when (and force (null loadhints-type))
> + (user-error "Loadhints not avaliable for `loadhints-type' is nil"))
> + (when (and loadhints-type
> + (or force
> + loadhints--modified
> + (null (locate-file loadhints-cache-file '("/")
> + (get-load-suffixes)))))
> + (let ((res (make-hash-table :test 'equal))
> + (filepath (concat loadhints-cache-file ".el")))
> + (cl-loop for (path . rest) in load-history
> + do (when-let ((x (cl-find 'provide rest
> + :test (lambda (a b)
> + (and (consp b)
> + (eq a (car b)))))))
> + (puthash (cdr x) path res)))
> + (with-temp-file filepath
> + (insert (format "(setq loadhints--cache %S)" res)))
> + (if async
> + (async-byte-compile-file filepath)
> + (byte-compile-file filepath)))))
> +
> +;;;###autoload
> +(defun loadhints-init (&optional type)
> + "Setup the advice for `require' and load the cached hints."
> + (when type
> + (setopt loadhints-type type))
> +
> + (when loadhints-type
> + (define-advice require (:around (fn feature &optional filename noerror))
> + (when-let (((null filename))
> + ((null (featurep feature)))
> + (loadhints--cache)
> + (path (gethash feature loadhints--cache)))
> + (if (not (file-exists-p path))
> + (setq loadhints--modified t)
> + (setq filename path)))
> + (funcall fn feature filename noerror))
> +
> + (when-let ((filepath (locate-file loadhints-cache-file '("/")
> + (get-load-suffixes))))
> + (load filepath))
> +
> + (cond ((eq loadhints-type 'startup)
> + (add-hook 'after-init-hook #'(lambda ()
> + (loadhints-refresh-maybe nil t))))
> + ((eq loadhints-type 'aggressive)
> + (add-hook 'kill-emacs-hook #'loadhints-refresh-maybe)))))
> +
> +(provide 'loadhints)
> +;;; loadhints.el ends here
> --
> 2.34.1
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 10:47:02 GMT)
Full text and
rfc822 format available.
Message #54 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Stefan Kangas <stefankangas <at> gmail.com>
> Date: Sun, 13 Oct 2024 09:50:42 +0000
> Cc: Andrea Corallo <acorallo <at> gnu.org>, 41646 <at> debbugs.gnu.org,
> Stefan Monnier <monnier <at> gnu.org>
>
> Why should this be MS-Windows specific, BTW? Is slow startup time with
> many packages much less of an issue on other operating systems? Is disk
> access somehow slower on MS-Windows?
I don't think this is Windows-specific, it's just that on Windows the
gains might be higher. Disk access is somewhat slower on Windows,
yes.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 14:50:02 GMT)
Full text and
rfc822 format available.
Message #57 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Stefan Kangas <stefankangas <at> gmail.com>
> > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > Cc: Andrea Corallo <acorallo <at> gnu.org>, 41646 <at> debbugs.gnu.org,
> > Stefan Monnier <monnier <at> gnu.org>
> >
> > Why should this be MS-Windows specific, BTW? Is slow startup time with
> > many packages much less of an issue on other operating systems? Is disk
> > access somehow slower on MS-Windows?
>
> I don't think this is Windows-specific, it's just that on Windows the
> gains might be higher. Disk access is somewhat slower on Windows,
> yes.
Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
(383 packages).
On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas <at> gmail.com> wrote:
>
> Here's a suggestion:
>
> Since this is about speeding up load time during startup, how about
> adding a new defvar that enables a cache for `require`, and then set
> that to t during startup and nil after? That would make the feature
> work as expected without any user customization. In general, this is
> preferable, as this reduces the overall complexity of Emacs, both for
> users and developers.
The "loadhints-cache" is not only for startup, actually it also works
anytime when requiring a feature but the `exec-path' has hundreds of
entries.
Agree, using the "loadhints-cache" in the `require' function native
code will be better than defeine-advice, I'm implementing it now, and
will post a patch later.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 15:25:02 GMT)
Full text and
rfc822 format available.
Message #60 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Sun, 13 Oct 2024 14:47:46 +0000
> Cc: Stefan Kangas <stefankangas <at> gmail.com>, acorallo <at> gnu.org, 41646 <at> debbugs.gnu.org,
> monnier <at> gnu.org
>
> On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > From: Stefan Kangas <stefankangas <at> gmail.com>
> > > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > > Cc: Andrea Corallo <acorallo <at> gnu.org>, 41646 <at> debbugs.gnu.org,
> > > Stefan Monnier <monnier <at> gnu.org>
> > >
> > > Why should this be MS-Windows specific, BTW? Is slow startup time with
> > > many packages much less of an issue on other operating systems? Is disk
> > > access somehow slower on MS-Windows?
> >
> > I don't think this is Windows-specific, it's just that on Windows the
> > gains might be higher. Disk access is somewhat slower on Windows,
> > yes.
>
> Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
> (383 packages).
>
>
> On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas <at> gmail.com> wrote:
> >
> > Here's a suggestion:
> >
> > Since this is about speeding up load time during startup, how about
> > adding a new defvar that enables a cache for `require`, and then set
> > that to t during startup and nil after? That would make the feature
> > work as expected without any user customization. In general, this is
> > preferable, as this reduces the overall complexity of Emacs, both for
> > users and developers.
>
> The "loadhints-cache" is not only for startup, actually it also works
> anytime when requiring a feature but the `exec-path' has hundreds of
> entries.
Is it possible to extend filecache.el to cover this use case as well?
For example, by adding to filecache.el the ability to save the cache,
and perhaps also the ability to collect the cached files by hooking
into 'load'?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 15:46:02 GMT)
Full text and
rfc822 format available.
Message #63 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> Is it possible to extend filecache.el to cover this use case as well?
> For example, by adding to filecache.el the ability to save the cache,
> and perhaps also the ability to collect the cached files by hooking
> into 'load'?
It's doable but I'm not sure how to maintain the feature-->filepath
maps to cover both file system changes and user preferred exec-path
order, or if it could break the user experience?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 15:52:02 GMT)
Full text and
rfc822 format available.
Message #66 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> [I have unarchived Bug#41646 so that this discussion is archived in the
> bug tracker.]
[ Note: I haven't seen the earlier part of this discussion (but I did read the
old bug#41646 thread). ]
>> Totally understand, and thanks for inviting Stefan and Andrea in the
>> conversation, to let new participants know the context quickly, I had
>> post a patch to speed up Emacs for the scenario that Emacs will very
>> slow on startup with hundreds packages (>300) installed, the keypoint
>> is hundreds packages will add hundreds paths into the `load-path',
>> then a simple `(require X)' may trigger hundreds searching according
>> to the `load-path',
Yup, that's a known problem which we've been not solving for
a long time.
The "latest" workaround is `package-quickstart`, which reduces some of
the associated pain.
If you're not using `package-quickstart` and are bothered by a slow
startup, then I'd recommend you go back and enable `package-quickstart`.
Similarly in the old bug#41646 thread, I see mentions of 40s startup
time, 8s of which are *not* spent in openp: 8s is still quite slow, so
it might be worth looking at what the startup file is doing and see if
we could do less at startup (e.g. load things more lazily).
Of course, the "long load-path" problem will still end up biting in
some cases. Until now we've managed to make those cases rare enough
that we haven't had to actually solve it.
>> so the patch build a map for feature --> filepath from the variable
>> `load-history' and store to disk, after that loading
I don't think we want a cache that's stored on disk: it would take too
much effort to create it, load it, ensure it's not damaged if several
Emacs sessions try to write it at the same time, make sure it's fresh,
etc...
Especially since I believe that building the cache shouldn't take very
long: longer than a single "look for file F in `load-path`", maybe, but
not by much.
IOW we could keep a cache that's populated on-the-fly the first time we
`load` something, and that's then automatically refreshed when `load`
sees a new `load-path`.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 15:57:01 GMT)
Full text and
rfc822 format available.
Message #69 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Sun, 13 Oct 2024 15:43:39 +0000
> Cc: stefankangas <at> gmail.com, acorallo <at> gnu.org, 41646 <at> debbugs.gnu.org,
> monnier <at> gnu.org
>
> On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > Is it possible to extend filecache.el to cover this use case as well?
> > For example, by adding to filecache.el the ability to save the cache,
> > and perhaps also the ability to collect the cached files by hooking
> > into 'load'?
> It's doable but I'm not sure how to maintain the feature-->filepath
> maps to cover both file system changes and user preferred exec-path
> order, or if it could break the user experience?
Sorry, I don't understand: what does exec-path have to do with this?
I thought this was about speeding up loading of Lisp files at startup?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 16:05:01 GMT)
Full text and
rfc822 format available.
Message #72 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Lin Sun <sunlin7.mail <at> gmail.com>
> >
> > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > >
> > > Is it possible to extend filecache.el to cover this use case as well?
> > > For example, by adding to filecache.el the ability to save the cache,
> > > and perhaps also the ability to collect the cached files by hooking
> > > into 'load'?
> > It's doable but I'm not sure how to maintain the feature-->filepath
> > maps to cover both file system changes and user preferred exec-path
> > order, or if it could break the user experience?
>
> Sorry, I don't understand: what does exec-path have to do with this?
> I thought this was about speeding up loading of Lisp files at startup?
Sorry I mean the `load-path' (not the exec-path), my typo. Thanks
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sun, 13 Oct 2024 16:40:01 GMT)
Full text and
rfc822 format available.
Message #75 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Sun, 13 Oct 2024 16:03:16 +0000
> Cc: stefankangas <at> gmail.com, acorallo <at> gnu.org, 41646 <at> debbugs.gnu.org,
> monnier <at> gnu.org
>
> On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > From: Lin Sun <sunlin7.mail <at> gmail.com>
> > >
> > > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > > >
> > > > Is it possible to extend filecache.el to cover this use case as well?
> > > > For example, by adding to filecache.el the ability to save the cache,
> > > > and perhaps also the ability to collect the cached files by hooking
> > > > into 'load'?
> > > It's doable but I'm not sure how to maintain the feature-->filepath
> > > maps to cover both file system changes and user preferred exec-path
> > > order, or if it could break the user experience?
> >
> > Sorry, I don't understand: what does exec-path have to do with this?
> > I thought this was about speeding up loading of Lisp files at startup?
> Sorry I mean the `load-path' (not the exec-path), my typo. Thanks
OK, but please elaborate on the difficulty, because I don't think I
understand it.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Wed, 16 Oct 2024 07:54:01 GMT)
Full text and
rfc822 format available.
Message #78 received at 41646 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Eli,
I wrote the code and tested it, the patch file attached.
In the patch, a new "load-hints" is introduced with a default value nil.
When its value is nil, everything works as before.
When its value is set as a list of hints, the openp function calling
count is reduced a lot.
I had an example for require "org", the "load-hints" can reduce the
access attempts from 123 to 6, here is the test command lines, strace
the emacs in batch mode, a single (require 'org) trigger more than
100+ open attempts; with correct load-hints, the count reduce to 6.
> strace src/emacs -batch -eval "(require 'org)" 2>&1| grep 'open.*/org\.' | wc -l
> 123
> strace src/emacs -batch -eval "(let ((load-hints '((\"org*\" \"~/tmp/emacs.debug/lisp/org/\"))))(require 'org))" 2>&1| grep 'open.*/org\.' | wc -l
> 6
And you had mentioned scan the load-path to build a full hints list,
it absolutely should work; I'm looking to change package.el to
generate the "<package>-autoloads.el" work with "load-hints",
currently the autoloads.el will add its folder into "load-path",
just change it to add its path into "load-hints" should work, then we
do NOT need to provide a function to build the "load-hints" for the
packages installed by package.el won't bother the "load-path".
[0001-New-variable-load-hints-to-speedup-searching-file-fo.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 21 Oct 2024 04:12:02 GMT)
Full text and
rfc822 format available.
Message #81 received at 41646 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Eli,
I had patched the package.el also, and now the `load-hints` can boost
the emacs, especially for the emacs on windows.
The `load-hints` can reduce searching attempts by putting the matched
paths on the top of `load-path`, it won't break the original
load-path; and the patch for the `package.el` will put the installed
files into the `load-hints`.
I tested on both Ubuntu 20.04 and Windows 11, I tested the patches
with `load-hints` enabled/disabled, based on the Spacemacs
distribution, 276 packages installed, test cli is: emacs -nw --eval
"(require 'helm)",
1. On my local linux test env, disable load-hints on the package.el,
the test cli spends 6.327s; and enable load-hints then it spends
5.392s.
2. On my local Windows test env, disable load-hints on the package.el,
the test cli spends 11.769s, and enable load-hints then it spends
7.279s.
It's very effective for windows, without any break changes.
Please review the patches. Thanks.
Best Regards, Lin
[0002-lisp-emacs-lisp-package.el-Support-the-load-hints.patch (text/x-patch, attachment)]
[0001-New-variable-load-hints-to-speedup-searching-file-fo.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 21 Oct 2024 14:36:02 GMT)
Full text and
rfc822 format available.
Message #84 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> The `load-hints` can reduce searching attempts by putting the matched
> paths on the top of `load-path`, it won't break the original
> load-path; and the patch for the `package.el` will put the installed
> files into the `load-hints`.
The downside is that it can break existing setups for users who use
`package.el` but also modify their `load-path` "by hand" in the
init file, and it doesn't help users who don't use `package.el`.
Note also that your `load-hints` could grow large, so scanning it could
take a significant amount of time. Maybe it would make sense to turn it
into a hash table for those entries that don't use the "*" special
thingy (and maybe use a radix-tree for those entries using the "*"
special thingy)?
But your prefix idea makes me think maybe we can aim for a significantly
smaller table, where we basically record only one entry per
package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
record "helm" because all the `.el` files share the "helm" prefix.
I.e. keep for each dir the corresponding longest-common-prefix.
If we're careful to consider only those files with a `.el` suffix, then
I think we can reduce the hint to such a longest-common-prefix. I.e. an
info which doesn't say just "you can find FOO* files here" but "you can
find *only* FOO* files here".
Then we should be able to create quickly (so it can be recomputed on the
fly whenever `load-path` changes) a radix-tree that maps a relative
file name to the list of directories from `load-path` where it is
worthwhile to look (by filtering out those dirs whose
longest-common-prefix doesn't match). We'd only do it for MUST_SUFFIX
is specified, of course.
> 1. On my local linux test env, disable load-hints on the package.el,
> the test cli spends 6.327s; and enable load-hints then it spends
> 5.392s.
>
> 2. On my local Windows test env, disable load-hints on the package.el,
> the test cli spends 11.769s, and enable load-hints then it spends
> 7.279s.
Is that with or without using `package-quickstart`?
BTW, in your patch, you change `locate-file-internal` which seems wrong,
since that function is not specific to loading ELisp files, it's also
used for $MANPATH, $PATH, and things like that.
Similarly, I wasn't able to convince myself that your patch does the
right thing when `require` or `load` is used such that MUST_SUFFIX is
not specified.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 21 Oct 2024 17:13:02 GMT)
Full text and
rfc822 format available.
Message #87 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Mon, Oct 21, 2024 at 2:34 PM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
>
> > The `load-hints` can reduce searching attempts by putting the matched
> > paths on the top of `load-path`, it won't break the original
> > load-path; and the patch for the `package.el` will put the installed
> > files into the `load-hints`.
>
> The downside is that it can break existing setups for users who use
> `package.el` but also modify their `load-path` "by hand" in the
> init file, and it doesn't help users who don't use `package.el`.
The `load-hints' do nothing with its default value nil.
The package manager can use the `load-hints' or ignore it. The
"package.el" is emacs builtin package manager, can support or ignore
the `load-hints' by setting `package-enable-load-hints` to t or nil.
Other package managers can continue without any change, or do some
work to get performance benefits by supporting the `load-hints'.
> Note also that your `load-hints` could grow large, so scanning it could
> take a significant amount of time. Maybe it would make sense to turn it
> into a hash table for those entries that don't use the "*" special
> thingy (and maybe use a radix-tree for those entries using the "*"
> special thingy)?
The `load-hints` could grow as the `load-path', or larger than the
`load-path', but it should not be too much.
For the hash table, it can not support the '*', or the hash-table
requires every file to have an entry explicitly, which costs a lot on
building the table.
I had checked the radix-tree at the beginning, it's not user-friendly,
or it's not easy to dump the radix tree for an end user to understand
which is obviously matching the entry or not.
The `load-hints' in the list are easy to understand / maintain by the end user.
> But your prefix idea makes me think maybe we can aim for a significantly
> smaller table, where we basically record only one entry per
> package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
> record "helm" because all the `.el` files share the "helm" prefix.
> I.e. keep for each dir the corresponding longest-common-prefix.
> If we're careful to consider only those files with a `.el` suffix, then
> I think we can reduce the hint to such a longest-common-prefix. I.e. an
> info which doesn't say just "you can find FOO* files here" but "you can
> find *only* FOO* files here".
I had searched all 200+ packages in my test env, most of the packages
use their feature name as the prefix, only 11 packages have
exceptions. But I didn't understand how it works toward the
`load-hints'.
> Then we should be able to create quickly (so it can be recomputed on the
> fly whenever `load-path` changes) a radix-tree that maps a relative
> file name to the list of directories from `load-path` where it is
> worthwhile to look (by filtering out those dirs whose
> longest-common-prefix doesn't match). We'd only do it for MUST_SUFFIX
> is specified, of course.
>
> > 1. On my local linux test env, disable load-hints on the package.el,
> > the test cli spends 6.327s; and enable load-hints then it spends
> > 5.392s.
> >
> > 2. On my local Windows test env, disable load-hints on the package.el,
> > the test cli spends 11.769s, and enable load-hints then it spends
> > 7.279s.
>
> Is that with or without using `package-quickstart`?
The `package-quickstart' does not help in this scenario, the key point
is the `load-path` count will times the read attempt.
A simple "(require 'X)" will lead the emacs walks through the
`load-path' to attempt opening the "X.so, X.so.gz, X.elc, X.elc.gz,
X.el, X.el.gz" one by one, if the `load-path' has 200 entries, emacs
will try search 200x6=1200 times for the worst case.
The `load-hints` will help put the matched path to the top of
`load-path` then emacs can find the X on the top entries of
`load-path' then returns shortly.
> BTW, in your patch, you change `locate-file-internal` which seems wrong,
> since that function is not specific to loading ELisp files, it's also
> used for $MANPATH, $PATH, and things like that.
>
> Similarly, I wasn't able to convince myself that your patch does the
> right thing when `require` or `load` is used such that MUST_SUFFIX is
> not specified.
I'm going to search the cases carefully. Thank you for all the
comments, appreciate it.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Mon, 21 Oct 2024 19:56:02 GMT)
Full text and
rfc822 format available.
Message #90 received at 41646 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Mon, Oct 21, 2024 at 2:34 PM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
> BTW, in your patch, you change `locate-file-internal` which seems wrong,
> since that function is not specific to loading ELisp files, it's also
> used for $MANPATH, $PATH, and things like that.
You're right, the change in `locate-file-internal' didn't merge the
original path.
I attached the patch to append the `dirs' with `path' together.
> Similarly, I wasn't able to convince myself that your patch does the
> right thing when `require` or `load` is used such that MUST_SUFFIX is
> not specified.
The `load-hints` just put the matched paths on the top of `load-path`,
still following the `load-path' mechanism, and won't affect any other
features (Or someone already has some code to adjust the `load-path'
orders, can just ignore the `load-hints', everything work like before,
no break changes).
[0001-New-variable-load-hints-to-speedup-searching-file-fo.patch (text/x-patch, attachment)]
[0002-lisp-emacs-lisp-package.el-Support-the-load-hints.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Thu, 31 Oct 2024 15:05:02 GMT)
Full text and
rfc822 format available.
Message #93 received at 41646 <at> debbugs.gnu.org (full text, mbox):
>> The downside is that it can break existing setups for users who use
>> `package.el` but also modify their `load-path` "by hand" in the
>> init file, and it doesn't help users who don't use `package.el`.
> The `load-hints' do nothing with its default value nil.
No, but if `package.el` adds elements to `load-hints` and then in your
init file you add more elements to `load-path`, the `load-hints` may end
up taking precedence over the new elements of `load-path` in some cases
and thus change which file shadows which.
> I had checked the radix-tree at the beginning, it's not user-friendly,
> or it's not easy to dump the radix tree for an end user to understand
> which is obviously matching the entry or not.
You're not "supposed" to look at the representation of the radix tree,
indeed (unless you're working on `radix-tree.el`, of course).
Feel free to ask for help using it.
> The `load-hints' in the list are easy to understand / maintain by the
> end user.
I don't think we need to burden end users with load hints.
>> But your prefix idea makes me think maybe we can aim for a significantly
>> smaller table, where we basically record only one entry per
>> package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
>> record "helm" because all the `.el` files share the "helm" prefix.
>> I.e. keep for each dir the corresponding longest-common-prefix.
>> If we're careful to consider only those files with a `.el` suffix, then
>> I think we can reduce the hint to such a longest-common-prefix. I.e. an
>> info which doesn't say just "you can find FOO* files here" but "you can
>> find *only* FOO* files here".
>
> I had searched all 200+ packages in my test env, most of the packages
> use their feature name as the prefix, only 11 packages have
> exceptions. But I didn't understand how it works toward the
> `load-hints'.
Here's the idea: the `<PKG>-autoloads.el` file can registers the longest
common prefix of all the `.el` files for its own `load-path` entry, with
say:
(load-prefix-register <DIR> <PREFIX>)
where we'd define this function along the lines of
(defconst load-prefix-directories (make-hash-table :test 'equal))
"Set of entries from `load-path` for which we have prefix info.")
(defconst load-prefix-map radix-tree-empty
"Table associating file prefixes to directories.")
(defun load-prefix-register (dir prefix)
(puthash dir t load-prefix-directories)
(let ((dirs (radix-tree-lookup load-prefix-map prefix)))
(unless (member dir dirs)
(setq load-prefix-map (radix-tree-insert load-prefix-map prefix
(cons dir dirs))))))
(defun load-prefix-trim-load-path (file)
"Return a trimmed `load-path` to use for FILE."
(if (file-name-directory file)
;; If there's a `/` in FILE, fallback on the safe default.
load-path
(let* ((prefixes (radix-tree-prefixes load-prefix-map file))
(dirs (apply #'append (mapcar #'cdr prefixes))))
;; Remove from `load-path` the entries which can't possibly
;; have FILE because their prefixes doesn't match.
(cl-remove-if (lambda (dir)
(and (gethash dir load-prefix-directories)
(not (member dir dirs))))
load-path))))
and then `load` can use `load-prefix-trim-load-path` to iterate on
a much shorter `load-path`.
I'm not completely sure if it's a good idea, tho: I'd really prefer
a solution that doesn't require any change to any package management
code, which instead uses a cache (updated/filled automatically) of all
the files found in all the `load-path` directories.
>> Is that with or without using `package-quickstart`?
> The `package-quickstart' does not help in this scenario,
AFAICT your scenario includes Emacs startup with packages installed, so
`package-quickstart' can definitely make a difference. But maybe you're
right that it will affect both cases equally.
>> Similarly, I wasn't able to convince myself that your patch does the
>> right thing when `require` or `load` is used such that MUST_SUFFIX is
>> not specified.
> The `load-hints` just put the matched paths on the top of `load-path`,
> still following the `load-path' mechanism, and won't affect any other
> features (Or someone already has some code to adjust the `load-path'
> orders, can just ignore the `load-hints', everything work like before,
> no break changes).
But if FOO is in one dir and FOO.el is in another dir, adding entries
to the load path can change which file we end up loading.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 07:20:02 GMT)
Full text and
rfc822 format available.
Message #96 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Thu, Oct 31, 2024 at 3:04 PM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
>
> >> The downside is that it can break existing setups for users who use
> >> `package.el` but also modify their `load-path` "by hand" in the
> >> init file, and it doesn't help users who don't use `package.el`.
> > The `load-hints' do nothing with its default value nil.
>
> No, but if `package.el` adds elements to `load-hints` and then in your
> init file you add more elements to `load-path`, the `load-hints` may end
> up taking precedence over the new elements of `load-path` in some cases
> and thus change which file shadows which.
Yes, introducing this new variable will increase the complexity to the end user.
> > I had checked the radix-tree at the beginning, it's not user-friendly,
> > or it's not easy to dump the radix tree for an end user to understand
> > which is obviously matching the entry or not.
>
> You're not "supposed" to look at the representation of the radix tree,
> indeed (unless you're working on `radix-tree.el`, of course).
> Feel free to ask for help using it.
>
> > The `load-hints' in the list are easy to understand / maintain by the
> > end user.
>
> I don't think we need to burden end users with load hints.
>
> >> But your prefix idea makes me think maybe we can aim for a significantly
> >> smaller table, where we basically record only one entry per
> >> package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
> >> record "helm" because all the `.el` files share the "helm" prefix.
> >> I.e. keep for each dir the corresponding longest-common-prefix.
> >> If we're careful to consider only those files with a `.el` suffix, then
> >> I think we can reduce the hint to such a longest-common-prefix. I.e. an
> >> info which doesn't say just "you can find FOO* files here" but "you can
> >> find *only* FOO* files here".
> >
> > I had searched all 200+ packages in my test env, most of the packages
> > use their feature name as the prefix, only 11 packages have
> > exceptions. But I didn't understand how it works toward the
> > `load-hints'.
>
> Here's the idea: the `<PKG>-autoloads.el` file can registers the longest
> common prefix of all the `.el` files for its own `load-path` entry, with
> say:
>
> (load-prefix-register <DIR> <PREFIX>)
>
> where we'd define this function along the lines of
>
> (defconst load-prefix-directories (make-hash-table :test 'equal))
> "Set of entries from `load-path` for which we have prefix info.")
>
> (defconst load-prefix-map radix-tree-empty
> "Table associating file prefixes to directories.")
>
> (defun load-prefix-register (dir prefix)
> (puthash dir t load-prefix-directories)
> (let ((dirs (radix-tree-lookup load-prefix-map prefix)))
> (unless (member dir dirs)
> (setq load-prefix-map (radix-tree-insert load-prefix-map prefix
> (cons dir dirs))))))
>
> (defun load-prefix-trim-load-path (file)
> "Return a trimmed `load-path` to use for FILE."
> (if (file-name-directory file)
> ;; If there's a `/` in FILE, fallback on the safe default.
> load-path
> (let* ((prefixes (radix-tree-prefixes load-prefix-map file))
> (dirs (apply #'append (mapcar #'cdr prefixes))))
> ;; Remove from `load-path` the entries which can't possibly
> ;; have FILE because their prefixes doesn't match.
> (cl-remove-if (lambda (dir)
> (and (gethash dir load-prefix-directories)
> (not (member dir dirs))))
> load-path))))
>
> and then `load` can use `load-prefix-trim-load-path` to iterate on
> a much shorter `load-path`.
>
> I'm not completely sure if it's a good idea, tho: I'd really prefer
> a solution that doesn't require any change to any package management
> code, which instead uses a cache (updated/filled automatically) of all
> the files found in all the `load-path` directories.
If that, we have to track the file/path changes in each entry of
load-path, it may not be possible for all the supported OSs.
> >> Is that with or without using `package-quickstart`?
> > The `package-quickstart' does not help in this scenario,
>
> AFAICT your scenario includes Emacs startup with packages installed, so
> `package-quickstart' can definitely make a difference. But maybe you're
> right that it will affect both cases equally.
I had tried the package-quickstart, it does NOT help on windows, like
after load the quickstart.el, it will add all the packages directories
into the load-path, the load-path may have ~300 entries, then a simple
"(require 'org)" will trigger the emacs walk on the load-path one by
one to try open (org.so, org.so.gz, org.elc, org.elc.gz, org.el,
org.el.gz) for 6 times, while the org actually is on the bottom of
load-path, so the emacs tried near 1800 (300x6) times to load one org
file (its depends need similar counts), on windows systems, it's very
slow.
> >> Similarly, I wasn't able to convince myself that your patch does the
> >> right thing when `require` or `load` is used such that MUST_SUFFIX is
> >> not specified.
> > The `load-hints` just put the matched paths on the top of `load-path`,
> > still following the `load-path' mechanism, and won't affect any other
> > features (Or someone already has some code to adjust the `load-path'
> > orders, can just ignore the `load-hints', everything work like before,
> > no break changes).
>
> But if FOO is in one dir and FOO.el is in another dir, adding entries
> to the load path can change which file we end up loading.
Yeh, that may happen, and the end user will get confused.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 07:51:01 GMT)
Full text and
rfc822 format available.
Message #99 received at 41646 <at> debbugs.gnu.org (full text, mbox):
Hi Eli, Stefan,
To speedup the Emacs (especially for emacs on windows), the key point
is reducing the opening file attempts.
I have two options,
1. Add a new variable like "load-hints", it holds the file list for
directories, may be organized in radix-tree or simple list.
But it may break the exists behavior, confusing the end user on
load-hints/load-path. Or
2. extend the load-path to be a directory with its file list. That
means one "load-path" entry can be a string to represent a directory
path, or a directory with its files. Here is an example as below, the
first entry is a list, path1 and its loadable files (without
extensions); the second entry is a string for path, a traditional
entry of "load-path".
'( ("<path1>" "file1" "file2)
"<path2>")
It works on "load-path", compatible with traditional "load-path".
So, the first option will introduce a new variable, which may affect
existing "load-path"; the second option will extend the "load-path",
no new variable, but maybe not compatible with some existing code.
Please comment on the option, and if we agree on one of the options,
I'll work on it. Thanks
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 08:18:01 GMT)
Full text and
rfc822 format available.
Message #102 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Fri, 1 Nov 2024 07:49:00 +0000
> Cc: Eli Zaretskii <eliz <at> gnu.org>, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> 41646 <at> debbugs.gnu.org, monnier <at> gnu.org
>
> 2. extend the load-path to be a directory with its file list. That
> means one "load-path" entry can be a string to represent a directory
> path, or a directory with its files. Here is an example as below, the
> first entry is a list, path1 and its loadable files (without
> extensions); the second entry is a string for path, a traditional
> entry of "load-path".
>
> '( ("<path1>" "file1" "file2)
> "<path2>")
How will the '"file1" "file2" ...' part be created?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 13:12:01 GMT)
Full text and
rfc822 format available.
Message #105 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> Yes, introducing this new variable will increase the complexity to the
> end user.
Exactly, and I think we should first try to solve the problem without
exposing the user to such complexity.
>> Here's the idea: the `<PKG>-autoloads.el` file can registers the longest
>> common prefix of all the `.el` files for its own `load-path` entry, with
>> say:
>>
>> (load-prefix-register <DIR> <PREFIX>)
>>
>> where we'd define this function along the lines of
>>
>> (defconst load-prefix-directories (make-hash-table :test 'equal))
>> "Set of entries from `load-path` for which we have prefix info.")
>>
>> (defconst load-prefix-map radix-tree-empty
>> "Table associating file prefixes to directories.")
>>
>> (defun load-prefix-register (dir prefix)
>> (puthash dir t load-prefix-directories)
>> (let ((dirs (radix-tree-lookup load-prefix-map prefix)))
>> (unless (member dir dirs)
>> (setq load-prefix-map (radix-tree-insert load-prefix-map prefix
>> (cons dir dirs))))))
>>
>> (defun load-prefix-trim-load-path (file)
>> "Return a trimmed `load-path` to use for FILE."
>> (if (file-name-directory file)
>> ;; If there's a `/` in FILE, fallback on the safe default.
>> load-path
>> (let* ((prefixes (radix-tree-prefixes load-prefix-map file))
>> (dirs (apply #'append (mapcar #'cdr prefixes))))
>> ;; Remove from `load-path` the entries which can't possibly
>> ;; have FILE because their prefixes doesn't match.
>> (cl-remove-if (lambda (dir)
>> (and (gethash dir load-prefix-directories)
>> (not (member dir dirs))))
>> load-path))))
>>
>> and then `load` can use `load-prefix-trim-load-path` to iterate on
>> a much shorter `load-path`.
Note that this above proposal should be transparent to the end user (tho
it requires extra work on the `package.el` side): e.g. funny changes
to `load-path` would be handled without fuss.
>> I'm not completely sure if it's a good idea, tho: I'd really prefer
>> a solution that doesn't require any change to any package management
>> code, which instead uses a cache (updated/filled automatically) of all
>> the files found in all the `load-path` directories.
> If that, we have to track the file/path changes in each entry of
> load-path, it may not be possible for all the supported OSs.
We can easily detect changes to `load-path` itself, of course, but as
for changes to the content of the directories in `load-path` that would
be more difficult&costly, admittedly. My plan was to do nothing
about it (i.e. allow the cache to go stale): if we use the cache only to
tell `load` in which directory to look for the file, it should usually
be safe because IME it's rare for files to be added/removed from
directories such that it changes from which directory a given ELisp file
is loaded. But of course we could also make efforts to try and keep our
cache consistent, e.g. via OS-level notification infrastructure or
by flushing the cache after a N seconds.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 16:59:02 GMT)
Full text and
rfc822 format available.
Message #108 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Fri, Nov 1, 2024 at 8:17 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> ...
> How will the '"file1" "file2" ...' part be created?
For the Emacs built in paths, we can create the (<path> [files])
during bootstrap and write to the "subdirs.el", then it will push the
extended (<path>, files...) into `load-path'.
On Fri, Nov 1, 2024 at 1:11 PM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
>
> > Yes, introducing this new variable will increase the complexity to the
> > end user.
>
> Exactly, and I think we should first try to solve the problem without
> exposing the user to such complexity.
> ...
> >> and then `load` can use `load-prefix-trim-load-path` to iterate on
> >> a much shorter `load-path`.
>
> Note that this above proposal should be transparent to the end user (tho
> it requires extra work on the `package.el` side): e.g. funny changes
> to `load-path` would be handled without fuss.
>
> >> I'm not completely sure if it's a good idea, tho: I'd really prefer
> >> a solution that doesn't require any change to any package management
> >> code, which instead uses a cache (updated/filled automatically) of all
> >> the files found in all the `load-path` directories.
> > If that, we have to track the file/path changes in each entry of
> > load-path, it may not be possible for all the supported OSs.
>
> We can easily detect changes to `load-path` itself, of course, but as
> for changes to the content of the directories in `load-path` that would
> be more difficult&costly, admittedly. My plan was to do nothing
> about it (i.e. allow the cache to go stale): if we use the cache only to
> tell `load` in which directory to look for the file, it should usually
> be safe because IME it's rare for files to be added/removed from
> directories such that it changes from which directory a given ELisp file
> is loaded. But of course we could also make efforts to try and keep our
> cache consistent, e.g. via OS-level notification infrastructure or
> by flushing the cache after a N seconds.
Agree the changes are difficult and costly.
The inotify way is hard to work on all OSes, and the cache need very
carefully maintenance policy.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 17:09:01 GMT)
Full text and
rfc822 format available.
Message #111 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Fri, 1 Nov 2024 16:56:59 +0000
> Cc: Eli Zaretskii <eliz <at> gnu.org>, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> 41646 <at> debbugs.gnu.org, monnier <at> gnu.org
>
> On Fri, Nov 1, 2024 at 8:17 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > ...
> > How will the '"file1" "file2" ...' part be created?
>
> For the Emacs built in paths, we can create the (<path> [files])
> during bootstrap and write to the "subdirs.el", then it will push the
> extended (<path>, files...) into `load-path'.
I don't understand: isn't this supposed to speed up primarily users
who have many 3rd-party packages installed? For them, what happens
during bootstrap is not relevant.
If all we want is to record the places where bundled files live,
that's a much easier problem.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 18:00:02 GMT)
Full text and
rfc822 format available.
Message #114 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Fri, Nov 1, 2024 at 5:08 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Lin Sun <sunlin7.mail <at> gmail.com>
> > Date: Fri, 1 Nov 2024 16:56:59 +0000
> > Cc: Eli Zaretskii <eliz <at> gnu.org>, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> > 41646 <at> debbugs.gnu.org, monnier <at> gnu.org
> >
> > On Fri, Nov 1, 2024 at 8:17 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > > ...
> > > How will the '"file1" "file2" ...' part be created?
> >
> > For the Emacs built in paths, we can create the (<path> [files])
> > during bootstrap and write to the "subdirs.el", then it will push the
> > extended (<path>, files...) into `load-path'.
>
> I don't understand: isn't this supposed to speed up primarily users
> who have many 3rd-party packages installed? For them, what happens
> during bootstrap is not relevant.
>
> If all we want is to record the places where bundled files live,
> that's a much easier problem.
Sorry for the fuzz "bootstrap", the "bootstrap" I wanted to say is
part of building steps, like "make bootstrap", then we can build the
files list into the "subdir.el".
Both the startup time and running time will be affected by too many
"load-path" entries.
Like the "package.el", it will add all 300+ packages' paths on the top
of "load-path" at the beginning of startup, the builtin paths will be
on the bottom of "load-path", after that a simple "(require
'<builtin-feature>)" will trigger emacs to walk through the
"load-path" from top to bottom, that leads thouthands
open-attemptions, that happened during emacs startup, or during
running time.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 18:17:01 GMT)
Full text and
rfc822 format available.
Message #117 received at 41646 <at> debbugs.gnu.org (full text, mbox):
How about this: I write some windows specific code only available on
the windows to cache the files for the load-path. It will NOT affect
other platforms. Like a flag "load-path-cache-expired 60" (nil/t) for
windows users who won't change path/files a lot, then Emacs can cache
the files list for path entries for 60 seconds. As I pasted, with the
files cache, it can reduce the startup time from ~16s to ~5s, and also
improve the running time performance. That will save a lot of emacs
windows users.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 19:19:02 GMT)
Full text and
rfc822 format available.
Message #120 received at 41646 <at> debbugs.gnu.org (full text, mbox):
>> We can easily detect changes to `load-path` itself, of course, but as
>> for changes to the content of the directories in `load-path` that would
>> be more difficult&costly, admittedly. My plan was to do nothing
>> about it (i.e. allow the cache to go stale): if we use the cache only to
>> tell `load` in which directory to look for the file, it should usually
>> be safe because IME it's rare for files to be added/removed from
>> directories such that it changes from which directory a given ELisp file
>> is loaded. But of course we could also make efforts to try and keep our
>> cache consistent, e.g. via OS-level notification infrastructure or
>> by flushing the cache after a N seconds.
> Agree the changes are difficult and costly.
I don't understand: I didn't say (nor do I think) that any of that is
complex or costly.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 19:32:02 GMT)
Full text and
rfc822 format available.
Message #123 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Fri, Nov 1, 2024 at 7:18 PM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
>
> >> We can easily detect changes to `load-path` itself, of course, but as
> >> for changes to the content of the directories in `load-path` that would
> >> be more difficult&costly, admittedly. My plan was to do nothing
> >> about it (i.e. allow the cache to go stale): if we use the cache only to
> >> tell `load` in which directory to look for the file, it should usually
> >> be safe because IME it's rare for files to be added/removed from
> >> directories such that it changes from which directory a given ELisp file
> >> is loaded. But of course we could also make efforts to try and keep our
> >> cache consistent, e.g. via OS-level notification infrastructure or
> >> by flushing the cache after a N seconds.
> > Agree the changes are difficult and costly.
>
> I don't understand: I didn't say (nor do I think) that any of that is
> complex or costly.
>
Apologize for my misunderstanding of the "difficult&costly", will read
the comments more precisely.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 19:43:02 GMT)
Full text and
rfc822 format available.
Message #126 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> I had searched all 200+ packages in my test env, most of the packages
> use their feature name as the prefix, only 11 packages have
> exceptions.
FWIW, I just looked at the longest common prefix for .el files in
(Non)GNU ELPA packages and the ratio is a bit worse, but still good
enough that it should give good speedups:
(let ((default-directory "../nongnu/packages/"))
(mapcar (lambda (dir)
(when (file-directory-p dir)
(let ((completion-regexp-list '("\\`[^.].*\\.el")))
(list (intern dir)
(file-name-completion
"" dir)))))
(directory-files "."))))
In both GNU ELPA and NonGNU ELPA only about 10% of the packages had
a non-empty common prefix, which means that `load` would be able to
skip about 90% of the package directories.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 19:50:02 GMT)
Full text and
rfc822 format available.
Message #129 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Fri, 1 Nov 2024 17:58:01 +0000
> Cc: monnier <at> iro.umontreal.ca, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> 41646 <at> debbugs.gnu.org, monnier <at> gnu.org
>
> > > > How will the '"file1" "file2" ...' part be created?
> > >
> > > For the Emacs built in paths, we can create the (<path> [files])
> > > during bootstrap and write to the "subdirs.el", then it will push the
> > > extended (<path>, files...) into `load-path'.
> >
> > I don't understand: isn't this supposed to speed up primarily users
> > who have many 3rd-party packages installed? For them, what happens
> > during bootstrap is not relevant.
> >
> > If all we want is to record the places where bundled files live,
> > that's a much easier problem.
> Sorry for the fuzz "bootstrap", the "bootstrap" I wanted to say is
> part of building steps, like "make bootstrap", then we can build the
> files list into the "subdir.el".
I still don't understand: below you are talking about installing 300+
packages using package.el, so "make bootstrap" is not relevant.
> Both the startup time and running time will be affected by too many
> "load-path" entries.
> Like the "package.el", it will add all 300+ packages' paths on the top
> of "load-path" at the beginning of startup, the builtin paths will be
> on the bottom of "load-path", after that a simple "(require
> '<builtin-feature>)" will trigger emacs to walk through the
> "load-path" from top to bottom, that leads thouthands
> open-attemptions, that happened during emacs startup, or during
> running time.
So now we are talking not only about startup, but also about what
happens after that?
It is hard to discuss a feature whose goal and the problems it
attempts to solve shift around all the time. Can we please formulate
the goals and not change them afterwards?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Fri, 01 Nov 2024 20:00:02 GMT)
Full text and
rfc822 format available.
Message #132 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Fri, 1 Nov 2024 18:15:23 +0000
> Cc: monnier <at> iro.umontreal.ca, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> 41646 <at> debbugs.gnu.org, monnier <at> gnu.org
>
> How about this: I write some windows specific code only available on
> the windows to cache the files for the load-path. It will NOT affect
> other platforms. Like a flag "load-path-cache-expired 60" (nil/t) for
> windows users who won't change path/files a lot, then Emacs can cache
> the files list for path entries for 60 seconds. As I pasted, with the
> files cache, it can reduce the startup time from ~16s to ~5s, and also
> improve the running time performance. That will save a lot of emacs
> windows users.
I'd like us first to have a good understanding of the problems, their
aspects, and the goals of the proposed changes. I'd also would like
to have some agreed-upon timing of searching load-path, including its
dependence on the number of directories in load-path, and the relative
part of the startup that this particular aspect explains. (Are you
really saying that it takes Emacs 16 sec to search 300 directories for
300 files?) Only then we will have enough information to discuss
potential solutions and their effects on the startup time.
In general, a Windows-only feature is something I'd prefer not to
have, because quite a few users use Emacs on both Windows and
GNU/Linux, so having OS specific features should be avoided. But it
can be acceptable if no better alternative exists. But this is
something we should talk about much later, not now.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 02 Nov 2024 04:36:01 GMT)
Full text and
rfc822 format available.
Message #135 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Fri, Nov 1, 2024 at 7:58 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> I'd like us first to have a good understanding of the problems, their
> aspects, and the goals of the proposed changes. I'd also would like
> to have some agreed-upon timing of searching load-path, including its
> dependence on the number of directories in load-path, and the relative
> part of the startup that this particular aspect explains. (Are you
> really saying that it takes Emacs 16 sec to search 300 directories for
> 300 files?) Only then we will have enough information to discuss
> potential solutions and their effects on the startup time.
I use Spacemacs which will install ~380 packages for emacs. And on
windows without my patch, it takes ~16 seconds on startup, takes more
than 3 seconds to open a *.org file; as a comparison, similar
configuration on Ubuntu, the startup time is less than 4 seconds, and
takes less than 1 second to open a *.org file.
After applying the "load-hints" patch, on Windows the startup time is
reduced from ~16 seconds to ~5 seconds, taking ~1.5 seconds to open a
*.org file. It does not imply Emacs spends 16sec to search
directories for 300 files on windows.
> In general, a Windows-only feature is something I'd prefer not to
> have, because quite a few users use Emacs on both Windows and
> GNU/Linux, so having OS specific features should be avoided. But it
> can be acceptable if no better alternative exists. But this is
> something we should talk about much later, not now.
Totally agree, really look forward to an elegant solution.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 02 Nov 2024 08:25:02 GMT)
Full text and
rfc822 format available.
Message #138 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Sat, 2 Nov 2024 04:34:01 +0000
> Cc: monnier <at> iro.umontreal.ca, stefankangas <at> gmail.com, acorallo <at> gnu.org,
> 41646 <at> debbugs.gnu.org
>
> On Fri, Nov 1, 2024 at 7:58 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > I'd like us first to have a good understanding of the problems, their
> > aspects, and the goals of the proposed changes. I'd also would like
> > to have some agreed-upon timing of searching load-path, including its
> > dependence on the number of directories in load-path, and the relative
> > part of the startup that this particular aspect explains. (Are you
> > really saying that it takes Emacs 16 sec to search 300 directories for
> > 300 files?) Only then we will have enough information to discuss
> > potential solutions and their effects on the startup time.
>
> I use Spacemacs which will install ~380 packages for emacs. And on
> windows without my patch, it takes ~16 seconds on startup, takes more
> than 3 seconds to open a *.org file; as a comparison, similar
> configuration on Ubuntu, the startup time is less than 4 seconds, and
> takes less than 1 second to open a *.org file.
>
> After applying the "load-hints" patch, on Windows the startup time is
> reduced from ~16 seconds to ~5 seconds, taking ~1.5 seconds to open a
> *.org file. It does not imply Emacs spends 16sec to search
> directories for 300 files on windows.
Thanks, but that's not what I asked for. I asked for timing just the
part of searching the load-path. Emacs startup includes a lot of
stuff, and we here are talking about speeding up the search of
load-path to find a Lisp file.
For that matter, what kind of speedup does the change suggested by
Stefan give in this case?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 02 Nov 2024 14:37:01 GMT)
Full text and
rfc822 format available.
Message #141 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> In both GNU ELPA and NonGNU ELPA only about 10% of the packages had
> a non-empty common prefix, which means that `load` would be able to
> skip about 90% of the package directories.
Hmm.. so I tried to build a table (represented as a radix tree) that
maps prefixes to the corresponding list of directories where such files
can be found:
(defun load--build-filter-map (path suffixes)
(let ((prefix-map radix-tree-empty)
(completion-regexp-list
;; Note 1: Ignore hidden files to weed out `.dir-locals.el'.
(list (concat "\\`[^.].*" (regexp-opt suffixes) "\\'"))))
(dolist (dir (reverse path))
(when (file-directory-p dir)
;; FIXME: `prefix' may be *too long* because it includes a suffix,
;; as is the case for `leim-list.el' in `lisp/leim'!
(let* ((prefix (or (ignore-errors (file-name-completion "" dir))
""))
(subtree (radix-tree-subtree prefix-map prefix)))
(unless (radix-tree-lookup subtree "")
(let ((ps (car (radix-tree-prefixes prefix-map prefix))))
(unless (or (null ps) (< (length (car ps)) (length prefix)))
(message "prefix=%S ps = %S" prefix ps))
(setq prefix-map (radix-tree-insert prefix-map prefix
(cons dir (cdr ps))))))
(radix-tree-iter-mappings
subtree
(lambda (subprefix dirs)
(setq prefix-map (radix-tree-insert
prefix-map (concat prefix subprefix)
(cons dir dirs))))))))
prefix-map))
That seems to work. E.g. in `emacs -Q` we see that the table has a few
different prefixes ("o" for org, "u" for url, "calc" for calc, "mh-" for
mh-e, "e" for eshell, and "" for the rest) and e.g. the prefix "" is
mapped to the list of all dirs except those with a non-empty prefix,
whereas the "o" prefix is mapped to that same list plus the org
subdirectory, ... Then `load` can use this table to quickly (a single
lookup) replace the full `load-path` with a shorter list.
In my "normal" Emacs, where I have more than 800 entries in `load-path`,
most prefixes are mapped to lists of ~90 entries, which is
a nice improvement.
But there are two problems:
- The above function takes too much time to build the table. With my
800 entries it takes ~0.5s to build the table. This might be
acceptable if we can make sure it's run only once, but if the table is
built "on demand" and flushed whenever `load-path` changes, then in my
config it's called 10 times in a normal startup, presumably because
it's called at different stages of building `load-path` (I have not
dug deeper to find the actual source).
- The above table doesn't scale nicely. The problem is that the number
of prefixes is more or less proportional to the number of directories,
but the length of the directories lists associated to each prefix is
also basically proportional to the number of directories. So in an
`emacs -Q` (with a 24-element `load-path`), that table occupies a mere
4kB but in my full config with 800 elements in `load-path` that grows
to 500kB.
Maybe the "full table" approach will work better:
(defun load--build-file-map (path suffixes)
(let ((map (make-hash-table :test 'equal))
(completion-regexp-list
;; Note 1: Ignore hidden files to weed out `.dir-locals.el'.
(list (concat "\\`[^.].*" (regexp-opt suffixes) "\\'"))))
(dolist (dir (reverse path))
(when (file-directory-p dir)
;; FIXME: `prefix' may be *too long* because it includes a suffix,
;; as is the case for `leim-list.el' in `lisp/leim'!
(let* ((files (ignore-errors (file-name-all-completions "" dir)))
(dirs (list dir)))
(dolist (file files) (puthash file dirs map)))))
map))
this function doesn't try to mess with prefixes and instead collects all
the files in all the directories and builds a "reverse" table which maps
filenames to the directory in which they're found.
- It's much faster to build (~0.1s).
- It filters the `load-path` even more effectively: N hash-table lookups
(for each of the N suffixes) will return at most N directories where
the file can be found, and in most cases it returns N times the same
directory so `load` will typically need to look only at
a single directory.
- It takes more space: in an `emacs -Q` the table uses up already
around 180kB. But it scales better: with my full config the table
grows "only" to ~730kB.
I get the impression that these tables are too big, tho. So maybe
a better approach is one that leaves a bit more work to `load`,
e.g. where we build a table that just maps directories to their longest
common prefix (that can be done quickly and doesn't take up
significantly more space than `load-path` itself), and then in `load` or
`openp` we lookup this table to dynamically filter out the elements of
`load-path` that don't apply to the current file: i.e. we still have to
loop through the whole 800-element `load-path` but we'd only touch the
filesystem in the ~90 directories that are not excluded by
their prefix.
Stefan
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Sun, 01 Dec 2024 12:24:08 GMT)
Full text and
rfc822 format available.
bug unarchived.
Request was from
Stefan Monnier <monnier <at> iro.umontreal.ca>
to
control <at> debbugs.gnu.org.
(Sat, 10 May 2025 09:39:01 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 10 May 2025 09:46:01 GMT)
Full text and
rfc822 format available.
Message #148 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> An example is '(("/path1" "file1" "file2") ("/path2" "file3"
> "file4")). Then the function "load" will search the `load-hints` to
> find the file, reducing atempution to call "openp".
That sounds very similar to the hash-table used by Spencer's patch.
Can you try and reduce the differences between your code and his?
Especially w.r.t the C code (so that we can start integrating the
C part of the change, while people can try and play with different
approaches at the ELisp level)?
Also, you reported that when you tried Spencer's patch it made things
slower. It would be interesting to get more detailed timings of the
various elements to better understand what matters and what doesn't.
So it would be good to add timing measurements to separately count the
time spent constructing the auxiliary info (0s in the current code,
precomputed for your patch, and computed on the fly for Spencer's
patch), the time spent reducing `load-path` to a smaller number of
directories (0s in the current code), and then the time spent in
`openp` itself.
Also, Spencer sees a speedup with his patch, whereas you see a slowdown.
It would be good to investigate why that is (and the above measurements
might help do that).
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 10 May 2025 14:50:02 GMT)
Full text and
rfc822 format available.
Message #151 received at 41646 <at> debbugs.gnu.org (full text, mbox):
On Sat, May 10, 2025 at 7:40 AM Stefan Monnier <monnier <at> iro.umontreal.ca> wrote:
> > An example is '(("/path1" "file1" "file2") ("/path2" "file3"
> > "file4")). Then the function "load" will search the `load-hints` to
> > find the file, reducing atempution to call "openp".
>
> That sounds very similar to the hash-table used by Spencer's patch.
> Can you try and reduce the differences between your code and his?
> Especially w.r.t the C code (so that we can start integrating the
> C part of the change, while people can try and play with different
> approaches at the ELisp level)?
The hash version can not work towards the deleting /path-0 and adding
/path-1 case (e.g. "package-upgrade", deletes old files and directory,
adds new directory and files).
While this list version can be maintenanced simply with list add/del
functions, similar to the load-path.
> Also, you reported that when you tried Spencer's patch it made things
> slower. It would be interesting to get more detailed timings of the
> various elements to better understand what matters and what doesn't.
> So it would be good to add timing measurements to separately count the
> time spent constructing the auxiliary info (0s in the current code,
> precomputed for your patch, and computed on the fly for Spencer's
> patch), the time spent reducing `load-path` to a smaller number of
> directories (0s in the current code), and then the time spent in
> `openp` itself.
>
> Also, Spencer sees a speedup with his patch, whereas you see a slowdown.
> It would be good to investigate why that is (and the above measurements
> might help do that).
The hash version will scan the paths during Emacs startup to construct
the hash table, that's why it slows emacs startup speed; but after the
hash table is ready, it do speedup heavy loading like "(require
'org)".
This list version, the files list is written to *-autoloads.el during
package installation, on emacs startup, the *-autoloads.el just adds
the (/path files...) into the load-hints, no scanning.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 10 May 2025 15:42:02 GMT)
Full text and
rfc822 format available.
Message #154 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> From: Lin Sun <sunlin7.mail <at> gmail.com>
> Date: Sat, 10 May 2025 14:49:01 +0000
> Cc: 41646 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
> Stefan Kangas <stefankangas <at> gmail.com>
>
> > Also, Spencer sees a speedup with his patch, whereas you see a slowdown.
> > It would be good to investigate why that is (and the above measurements
> > might help do that).
>
> The hash version will scan the paths during Emacs startup to construct
> the hash table, that's why it slows emacs startup speed; but after the
> hash table is ready, it do speedup heavy loading like "(require
> 'org)".
Nothing prevents us from saving the hash-table to a file, right?
But the question is: what happens when the files and directories on
disk change? For example, some package is removed, or the entire tree
gets relocated?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 10 May 2025 15:56:02 GMT)
Full text and
rfc822 format available.
Message #157 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> The hash version can not [...]
You're looking at it wrong. Don't look at it as a competitor and try to
see what your patch can do which the other can't. Think about *how* to
merge them into something that combines the best of the two approaches.
There might be hard choices to make at some point of course (we can't
always have our cake and eat it too), but the above it definitely not
one of them: "the hash version" already adapts to changes in `load-path`
like removal or addition of elements. The data for old directory will
simply stay in the hash-table without being used any more.
> The hash version will scan the paths during Emacs startup to construct
> the hash table, that's why it slows Emacs startup speed; but after the
> hash table is ready, it do speedup heavy loading like "(require
> 'org)".
That's a possible explanation, but without investigating further we
shouldn't take it as a given, especially since the results are different
for your test and for Spencer's, which strongly suggests that our
intuition is not to be trusted.
And as Eli suggests, if that's indeed the problem a precomputed approach
like yours can pre-populate Spencer's hash to avoid scanning the
`load-path` directories during Emacs startup.
Stefan
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#41646; Package
emacs.
(Sat, 10 May 2025 16:02:01 GMT)
Full text and
rfc822 format available.
Message #160 received at 41646 <at> debbugs.gnu.org (full text, mbox):
> But the question is: what happens when the files and directories on
> disk change? For example, some package is removed, or the entire tree
> gets relocated?
As long as the directory name changes (in `load-path`), Spencer's patch
will handle that automatically. It will also correctly handle the case
where files are *removed* (or modified) from directories.
The changes it has trouble with is when files are *added* to one of the
directories in `load-path` (and even among those, not all of them:
e.g. addition of a `foo.elc` file alongside a `foo.el` file will usually
work just fine, except when we (load "foo.elc") explicitly).
Stefan
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Sun, 08 Jun 2025 11:24:10 GMT)
Full text and
rfc822 format available.
This bug report was last modified 32 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.