GNU bug report logs - #77745
Async native compilation runs every time I start Emacs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 77745 in the body.
You can then email your comments to 77745 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Async native compilation runs every time I start Emacs
Date: Fri, 11 Apr 2025 16:47:17 +0200

The native compiler does not realize that it has already compiled the
code so it recompiles it again every time I restart
Emacs. Unfortunately, the logs don't describe *why* the compiler
thinks the source is stale so debugging the problem is difficult. I
can trigger the recompilation easily just by eagerly loading a package
in my init.el. For example,

    (use-package magit)

will cause magit and all modules it depends on to be recompiled. I'm
using magit as an example, but the same thing happens with most other
packages too. I can prevent recompilation by using lazy loading:

    (use-package magit :defer t)

Then only the file "use-package-core-b234260b-cc1d969d.eln"
gets recompiled. However, I don't think eager loading should trigger
redundant recompilations.

The entirety of my init.el is:

    (setq user-emacs-directory "~/.config/emacs/lisp/")
    (setq package-user-dir "~/.config/emacs/packages/")

    (require 'package)
    (package-initialize)
    (use-package magit)

I've tested with both Emacs 30.1 and 31.0.50 from git and it's the
same problem. Something I noticed is that the "hash" seem to be cut
short during compilation. E.g., the temporary file is called
magit-commit-664f0701-bd4a7b87KHzrEe.eln.tmp but the destination file
magit-commit-664f0701-bd4a7b87.eln. And if I load magit after init.el
it doesn't get recompiled either.

My async compilation log:
https://gist.github.com/bjourne/8958a3918b6f0e61840a5fc10a546b4b


-- 
mvh/best regards Björn Lindqvist

Message #8 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Fri, 11 Apr 2025 19:27:09 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Fri, 11 Apr 2025 16:47:17 +0200
> 
> The native compiler does not realize that it has already compiled the
> code so it recompiles it again every time I restart
> Emacs. Unfortunately, the logs don't describe *why* the compiler
> thinks the source is stale so debugging the problem is difficult. I
> can trigger the recompilation easily just by eagerly loading a package
> in my init.el. For example,
> 
>     (use-package magit)
> 
> will cause magit and all modules it depends on to be recompiled. I'm
> using magit as an example, but the same thing happens with most other
> packages too.

You need to figure out why this happens, because it definitely isn't
happening for others.

Do you see the *.eln files created each time the native compilation is
triggered?  If so, are these files recreated using the same hashes in
the file name, or do the hashes change?  Are the sources moved between
the sessions, perhaps?  Are you always starting the same Emacs binary,
or do you rebuild Emacs between sessions?  Did you customize the value
of native-comp-eln-load-path from its default?

These and other questions need to be answered before we can understand
why you have this problem.

> I can prevent recompilation by using lazy loading:
> 
>     (use-package magit :defer t)
> 
> Then only the file "use-package-core-b234260b-cc1d969d.eln"
> gets recompiled. However, I don't think eager loading should trigger
> redundant recompilations.

What happens if you start "emacs -Q" and then load Magit manually by
"M-x load-library"? does it get recompiled every time in that case?

> 
> The entirety of my init.el is:
> 
>     (setq user-emacs-directory "~/.config/emacs/lisp/")
>     (setq package-user-dir "~/.config/emacs/packages/")
> 
>     (require 'package)
>     (package-initialize)
>     (use-package magit)
> 
> I've tested with both Emacs 30.1 and 31.0.50 from git and it's the
> same problem. Something I noticed is that the "hash" seem to be cut
> short during compilation. E.g., the temporary file is called
> magit-commit-664f0701-bd4a7b87KHzrEe.eln.tmp but the destination file
> magit-commit-664f0701-bd4a7b87.eln. And if I load magit after init.el
> it doesn't get recompiled either.
> 
> My async compilation log:
> https://gist.github.com/bjourne/8958a3918b6f0e61840a5fc10a546b4b

Didn't see any errors there, so it sounds like the compilations all
succeeded.

Is this a recent problem, or did that always happen for you?  If this
started happening only lately, can you tell when?

Message #11 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Fri, 11 Apr 2025 20:50:17 +0200

> Do you see the *.eln files created each time the native compilation is
> triggered?  If so, are these files recreated using the same hashes in
> the file name, or do the hashes change?  Are the sources moved between
> the sessions, perhaps?  Are you always starting the same Emacs binary,
> or do you rebuild Emacs between sessions?  Did you customize the value
> of native-comp-eln-load-path from its default?

Yes. Same hashes. No. Same binary. Yes! At the top of my init.el I have

    (setq user-emacs-directory "~/.config/emacs/lisp/")

causing *most* .eln files to be dumped in
~/.config/emacs/lisp/eln-cache/31.0.50-859de2ee. However, two
mysterious files are still being written to
~/.config/emacs/eln-cache/31.0.50-859de2ee (no lisp) in the path:

    subr--trampoline-6d616b652d70726f63657373_make_process_0.eln
    subr--trampoline-616c6c2d636f6d706c6574696f6e73_all_completions_0.eln

If I do not set user-emacs-directory, native-comp-eln-load-path stays
at its default value and the recompilation problem goes away. I'll use
that as my workaround since setting user-emacs-directory to a
non-default isn't important for me. It appears that
native-comp-eln-load-path shouldn't be touched from within init.el
because that causes weird side-effects.

As an aside, shouldn't the default native-comp-eln-load-path be
~/.cache/emacs/eln-cache? Cache artifacts aren't meant to be generated in
the configuration hierarchy.


-- 
mvh/best regards Björn Lindqvist

Message #14 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Fri, 11 Apr 2025 22:09:59 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Fri, 11 Apr 2025 20:50:17 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> > Do you see the *.eln files created each time the native compilation is
> > triggered?  If so, are these files recreated using the same hashes in
> > the file name, or do the hashes change?  Are the sources moved between
> > the sessions, perhaps?  Are you always starting the same Emacs binary,
> > or do you rebuild Emacs between sessions?  Did you customize the value
> > of native-comp-eln-load-path from its default?
> 
> Yes. Same hashes. No. Same binary. Yes! At the top of my init.el I have
> 
>     (setq user-emacs-directory "~/.config/emacs/lisp/")
> 
> causing *most* .eln files to be dumped in
> ~/.config/emacs/lisp/eln-cache/31.0.50-859de2ee.

Why are you setting the value of user-emacs-directory?

> However, two
> mysterious files are still being written to
> ~/.config/emacs/eln-cache/31.0.50-859de2ee (no lisp) in the path:
> 
>     subr--trampoline-6d616b652d70726f63657373_make_process_0.eln
>     subr--trampoline-616c6c2d636f6d706c6574696f6e73_all_completions_0.eln

Probably because that happens before your modified value of
user-emacs-directory becomes in effect.

> If I do not set user-emacs-directory, native-comp-eln-load-path stays
> at its default value and the recompilation problem goes away.

Then please refrain from changing the value of user-emacs-directory.

> As an aside, shouldn't the default native-comp-eln-load-path be
> ~/.cache/emacs/eln-cache? Cache artifacts aren't meant to be generated in
> the configuration hierarchy.

Sorry, I don't understand the question.  What is "the configuration
hierarchy"?

Message #17 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Fri, 11 Apr 2025 23:08:16 +0200

> Why are you setting the value of user-emacs-directory?

Because I want the directory emacs packages throw random stuff in to
be distinct from my actual configuration directory. Ephemeral stuff
like ido.last does not belong in the same directory as init.el.

> Then please refrain from changing the value of user-emacs-directory.

I won't. But it is not really clear from the documentation that
user-emacs-directory is not user modifiable. If you google it you can
find plenty of examples of people overriding it in init.el to, for
example, install competing emacs environments (DoomEmacs +
SpacEmacs):

https://emacs.stackexchange.com/questions/4253/how-to-start-emacs-with-a-custom-user-emacs-directory

I assume the same holds true for native-comp-eln-load-path and you are
not supposed to touch that variable either?

> Sorry, I don't understand the question.  What is "the configuration
> hierarchy"?

It's from the XDG Base Directory specification. It stipulates that to
make it easier for users and programs to find files the configuration
files of a program should be stored in ~/.config/<progname> and
"user-specific non-essential (cached) data" should be stored in
~/.cache/<progname>.  The advantage of this scheme is that if you want
to reclaim disk space you can simply rm -rf ~/.cache/<progname> and if
you want to reset a program to its factory default rm -rf
~/.config/<progname>.


-- 
mvh/best regards Björn Lindqvist

Message #20 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 12 Apr 2025 09:39:13 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Fri, 11 Apr 2025 23:08:16 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> > Why are you setting the value of user-emacs-directory?
> 
> Because I want the directory emacs packages throw random stuff in to
> be distinct from my actual configuration directory. Ephemeral stuff
> like ido.last does not belong in the same directory as init.el.
> 
> > Then please refrain from changing the value of user-emacs-directory.
> 
> I won't. But it is not really clear from the documentation that
> user-emacs-directory is not user modifiable. If you google it you can
> find plenty of examples of people overriding it in init.el to, for
> example, install competing emacs environments (DoomEmacs +
> SpacEmacs):
> 
> https://emacs.stackexchange.com/questions/4253/how-to-start-emacs-with-a-custom-user-emacs-directory
> 
> I assume the same holds true for native-comp-eln-load-path and you are
> not supposed to touch that variable either?

People who modify user-emacs-directory in their init files should read
the manual very carefully, and should really understand what they are
doing.  Here are some facts that you probably knew, but perhaps never
considered together to realize what would be the effect of what you
are doing:

  . user-emacs-directory is determined by the startup code, as a side
    effect of looking for your init files
  . the startup code then loads the init files and executes their code
  . native-comp-eln-load-path is computed by the startup code, and it
    has the value of user-emacs-directory pushed onto the list

What this means is that if your init files load packages, directly or
indirectly, before they modify user-emacs-directory, the native
compilation will start for those packages using the value of
native-comp-eln-load-path that is not what your session will
subsequently use.  And packages compiled after user-emacs-directory is
modified will have their *.eln files deposited in another directory.

IOW, the order in which you modify user-emacs-directory and load
packages that need native compilation matters _a_lot_, as you have
discovered.

So, if you really _must_ modify user-emacs-directory, you should do it
in early-init file, and do it the very first thing there.  (Of course,
this means that all the other init files should then live in the
directory which is the updated value of user-emacs-directory.)  We
recommend not to modify the value of user-emacs-directory, because
doing that might break various initializations Emacs performs during
startup.

> > Sorry, I don't understand the question.  What is "the configuration
> > hierarchy"?
> 
> It's from the XDG Base Directory specification. It stipulates that to
> make it easier for users and programs to find files the configuration
> files of a program should be stored in ~/.config/<progname> and
> "user-specific non-essential (cached) data" should be stored in
> ~/.cache/<progname>.  The advantage of this scheme is that if you want
> to reclaim disk space you can simply rm -rf ~/.cache/<progname> and if
> you want to reset a program to its factory default rm -rf
> ~/.config/<progname>.

The eln cache Emacs uses is not really a "cache" XDG spec had in mind.
The *.eln files are not "non-essential", since we want them to
persist, and removing that directory will make the next startup of
Emacs significantly longer and more expensive.  Therefore, we decided
to have the cache under user-emacs-directory, as Emacs generally does
for all of its files.

Message #23 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 12 Apr 2025 19:08:05 +0200

Hello again,

I tried setting native-comp-eln-load-path explicitly in init.el

    (setq native-comp-eln-load-path
      (list (expand-file-name "~/tmp/")
            "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))

and this works much better. The native compiler only
writes to ~/tmp/31.0.50-859de2ee/. Not in any directory under
~/.config/emacs. Only the files warnings-28e75f4d-f0ade81c.eln,
icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln gets
recompiled on startup.

> What this means is that if your init files load packages, directly or
> indirectly, before they modify user-emacs-directory, the native
> compilation will start for those packages using the value of
> native-comp-eln-load-path that is not what your session will
> subsequently use.  And packages compiled after user-emacs-directory is
> modified will have their *.eln files deposited in another directory.

> IOW, the order in which you modify user-emacs-directory and load
> packages that need native compilation matters _a_lot_, as you have
> discovered.

The first sexp of my init.el is

    (setq user-emacs-directory "~/.config/emacs/lisp/")

which is (?) executed *before* the native compiler kicks in. Unless
something odd happens during *parsing* of init.el, but that seems
far-fetched.

What I think happens is that the native compiler builds the
native-comp-eln-load-path based on the value of user-emacs-directory
before init.el is loaded. Then user-emacs-directory gets modified and
this changes the path it *writes* to, but does nothing to the paths it
*reads* from. So it compiles the files and writes them to a directory
it won't read from the next time I start Emacs. Indeed, if I log
native-comp-eln-load-path I can see that the value is:

    (~/.config/emacs/eln-cache/ /opt/emacs/lib/emacs/31.0.50/native-lisp/)

but the native compiler writes to ~/.config/emacs/lisp/eln-cache/
(note: /lisp/), contradicting the manual: "By default,
asynchronous native compilation writes the *.eln files it produces to
a subdirectory of the first writable directory specified by the
native-comp-eln-load-path variable".

So it must be a bug that setting user-emacs-directory updates the path
the native compiler writes to, but not the path(s) it reads from.

> We recommend not to modify the value of user-emacs-directory,
> because doing that might break various initializations Emacs
> performs during startup.

Are there technical reasons user-emacs-directory can't be changed?
Isn't code that relies on user-emacs-directory having a specific value
or to never change is buggy?

> The eln cache Emacs uses is not really a "cache" XDG spec had in mind.
> The *.eln files are not "non-essential", since we want them to
> persist, and removing that directory will make the next startup of
> Emacs significantly longer and more expensive.

I think what you're describing is exactly what a cache does and
exactly how other programs use $XDG_CACHE_HOME. Of course a program
would take longer to start after you blow up its cache.


--
mvh/best regards Björn Lindqvist

Message #26 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 12 Apr 2025 13:34:59 -0400

[Message part 1 (text/plain, inline)]

On Sat, Apr 12, 2025 at 1:09 PM Björn Lindqvist <bjourne <at> gmail.com> wrote:

> Hello again,
>
> I tried setting native-comp-eln-load-path explicitly in init.el
>
>     (setq native-comp-eln-load-path
>       (list (expand-file-name "~/tmp/")
>             "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))
>
> and this works much better. The native compiler only
> writes to ~/tmp/31.0.50-859de2ee/. Not in any directory under
> ~/.config/emacs. Only the files warnings-28e75f4d-f0ade81c.eln,
> icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln gets
> recompiled on startup.
>
> > What this means is that if your init files load packages, directly or
> > indirectly, before they modify user-emacs-directory, the native
> > compilation will start for those packages using the value of
> > native-comp-eln-load-path that is not what your session will
> > subsequently use.  And packages compiled after user-emacs-directory is
> > modified will have their *.eln files deposited in another directory.
>
> > IOW, the order in which you modify user-emacs-directory and load
> > packages that need native compilation matters _a_lot_, as you have
> > discovered.
>
> The first sexp of my init.el is
>
>     (setq user-emacs-directory "~/.config/emacs/lisp/")
>
> which is (?) executed *before* the native compiler kicks in. Unless
> something odd happens during *parsing* of init.el, but that seems
> far-fetched.
>
> What I think happens is that the native compiler builds the
> native-comp-eln-load-path based on the value of user-emacs-directory
> before init.el is loaded. Then user-emacs-directory gets modified and
> this changes the path it *writes* to, but does nothing to the paths it
> *reads* from. So it compiles the files and writes them to a directory
> it won't read from the next time I start Emacs. Indeed, if I log
> native-comp-eln-load-path I can see that the value is:
>
>     (~/.config/emacs/eln-cache/ /opt/emacs/lib/emacs/31.0.50/native-lisp/)
>
> but the native compiler writes to ~/.config/emacs/lisp/eln-cache/
> (note: /lisp/), contradicting the manual: "By default,
> asynchronous native compilation writes the *.eln files it produces to
> a subdirectory of the first writable directory specified by the
> native-comp-eln-load-path variable".
>
> So it must be a bug that setting user-emacs-directory updates the path
> the native compiler writes to, but not the path(s) it reads from.
>
> > We recommend not to modify the value of user-emacs-directory,
> > because doing that might break various initializations Emacs
> > performs during startup.
>
> Are there technical reasons user-emacs-directory can't be changed?
> Isn't code that relies on user-emacs-directory having a specific value
> or to never change is buggy?
>
> > The eln cache Emacs uses is not really a "cache" XDG spec had in mind.
> > The *.eln files are not "non-essential", since we want them to
> > persist, and removing that directory will make the next startup of
> > Emacs significantly longer and more expensive.
>
> I think what you're describing is exactly what a cache does and
> exactly how other programs use $XDG_CACHE_HOME. Of course a program
> would take longer to start after you blow up its cache.
>

setq is incorrect for overriding native-comp-eln-load-path.  Call
startup-redirect-eln-cache only in your early-init.el file.

As far as overriding user-emacs-directory is concerned, for best results
you should use the --init-directory command-line argument if you can't live
with the default.  The default does respect XDG_CONFIG_HOME and defaults to
XDG_CONFIG_HOME/emacs or ~/.config/emacs then the more traditional
locations, in that priority.  This assumes you do not specify --user.  If
you don't use this method, you'll have to do a census of the variables
dependent on user-emacs-directory and fix them all.  There currently is no
function that does that, and even if there were, if you ran it too late in
your startup cycle, external packages would be tainted with the old value.

[Message part 2 (text/html, inline)]

Message #29 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Ship Mints <shipmints <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 12 Apr 2025 20:14:50 +0200

> setq is incorrect for overriding native-comp-eln-load-path.  Call startup-redirect-eln-cache only in your early-init.el file.
>
> As far as overriding user-emacs-directory is concerned, for best results you should use the --init-directory command-line argument if you can't live with the default.  The default does respect XDG_CONFIG_HOME and defaults to XDG_CONFIG_HOME/emacs or ~/.config/emacs then the more traditional locations, in that priority.  This assumes you do not specify --user.  If you don't use this method, you'll have to do a census of the variables dependent on user-emacs-directory and fix them all.  There currently is no function that does that, and even if there were, if you ran it too late in your startup cycle, external packages would be tainted with the old value.

The manual says that you can modify native-comp-eln-load-path directly:

    Sometimes there could be a need to prevent the native compilation
    from writing its results, the *.eln files, into a subdirectory of
    user-emacs-directory (see The Init File). You can do that by
    either changing the value of native-comp-eln-load-path (see
    Native-Compilation Variables) or by temporarily pointing the HOME
    environment variable to a non-existing directory.

Regardless, it doesn't matter. Having

    (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln/"))

in early-init.el is exactly equivalent to having

    (setq native-comp-eln-load-path
      (list (expand-file-name "~/tmp/")
            "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))

in init.el. The native compiler correctly reads and writes most files
in ~/.cache/emacs/eln/ *except* for warnings-28e75f4d-f0ade81c.eln,
icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln which it
keeps recompiling every time I start emacs. The only way I've found to
get it to not recompile warnings, icons, and cl-lib is to keep the
default ~/.config/emacs/eln-cache path.


-- 
mvh/best regards Björn Lindqvist

Message #32 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 12 Apr 2025 14:49:41 -0400

[Message part 1 (text/plain, inline)]

On Sat, Apr 12, 2025 at 2:15 PM Björn Lindqvist <bjourne <at> gmail.com> wrote:

> > setq is incorrect for overriding native-comp-eln-load-path.  Call
> startup-redirect-eln-cache only in your early-init.el file.
> >
> > As far as overriding user-emacs-directory is concerned, for best results
> you should use the --init-directory command-line argument if you can't live
> with the default.  The default does respect XDG_CONFIG_HOME and defaults to
> XDG_CONFIG_HOME/emacs or ~/.config/emacs then the more traditional
> locations, in that priority.  This assumes you do not specify --user.  If
> you don't use this method, you'll have to do a census of the variables
> dependent on user-emacs-directory and fix them all.  There currently is no
> function that does that, and even if there were, if you ran it too late in
> your startup cycle, external packages would be tainted with the old value.
>
> The manual says that you can modify native-comp-eln-load-path directly:
>
>     Sometimes there could be a need to prevent the native compilation
>     from writing its results, the *.eln files, into a subdirectory of
>     user-emacs-directory (see The Init File). You can do that by
>     either changing the value of native-comp-eln-load-path (see
>     Native-Compilation Variables) or by temporarily pointing the HOME
>     environment variable to a non-existing directory.
>
> Regardless, it doesn't matter. Having
>
>     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln/"))
>
> in early-init.el is exactly equivalent to having
>
>     (setq native-comp-eln-load-path
>       (list (expand-file-name "~/tmp/")
>             "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))
>

Of course.  But you run the risk of incompatibilities later on.

in init.el. The native compiler correctly reads and writes most files
> in ~/.cache/emacs/eln/ *except* for warnings-28e75f4d-f0ade81c.eln,
> icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln which it
> keeps recompiling every time I start emacs. The only way I've found to
> get it to not recompile warnings, icons, and cl-lib is to keep the
> default ~/.config/emacs/eln-cache path.
>

I apologize that I missed what you're trying to accomplish.  If you want to
avoid compiler warnings, perhaps suppress warnings in early init, and then
reenable them in init where it makes sense if you want warnings for
external packages but not core packages?

The load logic is, I think, find the elc file (or make it, if byte
compilation isn't inhibited) along load-path, and if found, probe the eln
cache and swap the eln, if found.

As far as icons and cl-lib recompiles, not sure here.  I also redirect my
eln cache into user-emacs-directory/var/eln-cache and I start Emacs
sessions gobs of times per day both 30 and 31/master and I don't see this
behavior.  What platform are you on and does your build have icons and
cl-lib in its pre-compiled directory?  Are you building Emacs without aot?

[Message part 2 (text/html, inline)]

Message #35 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sun, 13 Apr 2025 09:34:50 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Sat, 12 Apr 2025 19:08:05 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> Hello again,
> 
> I tried setting native-comp-eln-load-path explicitly in init.el
> 
>     (setq native-comp-eln-load-path
>       (list (expand-file-name "~/tmp/")
>             "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))
> 
> and this works much better. The native compiler only
> writes to ~/tmp/31.0.50-859de2ee/. Not in any directory under
> ~/.config/emacs. Only the files warnings-28e75f4d-f0ade81c.eln,
> icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln gets
> recompiled on startup.

We have a function for that, startup-redirect-eln-cache, and you are
well advised to use that instead of constructing the list by hand.
That function's doc string also says it's best to do that in
early-init file, so that the list of directories is not changed
halfway through the startup process, causing problems.  Please be sure
to follow our recommendations, since they are the best practices we
decided upon based on analyzing the various use cases and issues whose
subtleties you may not be aware of.

> > What this means is that if your init files load packages, directly or
> > indirectly, before they modify user-emacs-directory, the native
> > compilation will start for those packages using the value of
> > native-comp-eln-load-path that is not what your session will
> > subsequently use.  And packages compiled after user-emacs-directory is
> > modified will have their *.eln files deposited in another directory.
> 
> > IOW, the order in which you modify user-emacs-directory and load
> > packages that need native compilation matters _a_lot_, as you have
> > discovered.
> 
> The first sexp of my init.el is
> 
>     (setq user-emacs-directory "~/.config/emacs/lisp/")
> 
> which is (?) executed *before* the native compiler kicks in.

You cannot know that.  Emacs executes quite a bit of code before it
reads the init file, see startup.el.  Unless you really know that code
through and through, you should be modifying the eln load path as
early as possible, if at all.

> What I think happens is that the native compiler builds the
> native-comp-eln-load-path based on the value of user-emacs-directory
> before init.el is loaded.

That's not accurate.  The startup code updates
native-comp-eln-load-path no less than 3 times during the startup
procedure, at certain strategic points, because various init files
could affect it.

Once again, the startup procedure, being a kind-of bootstrap process,
is very complicated and delicate, and you are advised to use the
functions and facilities it provides as we recommend using them,
because doing stuff on your own can very easily break your sessions.

> Then user-emacs-directory gets modified and
> this changes the path it *writes* to, but does nothing to the paths it
> *reads* from.

There are no two paths, just one.  It is used both for looking up
*.eln files and for writing newly-compiled ones.

> So it compiles the files and writes them to a directory
> it won't read from the next time I start Emacs. Indeed, if I log
> native-comp-eln-load-path I can see that the value is:
> 
>     (~/.config/emacs/eln-cache/ /opt/emacs/lib/emacs/31.0.50/native-lisp/)
> 
> but the native compiler writes to ~/.config/emacs/lisp/eln-cache/
> (note: /lisp/), contradicting the manual: "By default,
> asynchronous native compilation writes the *.eln files it produces to
> a subdirectory of the first writable directory specified by the
> native-comp-eln-load-path variable".

The manual describes what happens by default, not what happens when
you tinker with user-emacs-directory and/or native-comp-eln-load-path
during the startup.

Since you change user-emacs-directory halfway through, some of the
compiled files get written to the wrong place, and are not found when
they are needed upon the next startup.  Just don't do that!

> So it must be a bug that setting user-emacs-directory updates the path
> the native compiler writes to, but not the path(s) it reads from.

No, there's no bug.  The only problem is that you are tinkering with
this delicate process in ways that we don't support, and without a
good understanding of what that does.

> > We recommend not to modify the value of user-emacs-directory,
> > because doing that might break various initializations Emacs
> > performs during startup.
> 
> Are there technical reasons user-emacs-directory can't be changed?

I explained them in my last two messages.  And it isn't that it could
be modified, it's just that doing that "by hand" in the middle of the
startup process can easily break things.  As it did for you.

The supported way of affecting the value of user-emacs-directory is to
invoke Emacs with the --init-directory command-line option, but even
that has its subtle aspects, described in the manual.  That option was
added for some rare and very specific use cases, and in general my
advice it not to use even that unless really necessary.

> Isn't code that relies on user-emacs-directory having a specific value
> or to never change is buggy?

Since the value of user-emacs-directory is determined by Emacs as part
of the process of finding your init files, and since the value is
thereafter used to find other related directories and files, it should
be clear to you that modifying the value in arbitrary places during
the startup is dangerous.  You should only use the supported
interfaces and protocols for adjusting it, and preferably not touch it
at all.  If you do, you are on your own; I'm not interested in
complicating even more the already quite complex code in startup.el to
cater to such use cases, when better, more reliable ways of doing the
same already exist and are documented.

> > The eln cache Emacs uses is not really a "cache" XDG spec had in mind.
> > The *.eln files are not "non-essential", since we want them to
> > persist, and removing that directory will make the next startup of
> > Emacs significantly longer and more expensive.
> 
> I think what you're describing is exactly what a cache does and
> exactly how other programs use $XDG_CACHE_HOME. Of course a program
> would take longer to start after you blow up its cache.

This is not a cache.  This is JIT-compiled code of demand-loaded Emacs
packages.  Removing it is the same as removing executable binaries and
leaving the user with their source code, to be recompiled the next
time it is needed.  It is absurd to consider this a cache in the XDG
meaning.

Message #38 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sun, 13 Apr 2025 09:47:19 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Sat, 12 Apr 2025 20:14:50 +0200
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
> 
> > setq is incorrect for overriding native-comp-eln-load-path.  Call startup-redirect-eln-cache only in your early-init.el file.
> >
> > As far as overriding user-emacs-directory is concerned, for best results you should use the --init-directory command-line argument if you can't live with the default.  The default does respect XDG_CONFIG_HOME and defaults to XDG_CONFIG_HOME/emacs or ~/.config/emacs then the more traditional locations, in that priority.  This assumes you do not specify --user.  If you don't use this method, you'll have to do a census of the variables dependent on user-emacs-directory and fix them all.  There currently is no function that does that, and even if there were, if you ran it too late in your startup cycle, external packages would be tainted with the old value.
> 
> The manual says that you can modify native-comp-eln-load-path directly:
> 
>     Sometimes there could be a need to prevent the native compilation
>     from writing its results, the *.eln files, into a subdirectory of
>     user-emacs-directory (see The Init File). You can do that by
>     either changing the value of native-comp-eln-load-path (see
>     Native-Compilation Variables) or by temporarily pointing the HOME
>     environment variable to a non-existing directory.

That fragment explains how to _prevent_ native-compilation, by
changing the eln load path so it is basically invalid.  So your
reference to that as a justification for manually changing the value
is misguided.

> Regardless, it doesn't matter. Having
> 
>     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln/"))
> 
> in early-init.el is exactly equivalent to having
> 
>     (setq native-comp-eln-load-path
>       (list (expand-file-name "~/tmp/")
>             "/opt/emacs/lib/emacs/31.0.50/native-lisp/"))
> 
> in init.el.

Only if your original value had only two elements in the list.  If it
has more, the above will lose some elements.

And anyway, it could be equivalent today, but it might not be
equivalent tomorrow, if we decide something else is needed.  Using
public APIs makes your code future-proof, while doing the stuff by
hand requires you to adjust to any future changes, and on top of that
requires you to understand very well how the eln load path is used.
Why do all that when we already provide a public function whose use
avoids all of that?

> The native compiler correctly reads and writes most files
> in ~/.cache/emacs/eln/ *except* for warnings-28e75f4d-f0ade81c.eln,
> icons-eafe82eb-47a1ab50.eln, and cl-lib-8b938900-6d6e9142.eln which it
> keeps recompiling every time I start emacs.

Probably because something in your startup triggers delayed warnings
very early, and with eln load path modified halfway through the
startup, the compiled *.eln files for these two packages are not
found.

> The only way I've found to get it to not recompile warnings, icons,
> and cl-lib is to keep the default ~/.config/emacs/eln-cache path.

Which is what you should do in the first place.  You should in any
case avoid removing that directory from the eln load path.

Message #41 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Ship Mints <shipmints <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Mon, 14 Apr 2025 12:14:13 +0200

> Of course.  But you run the risk of incompatibilities later on.

I guess, but there is no indication in the manual that
native-comp-eln-load-path shouldn't be modified directly. Emacs has many
other variables for search paths (like load-path) which can be freely
modified. And what if you want to add multiple directories to
native-comp-eln-load-path? Then startup-redirect-eln-cache is not
sufficient.

> I apologize that I missed what you're trying to accomplish.  If you
> want to avoid compiler warnings, perhaps suppress warnings in early
> init, and then reenable them in init where it makes sense if you
> want warnings for external packages but not core packages?

No, what I want to accomplish is:

    1. Compilation cache in ~/.cache/emacs/eln,
    2. No redundant recompilations at startup.

> The load logic is, I think, find the elc file (or make it, if byte
> compilation isn't inhibited) along load-path, and if found, probe
> the eln cache and swap the eln, if found.

I think so too, but there appears to be a race condition
involved. First the compiler can't find foobar.eln on the
native-comp-eln-load-path, so it schedules it for
recompilation. Meanwhile, native-comp-eln-load-path is updated. Then
the compiler writes foobar.eln to the *new path*.

> What platform are you on and does your build have icons and cl-lib
> in its pre-compiled directory?  Are you building Emacs without aot?

I'm building Emacs with aot on Arch Linux. Recompilation happens in
both Emacs 30.1 and 31.0.50. icons, cl-lib, and warnings are not in
the system-wide /opt/emacs/lib/emacs/31.0.50/native-lisp/ directory.


-- 
mvh/best regards Björn Lindqvist

Message #44 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Mon, 14 Apr 2025 13:39:18 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Mon, 14 Apr 2025 12:14:13 +0200
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
> 
> > Of course.  But you run the risk of incompatibilities later on.
> 
> I guess, but there is no indication in the manual that
> native-comp-eln-load-path shouldn't be modified directly. Emacs has many
> other variables for search paths (like load-path) which can be freely
> modified. And what if you want to add multiple directories to
> native-comp-eln-load-path? Then startup-redirect-eln-cache is not
> sufficient.

Adding directories is okay, as long as you keep this requirement from
the doc string fulfilled:

  The last directory of this list is assumed to be the one holding
  the system *.eln files, which are the files produced when building
  Emacs.

Message #47 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Ship Mints <shipmints <at> gmail.com>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Mon, 14 Apr 2025 07:21:55 -0400

[Message part 1 (text/plain, inline)]

On Mon, Apr 14, 2025 at 6:14 AM Björn Lindqvist <bjourne <at> gmail.com> wrote:

> > Of course.  But you run the risk of incompatibilities later on.
>
> I guess, but there is no indication in the manual that
> native-comp-eln-load-path shouldn't be modified directly. Emacs has many
> other variables for search paths (like load-path) which can be freely
> modified. And what if you want to add multiple directories to
> native-comp-eln-load-path? Then startup-redirect-eln-cache is not
> sufficient.
>
> > I apologize that I missed what you're trying to accomplish.  If you
> > want to avoid compiler warnings, perhaps suppress warnings in early
> > init, and then reenable them in init where it makes sense if you
> > want warnings for external packages but not core packages?
>
> No, what I want to accomplish is:
>
>     1. Compilation cache in ~/.cache/emacs/eln,
>     2. No redundant recompilations at startup.
>
> > The load logic is, I think, find the elc file (or make it, if byte
> > compilation isn't inhibited) along load-path, and if found, probe
> > the eln cache and swap the eln, if found.
>
> I think so too, but there appears to be a race condition
> involved. First the compiler can't find foobar.eln on the
> native-comp-eln-load-path, so it schedules it for
> recompilation. Meanwhile, native-comp-eln-load-path is updated. Then
> the compiler writes foobar.eln to the *new path*.
>
> > What platform are you on and does your build have icons and cl-lib
> > in its pre-compiled directory?  Are you building Emacs without aot?
>
> I'm building Emacs with aot on Arch Linux. Recompilation happens in
> both Emacs 30.1 and 31.0.50. icons, cl-lib, and warnings are not in
> the system-wide /opt/emacs/lib/emacs/31.0.50/native-lisp/ directory.
>

If your build completed successfully, and your installation is correct, the
aot eln files for icons and cl-lib should be in the installed native
directory and found there along with the others.  Are they installed
correctly with correct file permissions?  What makes these two different
than the other aot files?

[Message part 2 (text/html, inline)]

Message #50 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Ship Mints <shipmints <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Tue, 22 Apr 2025 01:17:00 +0200

Hello again,

I've had some more time to debug now.

> If your build completed successfully, and your installation is
> correct, the aot eln files for icons and cl-lib should be in the
> installed native directory and found there along with the others.
> Are they installed correctly with correct file permissions?  What
> makes these two different than the other aot files?

If I build from source with --with-native-compilation=aot then those
modules are natively compiled and installed. Otherwise they are
not. In the Arch Linux package of Emacs that I was using they were
missing.

Afact, not natively compiling during compilation shouldn't cause any
issues. But it somehow does since cl-lib, icons, and warnings appear
to be loaded before early-init.el. So if you change the path the
native compiler uses for caching in early-init.el it will recompile
these files every time you start Emacs.

Btw, the --with-native-compilation option is a bit weird. Shouldn't
there be two options, one for whether you want to compile Emacs with
native compilation support and another one for whether you want to
precompile all Elisp modules?


-- 
mvh/best regards Björn Lindqvist

Message #53 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Tue, 22 Apr 2025 01:55:20 +0200

> We have a function for that, startup-redirect-eln-cache, and you are
> well advised to use that instead of constructing the list by hand.
> That function's doc string also says it's best to do that in
> early-init file, so that the list of directories is not changed
> halfway through the startup process, causing problems.

I tried doing exactly that. Here is the entirety of my early-init.el:

    (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))

warnings, icons, and cl-lib are still being recompiled every time
Emacs starts. So it doesn't matter how I modify the cache path, the
result is still redundant recompilation.

> > The first sexp of my init.el is
> >
> >     (setq user-emacs-directory "~/.config/emacs/lisp/")
> >
> > which is (?) executed *before* the native compiler kicks in.

> You cannot know that.  Emacs executes quite a bit of code before it
> reads the init file, see startup.el.

No, but the native compiler is running asynchronously and is delayed
in relation to the modules loaded at startup. My guess is that it
schedules those modules for recompilation that it cannot find in its
load path but then, crucially, writes them to the new path it won't
look through the next time Emacs starts.

> The manual describes what happens by default, not what happens when
> you tinker with user-emacs-directory and/or native-comp-eln-load-path
> during the startup.

The same thing happens whether I directly modify the load path
or use startup-redirect-eln-cache.

I cut off the rest of your message because, while your advice is
appreciated, it is not about solving the bug. We can argue whether the
.eln files are cache data or not, but it is irrelevant. For those
users who want the eln-cache to reside in ~/.cache/emacs it should be
possible to do so, without causing any ill side-effects.


-- 
mvh/best regards Björn Lindqvist

Message #56 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Tue, 22 Apr 2025 15:06:01 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Tue, 22 Apr 2025 01:17:00 +0200
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 77745 <at> debbugs.gnu.org
> 
> > If your build completed successfully, and your installation is
> > correct, the aot eln files for icons and cl-lib should be in the
> > installed native directory and found there along with the others.
> > Are they installed correctly with correct file permissions?  What
> > makes these two different than the other aot files?
> 
> If I build from source with --with-native-compilation=aot then those
> modules are natively compiled and installed. Otherwise they are
> not. In the Arch Linux package of Emacs that I was using they were
> missing.

This is expected.  Without "aot" only the preloaded Lisp files are
natively compiled as part of the build.

> Afact, not natively compiling during compilation shouldn't cause any
> issues. But it somehow does since cl-lib, icons, and warnings appear
> to be loaded before early-init.el.

I'm guessing that this happens because something in your init files
causes Emacs to emit a warning.  When that happens, Emacs loads
warnings.el and its prerequisites (cl-lib and icons), because the
warning commands are autoloaded.  If the corresponding Lisp files
don't have .eln files in the places where Emacs looks for them, Emacs
will launch async native compilation of those files.

> So if you change the path the native compiler uses for caching in
> early-init.el it will recompile these files every time you start
> Emacs.

Yes.

> Btw, the --with-native-compilation option is a bit weird. Shouldn't
> there be two options, one for whether you want to compile Emacs with
> native compilation support and another one for whether you want to
> precompile all Elisp modules?

Maybe I misunderstand you, but we already have that: there's
'--with-native-compilation' and '--with-native-compilation=aot'.

Message #59 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Tue, 22 Apr 2025 15:29:16 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Tue, 22 Apr 2025 01:55:20 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> > We have a function for that, startup-redirect-eln-cache, and you are
> > well advised to use that instead of constructing the list by hand.
> > That function's doc string also says it's best to do that in
> > early-init file, so that the list of directories is not changed
> > halfway through the startup process, causing problems.
> 
> I tried doing exactly that. Here is the entirety of my early-init.el:
> 
>     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> 
> warnings, icons, and cl-lib are still being recompiled every time
> Emacs starts. So it doesn't matter how I modify the cache path, the
> result is still redundant recompilation.

Why does Emacs need to emit some warnings during startup in your case?
That shouldn't normally happen.  If Emacs loads warnings.el, itr means
it needed to emit some warning.

> > > The first sexp of my init.el is
> > >
> > >     (setq user-emacs-directory "~/.config/emacs/lisp/")
> > >
> > > which is (?) executed *before* the native compiler kicks in.
> 
> > You cannot know that.  Emacs executes quite a bit of code before it
> > reads the init file, see startup.el.
> 
> No, but the native compiler is running asynchronously and is delayed
> in relation to the modules loaded at startup.

I don't understand what you mean by "delayed".  The async native
compilation is started as soon as some Lisp package is loaded which
has no .eln file that Emacs can find.  That compilation is performed
by a fresh Emacs session, which runs in batch mode and asynchronously
wrt your interactive session.  But the value of
native-comp-eln-load-path that the async compilation gets is the value
which was known to Emacs at the moment it started the async
compilation sub-process, and it doesn't change afterwards because
whatever changes in user-emacs-directory and other variables that the
interactive session does are not reflected to the compilation
sub-process.  And since the compilation sub-process runs with -Q, it
doesn't process your init files, so it uses the original unmodified
values of user-emacs-directory and native-comp-eln-load-path.

Does this explain what you see?

> My guess is that it schedules those modules for recompilation that
> it cannot find in its load path but then, crucially, writes them to
> the new path it won't look through the next time Emacs starts.

How many execution units does your CPU have? what does the function
num-processors return on that machine?

By default, Emacs will send up to half that number of async
compilations in one go.  So if num-processors returns, say, 8, Emacs
will start up to 4 compilations concurrently.  Thus, if loading
warnings.el requires to compile it and up to 3 additional packages,
they will all be compiled with the same value of
native-comp-eln-load-path.  If you have fewer CPUs and/or Emacs needs
to compile more than half that number of packages, then yes, different
compilations started at different times could use different values of
native-comp-eln-load-path if your init files change that halfway
through the startup process.

I hope you understand better now why changing values of
user-emacs-directory and native-comp-eln-load-path in the init files
is not the best idea.

> I cut off the rest of your message because, while your advice is
> appreciated, it is not about solving the bug.

What is the bug, in your opinion?

Message #62 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sun, 27 Apr 2025 21:52:59 +0200

> > If I build from source with --with-native-compilation=aot then those
> > modules are natively compiled and installed. Otherwise they are
> > not. In the Arch Linux package of Emacs that I was using they were
> > missing.
>
> This is expected.  Without "aot" only the preloaded Lisp files are
> natively compiled as part of the build.

What is preloaded Lisp?  Does the native compiler require that they be
AOT-compiled?

> > Afact, not natively compiling during compilation shouldn't cause any
> > issues. But it somehow does since cl-lib, icons, and warnings appear
> > to be loaded before early-init.el.
>
> I'm guessing that this happens because something in your init files
> causes Emacs to emit a warning.

The entirety of my early-init.el is:

    (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))

I can't see any warning messages in the logs. Why would this
sexp could cause a warning?

> > Btw, the --with-native-compilation option is a bit weird. Shouldn't
> > there be two options, one for whether you want to compile Emacs with
> > native compilation support and another one for whether you want to
> > precompile all Elisp modules?
>
> Maybe I misunderstand you, but we already have that: there's
> '--with-native-compilation' and '--with-native-compilation=aot'.

Yeah, but --with-native-compilation will eventually disappear since
when native compilation works well there is no need to build Emacs
without that feature. But you will still need an option for whether
you want to aot compile all Lisp files during the installation process
or not. I.e., the options are orthogonal.


--
mvh/best regards Björn Lindqvist

Message #65 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sun, 27 Apr 2025 22:32:50 +0200

> > I tried doing exactly that. Here is the entirety of my early-init.el:
> >
> >     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> >
> > warnings, icons, and cl-lib are still being recompiled every time
> > Emacs starts. So it doesn't matter how I modify the cache path, the
> > result is still redundant recompilation.
>
> Why does Emacs need to emit some warnings during startup in your case?
> That shouldn't normally happen.  If Emacs loads warnings.el, itr means
> it needed to emit some warning.

I don't know. In git Emacs I'm getting warnings like:

    Missing ‘lexical-binding’ cookie in "~/p/ConfigurationFiles/dot-emacs".

which I've never seen before. Emacs 30.1 does not emit lexical binding
cookie warnings, but the native compiler still recompiles the three
files I enumerated. Could it be that these three files are
gzip-compressed?

> > No, but the native compiler is running asynchronously and is delayed
> > in relation to the modules loaded at startup.
>
> I don't understand what you mean by "delayed".  The async native
> compilation is started as soon as some Lisp package is loaded which
> has no .eln file that Emacs can find.  That compilation is performed
> by a fresh Emacs session, which runs in batch mode and asynchronously
> wrt your interactive session.  But the value of
> native-comp-eln-load-path that the async compilation gets is the value
> which was known to Emacs at the moment it started the async
> compilation sub-process, and it doesn't change afterwards because
> whatever changes in user-emacs-directory and other variables that the
> interactive session does are not reflected to the compilation
> sub-process.  And since the compilation sub-process runs with -Q, it
> doesn't process your init files, so it uses the original unmodified
> values of user-emacs-directory and native-comp-eln-load-path.
>
> Does this explain what you see?

Yes and no. In particular, "it doesn't change afterwards because
whatever changes in user-emacs-directory and other variables that the
interactive session does, are not reflected to the compilation
sub-process", is, afaict, incorrect.

This is what I observe: Emacs starts and sees that there is no
warnings*.eln in either ~/.config/emacs/eln-cache/31.0.50/ or
$prefix/lib/emacs/31.0.50/native-lisp/ so it schedules warnings.el for
native compilation. Native compilation runs and then it writes the
warnings.eln file to ~/.cache/emacs/eln/31.0.50/.

So variables that the interactive session updates *must* be reflected
to the compilation subprocess. Otherwise it would have written
warnings.eln to the first entry of the original
native-comp-eln-load-path directory;
~/.config/emacs/eln-cache/31.0.50/

> > My guess is that it schedules those modules for recompilation that
> > it cannot find in its load path but then, crucially, writes them to
> > the new path it won't look through the next time Emacs starts.
>
> How many execution units does your CPU have? what does the function
> num-processors return on that machine?

16

> > I cut off the rest of your message because, while your advice is
> > appreciated, it is not about solving the bug.
>
> What is the bug, in your opinion?

Well, I think you agree that the path Emacs uses for the eln cache
should be user configurable? Some users want it in ~/.cache/emacs
others are fine with it in ~/.config/emacs. You should be able to set
it without causing redundant recompilation. I can think of two ways of
fixing this bug:

1. Ensure that the native compiler writes to the same directory it
reads from. E.g., it looked for warnings.eln in
~/.config/emacs/eln-cache and it will write to
~/.config/emacs/eln-cache, regardless of concurrent updates to
native-comp-eln-load-path (given what you wrote this seems to have
been the intention).

2. Ensure that early-init.el is evaluated before the eln cache is
accessed. This seems cleanest to me, but idk if it is practical.


-- 
mvh/best regards Björn Lindqvist

Message #68 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Mon, 28 Apr 2025 15:04:52 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Sun, 27 Apr 2025 21:52:59 +0200
> Cc: shipmints <at> gmail.com, 77745 <at> debbugs.gnu.org
> 
> > > If I build from source with --with-native-compilation=aot then those
> > > modules are natively compiled and installed. Otherwise they are
> > > not. In the Arch Linux package of Emacs that I was using they were
> > > missing.
> >
> > This is expected.  Without "aot" only the preloaded Lisp files are
> > natively compiled as part of the build.
> 
> What is preloaded Lisp?

The Lisp files we load in lisp/loadup.el, and which we then dump when
we build Emacs.  They are all the Lisp packages that are always
available in Emacs.

> Does the native compiler require that they be AOT-compiled?

Yes.  It isn't a requirement, it's just how building Emacs with
native-compilation works, because we decided it should work that way:
the preloaded Lisp packages are already native-compiled when Emacs is
dumped.

> > > Afact, not natively compiling during compilation shouldn't cause any
> > > issues. But it somehow does since cl-lib, icons, and warnings appear
> > > to be loaded before early-init.el.
> >
> > I'm guessing that this happens because something in your init files
> > causes Emacs to emit a warning.
> 
> The entirety of my early-init.el is:
> 
>     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> 
> I can't see any warning messages in the logs. Why would this
> sexp could cause a warning?

I don't know.  Indeed very strange.  But OTOH, why would Emacs
otherwise load warnings if it didn't need to display a warning?

> > > Btw, the --with-native-compilation option is a bit weird. Shouldn't
> > > there be two options, one for whether you want to compile Emacs with
> > > native compilation support and another one for whether you want to
> > > precompile all Elisp modules?
> >
> > Maybe I misunderstand you, but we already have that: there's
> > '--with-native-compilation' and '--with-native-compilation=aot'.
> 
> Yeah, but --with-native-compilation will eventually disappear since
> when native compilation works well there is no need to build Emacs
> without that feature.

No, it won't disappear.  We just changed it to be ON by default, but
we still support --with-native-compilation=no, and that will probably
stay supported forever.

> But you will still need an option for whether you want to aot
> compile all Lisp files during the installation process or not. I.e.,
> the options are orthogonal.

I think you lost me here.

The --with-native-compilation option (which is ON by default in EMacs
30 and later) means we natively compile only the Lisp files that
loadup.el loads.  The --with-native-compilation=aot option, which is
by default OFF, means all the rest of Lisp files are _also_ natively
compiled, and the installed Emacs will then include their *.eln files.
There's no option top build Emacs that is capable of native
compilation, but without natively-compiling at least the preloaded
files, if that what you wanted, because we didn't see how that would
make sense.

Message #71 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Mon, 28 Apr 2025 15:16:25 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Sun, 27 Apr 2025 22:32:50 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> > > I tried doing exactly that. Here is the entirety of my early-init.el:
> > >
> > >     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> > >
> > > warnings, icons, and cl-lib are still being recompiled every time
> > > Emacs starts. So it doesn't matter how I modify the cache path, the
> > > result is still redundant recompilation.
> >
> > Why does Emacs need to emit some warnings during startup in your case?
> > That shouldn't normally happen.  If Emacs loads warnings.el, itr means
> > it needed to emit some warning.
> 
> I don't know. In git Emacs I'm getting warnings like:
> 
>     Missing ‘lexical-binding’ cookie in "~/p/ConfigurationFiles/dot-emacs".
> 
> which I've never seen before. Emacs 30.1 does not emit lexical binding
> cookie warnings, but the native compiler still recompiles the three
> files I enumerated. Could it be that these three files are
> gzip-compressed?

No, I think it's a known issue with Emacs 31 which was recently (as in
"yesterday") fixed.  So maybe update from Git and see if the problem
goes away.

(The warning about the lack of lexical-binding cookie is new in Emacs
31.)

> > I don't understand what you mean by "delayed".  The async native
> > compilation is started as soon as some Lisp package is loaded which
> > has no .eln file that Emacs can find.  That compilation is performed
> > by a fresh Emacs session, which runs in batch mode and asynchronously
> > wrt your interactive session.  But the value of
> > native-comp-eln-load-path that the async compilation gets is the value
> > which was known to Emacs at the moment it started the async
> > compilation sub-process, and it doesn't change afterwards because
> > whatever changes in user-emacs-directory and other variables that the
> > interactive session does are not reflected to the compilation
> > sub-process.  And since the compilation sub-process runs with -Q, it
> > doesn't process your init files, so it uses the original unmodified
> > values of user-emacs-directory and native-comp-eln-load-path.
> >
> > Does this explain what you see?
> 
> Yes and no. In particular, "it doesn't change afterwards because
> whatever changes in user-emacs-directory and other variables that the
> interactive session does, are not reflected to the compilation
> sub-process", is, afaict, incorrect.
> 
> This is what I observe: Emacs starts and sees that there is no
> warnings*.eln in either ~/.config/emacs/eln-cache/31.0.50/ or
> $prefix/lib/emacs/31.0.50/native-lisp/ so it schedules warnings.el for
> native compilation. Native compilation runs and then it writes the
> warnings.eln file to ~/.cache/emacs/eln/31.0.50/.
> 
> So variables that the interactive session updates *must* be reflected
> to the compilation subprocess. Otherwise it would have written
> warnings.eln to the first entry of the original
> native-comp-eln-load-path directory;
> ~/.config/emacs/eln-cache/31.0.50/

So either I'm missing something, or the particular timings of this
async compilation in your case is such that it succeeds to notice the
changes in the variables which affect native-comp-eln-load-path.

> > What is the bug, in your opinion?
> 
> Well, I think you agree that the path Emacs uses for the eln cache
> should be user configurable?

Sometimes, yes.  It's an unusual thing to do.

More importantly, when modifying the value, one should do it so as not
to disrupt the assumptions of the native compiler.  For example, you
should generally prepend new directories to the existing ones, instead
of replacing them on the fly.

> Some users want it in ~/.cache/emacs others are fine with it in
> ~/.config/emacs.

Actually, most users don't (and should not) care.

> You should be able to set it without causing redundant
> recompilation.

I don't necessarily agree.  What you call "redundant recompilation" is
not a significant problem IMO, it's just a minor nuisance.  It doesn't
interfere with correct operation, for example.

> I can think of two ways of fixing this bug:
> 
> 1. Ensure that the native compiler writes to the same directory it
> reads from. E.g., it looked for warnings.eln in
> ~/.config/emacs/eln-cache and it will write to
> ~/.config/emacs/eln-cache, regardless of concurrent updates to
> native-comp-eln-load-path (given what you wrote this seems to have
> been the intention).

This is impossible because native-comp-eln-load-path is a list of
directories, not a single directory, and Emacs looks in all of them
one by one before it decides that it needs to compile.

> 2. Ensure that early-init.el is evaluated before the eln cache is
> accessed. This seems cleanest to me, but idk if it is practical.

This is generally what happens, but if there's some contingency, like
a warning triggered by the act of loading early-init.el itself, it
becomes a chicken-and-egg problem, which has no solution.

Message #76 received at 77745-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: bjourne <at> gmail.com
Cc: 77745-done <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 10 May 2025 12:49:39 +0300

> Cc: 77745 <at> debbugs.gnu.org
> Date: Mon, 28 Apr 2025 15:16:25 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Björn Lindqvist <bjourne <at> gmail.com>
> > Date: Sun, 27 Apr 2025 22:32:50 +0200
> > Cc: 77745 <at> debbugs.gnu.org
> > 
> > > > I tried doing exactly that. Here is the entirety of my early-init.el:
> > > >
> > > >     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> > > >
> > > > warnings, icons, and cl-lib are still being recompiled every time
> > > > Emacs starts. So it doesn't matter how I modify the cache path, the
> > > > result is still redundant recompilation.
> > >
> > > Why does Emacs need to emit some warnings during startup in your case?
> > > That shouldn't normally happen.  If Emacs loads warnings.el, itr means
> > > it needed to emit some warning.
> > 
> > I don't know. In git Emacs I'm getting warnings like:
> > 
> >     Missing ‘lexical-binding’ cookie in "~/p/ConfigurationFiles/dot-emacs".
> > 
> > which I've never seen before. Emacs 30.1 does not emit lexical binding
> > cookie warnings, but the native compiler still recompiles the three
> > files I enumerated. Could it be that these three files are
> > gzip-compressed?
> 
> No, I think it's a known issue with Emacs 31 which was recently (as in
> "yesterday") fixed.  So maybe update from Git and see if the problem
> goes away.
> 
> (The warning about the lack of lexical-binding cookie is new in Emacs
> 31.)
> 
> > > I don't understand what you mean by "delayed".  The async native
> > > compilation is started as soon as some Lisp package is loaded which
> > > has no .eln file that Emacs can find.  That compilation is performed
> > > by a fresh Emacs session, which runs in batch mode and asynchronously
> > > wrt your interactive session.  But the value of
> > > native-comp-eln-load-path that the async compilation gets is the value
> > > which was known to Emacs at the moment it started the async
> > > compilation sub-process, and it doesn't change afterwards because
> > > whatever changes in user-emacs-directory and other variables that the
> > > interactive session does are not reflected to the compilation
> > > sub-process.  And since the compilation sub-process runs with -Q, it
> > > doesn't process your init files, so it uses the original unmodified
> > > values of user-emacs-directory and native-comp-eln-load-path.
> > >
> > > Does this explain what you see?
> > 
> > Yes and no. In particular, "it doesn't change afterwards because
> > whatever changes in user-emacs-directory and other variables that the
> > interactive session does, are not reflected to the compilation
> > sub-process", is, afaict, incorrect.
> > 
> > This is what I observe: Emacs starts and sees that there is no
> > warnings*.eln in either ~/.config/emacs/eln-cache/31.0.50/ or
> > $prefix/lib/emacs/31.0.50/native-lisp/ so it schedules warnings.el for
> > native compilation. Native compilation runs and then it writes the
> > warnings.eln file to ~/.cache/emacs/eln/31.0.50/.
> > 
> > So variables that the interactive session updates *must* be reflected
> > to the compilation subprocess. Otherwise it would have written
> > warnings.eln to the first entry of the original
> > native-comp-eln-load-path directory;
> > ~/.config/emacs/eln-cache/31.0.50/
> 
> So either I'm missing something, or the particular timings of this
> async compilation in your case is such that it succeeds to notice the
> changes in the variables which affect native-comp-eln-load-path.
> 
> > > What is the bug, in your opinion?
> > 
> > Well, I think you agree that the path Emacs uses for the eln cache
> > should be user configurable?
> 
> Sometimes, yes.  It's an unusual thing to do.
> 
> More importantly, when modifying the value, one should do it so as not
> to disrupt the assumptions of the native compiler.  For example, you
> should generally prepend new directories to the existing ones, instead
> of replacing them on the fly.
> 
> > Some users want it in ~/.cache/emacs others are fine with it in
> > ~/.config/emacs.
> 
> Actually, most users don't (and should not) care.
> 
> > You should be able to set it without causing redundant
> > recompilation.
> 
> I don't necessarily agree.  What you call "redundant recompilation" is
> not a significant problem IMO, it's just a minor nuisance.  It doesn't
> interfere with correct operation, for example.
> 
> > I can think of two ways of fixing this bug:
> > 
> > 1. Ensure that the native compiler writes to the same directory it
> > reads from. E.g., it looked for warnings.eln in
> > ~/.config/emacs/eln-cache and it will write to
> > ~/.config/emacs/eln-cache, regardless of concurrent updates to
> > native-comp-eln-load-path (given what you wrote this seems to have
> > been the intention).
> 
> This is impossible because native-comp-eln-load-path is a list of
> directories, not a single directory, and Emacs looks in all of them
> one by one before it decides that it needs to compile.
> 
> > 2. Ensure that early-init.el is evaluated before the eln cache is
> > accessed. This seems cleanest to me, but idk if it is practical.
> 
> This is generally what happens, but if there's some contingency, like
> a warning triggered by the act of loading early-init.el itself, it
> becomes a chicken-and-egg problem, which has no solution.

No further comments, so I presume that the issues are now clear, and
I'm therefore closing this bug.

Message #79 received at 77745-done <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745-done <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Sat, 10 May 2025 21:16:54 +0200

It's not. But i haven't had time to reply.

Den lör 10 maj 2025 kl 11:49 skrev Eli Zaretskii <eliz <at> gnu.org>:
>
> > Cc: 77745 <at> debbugs.gnu.org
> > Date: Mon, 28 Apr 2025 15:16:25 +0300
> > From: Eli Zaretskii <eliz <at> gnu.org>
> >
> > > From: Björn Lindqvist <bjourne <at> gmail.com>
> > > Date: Sun, 27 Apr 2025 22:32:50 +0200
> > > Cc: 77745 <at> debbugs.gnu.org
> > >
> > > > > I tried doing exactly that. Here is the entirety of my early-init.el:
> > > > >
> > > > >     (startup-redirect-eln-cache (expand-file-name "~/.cache/emacs/eln"))
> > > > >
> > > > > warnings, icons, and cl-lib are still being recompiled every time
> > > > > Emacs starts. So it doesn't matter how I modify the cache path, the
> > > > > result is still redundant recompilation.
> > > >
> > > > Why does Emacs need to emit some warnings during startup in your case?
> > > > That shouldn't normally happen.  If Emacs loads warnings.el, itr means
> > > > it needed to emit some warning.
> > >
> > > I don't know. In git Emacs I'm getting warnings like:
> > >
> > >     Missing ‘lexical-binding’ cookie in "~/p/ConfigurationFiles/dot-emacs".
> > >
> > > which I've never seen before. Emacs 30.1 does not emit lexical binding
> > > cookie warnings, but the native compiler still recompiles the three
> > > files I enumerated. Could it be that these three files are
> > > gzip-compressed?
> >
> > No, I think it's a known issue with Emacs 31 which was recently (as in
> > "yesterday") fixed.  So maybe update from Git and see if the problem
> > goes away.
> >
> > (The warning about the lack of lexical-binding cookie is new in Emacs
> > 31.)
> >
> > > > I don't understand what you mean by "delayed".  The async native
> > > > compilation is started as soon as some Lisp package is loaded which
> > > > has no .eln file that Emacs can find.  That compilation is performed
> > > > by a fresh Emacs session, which runs in batch mode and asynchronously
> > > > wrt your interactive session.  But the value of
> > > > native-comp-eln-load-path that the async compilation gets is the value
> > > > which was known to Emacs at the moment it started the async
> > > > compilation sub-process, and it doesn't change afterwards because
> > > > whatever changes in user-emacs-directory and other variables that the
> > > > interactive session does are not reflected to the compilation
> > > > sub-process.  And since the compilation sub-process runs with -Q, it
> > > > doesn't process your init files, so it uses the original unmodified
> > > > values of user-emacs-directory and native-comp-eln-load-path.
> > > >
> > > > Does this explain what you see?
> > >
> > > Yes and no. In particular, "it doesn't change afterwards because
> > > whatever changes in user-emacs-directory and other variables that the
> > > interactive session does, are not reflected to the compilation
> > > sub-process", is, afaict, incorrect.
> > >
> > > This is what I observe: Emacs starts and sees that there is no
> > > warnings*.eln in either ~/.config/emacs/eln-cache/31.0.50/ or
> > > $prefix/lib/emacs/31.0.50/native-lisp/ so it schedules warnings.el for
> > > native compilation. Native compilation runs and then it writes the
> > > warnings.eln file to ~/.cache/emacs/eln/31.0.50/.
> > >
> > > So variables that the interactive session updates *must* be reflected
> > > to the compilation subprocess. Otherwise it would have written
> > > warnings.eln to the first entry of the original
> > > native-comp-eln-load-path directory;
> > > ~/.config/emacs/eln-cache/31.0.50/
> >
> > So either I'm missing something, or the particular timings of this
> > async compilation in your case is such that it succeeds to notice the
> > changes in the variables which affect native-comp-eln-load-path.
> >
> > > > What is the bug, in your opinion?
> > >
> > > Well, I think you agree that the path Emacs uses for the eln cache
> > > should be user configurable?
> >
> > Sometimes, yes.  It's an unusual thing to do.
> >
> > More importantly, when modifying the value, one should do it so as not
> > to disrupt the assumptions of the native compiler.  For example, you
> > should generally prepend new directories to the existing ones, instead
> > of replacing them on the fly.
> >
> > > Some users want it in ~/.cache/emacs others are fine with it in
> > > ~/.config/emacs.
> >
> > Actually, most users don't (and should not) care.
> >
> > > You should be able to set it without causing redundant
> > > recompilation.
> >
> > I don't necessarily agree.  What you call "redundant recompilation" is
> > not a significant problem IMO, it's just a minor nuisance.  It doesn't
> > interfere with correct operation, for example.
> >
> > > I can think of two ways of fixing this bug:
> > >
> > > 1. Ensure that the native compiler writes to the same directory it
> > > reads from. E.g., it looked for warnings.eln in
> > > ~/.config/emacs/eln-cache and it will write to
> > > ~/.config/emacs/eln-cache, regardless of concurrent updates to
> > > native-comp-eln-load-path (given what you wrote this seems to have
> > > been the intention).
> >
> > This is impossible because native-comp-eln-load-path is a list of
> > directories, not a single directory, and Emacs looks in all of them
> > one by one before it decides that it needs to compile.
> >
> > > 2. Ensure that early-init.el is evaluated before the eln cache is
> > > accessed. This seems cleanest to me, but idk if it is practical.
> >
> > This is generally what happens, but if there's some contingency, like
> > a warning triggered by the act of loading early-init.el itself, it
> > becomes a chicken-and-egg problem, which has no solution.
>
> No further comments, so I presume that the issues are now clear, and
> I'm therefore closing this bug.



-- 
mvh/best regards Björn Lindqvist

Message #82 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Björn Lindqvist <bjourne <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Thu, 15 May 2025 11:03:35 +0200

Hello again and sorry for the delay,

> > Does the native compiler require that they be AOT-compiled?

> Yes.  It isn't a requirement, it's just how building Emacs with
> native-compilation works, because we decided it should work that
> way: the preloaded Lisp packages are already native-compiled when
> Emacs is dumped. [...]

> There's no option top build Emacs that is capable of native
> compilation, but without natively-compiling at least the preloaded
> files, if that what you wanted, because we didn't see how that would
> make sense.

So the difference between --with-native-compilation=aot and
--with-native-compilation=yes is that the former AOT-compiles all lisp
files and the latter only a small subset? Wouldn't it make more sense
for --with-native-compilation=yes to not AOT-compile anything? My
preferred setup is to have all .eln files in ~/.cache/emacs/eln. Then
I could occasionally blow up the cache and not have to worry about
.eln files in multiple locations.

> > which I've never seen before. Emacs 30.1 does not emit lexical
> > binding cookie warnings, but the native compiler still recompiles
> > the three files I enumerated. Could it be that these three files
> > are gzip-compressed?

> No, I think it's a known issue with Emacs 31 which was recently (as
> in "yesterday") fixed.  So maybe update from Git and see if the
> problem goes away.

The problem persists even with a fresh checkout of Emacs built today.

> > Well, I think you agree that the path Emacs uses for the eln cache
> > should be user configurable?

> Sometimes, yes.  It's an unusual thing to do. [...]

> > Some users want it in ~/.cache/emacs others are fine with it in
> > ~/.config/emacs.

> Actually, most users don't (and should not) care.

Maybe it is unusual because native compilation is a recent feature
that not everyone knows about? In general I don't think changing the
paths where Emacs stores ephemeral data is unusual. Not being able to
customize the eln cache paths seem to fly in the face of most other
Emacs features.

I think you should decide:

* Changing the eln cache paths is not supported. Then the manual
  should be updated to not give the opposite impression.

* Changing the cache paths is supported. Then users should not
  encounter strange problems (such as modules being recompiled
  every time at startup) when changing the eln cache paths.

Anything in between is dissatisfying.

> > 1. Ensure that the native compiler writes to the same directory it
> > reads from. E.g., it looked for warnings.eln in
> > ~/.config/emacs/eln-cache and it will write to
> > ~/.config/emacs/eln-cache, regardless of concurrent updates to
> > native-comp-eln-load-path (given what you wrote this seems to have
> > been the intention).

> This is impossible because native-comp-eln-load-path is a list of
> directories, not a single directory, and Emacs looks in all of them
> one by one before it decides that it needs to compile.

It only writes to one of the directories. So at the same time it
decides to compile an eln file it could also decide where to store
it. The parent process/scheduler could tell the worker "please compile
foo.el and store the result in ~/some/directory/foo.eln".

> This is generally what happens, but if there's some contingency,
> like a warning triggered by the act of loading early-init.el itself,
> it becomes a chicken-and-egg problem, which has no solution.

Ah, yes, it is a loop. The system has to be loaded to read the
configuration to figure out how the system should be loaded. My
suggestion is to break the cycle like how bootstrapping compilers does
it. Have an interpreter/diminished version read the
configuration in an prologue phase before the full system is
loaded. I.e., run early-init.el (or some other initialization module)
in "interpreted mode" and defer native compilation until after the
module has run.

Another option could be an environment variable (ELN_CACHE_WRITE_PATH)
or command-line switch. But that is quite ugly.


--
mvh/best regards Björn Lindqvist

Message #85 received at 77745 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Björn Lindqvist <bjourne <at> gmail.com>
Cc: 77745 <at> debbugs.gnu.org
Subject: Re: bug#77745: Async native compilation runs every time I start Emacs
Date: Thu, 15 May 2025 15:29:22 +0300

> From: Björn Lindqvist <bjourne <at> gmail.com>
> Date: Thu, 15 May 2025 11:03:35 +0200
> Cc: 77745 <at> debbugs.gnu.org
> 
> > > Does the native compiler require that they be AOT-compiled?
> 
> > Yes.  It isn't a requirement, it's just how building Emacs with
> > native-compilation works, because we decided it should work that
> > way: the preloaded Lisp packages are already native-compiled when
> > Emacs is dumped. [...]
> 
> > There's no option top build Emacs that is capable of native
> > compilation, but without natively-compiling at least the preloaded
> > files, if that what you wanted, because we didn't see how that would
> > make sense.
> 
> So the difference between --with-native-compilation=aot and
> --with-native-compilation=yes is that the former AOT-compiles all lisp
> files and the latter only a small subset?

Yes.

> Wouldn't it make more sense for --with-native-compilation=yes to not
> AOT-compile anything?

That'd just cause them to be natively-compiled at the first
invocation, so it will have no effect except making the first
invocation slower and more resource-consuming.  I don't see any sense
in that.

> My preferred setup is to have all .eln files in
> ~/.cache/emacs/eln. Then I could occasionally blow up the cache and
> not have to worry about .eln files in multiple locations.

The other locations are all in the versioned subdirectories of
/usr/lib/emacs/, so I don't see how that would be a "worry".  When you
stop using some version of Emacs and remove it from the system, just
remove its versioned directory under /usr/lib/emacs/as well.  Like you
probably do with subdirectories of /usr/share/emacs/.

> > > which I've never seen before. Emacs 30.1 does not emit lexical
> > > binding cookie warnings, but the native compiler still recompiles
> > > the three files I enumerated. Could it be that these three files
> > > are gzip-compressed?
> 
> > No, I think it's a known issue with Emacs 31 which was recently (as
> > in "yesterday") fixed.  So maybe update from Git and see if the
> > problem goes away.
> 
> The problem persists even with a fresh checkout of Emacs built today.

Of which branch?  If that's the master branch, then please report a
separate bug with a recipe that starts with "emacs -Q".  People who
reported the problem I had in mind said it was fixed, so maybe yours
is a different one.

> > > Well, I think you agree that the path Emacs uses for the eln cache
> > > should be user configurable?
> 
> > Sometimes, yes.  It's an unusual thing to do. [...]
> 
> > > Some users want it in ~/.cache/emacs others are fine with it in
> > > ~/.config/emacs.
> 
> > Actually, most users don't (and should not) care.
> 
> Maybe it is unusual because native compilation is a recent feature
> that not everyone knows about?

It isn't very recent.  It was introduced in Emacs 28, two major
releases ago, and became the default in Emacs 30, released 3 months
ago.  I'd expect enough people to know about that by now, especially
that I regularly see bug reports from Emacs built with
native-compilation support.

> In general I don't think changing the paths where Emacs stores
> ephemeral data is unusual.

Not if Emacs needs to read their files during startup.  In that case,
the change should happen as early as possible, and we have
command-line options, environment variables, and the early-init file
for such purposes.

> I think you should decide:
> 
> * Changing the eln cache paths is not supported. Then the manual
>   should be updated to not give the opposite impression.
> 
> * Changing the cache paths is supported. Then users should not
>   encounter strange problems (such as modules being recompiled
>   every time at startup) when changing the eln cache paths.
> 
> Anything in between is dissatisfying.

Maybe so, but we already have several variables whose changes in
arbitrary stages of the startup could have unexpected results.  So
yes, we support redirections of the eln cache, but it should be done
with extra care, and preferably not at all.

> > > 1. Ensure that the native compiler writes to the same directory it
> > > reads from. E.g., it looked for warnings.eln in
> > > ~/.config/emacs/eln-cache and it will write to
> > > ~/.config/emacs/eln-cache, regardless of concurrent updates to
> > > native-comp-eln-load-path (given what you wrote this seems to have
> > > been the intention).
> 
> > This is impossible because native-comp-eln-load-path is a list of
> > directories, not a single directory, and Emacs looks in all of them
> > one by one before it decides that it needs to compile.
> 
> It only writes to one of the directories.

If the list in native-comp-eln-load-path changes during the session,
some files will be written to one directory and others to elsewhere.

> So at the same time it
> decides to compile an eln file it could also decide where to store
> it. The parent process/scheduler could tell the worker "please compile
> foo.el and store the result in ~/some/directory/foo.eln".

That's what Emacs does.  But your suggestion seems to be based on an
incorrect assumption: Emacs doesn't read from a directory to which it
writes the *.eln files.  Instead, it searches _all_ of the directories
in native-comp-eln-load-path one by one, and only decides to produce a
file if it was not found in any of them.  When it does decide to
produce a .eln file, it writes it to the first writable directory it
finds in that list.

If your suggestion is to write the *.eln files to the directories
where Emacs finds the corresponding *.el files, then this will not
work with a typical Emacs installation, where the *.el files which
came with Emacs are installed into a directory that is not generally
writable by users.  In addition, the *.eln files need to be in a
subdirectory whose name identifies the Emacs binary which can safely
load them (because loading incompatible machine code could crash
Emacs), so placing them where we keep Lisp source files would prevent
people from running several Emacs binaries on the same system,
something that we always supported and would want to continue
supporting even with native-compilation.

> > This is generally what happens, but if there's some contingency,
> > like a warning triggered by the act of loading early-init.el itself,
> > it becomes a chicken-and-egg problem, which has no solution.
> 
> Ah, yes, it is a loop. The system has to be loaded to read the
> configuration to figure out how the system should be loaded. My
> suggestion is to break the cycle like how bootstrapping compilers does
> it. Have an interpreter/diminished version read the
> configuration in an prologue phase before the full system is
> loaded. I.e., run early-init.el (or some other initialization module)
> in "interpreted mode" and defer native compilation until after the
> module has run.

You are talking about completely redesigning how Emacs starts up.  I
see no reason to throw away all of our startup code and reimplement it
anew, certainly not for this quite obscure use case.  It took us many
years to get where we are today; starting from scratch would not be
wise.

> Another option could be an environment variable (ELN_CACHE_WRITE_PATH)
> or command-line switch. But that is quite ugly.

Besides being ugly (and problematic, e.g. if you want to run one
version of Emacs as a subprocess of another), I don't quite see how it
would solve the problem in your case.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.