GNU bug report logs - #55305
28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 55305 in the body.
You can then email your comments to 55305 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rsw <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50: With async nativecomp, package manager fails to load
 hyperbole-autoloads.el before compilation
Date: Sat, 7 May 2022 16:05:17 -0400

[Message part 1 (text/plain, inline)]

Tested under Emacs 28.1 and a recent tip of the Emacs git repo for Emacs
29 with asynchronous native compilation enabled:

M-x package-install RET hyperbole RET

fails to load the hyperbole-autoloads.el file before the
async native compiler and byte compiler produce these errors since
the autoloaded var:append function is not defined:

Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-em-but.el: Error:
Symbol's function definition is void var:append Disable showing Disable
logging
Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-mouse.el: Error:
Symbol's function definition is void var:append Disable showing Disable
logging
Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hbut.el: Error: Symbol's
function definition is void var:append Disable showing Disable logging

The package manager definitely generates hyperbole-autoloads.el at some
point though I do not know if it is before these errors are produced.

Thanks for any help with this.

-- Bob

[Message part 2 (text/html, inline)]

Message #8 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rswgnu <at> gmail.com
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Sun, 08 May 2022 08:09:28 +0300

> From: Robert Weiner <rsw <at> gnu.org>
> Date: Sat, 7 May 2022 16:05:17 -0400
> 
> Tested under Emacs 28.1 and a recent tip of the Emacs git repo for Emacs
> 29 with asynchronous native compilation enabled:
> 
> M-x package-install RET hyperbole RET
> 
> fails to load the hyperbole-autoloads.el file before the
> async native compiler and byte compiler produce these errors since
> the autoloaded var:append function is not defined:

Hyperbole is not part of Emacs, so this problem should first be taken
up with the Hyperbole developers.

> Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-em-but.el: Error: Symbol's function definition is void
> var:append Disable showing Disable logging
> Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-mouse.el: Error: Symbol's function definition is void
> var:append Disable showing Disable logging
> Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hbut.el: Error: Symbol's function definition is void
> var:append Disable showing Disable logging
> 
> The package manager definitely generates hyperbole-autoloads.el at some
> point though I do not know if it is before these errors are produced.

Do the *.el files that produce the error 'require' or 'load'
hyperbole-autoloads?  If not, how would compilation know to load that
file?  Asynchronous native compilation runs in a separate pristine
Emacs session, so it needs every dependency explicitly spelled out, or
it will fail.

But again, I think this is a matter for the Hyperbole developers to
look into first.

Thanks.

Message #13 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rsw <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Thu, 12 May 2022 01:14:58 -0400

[Message part 1 (text/plain, inline)]

On Sun, May 8, 2022 at 1:09 AM Eli Zaretskii <eliz <at> gnu.org> wrote:

> > From: Robert Weiner <rsw <at> gnu.org>
> > Date: Sat, 7 May 2022 16:05:17 -0400
> >
> > Tested under Emacs 28.1 and a recent tip of the Emacs git repo for Emacs
> > 29 with asynchronous native compilation enabled:
> >
> > M-x package-install RET hyperbole RET
> >
> > fails to load the hyperbole-autoloads.el file before the
> > async native compiler and byte compiler produce these errors since
> > the autoloaded var:append function is not defined:
>
> Hyperbole is not part of Emacs, so this problem should first be taken
> up with the Hyperbole developers.
>

Hi Eli:

Thanks for the response.  Two initial points:

1. I am the lead Hyperbole developer, so I will discuss it with myself :-)

2. I long ago was told that Elpa packages are considered part of Emacs
although they don't ship with it; this is why you see a number of Hyperbole
issues brought up on this list.  It would be good if you and other core
Emacs developers all held the same yes/no opinion on this so others could
follow whatever is decided.


> > Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-em-but.el: Error:
> Symbol's function definition is void
> > var:append Disable showing Disable logging
> > Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-mouse.el: Error:
> Symbol's function definition is void
> > var:append Disable showing Disable logging
> > Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hbut.el: Error: Symbol's
> function definition is void
> > var:append Disable showing Disable logging
> >
> > The package manager definitely generates hyperbole-autoloads.el at some
> > point though I do not know if it is before these errors are produced.
>
> Do the *.el files that produce the error 'require' or 'load'
> hyperbole-autoloads?


No, the Emacs package manager installation process generates that file from
the ;;;###autoload annotations in Hyperbole lisp files (see the
'package--make-autoloads-and-stuff' function in "package.el").  The package
manager also sets up at package activation time to have those autoloads
loaded prior to loading any other code from the Hyperbole package.  It
would not make sense to require hyperbole-autoloads in other Lisp files
since that file does not exist when the package distribution is generated
and cannot be referenced prior to making a release; it would also defeat
the purpose of autoloads.

>   If not, how would compilation know to load that
> file?  Asynchronous native compilation runs in a separate pristine
> Emacs session, so it needs every dependency explicitly spelled out, or
> it will fail.
>

This is the issue that I am bringing up that I am surprised does not affect
or has not been reported for other packages.  There needs to be a
persistent, cross-session Emacs hook that runs prior to native compilation
of packages that loads the autoloads file for the package.


> But again, I think this is a matter for the Hyperbole developers to
> look into first.
>

I have resolved this for Hyperbole in an upcoming pre-release by moving a
number of previously autoloaded definitions into a file that can be
'required' by each Hyperbole module instead.  This was a good bit of work
and does not address the more general problem.  I hope you and others will
consider this a bit more and look into how it can be resolved for the good
of all packages, as it seems to be a disconnect between the Emacs package
manager and the native compilation code.

Best regards,

Bob

[Message part 2 (text/html, inline)]

Message #16 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rswgnu <at> gmail.com
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Thu, 12 May 2022 08:51:45 +0300

> From: Robert Weiner <rsw <at> gnu.org>
> Date: Thu, 12 May 2022 01:14:58 -0400
> Cc: 55305 <at> debbugs.gnu.org
> 
>  Do the *.el files that produce the error 'require' or 'load'
>  hyperbole-autoloads?
> 
> No, the Emacs package manager installation process generates that file from the ;;;###autoload annotations
> in Hyperbole lisp files (see the 'package--make-autoloads-and-stuff' function in "package.el").  The package
> manager also sets up at package activation time to have those autoloads loaded prior to loading any other
> code from the Hyperbole package.

Does this last fact mean there's an assumption in Hyperbole that the
package is always activated before its *.el files are compiled?  If
so, perhaps this is why it fails during native-compilation, where the
package is not activated prior to the compilation?

And why would such an assumption, if it exist, make sense?  It seems
to me like the ability to compile a .el file should require activation
of any package, or what am I missing?

Message #19 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rsw <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Thu, 12 May 2022 02:21:37 -0400

[Message part 1 (text/plain, inline)]

On Thu, May 12, 2022 at 1:51 AM Eli Zaretskii <eliz <at> gnu.org> wrote:

> > From: Robert Weiner <rsw <at> gnu.org>
> > Date: Thu, 12 May 2022 01:14:58 -0400
> > Cc: 55305 <at> debbugs.gnu.org
> >
> >  Do the *.el files that produce the error 'require' or 'load'
> >  hyperbole-autoloads?
> >
> > No, the Emacs package manager installation process generates that file
> from the ;;;###autoload annotations
> > in Hyperbole lisp files (see the 'package--make-autoloads-and-stuff'
> function in "package.el").  The package
> > manager also sets up at package activation time to have those autoloads
> loaded prior to loading any other
> > code from the Hyperbole package.
>
> Does this last fact mean there's an assumption in Hyperbole that the
> package is always activated before its *.el files are compiled?  If
> so, perhaps this is why it fails during native-compilation, where the
> package is not activated prior to the compilation?
>

Said another way, there is an assumption that the hyperbole-autoloads.el
file is loaded prior to any compilation, yes.  This is similar to
assumptions that loaddefs.el are loaded prior to their reference in other
Emacs Lisp files.  The point of the autoloads file is to include
definitions that must exist in the Lisp environment prior to their
reference in any Lisp files, whether this is during package use or package
build-time.


> And why would such an assumption, if it exist, make sense?  It seems
> to me like the ability to compile a .el file should require activation
> of any package, or what am I missing?
>

I am not saying that the package must be activated prior to compilation but
just that there must be an additional hook provided that forces loading of
the autoloads prior to any build/compilation of the package (whether byte
compilation or native compilation).  Otherwise the build process will
generate errors because the autoload definitions will not exist, e.g. maybe
an autoloaded variable meant to be global to the package is referenced at
the top-level of a package Lisp file.  Complex packages have complex
dependencies that I would say cannot all be handled with requires;
otherwise, there would be no need for the autoload mechanism.  Or am I
missing something?

Bob

[Message part 2 (text/html, inline)]

Message #22 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rswgnu <at> gmail.com
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Thu, 12 May 2022 10:22:47 +0300

> From: Robert Weiner <rsw <at> gnu.org>
> Date: Thu, 12 May 2022 02:21:37 -0400
> Cc: 55305 <at> debbugs.gnu.org
> 
>  Does this last fact mean there's an assumption in Hyperbole that the
>  package is always activated before its *.el files are compiled?  If
>  so, perhaps this is why it fails during native-compilation, where the
>  package is not activated prior to the compilation?
> 
> Said another way, there is an assumption that the hyperbole-autoloads.el file is loaded prior to any
> compilation, yes.  This is similar to assumptions that loaddefs.el are loaded prior to their reference in other
> Emacs Lisp files.

loaddefs.el is preloaded into Emacs when it is built, so the analogy
doesn't work in practice.

I think you should look at bundled packages like Calc.  Calc has
calc-loaddefs.el, but I just now forced Emacs 28.1 to native-compile
Calc and didn't see any problems.  And I see that calc.el does say
explicitly

  ;;;; (Autoloads here)
  (load "calc-loaddefs.el" nil t)

> The point of the autoloads file is to include definitions that must exist in the Lisp
> environment prior to their reference in any Lisp files, whether this is during package use or package
> build-time.

That is true, but AFAIU packages that have their own separate
autoloads file should proactively do something to make sure those
autoloads are loaded before they are needed.

And this is not related to native-compilation in any way: the same
will happen if one tries to byte-compile Hyperbole files without first
loading its autoloads.  Right?

Message #25 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rsw <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Sat, 14 May 2022 10:47:40 -0400

[Message part 1 (text/plain, inline)]

On Thu, May 12, 2022 at 3:22 AM Eli Zaretskii <eliz <at> gnu.org> wrote:

> > From: Robert Weiner <rsw <at> gnu.org>
> > Date: Thu, 12 May 2022 02:21:37 -0400
> > Cc: 55305 <at> debbugs.gnu.org
> >
> >  Does this last fact mean there's an assumption in Hyperbole that the
> >  package is always activated before its *.el files are compiled?  If
> >  so, perhaps this is why it fails during native-compilation, where the
> >  package is not activated prior to the compilation?
> >
> > Said another way, there is an assumption that the hyperbole-autoloads.el
> file is loaded prior to any
> > compilation, yes.  This is similar to assumptions that loaddefs.el are
> loaded prior to their reference in other
> > Emacs Lisp files.

Hi Eli:

Again, thanks for your feedback.  I don't expect any change to be made on
this in Emacs at this point but wanted to finish the discussion with a few
final thoughts.

> loaddefs.el is preloaded into Emacs when it is built, so the analogy
> doesn't work in practice.
>

Emacs loads autoloads from a file when it is built (prior to dumping its
image) and I am simply suggesting that both the package manager and the
native compiler do the same for packages.


> I think you should look at bundled packages like Calc.  Calc has
> calc-loaddefs.el, but I just now forced Emacs 28.1 to native-compile
> Calc and didn't see any problems.  And I see that calc.el does say
> explicitly
>
>   ;;;; (Autoloads here)
>   (load "calc-loaddefs.el" nil t)
>

Exactly, calc.el works around this missing feature by explicitly loading
the loaddefs and then having every other calc module require the 'calc'
library.  This is equivalent to a manual load of the autoloads in every
module of the package, i.e. there is no autoloading since the autoload
definitions are required everywhere.  Any package can do this but then
nothing is autoloaded at build time when a definition is referenced.  The
calc package goes further and adds a hack at the end of certain files:

;; Local variables:
;; generated-autoload-file: "calc-loaddefs.el"
;; End:

to force magic ;;;###autoload definitions to be written to the
calc-loaddefs.el file.  All of this is necessary because certain automated
handling of the default package autoloading file is missing from Emacs.


> > The point of the autoloads file is to include definitions that must
> exist in the Lisp
> > environment prior to their reference in any Lisp files, whether this is
> during package use or package
> > build-time.
>
> That is true, but AFAIU packages that have their own separate
> autoloads file should proactively do something to make sure those
> autoloads are loaded before they are needed.
>

My question is why?  If we want definitions within packages autoloaded just
as they can be outside of packages, why do we not want to simply fix the
issue so that each package's autoload file is actually autoloaded by the
package manager and the native compiler at both build and package
activation time (the latter already being done)?  We have a standardized
naming for such files, package-<package-name>.el.  They are generated by
the package manager but presently not autoloaded at build initialization.
It is immensely more work for each large package to require this in each of
their files and makes little sense since such a file does not exist at
least for Elpa packages until build time.

And this is not related to native-compilation in any way: the same
> will happen if one tries to byte-compile Hyperbole files without first
> loading its autoloads.  Right?
>

Yes, if you manually byte-compile a package file, you have to ensure its
autoloads have been loaded, but this is for a manual process.  I am
suggesting that in an automated context of package building that this too
should be automated and autoloads should automatically be loaded by the
build automation systems.  Otherwise, my argument is that these are not
treated as autoload files at the package level but are simply Lisp
libraries that have to be manually loaded by every library in the package,
a tedious affair for large packages.  Certainly not the end of the world
but difficult to manage and get right all the time.

[Message part 2 (text/html, inline)]

Message #28 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rswgnu <at> gmail.com
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Sat, 14 May 2022 18:05:44 +0300

> From: Robert Weiner <rsw <at> gnu.org>
> Date: Sat, 14 May 2022 10:47:40 -0400
> Cc: 55305 <at> debbugs.gnu.org
> 
> > loaddefs.el is preloaded into Emacs when it is built, so the analogy
> > doesn't work in practice.
> 
> Emacs loads autoloads from a file when it is built (prior to dumping its
> image) and I am simply suggesting that both the package manager and the
> native compiler do the same for packages.

How can a native compiler or a byte compiler know that a given .el
file needs some loadefs file to be loaded before it's compiled?  Most
Lisp files don't have and don't need any loaddefs files to be
compiled.

For that matter, how can a compiler know that a given .el files _has_
a loaddefs file, and if so, what is its name?

Loaddefs files are needed for when a file is loaded, not when it's
compiled.  For compilation, we have 'require' and 'eval-when-compile',
and always had them.  Are you saying that 'require' and
'eval-when-compile' should also be replaced by some automation? if so,
how can this work even in principle without some meta-data available
somewhere for the compiler to find and use?  And if we need meta-data,
what is wrong with having it in the form of 'require' and
'eval-when-compile' in the file itself?

> > I think you should look at bundled packages like Calc.  Calc has
> > calc-loaddefs.el, but I just now forced Emacs 28.1 to native-compile
> > Calc and didn't see any problems.  And I see that calc.el does say
> > explicitly
> >
> >   ;;;; (Autoloads here)
> >   (load "calc-loaddefs.el" nil t)
> >
> 
> Exactly, calc.el works around this missing feature by explicitly loading
> the loaddefs and then having every other calc module require the 'calc'
> library.

But it isn't just calc.el: every single FOO-loaddefs.el file in the
Emacs tree is loaded like that.  This is a de-facto standard in how we
use the separate loaddefs files of bundled packages.

And again, there's nothing new here related to the native compiler: it
works the same with the byte compiler, and will output the same
warnings if you fail to follow this paradigm.  It's nothing new.

If you are lobbying for having more automated discovery and loading of
loaddefs files, then you are asking for a new feature, not reporting a
bug in Emacs 28 that didn't exist in previous versions.  And I don't
think I agree that this feature is a good idea, for when a file is
compiled.  But we could discuss this feature request; my point is that
there's no bug here, but a normal behavior we have with any
compilation of any Lisp file in any Emacs version.

Message #31 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rsw <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Sat, 14 May 2022 18:40:03 -0400

[Message part 1 (text/plain, inline)]

On Sat, May 14, 2022 at 11:05 AM Eli Zaretskii <eliz <at> gnu.org> wrote:

> If you are lobbying for having more automated discovery and loading of
> loaddefs files, then you are asking for a new feature, not reporting a
> bug in Emacs 28 that didn't exist in previous versions.  And I don't
> think I agree that this feature is a good idea, for when a file is
> compiled.  But we could discuss this feature request; my point is that
> there's no bug here, but a normal behavior we have with any
> compilation of any Lisp file in any Emacs version.
>

Hi Eli:

I see what you are saying.  I wish Stefan would say
something here too with his perspective of dealing
with autoload files in Elpa packages.

I will close this as a bug.

Thanks,

Bob

[Message part 2 (text/html, inline)]

Message #34 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: rswgnu <at> gmail.com, Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager fails
 to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 08:15:00 +0300

> From: Robert Weiner <rsw <at> gnu.org>
> Date: Sat, 14 May 2022 18:40:03 -0400
> Cc: 55305 <at> debbugs.gnu.org
> 
> On Sat, May 14, 2022 at 11:05 AM Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>  If you are lobbying for having more automated discovery and loading of
>  loaddefs files, then you are asking for a new feature, not reporting a
>  bug in Emacs 28 that didn't exist in previous versions.  And I don't
>  think I agree that this feature is a good idea, for when a file is
>  compiled.  But we could discuss this feature request; my point is that
>  there's no bug here, but a normal behavior we have with any
>  compilation of any Lisp file in any Emacs version.
> 
> Hi Eli:
> 
> I see what you are saying.  I wish Stefan would say
> something here too with his perspective of dealing
> with autoload files in Elpa packages.

Stefan doesn't read this list.  I've CC'ed him now, so he could
comment.

Message #37 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Robert Weiner <rsw <at> gnu.org>
Cc: rswgnu <at> gmail.com, Andrea Corallo <akrl <at> sdf.com>, 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 11:59:05 -0400

First, sorry for not chiming in earlier (I don't subscribe to the
emacs-bugs list, so I only see those bugs that are explicitly forwarded
to me).

Robert Weiner [2022-05-07 16:05:17] wrote:
> Tested under Emacs 28.1 and a recent tip of the Emacs git repo for Emacs
> 29 with asynchronous native compilation enabled:
>
> M-x package-install RET hyperbole RET

Hmm... I tried to reproduce it here, with `emacs -Q` this gives me
(during the normal compilation), among a bunch of lesser warnings:

    Compiling file ~/.emacs.d/elpa/hyperbole-8.0.0/test/kexport-tests.el at Sun May 15 11:01:59 2022
    kexport-tests.el:20:2: Error: Cannot open load file: Aucun fichier ou dossier de ce type, el-mock

I also noticed the following warning in *Messages*:

    hibtypes:0: Warning: Not registering prefix "pa".  Affects: ("parse-label-and-file" "pathname" "pathname-line-and-column" "patch-msg")

which points at some namespace uncleanliness in your code.
Oh, and:

    Warning: Eager macro-expansion skipped due to cycle:
      … => (load "hbut.el") => (macroexpand-all …) => (macroexpand (eval-and-compile …)) => (load "hbdata.el") => (load "hgnus.el") => (load "hvar.el") => (load "hsettings.el") => (load "hui-em-but.el") => (load "hbut.el")
    Waiting for git... [2 times]

You might wan to try and fix this one.

> fails to load the hyperbole-autoloads.el file before the
> async native compiler and byte compiler produce these errors since
> the autoloaded var:append function is not defined:

Indeed.

> Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-em-but.el: Error:
> Symbol's function definition is void var:append Disable showing Disable
> logging

It took a bit of while to get there (many other things to
native-compile before this, apparently), but yes, I'm able to
reproduce it.

Looking at `comp-run-async-workers` in `comp.el`, I see that the async
compilation basically does:

    emacs -q -l <temp-file>

where <temp-file>'s content is basically the `expr` below:

         do (let* ((expr `((require 'comp)
                           ,(when (boundp 'backtrace-line-length)
                              `(setf backtrace-line-length ,backtrace-line-length))
                           (setf comp-file-preloaded-p ,comp-file-preloaded-p
                                 native-compile-target-directory ,native-compile-target-directory
                                 native-comp-speed ,native-comp-speed
                                 native-comp-debug ,native-comp-debug
                                 native-comp-verbose ,native-comp-verbose
                                 comp-libgccjit-reproducer ,comp-libgccjit-reproducer
                                 comp-async-compilation t
                                 native-comp-eln-load-path ',native-comp-eln-load-path
                                 native-comp-compiler-options
                                 ',native-comp-compiler-options
                                 native-comp-driver-options
                                 ',native-comp-driver-options
                                 load-path ',load-path
                                 warning-fill-column most-positive-fixnum)
                           ,native-comp-async-env-modifier-form
                           (message "Compiling %s..." ,source-file)
                           (comp--native-compile ,source-file ,(and load t))))

so the sync compilation is careful to preserve the current load-path
via:

                                 load-path ',load-path

which is why many of the files can be compiled correctly but it doesn't
load the packages's autoloads like a normal session does.

I suspect we should add a call to `package-activate-all` somewhere
in the above code (and probably preserve `package-directory-list` and
`package-user-dir` as well).

I just tried to re-trigger the problem after applying the patch below
[which also make this part of the code obey our 80-column convention,
while at it] and it appears to be fixed (e.g. `hui-em-but.el` was
successfully compiled).
Andrea, any comment?


        Stefan


diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 237de52884b..aa49607462c 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -3926,22 +3926,27 @@ comp-run-async-workers
                   (file-newer-than-file-p
                    source-file (comp-el-to-eln-filename source-file)))
          do (let* ((expr `((require 'comp)
-                           ,(when (boundp 'backtrace-line-length)
-                              `(setf backtrace-line-length ,backtrace-line-length))
-                           (setf comp-file-preloaded-p ,comp-file-preloaded-p
-                                 native-compile-target-directory ,native-compile-target-directory
-                                 native-comp-speed ,native-comp-speed
-                                 native-comp-debug ,native-comp-debug
-                                 native-comp-verbose ,native-comp-verbose
-                                 comp-libgccjit-reproducer ,comp-libgccjit-reproducer
-                                 comp-async-compilation t
-                                 native-comp-eln-load-path ',native-comp-eln-load-path
-                                 native-comp-compiler-options
-                                 ',native-comp-compiler-options
-                                 native-comp-driver-options
-                                 ',native-comp-driver-options
-                                 load-path ',load-path
-                                 warning-fill-column most-positive-fixnum)
+                           (setq comp-async-compilation t)
+                           (setq warning-fill-column most-positive-fixnum)
+                           ,(let ((set (list 'setq)))
+                              (dolist (var '(comp-file-preloaded-p
+                                             native-compile-target-directory
+                                             native-comp-speed
+                                             native-comp-debug
+                                             native-comp-verbose
+                                             comp-libgccjit-reproducer
+                                             native-comp-eln-load-path
+                                             native-comp-compiler-options
+                                             native-comp-driver-options
+                                             load-path
+                                             backtrace-line-length
+                                             package-user-dir
+                                             package-directory-list))
+                                (when (boundp var)
+                                  (push var set)
+                                  (push `',(symbol-value var) set)))
+                              (nreverse set))
+                           (package-activate-all)
                            ,native-comp-async-env-modifier-form
                            (message "Compiling %s..." ,source-file)
                            (comp--native-compile ,source-file ,(and load t))))
@@ -3994,7 +3999,7 @@ comp-run-async-workers
     (run-hooks 'native-comp-async-all-done-hook)
     (with-current-buffer (get-buffer-create comp-async-buffer-name)
       (save-excursion
-        (let ((buffer-read-only nil))
+        (let ((inhibit-read-only t))
           (goto-char (point-max))
           (insert "Compilation finished.\n"))))
     ;; `comp-deferred-pending-h' should be empty at this stage.

Message #40 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 55305 <at> debbugs.gnu.org, rswgnu <at> gmail.com, akrl <at> sdf.com, rsw <at> gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Sun, 15 May 2022 19:17:05 +0300

> Cc: rswgnu <at> gmail.com, Andrea Corallo <akrl <at> sdf.com>, 55305 <at> debbugs.gnu.org
> Date: Sun, 15 May 2022 11:59:05 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> I suspect we should add a call to `package-activate-all` somewhere
> in the above code (and probably preserve `package-directory-list` and
> `package-user-dir` as well).

I don't see why comp.el should call package-related functions (or
indeed know anything about packages and distinguish between packages
and other Lisp files).  It makes no sense to me.  Compilation should
not consider user customizations or be dependent on them.

I could support some general infrastructure to detect whether a given
file has separate autoloads, and perhaps load them when compiling, but
that's all.  And even this should be discussed, because I don't think
I like the idea of a compilation always loading the autoloads, it's in
many/most cases an overkill IMNSHO.

> I just tried to re-trigger the problem after applying the patch below
> [which also make this part of the code obey our 80-column convention,
> while at it] and it appears to be fixed (e.g. `hui-em-but.el` was
> successfully compiled).
> Andrea, any comment?

I'm firmly against this, sorry.  Let's look for more elegant ways;
this one is too blunt, and most Lisp files don't need it.  Moreover,
activating the packages will make every compilation dependent on the
current user's customizations and installed packages, which is the
antithesis of batch-mode compilation: it isn't a coincidence that
"-batch" implies "-Q".

Message #43 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: akrl <at> sdf.org
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, monnier <at> iro.umontreal.ca,
 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Sun, 15 May 2022 19:22:02 +0300

[Resending for Andrea, whose address was incorrect.]

> Cc: rswgnu <at> gmail.com, Andrea Corallo <akrl <at> sdf.com>, 55305 <at> debbugs.gnu.org
> Date: Sun, 15 May 2022 11:59:05 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> I suspect we should add a call to `package-activate-all` somewhere
> in the above code (and probably preserve `package-directory-list` and
> `package-user-dir` as well).

I don't see why comp.el should call package-related functions (or
indeed know anything about packages and distinguish between packages
and other Lisp files).  It makes no sense to me.  Compilation should
not consider user customizations or be dependent on them.

I could support some general infrastructure to detect whether a given
file has separate autoloads, and perhaps load them when compiling, but
that's all.  And even this should be discussed, because I don't think
I like the idea of a compilation always loading the autoloads, it's in
many/most cases an overkill IMNSHO.

> I just tried to re-trigger the problem after applying the patch below
> [which also make this part of the code obey our 80-column convention,
> while at it] and it appears to be fixed (e.g. `hui-em-but.el` was
> successfully compiled).
> Andrea, any comment?

I'm firmly against this, sorry.  Let's look for more elegant ways;
this one is too blunt, and most Lisp files don't need it.  Moreover,
activating the packages will make every compilation dependent on the
current user's customizations and installed packages, which is the
antithesis of batch-mode compilation: it isn't a coincidence that
"-batch" implies "-Q".

Message #46 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 12:47:00 -0400

>> Cc: rswgnu <at> gmail.com, Andrea Corallo <akrl <at> sdf.com>, 55305 <at> debbugs.gnu.org
>> Date: Sun, 15 May 2022 11:59:05 -0400
>> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>> 
>> I suspect we should add a call to `package-activate-all` somewhere
>> in the above code (and probably preserve `package-directory-list` and
>> `package-user-dir` as well).
>
> I don't see why comp.el should call package-related functions (or
> indeed know anything about packages and distinguish between packages
> and other Lisp files).  It makes no sense to me.  Compilation should
> not consider user customizations or be dependent on them.

Compiling a `.el` file requires loading files, running macros, and
calling functions, all of which may not come with Emacs and may depend
on the user's specific customizations and set of installed packages
(regardless of whether they're installed via package.el or some other
way).

So in order for the compilation to happen correctly, our async workers
need to mimic to some extent the currently running Emacs session.

The current code only does that to the extent that it preserves the
`load-path`, but this is not always sufficient.

My suggested patch adds the call to `package-activate-all` which is
usually executed in `startup.el` between loading `early-init.el` and
`init.el` and whose intention is to initialize things for the
user-installed packages much like things are unconditionally initialized
for the bundled packages (i.e. enough so it's easy to use them, but
sufficiently little that it doesn't get in the way if the user doesn't
want to use the package).

Just like the current code, my proposed change can fail to do the right
in some circumstances and I don't think there is a way to setup the sync
workers such that they'll always do the right thing.  My proposed patch
just gets a bit closer to reproducing the user's setup in "the usual
case" so I think it's less problematic than what we have.

AFAIK the only way to make async compilation work reliably is to make it
generate the `.eln` file without using the `.el` file (i.e. using the
`.elc` file instead, which can be compiled without having to load any
user-installed file, expand any macro, or run any user-installed
function).  That will also save us from mis-compiling file and from
(re)emitting compilation warnings.

But Someone™ would have to work on that (I can't think of any reason why
it should be difficult, tho it might require a few changes to the .elc
files we generate to preserve a bit more info from the source code, but
it wouldn't require a new .elc format of anything complicated like that).


        Stefan

Message #49 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 20:01:53 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: akrl <at> sdf.org,  55305 <at> debbugs.gnu.org,  rswgnu <at> gmail.com,  rsw <at> gnu.org
> Date: Sun, 15 May 2022 12:47:00 -0400
> 
> > I don't see why comp.el should call package-related functions (or
> > indeed know anything about packages and distinguish between packages
> > and other Lisp files).  It makes no sense to me.  Compilation should
> > not consider user customizations or be dependent on them.
> 
> Compiling a `.el` file requires loading files, running macros, and
> calling functions, all of which may not come with Emacs and may depend
> on the user's specific customizations and set of installed packages
> (regardless of whether they're installed via package.el or some other
> way).
> 
> So in order for the compilation to happen correctly, our async workers
> need to mimic to some extent the currently running Emacs session.

That was never the way byte-compilation worked in Emacs.  We have all
those 'require' and 'eval-when-compile' things precisely so a file can
tell the compiler what is needed for the compilation.  And we _need_ a
way to make the compilation be completely independent of any local
customizations or installed packages.

> My suggested patch adds the call to `package-activate-all` which is
> usually executed in `startup.el` between loading `early-init.el` and
> `init.el` and whose intention is to initialize things for the
> user-installed packages much like things are unconditionally initialized
> for the bundled packages (i.e. enough so it's easy to use them, but
> sufficiently little that it doesn't get in the way if the user doesn't
> want to use the package).

We don't call package-activate-all at startup when Emacs is told to
ignore user and site customizations.  That is NOT an accident, that is
the only way to have *.elc and *.eln files that can be copied to
another system and still work the same.  Changing this makes no sense.

I'm firmly against doing this.

> AFAIK the only way to make async compilation work reliably is to make it
> generate the `.eln` file without using the `.el` file (i.e. using the
> `.elc` file instead, which can be compiled without having to load any
> user-installed file, expand any macro, or run any user-installed
> function).  That will also save us from mis-compiling file and from
> (re)emitting compilation warnings.

This is unrelated, and is an entirely different discussion (which
comes up from time to time, and we didn't yet find a way around the
obstacles).

Message #52 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: monnier <at> iro.umontreal.ca
Cc: akrl <at> sdf.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, rsw <at> gnu.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Sun, 15 May 2022 20:15:24 +0300

> Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
> Date: Sun, 15 May 2022 20:01:53 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > My suggested patch adds the call to `package-activate-all` which is
> > usually executed in `startup.el` between loading `early-init.el` and
> > `init.el` and whose intention is to initialize things for the
> > user-installed packages much like things are unconditionally initialized
> > for the bundled packages (i.e. enough so it's easy to use them, but
> > sufficiently little that it doesn't get in the way if the user doesn't
> > want to use the package).
> 
> We don't call package-activate-all at startup when Emacs is told to
> ignore user and site customizations.  That is NOT an accident, that is
> the only way to have *.elc and *.eln files that can be copied to
> another system and still work the same.  Changing this makes no sense.

Btw, having a package load its loaddefs (perhaps with a non-nil 2nd
arg) is a much easier solution that doesn't need any changes at all,
and will immediately resolve any problems with compilation.  As you
know only very well, that's what we do in core.

Message #55 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 16:12:48 -0400

>> So in order for the compilation to happen correctly, our async workers
>> need to mimic to some extent the currently running Emacs session.
> That was never the way byte-compilation worked in Emacs.

It has never been officially documented, you're right.

But all the files bundled in Emacs are byte-compiled in an Emacs session
where `lisp/loaddefs.el` has already been loaded and many packages rely
on that to minimize the amount of explicit `require` they use (and to
break some cyclic dependencies).

In ELPA packages, that translates into an assumption that the package's
`<pkg>-autoloads.el` has already been loaded (which is indeed the case
during `package-install` and is also the case when compiled via the code
in GNU ELPA's `elpa-admin.el`).

> We have all those 'require' and 'eval-when-compile' things precisely
> so a file can tell the compiler what is needed for the compilation.
> And we _need_ a way to make the compilation be completely independent
> of any local customizations or installed packages.

When a package uses some other package's macro, it necessarily depends on
the locally installed packages to be compiled correctly.

Until now `comp.el` limits the support for "local customizations or
installed packages" to the act of propagating the current session's
`load-path`.  In theory, it could be sufficient.  But this is not the
same as what has been provided for the last ten years when compiling
ELPA packages, so it will inevitably bump into packages for which it
breaks compilation (such as Hyperbole).

I'm not claiming that calling `package-activate-all` is right for
reasons of principle.  We sadly never clearly defined what it is that
a package can count on. In practice ELPA packages have been able to
count on the fact that their autoloads (and their dependencies's
autoloads) have all been loaded (which also implies that all those
packages have been added to the `load-path`).

> I'm firmly against doing this.

OK.

>> AFAIK the only way to make async compilation work reliably is to make it
>> generate the `.eln` file without using the `.el` file (i.e. using the
>> `.elc` file instead, which can be compiled without having to load any
>> user-installed file, expand any macro, or run any user-installed
>> function).  That will also save us from mis-compiling file and from
>> (re)emitting compilation warnings.
> This is unrelated,

I firmly disagree with this, but I agree that it's a side discussion
because we haven't yet found anyone motivated to tackle this problem in
our native-comp pipeline.  So in the mean time we have to paper over the
consequences.


        Stefan

Message #58 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Sun, 15 May 2022 16:39:07 -0400

> We don't call package-activate-all at startup when Emacs is told to
> ignore user and site customizations.  That is NOT an accident, that is
> the only way to have *.elc and *.eln files that can be copied to
> another system and still work the same.  Changing this makes no sense.

I think I fully agree with you when it comes to compiling those files
which are bundled with Emacs.  But when compiling files from ELPA
packages, this would be an incompatible change, so I'm not sure it's
the right move.


        Stefan

Message #61 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 05:31:01 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: akrl <at> sdf.org,  55305 <at> debbugs.gnu.org,  rswgnu <at> gmail.com,  rsw <at> gnu.org
> Date: Sun, 15 May 2022 16:12:48 -0400
> 
> > We have all those 'require' and 'eval-when-compile' things precisely
> > so a file can tell the compiler what is needed for the compilation.
> > And we _need_ a way to make the compilation be completely independent
> > of any local customizations or installed packages.
> 
> When a package uses some other package's macro, it necessarily depends on
> the locally installed packages to be compiled correctly.

My worry is about packages that do NOT depend on such macros.  Calling
package-activate-all will activate all the packages on the user's
system, and there's no way of knowing what those packages do at
activation time.  They can change variables, advise functions,
redefine commands, etc.  We have no idea what will be the state of the
session after doing that.

> I'm not claiming that calling `package-activate-all` is right for
> reasons of principle.  We sadly never clearly defined what it is that
> a package can count on.

Then we should do that _before_ we propose solutions that rely on
what's there, knowing that what's there was never intended to solve
this particular issue.  This cure is worse than the disease.

Message #64 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 05:33:24 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: akrl <at> sdf.org,  55305 <at> debbugs.gnu.org,  rswgnu <at> gmail.com,  rsw <at> gnu.org
> Date: Sun, 15 May 2022 16:39:07 -0400
> 
> > We don't call package-activate-all at startup when Emacs is told to
> > ignore user and site customizations.  That is NOT an accident, that is
> > the only way to have *.elc and *.eln files that can be copied to
> > another system and still work the same.  Changing this makes no sense.
> 
> I think I fully agree with you when it comes to compiling those files
> which are bundled with Emacs.  But when compiling files from ELPA
> packages, this would be an incompatible change, so I'm not sure it's
> the right move.

Your proposal was to modify comp.el, so it will affect any
native-compilation, including the compilation of core Emacs files.

And even for files that are not bundled, think about downstream Emacs
distros that would want to provide some of such files as part of the
distributions.  They should be compiled in a clean environment.

Message #67 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>
Cc: 55305 <at> debbugs.gnu.org, rswgnu <at> gmail.com,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, Andrea Corallo <akrl <at> sdf.com>,
 Robert Weiner <rsw <at> gnu.org>
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 09:34:02 +0000

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs <at> gnu.org> writes:

> First, sorry for not chiming in earlier (I don't subscribe to the
> emacs-bugs list, so I only see those bugs that are explicitly forwarded
> to me).
>
> Robert Weiner [2022-05-07 16:05:17] wrote:
>> Tested under Emacs 28.1 and a recent tip of the Emacs git repo for Emacs
>> 29 with asynchronous native compilation enabled:
>>
>> M-x package-install RET hyperbole RET
>
> Hmm... I tried to reproduce it here, with `emacs -Q` this gives me
> (during the normal compilation), among a bunch of lesser warnings:
>
>     Compiling file ~/.emacs.d/elpa/hyperbole-8.0.0/test/kexport-tests.el at Sun May 15 11:01:59 2022
>     kexport-tests.el:20:2: Error: Cannot open load file: Aucun fichier ou dossier de ce type, el-mock
>
> I also noticed the following warning in *Messages*:
>
>     hibtypes:0: Warning: Not registering prefix "pa".  Affects: ("parse-label-and-file" "pathname" "pathname-line-and-column" "patch-msg")
>
> which points at some namespace uncleanliness in your code.
> Oh, and:
>
>     Warning: Eager macro-expansion skipped due to cycle:
>       … => (load "hbut.el") => (macroexpand-all …) => (macroexpand (eval-and-compile …)) => (load "hbdata.el") => (load "hgnus.el") => (load "hvar.el") => (load "hsettings.el") => (load "hui-em-but.el") => (load "hbut.el")
>     Waiting for git... [2 times]
>
> You might wan to try and fix this one.
>
>> fails to load the hyperbole-autoloads.el file before the
>> async native compiler and byte compiler produce these errors since
>> the autoloaded var:append function is not defined:
>
> Indeed.
>
>> Warning (comp): ~/.emacs.d/elpa/hyperbole-8.0.0/hui-em-but.el: Error:
>> Symbol's function definition is void var:append Disable showing Disable
>> logging
>
> It took a bit of while to get there (many other things to
> native-compile before this, apparently), but yes, I'm able to
> reproduce it.
>
> Looking at `comp-run-async-workers` in `comp.el`, I see that the async
> compilation basically does:
>
>     emacs -q -l <temp-file>
>
> where <temp-file>'s content is basically the `expr` below:
>
>          do (let* ((expr `((require 'comp)
>                            ,(when (boundp 'backtrace-line-length)
>                               `(setf backtrace-line-length ,backtrace-line-length))
>                            (setf comp-file-preloaded-p ,comp-file-preloaded-p
>                                  native-compile-target-directory ,native-compile-target-directory
>                                  native-comp-speed ,native-comp-speed
>                                  native-comp-debug ,native-comp-debug
>                                  native-comp-verbose ,native-comp-verbose
>                                  comp-libgccjit-reproducer ,comp-libgccjit-reproducer
>                                  comp-async-compilation t
>                                  native-comp-eln-load-path ',native-comp-eln-load-path
>                                  native-comp-compiler-options
>                                  ',native-comp-compiler-options
>                                  native-comp-driver-options
>                                  ',native-comp-driver-options
>                                  load-path ',load-path
>                                  warning-fill-column most-positive-fixnum)
>                            ,native-comp-async-env-modifier-form
>                            (message "Compiling %s..." ,source-file)
>                            (comp--native-compile ,source-file ,(and load t))))
>
> so the sync compilation is careful to preserve the current load-path
> via:
>
>                                  load-path ',load-path
>
> which is why many of the files can be compiled correctly but it doesn't
> load the packages's autoloads like a normal session does.
>
> I suspect we should add a call to `package-activate-all` somewhere
> in the above code (and probably preserve `package-directory-list` and
> `package-user-dir` as well).
>
> I just tried to re-trigger the problem after applying the patch below
> [which also make this part of the code obey our 80-column convention,
> while at it] and it appears to be fixed (e.g. `hui-em-but.el` was
> successfully compiled).
> Andrea, any comment?
>
>
>         Stefan

Hi Stefan,

thanks for having debugged this.

I see no harm in propagating `package-user-dir' `package-directory-list'
to the async worker if it's useful, but I'm with Eli in not calling
`package-activate-all' when compiling.

An idea would be that if the code being compiled needs that it could
leverage `native-comp-async-env-modifier-form' for that.

Best Regards

  Andrea

Message #73 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 12:40:41 -0400

Eli Zaretskii [2022-05-16 05:31:01] wrote:
> My worry is about packages that do NOT depend on such macros.  Calling
> package-activate-all will activate all the packages on the user's
> system, and there's no way of knowing what those packages do at
> activation time.  They can change variables, advise functions,
> redefine commands, etc.

Yup.   I agree it can be worrisome, but:
- Those packages are presumably already activated in the current session
  and hence similarly affected the generation of the `.elc` files if the
  `.elc` files were generated by `package-install`.
- Preserving the `load-path` like we already do also exposes similar
  problems (some files may shadow the ones we need).

> We have no idea what will be the state of the session after
> doing that.

After the files are compiled, the session is killed (we're talking about
an Emacs sub-process performing native compilation asynchronously), so
I don't think we need to worry too much about that.

>> I'm not claiming that calling `package-activate-all` is right for
>> reasons of principle.  We sadly never clearly defined what it is that
>> a package can count on.
> Then we should do that _before_ we propose solutions that rely on
> what's there, knowing that what's there was never intended to solve
> this particular issue.

I don't understand what you're saying, here.

There is a de-facto definition of what a package can count on (which is
that its autoloads file has been loaded), which has been used ever since
the inception of `package.el`.

We can't go back to a time before that.

I don't know enough of what you mean by "this particular issue" to judge
whether this design was made to solve this issue or not, but I do know
that the autoloads file is loaded on purpose before compiling the `.elc`
files so as to make it more convenient to write the code of a package,
just like we happily rely on `loaddefs.el` being loaded when working on
Emacs's bundled code.

> This cure is worse than the disease.

My proposed patch is not a cure.  The cure would be to compile from the
`.elc` file (since the root of the problem is that we need those
autoloads and the `load-path` to macroexpand the code, and the only
reason why the native-compiler needs to macroexpand the code is because
it (re)starts from the `.el` file instead of using the `.elc` file
where no macro-expansion need to take place).


        Stefan

Message #76 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 55305 <at> debbugs.gnu.org, "Stefan Monnier via Bug reports for GNU Emacs,
 the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>, rswgnu <at> gmail.com,
 Andrea Corallo <akrl <at> sdf.com>, Robert Weiner <rsw <at> gnu.org>
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 12:42:44 -0400

> I see no harm in propagating `package-user-dir' `package-directory-list'
> to the async worker if it's useful, but I'm with Eli in not calling
> `package-activate-all' when compiling.

The currently discussed error happens because `hyperbole-autoloads.el`
was not loaded before the compilation.  Setting `package-user-dir` and
`package-directory-list` won't help.

> An idea would be that if the code being compiled needs that it could
> leverage `native-comp-async-env-modifier-form' for that.

Ewww!


        Stefan

Message #82 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 19:57:00 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: akrl <at> sdf.org,  55305 <at> debbugs.gnu.org,  rswgnu <at> gmail.com,  rsw <at> gnu.org
> Date: Mon, 16 May 2022 12:40:41 -0400
> 
> Eli Zaretskii [2022-05-16 05:31:01] wrote:
> > My worry is about packages that do NOT depend on such macros.  Calling
> > package-activate-all will activate all the packages on the user's
> > system, and there's no way of knowing what those packages do at
> > activation time.  They can change variables, advise functions,
> > redefine commands, etc.
> 
> Yup.   I agree it can be worrisome, but:
> - Those packages are presumably already activated in the current session
>   and hence similarly affected the generation of the `.elc` files if the
>   `.elc` files were generated by `package-install`.

Two wrongs don't make one right.

Besides, as long as the *.elc files are produced by batch-compilation,
this will not happen.  And I'm talking only about batch compilation,
not about interactive compilation that the user does from a live
session.

> - Preserving the `load-path` like we already do also exposes similar
>   problems (some files may shadow the ones we need).

The effect of load-path is a much smaller one, and not propagating
load-path would cause much graver problems.

> > We have no idea what will be the state of the session after
> > doing that.
> 
> After the files are compiled, the session is killed (we're talking about
> an Emacs sub-process performing native compilation asynchronously), so
> I don't think we need to worry too much about that.

My worry is about the effects on the produced *.elc/*.eln files, so
killing the session doesn't help.

> >> I'm not claiming that calling `package-activate-all` is right for
> >> reasons of principle.  We sadly never clearly defined what it is that
> >> a package can count on.
> > Then we should do that _before_ we propose solutions that rely on
> > what's there, knowing that what's there was never intended to solve
> > this particular issue.
> 
> I don't understand what you're saying, here.

I'm sating we should define what is it the package can count on before
suggesting solutions based on lack of such a definition.

> There is a de-facto definition of what a package can count on (which is
> that its autoloads file has been loaded), which has been used ever since
> the inception of `package.el`.

That's when the package is loaded, not when it's compiled.

> I don't know enough of what you mean by "this particular issue" to judge
> whether this design was made to solve this issue or not, but I do know
> that the autoloads file is loaded on purpose before compiling the `.elc`
> files so as to make it more convenient to write the code of a package,
> just like we happily rely on `loaddefs.el` being loaded when working on
> Emacs's bundled code.

Loading the loaddefs file by some (as yet nonexistent) mechanism is
something I'm prepared to discuss with the purpose of maybe finding
some reasonable solution (though it isn't yet clear to me how to do
that ion general, for the reasons I explained several messages
up-thread).  What I'm NOT interested in considering is a call to
package-activate-all as part of compiling every Lisp file.

> > This cure is worse than the disease.
> 
> My proposed patch is not a cure.  The cure would be to compile from the
> `.elc` file (since the root of the problem is that we need those
> autoloads and the `load-path` to macroexpand the code, and the only
> reason why the native-compiler needs to macroexpand the code is because
> it (re)starts from the `.el` file instead of using the `.elc` file
> where no macro-expansion need to take place).

I'm also okay with leaving this issue unsolved until such time as we
have native compilation based on .elc files, and revisiting this then.

Message #85 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.com,
 akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Mon, 16 May 2022 19:59:13 +0300

> Cc: 55305 <at> debbugs.gnu.org, rswgnu <at> gmail.com, akrl <at> sdf.com, rsw <at> gnu.org
> Date: Mon, 16 May 2022 12:42:44 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> > I see no harm in propagating `package-user-dir' `package-directory-list'
> > to the async worker if it's useful, but I'm with Eli in not calling
> > `package-activate-all' when compiling.
> 
> The currently discussed error happens because `hyperbole-autoloads.el`
> was not loaded before the compilation.  Setting `package-user-dir` and
> `package-directory-list` won't help.

But loading hyperbole-autoloads.el from the files that need it will
help, and will solve this problem simply and without any
complications.  Look how much energy we invested in discussing a
problem that takes a trivial one-liner to solve.

Message #88 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, rswgnu <at> gmail.com, 55305 <at> debbugs.gnu.org, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Mon, 16 May 2022 13:17:14 -0400

> Besides, as long as the *.elc files are produced by batch-compilation,
> this will not happen.

AFAIK `.elc` files generated by batch-compilation is the rule for
Emacs's bundled files but the exception for ELPA-installed packages.
[ Many packages fail to compile properly when compiled in a batch
  session ;-)  This has improved somewhat over the years, tho.  ]

>> - Preserving the `load-path` like we already do also exposes similar
>>   problems (some files may shadow the ones we need).
> The effect of load-path is a much smaller one,
> and not propagating load-path would cause much graver problems.

Agreed.

>> I don't understand what you're saying, here.
> I'm sating we should define what is it the package can count on before
> suggesting solutions based on lack of such a definition.

My solution is not based on lack of definition.  It's based on the
de-facto definition that's been in use for the last 10 years.

>> There is a de-facto definition of what a package can count on (which is
>> that its autoloads file has been loaded), which has been used ever since
>> the inception of `package.el`.
> That's when the package is loaded, not when it's compiled.

No, I'm talking about the compilation of the package's `.el` files into
`.elc` files.

> I'm also okay with leaving this issue unsolved until such time as we
> have native compilation based on .elc files, and revisiting this then.

Might be a good motivation to get this `.elc` compilation sooner rather
than later.


        Stefan

Message #91 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rswgnu <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rsw <at> gnu.org, 55305 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, akrl <at> sdf.com, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Mon, 16 May 2022 18:27:13 -0400

Just FYI, Hyperbole used to generate the hyperbole-autoloads.el file itself and include it in the packaged release but this broke the Elpa build process.

-- Bob

> On May 16, 2022, at 12:59 PM, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> 
>> 
>> Cc: 55305 <at> debbugs.gnu.org, rswgnu <at> gmail.com, akrl <at> sdf.com, rsw <at> gnu.org
>> Date: Mon, 16 May 2022 12:42:44 -0400
>> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
>> 
>>> I see no harm in propagating `package-user-dir' `package-directory-list'
>>> to the async worker if it's useful, but I'm with Eli in not calling
>>> `package-activate-all' when compiling.
>> 
>> The currently discussed error happens because `hyperbole-autoloads.el`
>> was not loaded before the compilation.  Setting `package-user-dir` and
>> `package-directory-list` won't help.
> 
> But loading hyperbole-autoloads.el from the files that need it will
> help, and will solve this problem simply and without any
> complications.  Look how much energy we invested in discussing a
> problem that takes a trivial one-liner to solve.

Message #94 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Robert Weiner <rswgnu <at> gmail.com>
Cc: rsw <at> gnu.org, 55305 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca, akrl <at> sdf.com,
 akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation
Date: Tue, 17 May 2022 05:27:42 +0300

> From: Robert Weiner <rswgnu <at> gmail.com>
> Date: Mon, 16 May 2022 18:27:13 -0400
> Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, akrl <at> sdf.org,
>  55305 <at> debbugs.gnu.org, akrl <at> sdf.com, rsw <at> gnu.org
> 
> Just FYI, Hyperbole used to generate the hyperbole-autoloads.el file itself and include it in the packaged release but this broke the Elpa build process.

Then perhaps the ELPA build process needs to be augmented not to break
in those cases.

Message #101 received at 55305-done <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <acorallo <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 55305-done <at> debbugs.gnu.org, akrl <at> sdf.com, rsw <at> gnu.org,
 monnier <at> iro.umontreal.ca, Robert Weiner <rswgnu <at> gmail.com>, akrl <at> sdf.org
Subject: Re: bug#55305: 28.0.50: With async nativecomp, package manager
 fails to load hyperbole-autoloads.el before compilation
Date: Wed, 07 Jun 2023 17:36:04 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Robert Weiner <rswgnu <at> gmail.com>
>> Date: Mon, 16 May 2022 18:27:13 -0400
>> Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>, akrl <at> sdf.org,
>>  55305 <at> debbugs.gnu.org, akrl <at> sdf.com, rsw <at> gnu.org
>> 
>> Just FYI, Hyperbole used to generate the hyperbole-autoloads.el file itself and include it in the packaged release but this broke the Elpa build process.
>
> Then perhaps the ELPA build process needs to be augmented not to break
> in those cases.

I'm closing this old bug as the outcome was that the issue is not a
compiler bug.  If the ELPA infrastructure needs some improvment we
probably need a dedicated bug for that.

Happy to reopen anyway if necessary.

Bests

  Andrea

Message #104 received at 55305 <at> debbugs.gnu.org (full text, mbox):

From: Robert Weiner <rswgnu <at> gmail.com>
To: 55305 <at> debbugs.gnu.org
Subject: Re: bug#55305: closed (Re: bug#55305: 28.0.50: With async nativecomp,
 package manager fails to load hyperbole-autoloads.el before
 compilation)
Date: Wed, 7 Jun 2023 22:03:56 -0400

Fyi, this was fixed long ago in the Hyerbole elpa-devel release, we just did not realize there was a bug still open on it.

-- Bob

> On Jun 7, 2023, at 5:37 PM, help-debbugs <at> gnu.org wrote:
> 
> Your bug report
> 
> #55305: 28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation
> 
> which was filed against the emacs package, has been closed.
> 
> The explanation is attached below, along with your original report.
> If you require more details, please reply to 55305 <at> debbugs.gnu.org.
> 
> -- 
> 55305: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=55305
> GNU Bug Tracking System
> Contact help-debbugs <at> gnu.org with problems
> <mime-attachment>
> <mime-attachment>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.