GNU bug report logs - #49270
28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 49270 in the body.
You can then email your comments to 49270 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jim Myhrberg <contact <at> jimeh.me>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50: native-comp: macOS self-contained .app does not use bundled
 *.eln files
Date: Tue, 29 Jun 2021 12:51:48 +0100

When performing a self-contained Emacs.app build, *.eln files for
Emacs' built-in lisp files stored within the Emacs.app bundle are not
used.

I believe commit 5dd2d50 which moved *.eln files and various paths
around a bit for macOS builds, has re-introduced an old bug from last
year. Basically the checksums that makes up part of the *.eln file
names uses the absolute path of the .el file in question, but for self
contained .app builds it needs to just use the relative path to the
app itself.

In short, any *.el files which contain ".app/Contents/" within their
absolute file path, need to have anything before ".app/Contents/"
removed before the *.eln filename checksum is calculated. Potentially
some extra logic might be needed for build-time native-comp, as the
checksums in the *.eln files bundled into the app right now are not
based on the final location of lisp files within the .app bundle.

In my testing, of the two checksums which are part of all *.eln
filenames, the first checksum is different in the newly async
generated *.eln files compared to those within the application bundle.
The second checksum is identical. Also moving Emacs.app to different
locations on the file system will cause it to re-compile *.eln files
again, with the first checksum changing again.

For the sake of any doubt, this issue occurs both with and without
NATIVE_FULL_AOT=1.

I've seen this behavior with all builds from the master branch that
I've done since the 27th of June.

Hopefully I've provided enough information here, please don't hesitate
to ask if anything more is needed.

P.S. I'm pretty new to emacs bug reporting, so apologies if I'm doing
something wrong, and please let me know if so, and what I should be
doing instead :)

Message #8 received at 49270 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jim Myhrberg <contact <at> jimeh.me>
Cc: 49270 <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Tue, 29 Jun 2021 15:25:41 +0300

> From: Jim Myhrberg <contact <at> jimeh.me>
> Date: Tue, 29 Jun 2021 12:51:48 +0100
> 
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.

Thanks, but how do you see that?

Message #11 received at 49270 <at> debbugs.gnu.org (full text, mbox):

From: Jim Myhrberg <contact <at> jimeh.me>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 49270 <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Tue, 29 Jun 2021 14:07:08 +0100

On Tue, Jun 29, 2021 at 1:25 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Jim Myhrberg <contact <at> jimeh.me>
> > Date: Tue, 29 Jun 2021 12:51:48 +0100
> >
> > When performing a self-contained Emacs.app build, *.eln files for
> > Emacs' built-in lisp files stored within the Emacs.app bundle are not
> > used.
>
> Thanks, but how do you see that?

For better or worse, in a rather unsophisticated way; by launching Emacs.app and
seeing the async native-comp kick off for all of Emacs' built-in lisp files. And
by doing some additional experiments with *.eln files just now.

The Emacs.app bundle I built earlier today includes
"bytecomp-12882072-3f2b96ca.eln" in
"Emacs.app/Contents/MacOS/lib/emacs/28.0.50/native-lisp", but when launching
Emacs.app it performs async native-comp on bytecomp.el, and produced
"bytecomp-a3d51747-3f2b96ca.eln" in my user eln cache folder. If I just relaunch
Emacs again it does not re-compile bytecomp.el.

If I move the newly created "bytecomp-a3d51747-3f2b96ca.eln" from my user eln
cache folder into the Emacs.app bundle next to the original bytecomp-*.eln, and
launch Emacs.app, it also does not re-compile bytecomp.el.

However, if I then move Emacs.app itself (from /Users/jimeh/Downloads to
/Users/jimeh/Desktop in this case), it will re-compile bytecomp.el again on
launch, producing "bytecomp-30860bee-3f2b96ca.eln" in my user eln cache folder.

At the end I have three separate bytecomp-*.eln files:

- "bytecomp-12882072-3f2b96ca.eln" - produced at Emacs build time and bundled
  into the app
- "bytecomp-a3d51747-3f2b96ca.eln" - produced via async native-comp at launch of
  Emacs.app while it was located in "/Users/jimeh/Downloads/", with absolute
  path to bytecomp.el.gz being
  "/Users/jimeh/Downloads/Emacs.app/Contents/Resources/lisp/emacs-lisp/bytecomp.el.gz"
- "bytecomp-30860bee-3f2b96ca.eln" - produced via async native-comp at launch of
  Emacs.app while it was located in "/Users/jimeh/Desktop/", with absolute path
  to bytecomp.el.gz being
  "/Users/jimeh/Desktop/Emacs.app/Contents/Resources/lisp/emacs-lisp/bytecomp.el.gz"

And this is why I believe the first checksum within *.eln file names is based on
the absolute path even when the source .el/.el.gz files are located within a
.app bundle.

Also, I forgot mention earlier, but the previous bug for this issue from last
year is bug#43532, potentially that might give some additional context.

Message #14 received at 49270 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Jim Myhrberg <contact <at> jimeh.me>
Cc: 49270 <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Tue, 29 Jun 2021 21:11:58 +0300

> From: Jim Myhrberg <contact <at> jimeh.me>
> Date: Tue, 29 Jun 2021 12:51:48 +0100
> 
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.
> 
> I believe commit 5dd2d50 which moved *.eln files and various paths
> around a bit for macOS builds, has re-introduced an old bug from last
> year. Basically the checksums that makes up part of the *.eln file
> names uses the absolute path of the .el file in question, but for self
> contained .app builds it needs to just use the relative path to the
> app itself.

That's not (exactly) how this stuff is supposed to work, AFAIR, not
after the recent changes.  Please look at the code and the comments in
comp.c:comp-el-to-eln-rel-filename, and tell if you understand why
this isn't working for you.

Message #17 received at 49270 <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Jim Myhrberg <contact <at> jimeh.me>
Cc: 49270 <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Tue, 29 Jun 2021 22:06:43 +0100

[Message part 1 (text/plain, inline)]

On Tue, Jun 29, 2021 at 12:51:48PM +0100, Jim Myhrberg wrote:
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.
> 
> I believe commit 5dd2d50 which moved *.eln files and various paths
> around a bit for macOS builds, has re-introduced an old bug from last
> year. Basically the checksums that makes up part of the *.eln file
> names uses the absolute path of the .el file in question, but for self
> contained .app builds it needs to just use the relative path to the
> app itself.

I don't think this is the same bug (although I guess it has the same
symptoms).

Please try the attached patch, it fixes it here.
-- 
Alan Third

[0001-Fix-NS-native-comp-search-path-bug-49270.patch (text/plain, attachment)]

Message #20 received at 49270 <at> debbugs.gnu.org (full text, mbox):

From: Jim Myhrberg <contact <at> jimeh.me>
To: Alan Third <alan <at> idiocy.org>, Jim Myhrberg <contact <at> jimeh.me>,
 49270 <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Wed, 30 Jun 2021 10:32:09 +0100

On Tue, Jun 29, 2021 at 10:06 PM Alan Third <alan <at> idiocy.org> wrote:
> Please try the attached patch, it fixes it here.

Awesome, thanks. I've just tested the patch and can confirm it fixes
the issue on my end too. All *.eln files in the .app bundle are now
located correctly, even when moving the app around the filesystem :)

Message #25 received at 49270-done <at> debbugs.gnu.org (full text, mbox):

From: Alan Third <alan <at> idiocy.org>
To: Jim Myhrberg <contact <at> jimeh.me>
Cc: 49270-done <at> debbugs.gnu.org
Subject: Re: bug#49270: 28.0.50: native-comp: macOS self-contained .app does
 not use bundled *.eln files
Date: Thu, 1 Jul 2021 22:16:56 +0100

On Wed, Jun 30, 2021 at 10:32:09AM +0100, Jim Myhrberg wrote:
> On Tue, Jun 29, 2021 at 10:06 PM Alan Third <alan <at> idiocy.org> wrote:
> > Please try the attached patch, it fixes it here.
> 
> Awesome, thanks. I've just tested the patch and can confirm it fixes
> the issue on my end too. All *.eln files in the .app bundle are now
> located correctly, even when moving the app around the filesystem :)

Pushed to master.
-- 
Alan Third

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.