GNU bug report logs - #43137
28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 43137 in the body.
You can then email your comments to 43137 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 28.0.50; [feature/native-comp] .eln path fixup confused using
 relative paths
Date: Mon, 31 Aug 2020 11:11:34 -0400

[Message part 1 (text/plain, inline)]

On macos (if you use --with-ns), there are two ways the emacs binary
gets installed:

* as <prefix>/bin/emacs, as normal
* as <prefix>/Application/Emacs.app/Contents/MacOS/Emacs, in a macOS app
  bundle.

Both of these cause problems, unfortunately:

The "normal" emacs binary you can't invoke from $PATH:

$ emacs -q --batch --eval '(message "hi")'
emacs:
dlopen(../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0/lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
1): image not found

But it does work when run with an absolute path:

$
/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/bin/emacs
-q --batch --eval '(message "hi")'
hi


The app bundle Emacs, on the other hand, doesn't like the relative path
at all either:

$
/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/Emacs.app/Contents/MacOS/Emacs
emacs:
dlopen(/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/Emacs.app/Contents/MacOS/../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0/lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
1): image not found

I have traced both of these down to the relocation logic in pdumper.c:
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/pdumper.c#L5255-L5277

This code (and the corresponding bit in
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/lisp/loadup.el#L471-L477
both seem to take the absolute installation directory and turn them into
relative directories that then fail to work. That the plain "emacs"
binary fails to run might be due to the fact that `invocation-directory`
is not set in that case, and so it has nothing to prepend to the
relative name.

In nix, an installation never moves - it's a content-addressed store, so
there's a very fixed pathname that everything lives at forever; is
it possible to turn off that pathname relativization?

Alternatively, I guess for darwin I'd also be good if we could get the
correct relative names in the application bundle; There is just one
`../Resources` too few in there. I think to handle this right, we'd have
to add another case to the installation_state enum in pdumper.c, is
that right?

Cheers,
Andreas.



In GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin19.6.0, NS appkit-1894.60
Version 10.15.6 (Build 19G2021))
Repository revision: aa526c9470d679e9144af55d9e56928a111d2ceb
Repository branch: master
Windowing system distributor 'Apple', version 10.3.1894
System Description:  Mac OS X 10.15.6

Configured using:
 'configure
--prefix=/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0
--disable-build-details --with-modules --with-ns
--disable-ns-self-contained --with-nativecomp
CFLAGS=-DMAC_OS_X_VERSION_MAX_ALLOWED=101200'


--
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[Message part 2 (text/html, inline)]

Message #8 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andreas Fuchs <asf <at> boinkor.net>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50;
 [feature/native-comp] .eln path fixup confused using relative paths
Date: Mon, 31 Aug 2020 16:32:01 +0000

Andreas Fuchs <asf <at> boinkor.net> writes:

> On macos (if you use --with-ns), there are two ways the emacs binary
> gets installed:
>
> * as <prefix>/bin/emacs, as normal
> * as <prefix>/Application/Emacs.app/Contents/MacOS/Emacs, in a macOS
> app
>   bundle.
>
> Both of these cause problems, unfortunately:
>
> The "normal" emacs binary you can't invoke from $PATH:
>
> $ emacs -q --batch --eval '(message "hi")'
> emacs: dlopen(../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0
> /
> lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
> 1): image not found
>
> But it does work when run with an absolute path:
>
> $ /nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/
> bin/emacs -q --batch --eval '(message "hi")'
> hi
>
>
> The app bundle Emacs, on the other hand, doesn't like the relative
> path
> at all either:
>
> $ /nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/
> Applications/Emacs.app/Contents/MacOS/Emacs
> emacs: dlopen(/nix/store/
> a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/
> Emacs.app/Contents/MacOS/../eln-cache/
> x86_64-apple-darwin19.6.0-9cab85d51a8656a0/
> lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
> 1): image not found
>  
> I have traced both of these down to the relocation logic in
> pdumper.c:
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> pdumper.c#L5255-L5277
>
> This code (and the corresponding bit in
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/lisp/
> loadup.el#L471-L477
> both seem to take the absolute installation directory and turn them
> into
> relative directories that then fail to work. That the plain "emacs"
> binary fails to run might be due to the fact that
> `invocation-directory`
> is not set in that case, and so it has nothing to prepend to the
> relative name.
>
> In nix, an installation never moves - it's a content-addressed store,
> so
> there's a very fixed pathname that everything lives at forever; is
> it possible to turn off that pathname relativization?
>
> Alternatively, I guess for darwin I'd also be good if we could get
> the
> correct relative names in the application bundle; There is just one
> `../Resources` too few in there. I think to handle this right, we'd
> have
> to add another case to the installation_state enum in pdumper.c, is
> that right?
>
> Cheers,
> Andreas.

Hi Andreas,

I think we might have two issues here.

The first is to provide the correct eln destination directory during the
build so it can be used correctly by the logic that starts at
loadup.el:452.  This should fixup the filenames so afterwards when
resurrecting the logic in pdumper.c can work correctly.

About this there's a branch in feature/native-comp-macos-fixes (by
Vibhav Pant Cc'ed) with a fix.  See also [1].  Maybe you like to give it
a go.

The other possible source of problems may be Vinvocation_directory still
not set when we pass into pdumper.c:5270, I can't verify that as I don't
have use macos but should be realitively easy to verify.

Ciao

  Andrea

[1] <https://lists.gnu.org/archive/html/emacs-devel/2020-08/msg00700.html>

Message #11 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Tue, 1 Sep 2020 10:15:50 -0400

[Message part 1 (text/plain, inline)]

Andrea Corallo <akrl <at> sdf.org> writes:
> The first is to provide the correct eln destination directory during the
> build so it can be used correctly by the logic that starts at
> loadup.el:452.  This should fixup the filenames so afterwards when
> resurrecting the logic in pdumper.c can work correctly.
>
> About this there's a branch in feature/native-comp-macos-fixes (by
> Vibhav Pant Cc'ed) with a fix.  See also [1].  Maybe you like to give it
> a go.

Oooh, that's a good catch. If I read the diff correctly, with a
not-only-self-contained build, it will set things up such that the
Application bundle works, but the (also installed) emacs binary won't.

A solution here might be to build two pdmp files - one for the
relocatable app bundle and one for commandline usage. I'll try to build
that.

> The other possible source of problems may be Vinvocation_directory still
> not set when we pass into pdumper.c:5270, I can't verify that as I don't
> have use macos but should be realitively easy to verify.

This makes sense too. I'll investigate when `Vinvocation_directory` gets
set. Does sound like that doesn't happen early enough, maybe we can pull
it forward a bit.

Thanks,
-- 
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[Message part 2 (text/html, inline)]

Message #14 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Tue, 1 Sep 2020 15:32:05 -0400

[Message part 1 (text/plain, inline)]

I <asf <at> boinkor.net> wrote:
> This makes sense too. I'll investigate when `Vinvocation_directory` gets
> set. Does sound like that doesn't happen early enough, maybe we can pull
> it forward a bit.

I dug into this, and the way main() progresses is that it first tries to
load the pdump (via load_pdump in line 1076,
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/emacs.c#L1076
),
and only much later, sets invocation-directory and other cmdargs via
set_invocation_vars called from init_cmdargs (called in main on line 1794,
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/emacs.c#L1794
).

I think this should be failing cross-platform; is it working for you?

Thanks,
--
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[Message part 2 (text/html, inline)]

Message #17 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andreas Fuchs <asf <at> boinkor.net>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50;
 [feature/native-comp] .eln path fixup confused using relative paths
Date: Tue, 01 Sep 2020 19:51:18 +0000

Andreas Fuchs <asf <at> boinkor.net> writes:

> I <asf <at> boinkor.net> wrote:
>> This makes sense too. I'll investigate when `Vinvocation_directory`
> gets
>> set. Does sound like that doesn't happen early enough, maybe we can
> pull
>> it forward a bit.
>
> I dug into this, and the way main() progresses is that it first tries
> to
> load the pdump (via load_pdump in line 1076,
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> emacs.c#L1076),
> and only much later, sets invocation-directory and other cmdargs via
> set_invocation_vars called from init_cmdargs (called in main on line
> 1794,
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> emacs.c#L1794).
>
> I think this should be failing cross-platform; is it working for you?
>
> Thanks,

Hi Andreas,

yes for me is set before.  I've made set_invocation_vars to be called
before doing what I've called LATE_RELOCS in the pdumper.

Here is my backtrace of where gdb stops just after Vinvocation_directory
is set.

 (gdb) bt
 #0  set_invocation_vars (argv0=0x7fffffffe693 "/home/andcor03/emacs2/src/emacs",
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at emacs.c:455
 #1  0x000000000057ade2 in pdumper_load (
     dump_filename=0xacb750 "/home/andcor03/emacs2/src/emacs.pdmp",
     argv0=0x7fffffffe693 "/home/andcor03/emacs2/src/emacs",
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at pdumper.c:5593
 #2  0x00000000004e3a2f in load_pdump (argc=2, argv=0x7fffffffe428,
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at emacs.c:868
 #3  0x00000000004e3f4d in main (argc=2, argv=0x7fffffffe428) at emacs.c:1076

Is it possible that set_invocation_vars on macos fails to set the
correct value?  I'd look into it as I guess something may go wrong
there.

  Andrea

Message #20 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Tue, 1 Sep 2020 20:51:43 -0400

[Message part 1 (text/plain, inline)]

I think I found the solution: set_invocation_vars requires Vexec_path to be
initialized correctly, via init_callproc_1. Somehow, mine ends up being
something other than $PATH. If I set init_callproc_1 up with a
double_run_guard the same way set_invocation_var is and call it from
pdumper_load, it results in a correctly-set Vexec_path variable and a
correctly-set Vinvocation_directory, as well.

This is really curious, but I guess it would explain why the invocation
directory variable is empty! What it doesn't explain is, why does this work
for you? Maybe your $PATH variable is set up differently? (Though I don't
see where exec-path would be set otherwise...)

[Message part 2 (text/html, inline)]

Message #23 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 43137 <at> debbugs.gnu.org
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Wed, 2 Sep 2020 00:41:46 -0400

[Message part 1 (text/plain, inline)]

For the exec-path issue, I have a patch here:
https://github.com/emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;
this ensures running `emacs` from PATH on my system finds the right binary,
and lets emacs start up (there is a weird delay though, which may well be
my system being creaky).

The same branch that this change is on, I also have a change to build a
separate .pdmp file for the macOS app bundle - but that set of changes
isn't yet correct. It still picks up the wrong file name, still.

However, good news: One part of the bug is dead, at least (:

[Message part 2 (text/html, inline)]

Message #26 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andreas Fuchs <asf <at> boinkor.net>
Cc: 43137 <at> debbugs.gnu.org, Vibhav Pant <vibhavp <at> gmail.com>
Subject: Re: bug#43137: 28.0.50;
 [feature/native-comp] .eln path fixup confused using relative paths
Date: Wed, 02 Sep 2020 07:28:49 +0000

Andreas Fuchs <asf <at> boinkor.net> writes:

> I think I found the solution: set_invocation_vars requires Vexec_path
> to be initialized correctly, via init_callproc_1. Somehow, mine ends
> up being something other than $PATH. If I set init_callproc_1 up with
> a double_run_guard the same way set_invocation_var is and call it
> from pdumper_load, it results in a correctly-set Vexec_path variable
> and a correctly-set Vinvocation_directory, as well.
> This is really curious, but I guess it would explain why
> the invocation directory variable is empty! What it doesn't explain
> is, why does this work for you? Maybe your $PATH variable is set up
> differently? (Though I don't see where exec-path would be set
> otherwise...)

I guess is beacuse argv0 is not set on macos so raw_name is empty.
Vexec_path comes into play only in this case.

  Andrea

Message #29 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andreas Fuchs <asf <at> boinkor.net>
Cc: 43137 <at> debbugs.gnu.org
Subject: Re: bug#43137: 28.0.50;
 [feature/native-comp] .eln path fixup confused using relative paths
Date: Wed, 02 Sep 2020 07:40:12 +0000

Andreas Fuchs <asf <at> boinkor.net> writes:

> For the exec-path issue, I have a patch here: https://github.com/
> emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;

I guess the best at this point would be to move out the invocation of
'set_invocation_vars' from 'init_cmdargs' and have 'set_invocation_vars'
and 'init_callproc_1' under a same wrapper with only one double
invocation guard.

Have you already done the copyright paperwork?  If not would you like to
do so?

  Andrea

Message #32 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Fuchs <asf <at> boinkor.net>
To: Andrea Corallo <akrl <at> sdf.org>
Cc: 43137 <at> debbugs.gnu.org
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Wed, 2 Sep 2020 09:49:04 -0400

[Message part 1 (text/plain, inline)]

On Wed, Sep 2, 2020 at 3:40 AM Andrea Corallo <akrl <at> sdf.org> wrote:

> Andreas Fuchs <asf <at> boinkor.net> writes
> > For the exec-path issue, I have a patch here: https://github.com/
> > emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;
>
> I guess the best at this point would be to move out the invocation of
> 'set_invocation_vars' from 'init_cmdargs' and have 'set_invocation_vars'
> and 'init_callproc_1' under a same wrapper with only one double
> invocation guard.
>

That makes sense - they're a bit far removed from each other in `main' but
are somewhat deeply intertwined, so I punted on this for now... your
suggestion is the right thing to do, of course (:


> Have you already done the copyright paperwork?  If not would you like to
> do so?


I've done my assignment for past & future contributions to GNU Emacs, a few
weeks ago! (:

Cheers,
-- 
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[Message part 2 (text/html, inline)]

Message #35 received at 43137 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andreas Fuchs <asf <at> boinkor.net>
Cc: 43137 <at> debbugs.gnu.org
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Sat, 26 Sep 2020 13:51:57 +0000

Hi Andreas,

I think
"dc0cf16c7a Always set 'Vexec_path' before 'Vinvocation_directory' (bug#43137)"
should fix the issue.  Could you have a look if it works for you?

Thanks!

  Andrea

Message #40 received at 43137-done <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <akrl <at> sdf.org>
To: Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>
Cc: Andreas Fuchs <asf <at> boinkor.net>, 43137-done <at> debbugs.gnu.org
Subject: Re: bug#43137: 28.0.50; [feature/native-comp] .eln path fixup
 confused using relative paths
Date: Fri, 02 Oct 2020 07:59:53 +0000

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs <at> gnu.org> writes:

> Hi Andreas,
>
> I think
> "dc0cf16c7a Always set 'Vexec_path' before 'Vinvocation_directory' (bug#43137)"
> should fix the issue.  Could you have a look if it works for you?
>

I'm closing this as I believe this is solved.  Please follow-up if some
problem shows-up so we can repoen it.

Thanks

  Andrea

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.