GNU bug report logs - #34322
reproducibility: absolute file name in tramp.elc

Previous Next

To reply to this bug, email your comments to 34322 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #3 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: submit <at> debbugs.gnu.org
Subject: reproducibility: absolute file name in tramp.elc
Date: Mon, 04 Feb 2019 17:58:50 -0500

Package: emacs
Version: 26.1.91
Severity: minor

Perhaps the same issue as https://debbugs.gnu.org/34321 .
The definition of tramp-lookup-syntax in tramp.elc contains the absolute
file name of the build directory, making the file non-reproducible.
Eg in the 26.1.91 pretest tarfile, it contains
"/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

Message #6 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 10:00:15 +0100

Glenn Morris <rgm <at> gnu.org> writes:

> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

Don't know how to fix it. For the records, in the emacs-26 branch,
tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\"\207" [tramp-syntax #[0 "\301\267\202\n\302\207\303\207\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
--8<---------------cut here---------------end--------------->8---

OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\301\267\202\302\202\303\202\236A\206\304\305\"\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10)) default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
--8<---------------cut here---------------end--------------->8---

I have no idea where the absolute file name in the former comes
from. Both repositories use the same Emacs 26.1.91 binary for the build.

Best regards, Michael.

Message #9 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Glenn Morris <rgm <at> gnu.org>,
    Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 18:14:54 +0200

> From: Glenn Morris <rgm <at> gnu.org>
> Date: Mon, 04 Feb 2019 17:58:50 -0500
> 
> Package: emacs
> Version: 26.1.91
> Severity: minor
> 
> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

It seems to be a reference to the place in tramp-compat.elc where
tramp-compat-tramp-syntax is defined.  Maybe some byte-compiler guru
could explain or guess when does the byte compiler use such
references.  Stefan, any ideas?

Message #12 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rgm <at> gnu.org, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 18:18:32 +0200

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Date: Tue, 05 Feb 2019 10:00:15 +0100
> Cc: 34322 <at> debbugs.gnu.org
> 
> Don't know how to fix it. For the records, in the emacs-26 branch,
> tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\"\207" [tramp-syntax #[0 "\301\267\202\n\302\207\303\207\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
> --8<---------------cut here---------------end--------------->8---
> 
> OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\301\267\202 \302\202 \303\202 \236A\206 \304\305\"\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10)) default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
> --8<---------------cut here---------------end--------------->8---
> 
> I have no idea where the absolute file name in the former comes
> from.

I tried to explain what it means in another message.

> Both repositories use the same Emacs 26.1.91 binary for the build.

I think whether there is or isn't such a reference depends on whether
tramp-compat.elc exists when you byte-compile tramp.c.  Try removing
the former or renaming it, and you get the contents you have in the
second excerpt.  So perhaps whether the reference exists or not
depends on the order of the compilation, and maybe also on how many
compilation jobs in parallel are run when boostrapping Emacs.

Message #15 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: rgm <at> gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 11:36:29 -0500

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\"\207"
>> [tramp-syntax #[0 "\301\267\202\n\302\207\303\207\207" [tramp-syntax
>> #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125
>> purecopy t data (ftp 6 sep 8)) default separate]
>> 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error
>> "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
>> --8<---------------cut here---------------end--------------->8---

Hmm... looks like a bug in the implementation of defsubst: it works by
replacing references to tramp-compat-tramp-syntax with a copy of the
function's byte-code (this is the #[0 ...] object above, which includes
the source file name info), and later on those copied bytecode objects
are normally spliced into the surrounding bytecode (at which point the
extra source info is dropped), but here it seems like it hasn't been
spliced as it should.

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\301\267\202 \302\202 \303\202
>> \236A\206 \304\305\"\207" [tramp-syntax #s(hash-table size 2 test eq
>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>> --8<---------------cut here---------------end--------------->8---

Here you can see that there is no nested bytecode object (i.e. no
#[...] with the main #[...]) so the call has been correctly inlined.

> I think whether there is or isn't such a reference depends on whether
> tramp-compat.elc exists when you byte-compile tramp.c.  Try removing

I don't think we've changed anything significant in the implementation
of defsubst so indeed it's likely triggered by something like the order
of compilation.


        Stefan

Message #18 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: rgm <at> gnu.org, Eli Zaretskii <eliz <at> gnu.org>, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 18:45:34 +0100

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>>> --8<---------------cut here---------------start------------->8---
>>> (defalias 'tramp-lookup-syntax #[257 "\301\267\202 \302\202 \303\202
>>> \236A\206 \304\305\"\207" [tramp-syntax #s(hash-table size 2 test eq
>>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>>> --8<---------------cut here---------------end--------------->8---
>
> Here you can see that there is no nested bytecode object (i.e. no
> #[...] with the main #[...]) so the call has been correctly inlined.

Given, that the absolute file name is not needed in the bytecode, I'm
wondering why we insert it there. Couldn't we live without?

>         Stefan

Best regards, Michael.

Message #21 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: rgm <at> gnu.org, Eli Zaretskii <eliz <at> gnu.org>, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Tue, 05 Feb 2019 15:41:44 -0500

>> Here you can see that there is no nested bytecode object (i.e. no
>> #[...] with the main #[...]) so the call has been correctly inlined.
> Given, that the absolute file name is not needed in the bytecode,

It's present in the in-memory bytecode object so that `C-h f` can jump
to the source upon demand, so it is needed.

> I'm wondering why we insert it there.

It's because we just insert the bytecode object wholesale (not a copy of
the object), so it comes with all its fields.

> Couldn't we live without?

We'd have to make a copy of the bytecode object, skipping the
source-code reference.

But really, this is actually a side-problem: the inlined bytecode is not
spliced the way it should, so the inlining optimization is basically
missing.  Such a half-assed inlining gives you all the downsides of
inlining without its upsides.  Once we fix that, the reproducibility
problem will also be fixed.


        Stefan

Message #26 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 34321 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>, 34322 <at> debbugs.gnu.org,
 Michael Albinus <michael.albinus <at> gmx.de>, rgm <at> gnu.org
Subject: Re: bug#34321: reproducibility: absolute file name in
 newst-treeview.elc
Date: Wed, 11 Aug 2021 21:14:47 +0200

Stefan Monnier <monnier <at> IRO.UMontreal.CA> writes:

> But really, this is actually a side-problem: the inlined bytecode is not
> spliced the way it should, so the inlining optimization is basically
> missing.  Such a half-assed inlining gives you all the downsides of
> inlining without its upsides.  Once we fix that, the reproducibility
> problem will also be fixed.

It's two years later, so I checked whether this problem is still present
on the trunk, and indeed:

(defalias 'tramp-lookup-syntax #[257 "..." [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/home/larsi/src/emacs/trunk/lisp/net/tramp-compat.elc" . 7627)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 29248)])

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #31 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 34321 <at> debbugs.gnu.org, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Sat, 29 Jan 2022 16:15:30 +0100

Glenn Morris <rgm <at> gnu.org> writes:

> For some reason, the definition of newsticker--group-shift in the
> compiled file newst-treeview.elc ends up containing a string that points
> to the build directory. Eg in the 26.1.91 pretest tarfile, it is:
>
> "/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"
>
> This means that newst-treeview.elc is non-reproducible (contents change
> as build directory changes).

I grepped through the news*.elc files for /home, but couldn't find any.
So has this problem fixed itself since it was reported?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #38 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 34321 <at> debbugs.gnu.org, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Sat, 29 Jan 2022 12:40:29 -0500

Lars Ingebrigtsen wrote:

> So has this problem fixed itself since it was reported?

No it has not.

find . -name '*.elc' | xargs grep /scratch
Binary file ./lisp/org/ox-odt.elc matches
Binary file ./lisp/org/ox.elc matches
Binary file ./lisp/progmodes/cc-mode.elc matches

Even if that didn't return any matches, the onus would still be there to
verify that it was due to a genuine bytecode fix rather than
coincidental changes in the source files or build ordering.

Message #41 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Glenn Morris <rgm <at> gnu.org>
Cc: 34321 <at> debbugs.gnu.org, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Sun, 30 Jan 2022 16:54:25 +0100

Glenn Morris <rgm <at> gnu.org> writes:

>> So has this problem fixed itself since it was reported?
>
> No it has not.
>
> find . -name '*.elc' | xargs grep /scratch
> Binary file ./lisp/org/ox-odt.elc matches
> Binary file ./lisp/org/ox.elc matches
> Binary file ./lisp/progmodes/cc-mode.elc matches

Ah, right, I also get my complete path in those .elc files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #44 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Glenn Morris <rgm <at> gnu.org>, 34321 <at> debbugs.gnu.org, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Sun, 30 Jan 2022 18:03:59 +0100

On Jan 30 2022, Lars Ingebrigtsen wrote:

> Glenn Morris <rgm <at> gnu.org> writes:
>
>>> So has this problem fixed itself since it was reported?
>>
>> No it has not.
>>
>> find . -name '*.elc' | xargs grep /scratch
>> Binary file ./lisp/org/ox-odt.elc matches
>> Binary file ./lisp/org/ox.elc matches
>> Binary file ./lisp/progmodes/cc-mode.elc matches
>
> Ah, right, I also get my complete path in those .elc files.

That appears to be a bug in the bytecode inliner.  In ox.elc the
function org-element-class (a defsubst) is supposed to be inlined into
org-export-data and org-export-expand, but instead the original bytecode
of org-element-class is put into the bytecode, including its doc string
reference (pointing to org-element.elc, which would normally be written
as $# if it appeared there instead of in ex.elc).

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #47 received at 34322 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 Glenn Morris <rgm <at> gnu.org>, 34321 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, 34322 <at> debbugs.gnu.org
Subject: Re: bug#34322: reproducibility: absolute file name in tramp.elc
Date: Sun, 01 May 2022 11:03:06 +0200

Andreas Schwab <schwab <at> linux-m68k.org> writes:

> That appears to be a bug in the bytecode inliner.  In ox.elc the
> function org-element-class (a defsubst) is supposed to be inlined into
> org-export-data and org-export-expand, but instead the original bytecode
> of org-element-class is put into the bytecode, including its doc string
> reference (pointing to org-element.elc, which would normally be written
> as $# if it appeared there instead of in ex.elc).

Hm, yes.

(As an aside, org-element-class should probably not be a defsubst,
because three other org files declare-function it because they don't
load it at compile time.  But ox.el does load it when compiling.)

Perhaps Stefan or Mattias have some input on what's going wrong with
inlining here; added to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.