GNU bug report logs - #57152
29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 57152 in the body.
You can then email your comments to 57152 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified
Date: Fri, 12 Aug 2022 10:03:44 +0300

The current Makefile's still exhibit the problem that when loaddefs.el
is updated, the Emacs executable is not re-dumped to reflect those
updates.  This leads to an annoying and error-prone practice that one
needs to watch the build, detect the telltale "ELC loaddefs.elc"
command (which means loaddefs.el was actually modified), and then
manually invoke "make" again to re-dump Emacs.

AFAICT, the root cause of this is that src/Makefile.in doesn't know
about the dependencies of loaddefs.el, which are spelled out in
lisp/Makefile.in, and therefore it doesn't realize that loaddefs.el
will be modified as part of the current build.



In GNU Emacs 29.0.50 (build 1694, i686-pc-mingw32)
 of 2022-08-12 built on HOME-C4E4A596F7
Repository revision: 687fcc2c40d693bbe342d938e37a4c9a245a1b02
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process emacs)

Memory information:
((conses 16 42199 12256)
 (symbols 48 6234 0)
 (strings 16 16487 2980)
 (string-bytes 1 396347)
 (vectors 16 9214)
 (vector-slots 8 144024 12454)
 (floats 8 23 45)
 (intervals 40 268 96)
 (buffers 888 10))

Message #8 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Fri, 12 Aug 2022 17:14:26 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> The current Makefile's still exhibit the problem that when loaddefs.el
> is updated, the Emacs executable is not re-dumped to reflect those
> updates.  This leads to an annoying and error-prone practice that one
> needs to watch the build, detect the telltale "ELC loaddefs.elc"
> command (which means loaddefs.el was actually modified), and then
> manually invoke "make" again to re-dump Emacs.
>
> AFAICT, the root cause of this is that src/Makefile.in doesn't know
> about the dependencies of loaddefs.el, which are spelled out in
> lisp/Makefile.in, and therefore it doesn't realize that loaddefs.el
> will be modified as part of the current build.

Yes.  So the next time you say "make", it'll rebuild the executable.

In one way, this is slightly better than what we had before, where
loaddefs.el wouldn't be rebuilt until after VCWITNESS had changed, while
it's now actually updated all the time (that something changes).  It's
more... regular?  But on the other hand, these updates are seldom
actually necessary, so we're doing it a bit too much now, perhaps.

Anyway, back to the problem -- I've poked at this quite a bit, but no
solutions seem really...  nice?  If we updated the loaddefs.el file a
bit earlier (before we build the Emacs executable), that'd fix the
problem, but I could find a way to do that without breaking something
else.  There's probably something trivial I'm overlooking here.

Message #11 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Fri, 12 Aug 2022 18:29:43 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: 57152 <at> debbugs.gnu.org
> Date: Fri, 12 Aug 2022 17:14:26 +0200
> 
> Anyway, back to the problem -- I've poked at this quite a bit, but no
> solutions seem really...  nice?  If we updated the loaddefs.el file a
> bit earlier (before we build the Emacs executable), that'd fix the
> problem, but I could find a way to do that without breaking something
> else.  There's probably something trivial I'm overlooking here.

Something similar to what we do with $(shortlisp), perhaps?

Message #14 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Sat, 13 Aug 2022 13:42:52 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

> Something similar to what we do with $(shortlisp), perhaps?

Yes, probably.  loaddefs.el is already in $(shortlisp), but the problem
is that we may later update loaddefs.el again (while doing the scan).

Hm; there must be some easy way to sort of these dependencies.
Unfortunately I don't have much time this weekend, but I'll poke away at
it again next week.

Message #17 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Mon, 15 Aug 2022 12:07:15 +0200

I've spent an hour poking at this.  Here's the recipe to reproduce:

echo ";;;###autoload ;;(+ 1 2)" >> lisp/foo.el
make

This will update loaddefs.el(c), but won't rebuild src/emacs.pdmp.
Then, if you say

make

it'll rebuild src/emacs.pdmp.

I've tried various things, like making autoloads an order-only
prerequisite for $(pdmp) (and various other targets), but the main
problem seems to be that Make has already computed the
timestamps/dependencies at this point, so even if we're running the
update at the "right" time, we don't trigger the $(pdmp) rule.

And moving this earlier results in problems with the boot build.

I think it'd be helpful if somebody with fresh eyes could take a look at
this.

Message #20 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Mon, 15 Aug 2022 14:58:07 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: 57152 <at> debbugs.gnu.org
> Date: Mon, 15 Aug 2022 12:07:15 +0200
> 
> I've tried various things, like making autoloads an order-only
> prerequisite for $(pdmp) (and various other targets), but the main
> problem seems to be that Make has already computed the
> timestamps/dependencies at this point, so even if we're running the
> update at the "right" time, we don't trigger the $(pdmp) rule.

That cannot be the reason, AFAIK.  Make begins by reading all the
rules and dependencies in the Makefile and producing a DAG for
rebuilding the targets; then it walks the DAG invoking commands as it
goes.  Thus, a target that is going to be rebuilt will automatically
trigger the rules of all the targets that depend on it, as long as the
dependencies are known to Make in full.

IOW, Make doesn't update the timestamps of files during the build, it
figures it all out in advance.

So I think the problem here is that the rules which trigger
regeneration of loaddefs.el are in lisp/Makefile, while the rules for
building $(pdmp) are in src/Makefile.

So maybe moving or duplicating the loaddefs.el rules in src/Makefile
will do the trick.

Message #23 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Mon, 15 Aug 2022 15:32:43 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi <at> gnus.org>
>> Cc: 57152 <at> debbugs.gnu.org
>> Date: Mon, 15 Aug 2022 12:07:15 +0200
>> 
>> I've tried various things, like making autoloads an order-only
>> prerequisite for $(pdmp) (and various other targets), but the main
>> problem seems to be that Make has already computed the
>> timestamps/dependencies at this point, so even if we're running the
>> update at the "right" time, we don't trigger the $(pdmp) rule.
>
> So maybe moving or duplicating the loaddefs.el rules in src/Makefile
> will do the trick.

Could it be that src/Makefile is simply not invoked after lisp/Makefile
has built loaddefs.*? In Makefile.in we have

SUBDIR = $(NTDIR) lib lib-src src lisp
...
all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)

That is src comes before lisp.  Haching something like a second 'make
-C' at the end seems to do something not entirely unreasonable.

Message #26 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Wed, 17 Aug 2022 12:34:13 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> I've tried various things, like making autoloads an order-only
>> prerequisite for $(pdmp) (and various other targets), but the main
>> problem seems to be that Make has already computed the
>> timestamps/dependencies at this point, so even if we're running the
>> update at the "right" time, we don't trigger the $(pdmp) rule.

[...]

> IOW, Make doesn't update the timestamps of files during the build, it
> figures it all out in advance.

I think we're saying the same thing.  😀

> So I think the problem here is that the rules which trigger
> regeneration of loaddefs.el are in lisp/Makefile, while the rules for
> building $(pdmp) are in src/Makefile.
>
> So maybe moving or duplicating the loaddefs.el rules in src/Makefile
> will do the trick.

There really are no rules for loaddefs.el -- Emacs itself (via
loaddefs-gen) figures out whether loaddefs.el needs updating.  I.e.,
Make knows nothing about the dependencies loaddefs.el really has.

We could make loaddefs.el depend on all .el files, but then we'd rebuild
the Emacs executable every time we change an .el file, and we don't want
to do that, either.

Message #29 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Wed, 17 Aug 2022 12:40:32 +0200

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> Could it be that src/Makefile is simply not invoked after lisp/Makefile
> has built loaddefs.*? In Makefile.in we have
>
> SUBDIR = $(NTDIR) lib lib-src src lisp
> ...
> all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
>
> That is src comes before lisp.  Haching something like a second 'make
> -C' at the end seems to do something not entirely unreasonable.

Hm, interesting...  but I think we might end up in a situation where we
first build the Emacs executable, then update the loaddefs.el, and then
build the Emacs executable again.

But perhaps that's OK -- while we're scanning for new loaddefs every
build, there's seldom any new ;;;###autoloads, so the loaddefs.el file
doesn't update all that often.

I'm not quite sure where the second "make -C" would go, though.

Message #32 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Wed, 17 Aug 2022 15:08:08 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Cc: 57152 <at> debbugs.gnu.org
> Date: Wed, 17 Aug 2022 12:34:13 +0200
> 
> We could make loaddefs.el depend on all .el files, but then we'd rebuild
> the Emacs executable every time we change an .el file, and we don't want
> to do that, either.

What about using move-if-change?  That's a standard technique of
avoiding large rebuilds because some dependency's rules cannot be
easily stated.

Message #35 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <akrl <at> sdf.org>,
 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Wed, 17 Aug 2022 15:50:35 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:
>
>> Could it be that src/Makefile is simply not invoked after lisp/Makefile
>> has built loaddefs.*? In Makefile.in we have
>>
>> SUBDIR = $(NTDIR) lib lib-src src lisp
>> ...
>> all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
>>
>> That is src comes before lisp.  Haching something like a second 'make
>> -C' at the end seems to do something not entirely unreasonable.
>
> Hm, interesting...  but I think we might end up in a situation where we
> first build the Emacs executable, then update the loaddefs.el, and then
> build the Emacs executable again.

I haven't observed that with your 'echo ...>>foo.el; make' example.

But wouldn't that be a hint that there is something fishy in
src/Makefile?  Or incomplete, or something?

>
> But perhaps that's OK -- while we're scanning for new loaddefs every
> build, there's seldom any new ;;;###autoloads, so the loaddefs.el file
> doesn't update all that often.
>
> I'm not quite sure where the second "make -C" would go, though.

I did it this way:

diff --git a/Makefile.in b/Makefile.in
index bf0f52b514..78103f897f 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -358,10 +358,17 @@ ELN_DESTDIR =
 
 gsettings_SCHEMAS = etc/org.gnu.emacs.defaults.gschema.xml
 
-all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
+all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid) src-depending-on-lisp
 
 .PHONY: all ${SUBDIR} blessmail epaths-force epaths-force-w32 epaths-force-ns-self-contained etc-emacsver
 
+# Changes in lisp may require us to reconsider the build in src.  For
+# example, if loaddefs.{el,elc} were built in lisp, we need a new
+# .pdmp containing the new autoloads.
+.PHONY: src-depending-on-lisp
+src-depending-on-lisp: lisp
+	${MAKE} -C src
+
 # If configure were to just generate emacsver.tex from emacsver.tex.in
 # in the normal way, the timestamp of emacsver.tex would always be
 # newer than that of the pdf files, which are prebuilt in release tarfiles.


The dependency on lisp in src-depending-on-lisp is necessary to make
sure it runs after lisp has been built.  Maybe there is a clever trick
to express "only if loaddefs has been regenerated" in some way, but
'make -C src' is quite fast already if it hasn't to do anything, at
least compared to other things.

There is one pretty strange thing though, which I can't explain.  I once
got an error "trying to load incoherent eln" which is mentioned in
bug#45103.  I can't say much else about this though, not even what the
cause of this might be.  It was a build with -j8 if that matters.  Maybe
elns are modified during dumping, so that a second dump cannot be done?
But that wouldn't explain why it succeeds most of the time.  No idea,
and not easily reproducible, alas.

CC'ing Andrea.

Message #38 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <akrl <at> sdf.org>,
 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Thu, 18 Aug 2022 14:46:55 +0200

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

>> I'm not quite sure where the second "make -C" would go, though.
>
> I did it this way:

[...]

> +all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid) src-depending-on-lisp

Ah, right.  I've now done some testing with your patch, and it fixes the
problem (without introducing any new extra src/emacs builds in a
bootstrap or otherwise).  So I just went ahead and pushed it.  :-)

> There is one pretty strange thing though, which I can't explain.  I once
> got an error "trying to load incoherent eln" which is mentioned in
> bug#45103.  I can't say much else about this though, not even what the
> cause of this might be.  It was a build with -j8 if that matters.  Maybe
> elns are modified during dumping, so that a second dump cannot be done?
> But that wouldn't explain why it succeeds most of the time.  No idea,
> and not easily reproducible, alas.

Yes, I've seen those, too, but very seldom (like -- once a month?), and
I haven't been able to reproduce it.

Message #43 received at 57152 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Andrea Corallo <akrl <at> sdf.org>,
 57152 <at> debbugs.gnu.org
Subject: Re: bug#57152: 29.0.50; Emacs executable isn't rebuilt when
 loaddefs.el is modified
Date: Fri, 19 Aug 2022 11:24:36 +0200

On 22-08-18 14:46 , Lars Ingebrigtsen wrote:
> So I just went ahead and pushed it. :-) 

Oh no :-).

> There is one pretty strange thing though, which I can't explain. I once
>> got an error "trying to load incoherent eln" which is mentioned in
>> bug#45103.  I can't say much else about this though, not even what the
>> cause of this might be.  It was a build with -j8 if that matters.  Maybe
>> elns are modified during dumping, so that a second dump cannot be done?
>> But that wouldn't explain why it succeeds most of the time.  No idea,
>> and not easily reproducible, alas.
> Yes, I've seen those, too, but very seldom (like -- once a month?), and
> I haven't been able to reproduce it.

If anyone reading this has any idea, be it just a gut feeling or 
something, how to reproduce this failure, even if not 100% reliable, I'd 
be grateful to hear it.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.