GNU bug report logs - #78995
[PATCH] ;;;autoload-expand for special macros

Previous Next

To reply to this bug, email your comments to 78995 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: [PATCH] ;;;autoload-expand for special macros
Date: Fri, 11 Jul 2025 15:28:03 -0400

[Message part 1 (text/plain, inline)]

As mentioned over on emacs-devel, it's sometimes a problem that `loaddefs-gen' hard-codes the list of special macros it is willing to expand:

(memq car '(easy-mmode-define-global-mode define-global-minor-mode
			    define-globalized-minor-mode defun defmacro
			    easy-mmode-define-minor-mode define-minor-mode
			    define-inline cl-defun cl-defmacro cl-defgeneric
			    cl-defstruct pcase-defmacro iter-defun cl-iter-defun))

This makes it challenging to wrap things like `define-minor-mode' into a new macro, and autoload the resulting forms.  This small patch adds a new variant of the autoload cookie:

;;;autoload-expand
(my-autoloadable-macro-to-expand ...)

which allows package authors to "opt-in" to macro expansion of the following form during autoload generation.  It would be useful primarily for special macros that are not in the hard-coded list above, but which expand to other special macros that are.

Docs can be added if this looks good.

[autoload-expand.patch (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

Message #8 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: JD Smith <jdtsmith <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Philip Kaludercic <philipk <at> posteo.net>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 09:10:05 +0300

> From: JD Smith <jdtsmith <at> gmail.com>
> Date: Fri, 11 Jul 2025 15:28:03 -0400
> 
>  (defun loaddefs-generate--make-autoload (form file &optional expansion)
>    "Turn FORM into an autoload or defvar for source file FILE.
> -Returns nil if FORM is not a special autoload form (i.e. a function definition
> -or macro definition or a defcustom).
> -If EXPANSION is non-nil, we're processing the macro expansion of an
> -expression, in which case we want to handle forms differently."
> +Returns nil if FORM is not a special autoload form (i.e. a function
> +definition or macro definition or a defcustom).  If EXPANSION is
> +non-nil, we're processing the macro expansion of an expression, in which
> +case we want to handle forms differently.  If it the symbol `force',
                                                 ^^^^^^^^^^^^^
"it is the symbol".

Also, I wonder why you decided to refill the existing text.  A doc
string is easier readable if each argument's description starts on a
new line, because then it's easier to scan the doc string if you are
looking for the description of a specific argument.

And finally, the last sentence, which you added, doesn't really
explain the effect of the special value 'force' -- can we do a better
job describing that?

> -                              (or local-outfile main-outfile))))
> +                              (or local-outfile main-outfile)))
> +		   (expansion (and (looking-at "-expand[ \t]*") 'force)))

And this part seems to be completely undocumented?

Message #11 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: JD Smith <jdtsmith <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Philip Kaludercic <philipk <at> posteo.net>, 78995 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 10:59:45 -0400

[Message part 1 (text/plain, inline)]

Thanks, improved doc-strings in the attached.  If this is looking good I can also update the `Autoload' section of the elisp manual, near "If you write a function definition with an unusual macro...".  I also noticed that `:autoload-end' is not mentioned there, and probably should be (this keyword marker halts further processing of forms in the macro expansion).

One question on nesting of macros: I'm wondering whether an ;;;###autoload-expand should result in using `EXPANSION=force' in all recursive calls to `loaddefs-generate--make-autoload'.  I.e. should all macros be _recursively_ expanded with `force'?  This would allow nesting, e.g. a macro wrapping a macro which itself wraps `define-minor-mode'.  But it would also expand all other macros encountered in the expansion, which could be problematic.  My instinct is that one level of such forced macro expansion is all that can be safely supported.

[autoload-expand.patch (application/octet-stream, attachment)]

Message #14 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 12:09:55 -0400

> This makes it challenging to wrap things like `define-minor-mode' into a new
> macro, and autoload the resulting forms.  This small patch adds a new
> variant of the autoload cookie:
>
> ;;;autoload-expand
> (my-autoloadable-macro-to-expand ...)

I suspect this will often fail to work because autoloads are often
generated by a fresh new session where `my-autoloadable-macro-to-expand`
will not yet be defined.

So at the very least, your patch should make sure to signal an error
when the cookie is of the form `;;;autoload-expand` by the head is not
currently defined as a macro.

I've been wanting to add the feature you propose, but this "macro is not
defined" has gotten in the way until now.  I think we need to add
something like a file-local `autoload-requires-for-macros`.


        Stefan

Message #17 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: JD Smith <jdtsmith <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 78995 <at> debbugs.gnu.org,
 Philip Kaludercic <philipk <at> posteo.net>
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 12:12:42 -0400

> One question on nesting of macros: I'm wondering whether
> an ;;;###autoload-expand should result in using `EXPANSION=force' in all
> recursive calls to `loaddefs-generate--make-autoload'.  I.e. should all
> macros be _recursively_ expanded with `force'?  This would allow nesting,
> e.g. a macro wrapping a macro which itself wraps `define-minor-mode'.
> But it would also expand all other macros encountered in the expansion,
> which could be problematic.  My instinct is that one level of such forced
> macro expansion is all that can be safely supported.

Another way might be to replace the `;;;###autoload-expand` marker with
an list of of macros which should be expanded (i.e. a way to extent the
hardcoded list).


        Stefan

Message #20 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 13:26:04 -0400

[Message part 1 (text/plain, inline)]

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> This makes it challenging to wrap things like `define-minor-mode' into a new
>> macro, and autoload the resulting forms.  This small patch adds a new
>> variant of the autoload cookie:
>>
>> ;;;autoload-expand
>> (my-autoloadable-macro-to-expand ...)
>
> I suspect this will often fail to work because autoloads are often
> generated by a fresh new session where `my-autoloadable-macro-to-expand`
> will not yet be defined.

Good point.  That chicken-and-egg problem was definitely in the back of
my mind, and is why I didn't attempt to create an expandable list to
replace the list of hard-coded special macros.

> I've been wanting to add the feature you propose, but this "macro is not
> defined" has gotten in the way until now.  I think we need to add
> something like a file-local `autoload-requires-for-macros`.

How about this solution?  If the car of the following form is not a
macro (and not a function), attempt to load the file itself (temporarily
expanding `load-path' to include the file's containing directory).  If
that fails to define the macro, warn the user, and simply proceed
without expansion.

[autoload-expand.patch (text/x-patch, inline)]

diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 0f136df1fe2..f7919a7a377 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -147,7 +147,10 @@ loaddefs-generate--make-autoload
 Returns nil if FORM is not a special autoload form (i.e. a function definition
 or macro definition or a defcustom).
 If EXPANSION is non-nil, we're processing the macro expansion of an
-expression, in which case we want to handle forms differently."
+expression, in which case we want to handle forms differently.  If
+EXPANSION is the symbol `force' and the following form is a macro call,
+it will be expanded prior to generating autoload forms, similar to what
+is done automatically for special macros like `define-minor-mode'."
   (let ((car (car-safe form)) expand)
     (cond
      ((and expansion (eq car 'defalias))
@@ -196,12 +199,13 @@ loaddefs-generate--make-autoload
                                       (cdr form)))))
           (when exps (cons 'progn exps)))))
 
-     ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '(easy-mmode-define-global-mode define-global-minor-mode
-                       define-globalized-minor-mode defun defmacro
-		       easy-mmode-define-minor-mode define-minor-mode
-                       define-inline cl-defun cl-defmacro cl-defgeneric
-                       cl-defstruct pcase-defmacro iter-defun cl-iter-defun))
+     ;; For complex cases and forced expansions, try again on the macro-expansion.
+     ((and (or (eq expansion 'force)
+	       (memq car '(easy-mmode-define-global-mode define-global-minor-mode
+			    define-globalized-minor-mode defun defmacro
+			    easy-mmode-define-minor-mode define-minor-mode
+			    define-inline cl-defun cl-defmacro cl-defgeneric
+			    cl-defstruct pcase-defmacro iter-defun cl-iter-defun)))
            (macrop car)
 	   (setq expand (let ((load-true-file-name file)
                               (load-file-name file))
@@ -381,6 +385,10 @@ loaddefs-generate--parse-file
 setting of `generated-autoload-file' in FILE, and
 by ;;;###foo-autoload statements.
 
+The special statement ;;;###[foo-]autoload-expand causes the following
+form to be macro-expanded prior to generating autoloads; see
+`loaddefs-generate--make-autoload'.
+
 If PACKAGE-DATA is `only', return only the package data.  If t,
 include the package data with the rest of the data.  Otherwise,
 don't include."
@@ -465,16 +473,38 @@ loaddefs-generate--parse-file
                                 (expand-file-name
                                  (concat aname "-loaddefs.el")
                                  (file-name-directory file))
-                              (or local-outfile main-outfile))))
+                              (or local-outfile main-outfile)))
+		   (expansion (and (looking-at "-expand[ \t]*") 'force)))
+	      (when expansion
+		(goto-char (match-end 0)))
               (if (eolp)
                   ;; We have a form following.
                   (let* ((form (prog1
                                    (read (current-buffer))
                                  (unless (bolp)
                                    (forward-line 1))))
-                         (autoload (or (loaddefs-generate--make-autoload
-                                        form load-name)
-                                       form)))
+                         (car (car form))
+                         autoload)
+                    (when expansion
+                      ;; We encountered ;;;###autoload-expand, so the
+                      ;; following form should be a macro.  Check that
+                      ;; it is defined.  If it is undefined (and not a
+                      ;; function), load the file to attempt to define
+                      ;; it.  If it remains undefined, warn and simply
+                      ;; ignore the expand request.
+                      (unless (or (macrop car) (functionp car))
+                        (let ((load-path (cons (file-name-directory file) load-path)))
+                         (message "loaddefs-gen: loading file %s" file)
+                         (condition-case e (load file)
+                           (error
+                            (warn "loaddefs-gen: load error\n\t%s" e)))))
+                      (unless (macrop car)
+                        (warn "loaddefs-gen: macro undefined, skipping expansion\n\t%s"
+                              car)
+                        (setq expansion nil)))
+                    (setq autoload (or (loaddefs-generate--make-autoload
+                                        form load-name expansion)
+                                       form))
                     ;; We get back either an autoload form, or a tree
                     ;; structure of `(progn ...)' things, so unravel that.
                     (let ((forms (if (eq (car autoload) 'progn)

Message #23 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 13:51:09 -0400

>> I've been wanting to add the feature you propose, but this "macro is not
>> defined" has gotten in the way until now.  I think we need to add
>> something like a file-local `autoload-requires-for-macros`.
> How about this solution?  If the car of the following form is not a
> macro (and not a function), attempt to load the file itself (temporarily
> expanding `load-path' to include the file's containing directory).  If
> that fails to define the macro, warn the user, and simply proceed
> without expansion.

I don't think it's good enough.  Many of the use-cases I've bumped into
use macros defined in another file (or even another package).

I think we should aim to provide a mechanism that lets us reduce the
size of the hardcoded list (ideally to nothing).


        Stefan

Message #26 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 15:59:44 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>>> I've been wanting to add the feature you propose, but this "macro is not
>>> defined" has gotten in the way until now.  I think we need to add
>>> something like a file-local `autoload-requires-for-macros`.
>> How about this solution?  If the car of the following form is not a
>> macro (and not a function), attempt to load the file itself (temporarily
>> expanding `load-path' to include the file's containing directory).  If
>> that fails to define the macro, warn the user, and simply proceed
>> without expansion.
>
> I don't think it's good enough.  Many of the use-cases I've bumped into
> use macros defined in another file (or even another package).

Doesn't a file usually need to require such files/packages in order to
make use of their macros?  Or is the concern that those other
files/packages would not be in the current directory or on the
load-path?  I think package-install installs all requirements first, so
that shouldn't happen for it.

Or perhaps the issue arises in cases like multi-file packages where
`declare-function' is used because you know the load order.  I.e. if:

1. A.el always requires B.el and is its entry point,
2. B.el uses a macro from A.el with an ;;;###autoload-expand, and
3. B.el does _not_ require A.el, only (possibly) mentioning it via
   `declare-function`.

Then loading B.el is, as you say, not good enough.  This is tricky to
work around.

> I think we should aim to provide a mechanism that lets us reduce the
> size of the hardcoded list (ideally to nothing).

There are actually _multiple_ hard-coded lists in
`loaddefs-generate--make-autoload':

1. Macros to expand (by name) and recurse on the expansion.
2. "Function-like operators", which get an `(autoload ...)' form
  created for them.
3. Macros which are known to produce interactive function definitions
   (to set the INTERACTIVE arg for the generated autoload).

What about just mentioning in the docs that using ;;;###autoload-expand
successfully requires that the macro to be expanded is defined upon
loading the containing file?  There is a prominent warning if loading
the file fails to define the macro, so developers should be aware (or
made aware by their users).  They can always, e.g., factor out their
macro to a common file they can require.  The "fallback" is equivalent
to the current behavior: copy the full form into the autoload file.

Otherwise, if a documentation solution is not good enough, we could
create an additional file-local variable that could be consulted if the
user of ;;;###autoload-expand needs to give it some help, e.g.:

- `autoload-macro-expansion-files': a file-local list of additional
  file(s) to load prior to macro expansion, if any macros in calls
  directly preceded by ;;;###autoload-expand are undefined.  A symbol
  `file' in this list indicates the current file.  Defaults to `(file)'.

Message #29 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 18:43:56 -0400

>> I don't think it's good enough.  Many of the use-cases I've bumped into
>> use macros defined in another file (or even another package).
> Doesn't a file usually need to require such files/packages in order to
> make use of their macros?

Duh!  You're right.

> Or perhaps the issue arises in cases like multi-file packages where
> `declare-function' is used because you know the load order.  I.e. if:

No, no, these are "evil" and they get what they deserve.

>> I think we should aim to provide a mechanism that lets us reduce the
>> size of the hardcoded list (ideally to nothing).
>
> There are actually _multiple_ hard-coded lists in
> `loaddefs-generate--make-autoload':
>
> 1. Macros to expand (by name) and recurse on the expansion.
> 2. "Function-like operators", which get an `(autoload ...)' form
>   created for them.
> 3. Macros which are known to produce interactive function definitions
>    (to set the INTERACTIVE arg for the generated autoload).

The only list that I've found a need to extend is the (1) above, so
I wouldn't worry too much about the other two.

[ Declaring specific macros as "to be expanded for autoload" (as opposed
  to marking specific autoloaded macro calls), would still be overall
  better, since it would solve the question of "what about macros that
  expand to something that also needs expanding".  ]


        Stefan

Message #32 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 22:25:05 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> The only list that I've found a need to extend is the (1) above, so
> I wouldn't worry too much about the other two.
>
> [ Declaring specific macros as "to be expanded for autoload" (as opposed
>   to marking specific autoloaded macro calls), would still be overall
>   better, since it would solve the question of "what about macros that
>   expand to something that also needs expanding".  ]

That's a good point.  It would relieve us of the binary choice of
"expand just this form" vs. "expand any and all macros this form expands
to", which isn't ideal.  But I wonder what the best approach is?  A
file-local variable, which extends the (1) list only for that file?

We _could_ do both: an ;;;###autoload-expand magic comment for "expand
this one", and a optional file-local variable, something like
`autoload-extra-expand-macros', naming additional macros to expand at
any depth[1] (if needed).  We could even allow elements of the list to
look like (MACRO . FILE) for an additional FILE to define MACRO load
(for the evil people), defaulting to `(file)' (meaning: this file).

That's some duplication of purpose, and you don't technically need the
magic form, but formatting file-local variables is a pain if you don't
need them.

[1] Would we possibly have to guard against recursive macro definitions?
    An advantage of "just this next form" expansion is you don't have to
    worry about that.

Message #35 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sat, 12 Jul 2025 22:47:44 -0400

>> The only list that I've found a need to extend is the (1) above, so
>> I wouldn't worry too much about the other two.
>>
>> [ Declaring specific macros as "to be expanded for autoload" (as opposed
>>   to marking specific autoloaded macro calls), would still be overall
>>   better, since it would solve the question of "what about macros that
>>   expand to something that also needs expanding".  ]
>
> That's a good point.  It would relieve us of the binary choice of
> "expand just this form" vs. "expand any and all macros this form expands
> to", which isn't ideal.  But I wonder what the best approach is?

FWIW, I've come to the opinion that what you propose and what I want
match different use-cases.  The one I care about is when a package
provides a (typically autoloaded) macro like `define-minor-mode`, for
use by other packages.  But what you propose matches the use-case where
a package defines its own macro for it own internal use, usually with
no intention to autoload it.

> A file-local variable, which extends the (1) list only for that file?

I'm thinking we should replace the hardcoded list with a symbol
property.  When defining macros like `define-minor-mode` we'd set this
`autoload-macroexpand` property to a non-nil value, and `autoload.el`
would simply check the property instead of looking inside the hardcoded
list with `memq`.

This would work only for predefined or autoloaded macros (where the
`autoload-macroexpand` setting would be also pre-defined/loaded), so
it wouldn't work conveniently for a locally-defined and locally-used
macro, since you'd then need to somehow set the property and define or
autoload the macro before `autoload.el` gets to the macro calls.

So I think your approach might be a good solution for the case of local
macros.  I don't know how often we need such a thing, tho.  I can't
remember bumping into it, but it might be because I'd intuitively avoid
it, knowing we don't have support for it.

I assume there's a concrete use-case which motivated your patch.
Do you mind sharing it?


        Stefan

Message #38 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 09:47:48 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:


>> A file-local variable, which extends the (1) list only for that file?
>
> I'm thinking we should replace the hardcoded list with a symbol
> property.  When defining macros like `define-minor-mode` we'd set this
> `autoload-macroexpand` property to a non-nil value, and `autoload.el`
> would simply check the property instead of looking inside the hardcoded
> list with `memq`.

Oh, this is much better!  It relieves the _user_ of the macro the burden
of remembering to "auto-load it correctly."  For macros which are
defined in every session, you're done.  No special autoload-expand magic
comment needed. It also makes it possible to entirely eliminate
hard-coded list #1.  You'd just need to insert, e.g.:

  (put 'defun 'autoload-macroexpand t)

where the special macros are _defined_.  This would be a a bit more
self-documenting too.

We'd leave the "function-like operator" hard-coded list, but it could
probably be trimmed down with the new `autoload-macroexpand' feature.
For example, `transient-define-prefix' boils down to a defalias, which
is already supported.

> This would work only for predefined or autoloaded macros (where the
> `autoload-macroexpand` setting would be also pre-defined/loaded), so
> it wouldn't work conveniently for a locally-defined and locally-used
> macro, since you'd then need to somehow set the property and define or
> autoload the macro before `autoload.el` gets to the macro calls.

We could be aggressive about it.  If the car of some form encountered
during file scanning (including expansion) is a macro, we check its
`autoload-macroexpand' property and act accordingly.  But if it not
already a known special-form, function, or macro, we load the file to
see if that defines it.  If it _remains_ undefined, we just include the
whole form in the autoload file, just as we do now (generating a
warning).  But if loading the file _did_ define it as a macro, we check
its `autoload-macroexpand' property, and proceed as before.

I was curious how many existing ;;;###autoloads this would affect, since
I almost always see it used with `defun' or `define-minor-mode'.  I
looked through the entire emacs-30 lisp codebase[1] and could only find
`define-erc-module' as the car of a top-level autoload form which wasn't
already a known function, macro, or special form.  I repeated this
exercise among my 171 installed packages, and found only:

  Unknown autoload car: cape-char--define
  Unknown autoload car: define-erc-module

Importantly, in all of these cases, they are actually using adorned
magic comments, like:

  ;;;###autoload (autoload 'cape-tex "cape-char" nil t)
  (cape-char--define tex "TeX" ?\\ ?^ ?_)

so not relevant to our proposal (except that they could be improved by
it.)

> I assume there's a concrete use-case which motivated your patch.
> Do you mind sharing it?

I have a large package in development which has a flexible extension
system.  I'd like to enable "3rd-party" extensions with a convenience
`define-extension' macro.  In addition to defining a minor mode, this
macro will generate various `(put some-extension-mode ...)' statements,
which themselves need to be auto-loaded, as they are used to determine
when to load the extension (including, possibly, at package load time).

[1] Here's the distribution of pure ;;;###autoload forms in the emacs-30
    codebase:

     defun: 3226
     defcustom: 181
     define-minor-mode: 173
     define-derived-mode: 157
     defmacro: 130
     defalias: 101
     put: 88
     defvar: 74
     define-obsolete-function-alias: 27
     defvar-local: 20
     define-overloadable-function: 19
     cl-defgeneric: 19
     defconst: 15
     define-globalized-minor-mode: 15
     progn: 12
     defclass: 8
     add-to-list: 8
     cl-defun: 8
     defsubst: 7
     cl-defmacro: 5
     define-mail-user-agent: 4
     or: 3
     pcase-defmacro: 3
     define-global-minor-mode: 2
     make-variable-buffer-local: 2
     eval-after-load: 2
     dolist: 2
     define-key: 2
     unless: 1
     autoload: 1
     def-edebug-elem-spec: 1
     defvar-keymap: 1
     define-inline: 1
     when: 1
     define-compilation-mode: 1
     define-skeleton: 1
     let: 1
     fset: 1
     setq: 1

   Created with:

   rg -IA 1 -t elisp '^;;;###autoload *$' | perl -ne '$AL{$1}++ if /^\(([^ ]+) /; END { foreach $k (reverse sort { $AL{$a} <=> $AL{$b} } keys %AL) {print "$k: $AL{$k}\n"}}' 

Message #41 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 11:19:08 -0400

>>> A file-local variable, which extends the (1) list only for that file?
>> I'm thinking we should replace the hardcoded list with a symbol
>> property.  When defining macros like `define-minor-mode` we'd set this
>> `autoload-macroexpand` property to a non-nil value, and `autoload.el`
>> would simply check the property instead of looking inside the hardcoded
>> list with `memq`.
>
> Oh, this is much better!  It relieves the _user_ of the macro the burden
> of remembering to "auto-load it correctly."  For macros which are
> defined in every session, you're done.  No special autoload-expand magic
> comment needed. It also makes it possible to entirely eliminate
> hard-coded list #1.  You'd just need to insert, e.g.:
>
>   (put 'defun 'autoload-macroexpand t)
>
> where the special macros are _defined_.  This would be a a bit more
> self-documenting too.

Yes, tho we'd probably define a `declare` thingy so you wouldn't write
the `put` but you'd write

    (defmacro define-minor-mode ...
      (declare (...
                (autoload macroexpand)
                ...))
      ...)

> We'd leave the "function-like operator" hard-coded list, but it could
> probably be trimmed down with the new `autoload-macroexpand' feature.
> For example, `transient-define-prefix' boils down to a defalias, which
> is already supported.

Ah, yes, I had not noticed that `transient-define-prefix` had been added
there, and now that I see it, it does make it seem like my impression
that it was unimportant may have been wrong.
Usually macroexpanding those things is enough (the macro can be defined
appropriately).  We could still refine the above with something like:

    (defmacro transient-define-prefix ...
      (declare (...
                (autoload function)
                ...))
      ...)

tho this probably needs more thought/design because we need more info
(where's the name of the defined thing, whether it's a macro or
a function, what's the list of arguments, ...).

>> This would work only for predefined or autoloaded macros (where the
>> `autoload-macroexpand` setting would be also pre-defined/loaded), so
>> it wouldn't work conveniently for a locally-defined and locally-used
>> macro, since you'd then need to somehow set the property and define or
>> autoload the macro before `autoload.el` gets to the macro calls.
>
> We could be aggressive about it.  If the car of some form encountered
> during file scanning (including expansion) is a macro, we check its
> `autoload-macroexpand' property and act accordingly.  But if it not
> already a known special-form, function, or macro, we load the file to
> see if that defines it.

Oh, yeah, neat.

> Importantly, in all of these cases, they are actually using adorned
> magic comments, like:
>
>   ;;;###autoload (autoload 'cape-tex "cape-char" nil t)
>   (cape-char--define tex "TeX" ?\\ ?^ ?_)
>
> so not relevant to our proposal (except that they could be improved by
> it.)

Yes, these are the ones that "document" a need for the new feature.


        Stefan


diff --git a/lisp/emacs-lisp/byte-run.el b/lisp/emacs-lisp/byte-run.el
index 6412c8cde22..654b67ada1b 100644
--- a/lisp/emacs-lisp/byte-run.el
+++ b/lisp/emacs-lisp/byte-run.el
@@ -286,6 +286,11 @@
 	    (list 'put (list 'quote name)
 		  ''edebug-form-spec (list 'quote spec)))))
 
+(defalias 'byte-run--set-autoload-macro
+  #'(lambda (name _args spec)
+     (list 'put (list 'quote name)
+	   ''autoload-macro (list 'quote spec))))
+
 (defalias 'byte-run--set-no-font-lock-keyword
   #'(lambda (name _args val)
       (list 'function-put (list 'quote name)
@@ -365,8 +370,10 @@ macro-declarations-alist
   (cons
    (list 'debug #'byte-run--set-debug)
    (cons
-    (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
-    defun-declarations-alist))
+    (list 'autoload-macro #'byte-run--set-autoload-macro)
+    (cons
+     (list 'no-font-lock-keyword #'byte-run--set-no-font-lock-keyword)
+     defun-declarations-alist)))
   "List associating properties of macros to their macro expansion.
 Each element of the list takes the form (PROP FUN) where FUN is a function.
 For each (PROP . VALUES) in a macro's declaration, the FUN corresponding
diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index 8a131bf885f..b8dfe095589 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -198,19 +198,20 @@
           (when exps (cons 'progn exps)))))
 
      ;; For complex cases, try again on the macro-expansion.
-     ((and (memq car '( define-globalized-minor-mode defun defmacro
+     ((and (or (eq 'expand (get car 'autoload-macro))
+               (memq car '( define-globalized-minor-mode defun defmacro
                         define-minor-mode define-inline
                         cl-defun cl-defmacro cl-defgeneric
                         cl-defstruct pcase-defmacro iter-defun cl-iter-defun
                         ;; Obsolete; keep until the alias is removed.
                         easy-mmode-define-global-mode
                         easy-mmode-define-minor-mode
-                        define-global-minor-mode))
+                            define-global-minor-mode)))
            (macrop car)
 	   (setq expand (let ((load-true-file-name file)
                               (load-file-name file))
                           (macroexpand form)))
-	   (memq (car expand) '(progn prog1 defalias)))
+	   (not (eq car (car expand))))
       ;; Recurse on the expansion.
       (loaddefs-generate--make-autoload expand file 'expansion))
 

Message #44 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 15:01:07 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>> You'd just need to insert, e.g.:
>>
>>   (put 'defun 'autoload-macroexpand t)
>>
>> where the special macros are _defined_.  This would be a a bit more
>> self-documenting too.
>
> Yes, tho we'd probably define a `declare` thingy so you wouldn't write
> the `put` but you'd write
>
>     (defmacro define-minor-mode ...
>       (declare (...
>                 (autoload macroexpand)

Wow, this is even nicer!  Never knew those `delare's were so simple. 

>> We'd leave the "function-like operator" hard-coded list, but it could
>> probably be trimmed down with the new `autoload-macroexpand' feature.
>> For example, `transient-define-prefix' boils down to a defalias, which
>> is already supported.
>
> Ah, yes, I had not noticed that `transient-define-prefix` had been added
> there, and now that I see it, it does make it seem like my impression
> that it was unimportant may have been wrong.
> Usually macroexpanding those things is enough (the macro can be defined
> appropriately).  We could still refine the above with something like:
>
>     (defmacro transient-define-prefix ...
>       (declare (...
>                 (autoload function)

I mean `defalias' is supported because `defun' expands to it.  Isn't
that enough?  I guess some macros are complex enough you want to take
the quick short cut and turn them right into an `(autoload ...)'
statement, without having to design them carefully?

> tho this probably needs more thought/design because we need more info
> (where's the name of the defined thing, whether it's a macro or
> a function, what's the list of arguments, ...).

The code in `--make-autoload' seems to think it knows how to handle all
those "function-like operators".  But list 1 and list 2 seem to be
disjoint.  E.g. `transient-define-prefix' is on list 2, but _not_ on
list 1.  So it would like to be treated as a function operator, but not
expanded.

Actually I'm confused about that logic.  Take `define-minor-mode'.  It's
on list 1 (expand the macro) and list 2 (treat it as a function
operator).  I don't see how it could ever get list 2 treatment, since it
always would have been expanded first by the list 1 logic above.  There
are many macros in the 1 & 2 category:

   ;; Expand but don't treat as a special function (1, not 2):
   (defun defmacro define-inline
     cl-defgeneric
     cl-defstruct
     pcase-defmacro
     iter-defun
     cl-iter-defun)
   ;; Only treat as a special function op (2, not 1):
   (define-skeleton
     define-derived-mode
     define-compilation-mode
     define-generic-mode
     defun*
     defmacro*
     define-overloadable-function
     transient-define-prefix
     transient-define-suffix
     transient-define-infix
     transient-define-argument)
   ;; Expand _and_ treat as a special function op (1 & 2):
   (define-skeleton
     define-derived-mode
     define-compilation-mode
     define-generic-mode
     easy-mmode-define-global-mode
     define-global-minor-mode
     define-globalized-minor-mode
     easy-mmode-define-minor-mode
     define-minor-mode cl-defun
     defun*
     cl-defmacro
     defmacro*
     define-overloadable-function
     transient-define-prefix
     transient-define-suffix
     transient-define-infix
     transient-define-argument
     defun defmacro define-inline
     cl-defgeneric
     cl-defstruct
     pcase-defmacro
     iter-defun
     cl-iter-defun)

What am I missing?

Message #47 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 18:49:30 -0400

>>     (defmacro define-minor-mode ...
>>       (declare (...
>>                 (autoload macroexpand)
>
> Wow, this is even nicer!  Never knew those `delare's were so simple. 

🙂

[ Of course, third-party code would have to use `put` in the short term
  for backward compatibility reasons.  🙁 ]

>>     (defmacro transient-define-prefix ...
>>       (declare (...
>>                 (autoload function)
>
> I mean `defalias' is supported because `defun' expands to it.  Isn't
> that enough?

Yes, I think we could mark `transient-define-prefix` as `expand`.
My understanding is that currently putting it in "list 1" wouldn't work
simply because it's not autoloaded.

> The code in `--make-autoload' seems to think it knows how to handle all
> those "function-like operators".  But list 1 and list 2 seem to be
> disjoint.  E.g. `transient-define-prefix' is on list 2, but _not_ on
> list 1.  So it would like to be treated as a function operator, but not
> expanded.

As mentioned above, I think it's just because of a lack of autoloading.

>    ;; Expand _and_ treat as a special function op (1 & 2):
>    (define-skeleton
>      define-derived-mode
>      define-compilation-mode
>      define-generic-mode
>      easy-mmode-define-global-mode
>      define-global-minor-mode
>      define-globalized-minor-mode
>      easy-mmode-define-minor-mode
>      define-minor-mode cl-defun
>      defun*
>      cl-defmacro
>      defmacro*
>      define-overloadable-function
>      transient-define-prefix
>      transient-define-suffix
>      transient-define-infix
>      transient-define-argument
>      defun defmacro define-inline
>      cl-defgeneric
>      cl-defstruct
>      pcase-defmacro
>      iter-defun
>      cl-iter-defun)

I don't see `cl-defstruct` in list 2 (same for several others), so
I think the above list isn't right.
But yes, some are in both lists and it's mostly "historical accidents".
IIRC `easy-mmode-define-minor-mode` started in list 2 and was later
added to list 1 so as to get a better result (e.g. more complete
docstring) in those cases where the macro happens to be defined.

But I think your trick of evaluating the file (to cause the macros
to be defined) should allow us to focus on list 1 and leave only the
strict minimum in list 2.


        Stefan

Message #50 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 20:05:07 -0400

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> I don't see `cl-defstruct` in list 2 (same for several others), so
> I think the above list isn't right.

Oops, wrong set function.  It's:

   ;; Expand and treat as function op:
   (easy-mmode-define-global-mode
     define-global-minor-mode
     define-globalized-minor-mode
     easy-mmode-define-minor-mode
     define-minor-mode cl-defun
     cl-defmacro)

> some are in both lists and it's mostly "historical accidents".

Makes sense.

> But I think your trick of evaluating the file (to cause the macros
> to be defined) should allow us to focus on list 1 and leave only the
> strict minimum in list 2.

I will work up a patch using this approach unless you'd like to (will be
a few days min).  I think we are in agreement that a simple (autoload
macroexpand) will suffice.  We could leave list 2 as-is for now, with a
note that it can be trimmed once `declare' for list 1 is fully
operational.

Probably would also need to loop in package authors, e.g. for transient,
to ensure they are aware of the change.

Message #53 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "J.D. Smith" <jdtsmith <at> gmail.com>
Cc: 78995 <at> debbugs.gnu.org
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Sun, 13 Jul 2025 22:31:58 -0400

> I will work up a patch using this approach

Sounds good.  🙂

> I think we are in agreement that a simple (autoload macroexpand) will
> suffice.  We could leave list 2 as-is for now, with a note that it can
> be trimmed once `declare' for list 1 is fully operational.
>
> Probably would also need to loop in package authors, e.g. for transient,
> to ensure they are aware of the change.

I think we can start by removing those entries that we can turn into
`declare` directly in Emacs (e.g. not those for Transient).
And then wait for the relevant package authors to start using the new
functionality before we remove the rest, bit by bit.


        Stefan

Message #56 received at 78995 <at> debbugs.gnu.org (full text, mbox):

From: "J.D. Smith" <jdtsmith <at> gmail.com>
To: 78995 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#78995: [PATCH] ;;;autoload-expand for special macros
Date: Tue, 15 Jul 2025 23:35:54 -0400

[Message part 1 (text/plain, inline)]

Here is an update on progress.  The attached is working reasonably well,
reproducing lisp/loaddefs.el with a few changes (see attached).  I went
with (declare ((autoload . macro-expand))) on the notion that there
might be other autoload-related declarations in the future, but open to
suggestion.

I've now fully eliminated list 1 in favor of declare statements, but
also had to handle the case of _aliased_ macros, e.g. all the
[easy-mmode]-define-global[ized]-minor-mode yada yada.

There are a couple things I need to check on.

1. Non-repeatable loaddefs:

When you re-run make and it needs to regenerate lisp/loaddefs.el, it
appears easy-mmode is _not_ loaded by whatever emacs process conducts
that subsequent autoload-generation round.  Consequently,
`define-minor-mode' isn't loaded, so calls to it produce only partial
autoload entries (via, I believe, list 2).  Maybe that's why list 2
duplicates entries from list 1, as a backup against this non-repeatable
build.  But it seems strange to have different packages loaded on
different passes of the same Makefile target.  You can confirm this by:

% mv lisp/loaddefs.el lisp/loaddefs_orig.el
% make
% diff -u lisp/loaddefs_orig.el lisp/loaddefs.el

I'm not clear if this has anything to do with this patch, or could even
be a pre-existing build bug.  Once you are in this situation, you
basically have to `make bootstrap' to get the full and complete autoload
statements to appear again in loaddefs.

2. Some files are now being loaded during autoloads generation:

I noticed a few files among bundled lisp packages that now get loaded,
because they autoload undefined macros:

   loaddefs-gen: loading file frameset (cl-defun)
   Loading frameset...
     INFO     Scraping 1565 files for loaddefs...15% 
   loaddefs-gen: loading file ede/cpp-root (defclass)
   Loading ede/cpp-root...
     INFO     Scraping 1565 files for loaddefs...25% 
     INFO     Scraping 1565 files for loaddefs...47% 
     INFO     Scraping 1565 files for loaddefs...64% 
   loaddefs-gen: loading file tramp-adb (tramp--with-startup)
   Loading tramp-adb...
     INFO     Scraping 1565 files for loaddefs...80% 
   loaddefs-gen: loading file grep (define-compilation-mode)
   Loading grep...

These mostly seem to be innocuous in that after loading them, it becomes
clear that these particular autoloaded macros don't request
macro-expansion, so nothing different happens.  But it does represent a
few new "internal" load calls during autoload generation.  I think list
1 is now "doing more work" because of this pre-load mechanism.

Thoughts and suggestions very welcome.

[autoload-expand_2.patch (text/x-patch, attachment)]

[loaddefs.diff (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.