GNU bug report logs - #79599
31.0.50; Troubles with macroexp-preserve-posification

Previous Next

To reply to this bug, email your comments to 79599 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eshel Yaron <me <at> eshelyaron.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 31.0.50; Troubles with macroexp-preserve-posification
Date: Wed, 08 Oct 2025 14:41:55 +0200

Hi Alan and Stefan,

I've encountered an issue with the new position-preserving
macro-expansion introduced in commit c44903b0:

As part of the ELisp semantic highlighting implementation in
feature/elisp-fontify-semantically, we analyze forms that we read with
read-positioning-symbols, and sometimes during that analysis we expand
macros that we encounter and then continue analyzing the expanded form.
But now the expanded form can contain "fake" position information added
by macroexp-preserve-posification, which results in wrong highlighting.
This happens when we expand a macro which itself invokes macro-expansion,
such as iter-defun.

I understand the motivation for preserving this position information for
byte-compilation diagnostics, but for other purposes it would be good to
have a way to distinguish "true" symbol positions.

WDYT about adding some variable that we could let-bind to nullify the
effect of macroexp-preserve-posification?  Better yet, make it opt-in,
so only the byte-compiler would enable it by let-binding some variable.


Thanks,

Eshel

Message #8 received at 79599 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Eshel Yaron <me <at> eshelyaron.com>
Cc: 79599 <at> debbugs.gnu.org, acm <at> muc.de,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#79599: 31.0.50; Troubles with macroexp-preserve-posification
Date: Thu, 9 Oct 2025 13:53:46 +0000

Hello, Eshel.

On Wed, Oct 08, 2025 at 14:41:55 +0200, Eshel Yaron wrote:

> Hi Alan and Stefan,

> I've encountered an issue with the new position-preserving
> macro-expansion introduced in commit c44903b0:

OK.  It's heartening to hear of other uses for symbols with position.
;-)

> As part of the ELisp semantic highlighting implementation in
> feature/elisp-fontify-semantically, we analyze forms that we read with
> read-positioning-symbols, and sometimes during that analysis we expand
> macros that we encounter and then continue analyzing the expanded form.
> But now the expanded form can contain "fake" position information added
> by macroexp-preserve-posification, which results in wrong highlighting.
> This happens when we expand a macro which itself invokes macro-expansion,
> such as iter-defun.

Understood.

> I understand the motivation for preserving this position information for
> byte-compilation diagnostics, but for other purposes it would be good to
> have a way to distinguish "true" symbol positions.

> WDYT about adding some variable that we could let-bind to nullify the
> effect of macroexp-preserve-posification?  Better yet, make it opt-in,
> so only the byte-compiler would enable it by let-binding some variable.

I've spent some time trying to come up with a better fix, without any
success.  So I have to agree with you that a new defvar is the best way
to fix the problem.

I suggest the new variable should go into macroexp.el with a name
something like macroexp-preserve-posification-flag, and a default value
of nil, meaning "don't preserve the calling position of the macro".  The
variable would need binding to t at all the entry points to the byte
compiler.  I think that's exactly what you suggested.

That's assuming we don't get any better suggestions from Stefan.

One of us will need to implement it.  Possibly it would be better for
you to do this, since you're fired up with more enthusiasm than me, and
it seems likely you've already implemented it in your own repository
copy anyway.  ;-)

If you would prefer me to do it, please just say so, and I will.

> Thanks,

> Eshel

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #11 received at 79599 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Alan Mackenzie <acm <at> muc.de>
Cc: 79599 <at> debbugs.gnu.org, Eshel Yaron <me <at> eshelyaron.com>
Subject: Re: bug#79599: 31.0.50; Troubles with macroexp-preserve-posification
Date: Thu, 09 Oct 2025 13:03:55 -0400

>> WDYT about adding some variable that we could let-bind to nullify the
>> effect of macroexp-preserve-posification?  Better yet, make it opt-in,
>> so only the byte-compiler would enable it by let-binding some variable.
> I've spent some time trying to come up with a better fix, without any
> success.  So I have to agree with you that a new defvar is the best way
> to fix the problem.

Mostly agreed.  There's the usual problem of dynbound "bleeding",
e.g. if Eshel's code ends up run while in the middle of a compilation
(e.g. inside an `eval-when-compile` or in a file that's `require`d,
...).

The "standard solution" for that in this specific context would be to
put the info inside `macroexpand-all-environment` (because we already
take care to manually bind/rebind that variable so its dynbound reach
is (hopefully) limited to the right places).

It also smells a bit hackish, and comes with its own downside, mostly
the fact that it would likely be slower because you'd need to do
a search through `macroexpand-all-environment` every time, instead of
consulting a boolean var.

It's not the first time we have such a need (e.g. some marker to let
`macroexp-compiling-p` do its job, or the alist of symbol-macros), so in
the long term it would be good to have a better answer.

In the mean time, a global boolean var sounds fine to me (in this
specific instance, we can hope that dynbound bleeding will be
vanishingly rare in practice).

> I suggest the new variable should go into macroexp.el with a name
> something like macroexp-preserve-posification-flag, and a default value
> of nil, meaning "don't preserve the calling position of the macro".

Hmm... the name suggests that if it's nil the output won't have any
position info, whereas that's not what will/would happen (nor what Eshel
needs).  So, maybe use a name with "inherit" or "synthesize" in it to
clarify what it controls?

Also, I dislike the "-flag" convention, so I'd just call it
`macroexp-preserve-posification` instead.

[ These are just bikeshed color preferences on my side, no need to try
  and convince me that another color is better.  Whoever codes it gets
  to pick the color.  ]

> That's assuming we don't get any better suggestions from Stefan.

No better suggestion from me, I'm afraid.


        Stefan

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eshel Yaron <me <at> eshelyaron.com>
To: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>
Cc: 79599 <at> debbugs.gnu.org, Alan Mackenzie <acm <at> muc.de>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#79599: 31.0.50; Troubles with macroexp-preserve-posification
Date: Fri, 10 Oct 2025 09:23:09 +0200

Hi,

Stefan Monnier writes:

>>> WDYT about adding some variable that we could let-bind to nullify the
>>> effect of macroexp-preserve-posification?  Better yet, make it opt-in,
>>> so only the byte-compiler would enable it by let-binding some variable.
>> I've spent some time trying to come up with a better fix, without any
>> success.  So I have to agree with you that a new defvar is the best way
>> to fix the problem.
>
> Mostly agreed.  There's the usual problem of dynbound "bleeding",
> e.g. if Eshel's code ends up run while in the middle of a compilation
> (e.g. inside an `eval-when-compile` or in a file that's `require`d,
> ...).
>
> The "standard solution" for that in this specific context would be to
> put the info inside `macroexpand-all-environment` (because we already
> take care to manually bind/rebind that variable so its dynbound reach
> is (hopefully) limited to the right places).
>
> It also smells a bit hackish, and comes with its own downside, mostly
> the fact that it would likely be slower because you'd need to do
> a search through `macroexpand-all-environment` every time, instead of
> consulting a boolean var.
>
> It's not the first time we have such a need (e.g. some marker to let
> `macroexp-compiling-p` do its job, or the alist of symbol-macros), so in
> the long term it would be good to have a better answer.
>
> In the mean time, a global boolean var sounds fine to me (in this
> specific instance, we can hope that dynbound bleeding will be
> vanishingly rare in practice).

Cool, thank you both.

>> I suggest the new variable should go into macroexp.el with a name
>> something like macroexp-preserve-posification-flag, and a default value
>> of nil, meaning "don't preserve the calling position of the macro".
>
> Hmm... the name suggests that if it's nil the output won't have any
> position info, whereas that's not what will/would happen (nor what Eshel
> needs).  So, maybe use a name with "inherit" or "synthesize" in it to
> clarify what it controls?
>
> Also, I dislike the "-flag" convention, so I'd just call it
> `macroexp-preserve-posification` instead.

Yeah, that's the quick draft I used for testing:

diff --git a/lisp/emacs-lisp/macroexp.el b/lisp/emacs-lisp/macroexp.el
index 2bf9e0eb451..35071c03d87 100644
--- a/lisp/emacs-lisp/macroexp.el
+++ b/lisp/emacs-lisp/macroexp.el
@@ -298,6 +298,9 @@ macroexp--posify-form
   (let ((new-form (macroexp--posify-form-1 form call-pos 10)))
     (or new-form form)))
 
+(defvar macroexp-preserve-posification t
+  "Whether to attach position of a macro call to the expanded form.")
+
 (defmacro macroexp-preserve-posification (pos-form &rest body)
   "Evaluate BODY..., posifying the result with POS-FORM's position, if any.
 If the result of body happens to have a position already, we do not
@@ -310,7 +313,7 @@ macroexp-preserve-posification
                     ((symbol-with-pos-p ,pos-form)
                      (symbol-with-pos-pos ,pos-form))))
          (new-value (progn ,@body)))
-     (if (and call-pos
+     (if (and macroexp-preserve-posification call-pos
               (not (or (and (consp new-value)
                             (symbol-with-pos-p (car new-value)))
                        (and (symbol-with-pos-p new-value)))))


Note that this sets the default value to t instead of nil, but that's
only because I wasn't sure that I got all the places in which we'd like
to let-bind it to t, so I was playing it safe.

Would the macroexpand-all call in byte-compile-preprocess cover it?
I see some calls to macroexp-preserve-posification in byte-opt.el that
might require another let-binding, somewhere?


Thanks,

Eshel

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.