GNU bug report logs - #76781
Fontification of expl3 code

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 76781 in the body.
You can then email your comments to 76781 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Arash Esbati <arash <at> gnu.org>
To: "auctex-bugs" <bug-auctex <at> gnu.org>
Subject: Fontification of expl3 code
Date: Thu, 06 Mar 2025 13:58:37 +0100

[Message part 1 (text/plain, inline)]

Hi all,

with the recent updates to style/expl3.el, I think we have an issue with
fontification.  For this small test file:

--8<---------------cut here---------------start------------->8---
% \iffalse
%<package>\NeedsTeXFormat{LaTeX2e}[2005/12/01]
%
%<*driver>
\documentclass{ltxdoc}
\usepackage{expl3}
\begin{document}
\DocInput{dtx-fontification.dtx}
\end{document}
%</driver>
% \fi
%
% \begin{macro}{\next}
%   Implement \cs{next} which is:
%    \begin{macrocode}
\cs_set:Npn \next:nn #1#2 { x #1~y #2 }
\cs_prefix_spec:N \next:nn

\cs_to_str:N \next
\end{macrocode}
% \end{macro}
%
% \endinput
% Local Variables:
% mode: docTeX
% TeX-master: t
% End:
--8<---------------cut here---------------end--------------->8---

The fontification looks like this with 'emacs -Q':

[before.png (image/png, inline)]

[Message part 3 (text/plain, inline)]

Note the broken fontification for \next:nn and \cs_prefix_spec:N,
\cs_to_str:N.  This is mostly because : and _ have the symbol syntax in
this case and font-latex looks for word boundaries (\\>).  I think we
can cure this with this small change:

--8<---------------cut here---------------start------------->8---
diff --git a/style/expl3.el b/style/expl3.el
index 9ce0e4b0..f35088df 100644
--- a/style/expl3.el
+++ b/style/expl3.el
@@ -38,9 +38,9 @@

 (defvar LaTeX-expl3-syntax-table
   (let ((st (copy-syntax-table LaTeX-mode-syntax-table)))
-    ;; Make _ and : symbol chars
-    (modify-syntax-entry ?\_ "_" st)
-    (modify-syntax-entry ?\: "_" st)
+    ;; Make _ and : word chars
+    (modify-syntax-entry ?\_ "w" st)
+    (modify-syntax-entry ?\: "w" st)
     st))

 (defun LaTeX-expl3--set-TeX-exit-mark (_optional &optional pos)
@@ -710,7 +710,10 @@ Pass OPTIONAL, PROMPT and NO-PARAM to `TeX-arg-expl3-macro', which see."

                                 ("cs_undefine:N" "\\")
                                 ("cs_undefine:c" "\\"))
-                              'function)))
+                              'function))
+   ;; Also tell font-lock to update its internals
+   (setq font-lock-major-mode nil)
+   (font-lock-set-defaults))
  TeX-dialect)

 (defvar LaTeX-expl3-package-options-list
--8<---------------cut here---------------end--------------->8---

which then gives:

[after.png (image/png, inline)]

[Message part 5 (text/plain, inline)]

Any comments?

Best, Arash

Message #8 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Arash Esbati <arash <at> gnu.org>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Fri, 07 Mar 2025 15:07:38 +0900

Hi Arash,

>>>>> Arash Esbati <arash <at> gnu.org> writes:
> The fontification looks like this with 'emacs -Q':
> x
> Note the broken fontification for \next:nn and \cs_prefix_spec:N,
> \cs_to_str:N.  This is mostly because : and _ have the symbol syntax in
> this case and font-latex looks for word boundaries (\\>).  I think we
> can cure this with this small change:
> diff --git a/style/expl3.el b/style/expl3.el
> index 9ce0e4b0..f35088df 100644
> --- a/style/expl3.el
> +++ b/style/expl3.el
> @@ -38,9 +38,9 @@

>  (defvar LaTeX-expl3-syntax-table
>    (let ((st (copy-syntax-table LaTeX-mode-syntax-table)))
> -    ;; Make _ and : symbol chars
> -    (modify-syntax-entry ?\_ "_" st)
> -    (modify-syntax-entry ?\: "_" st)
> +    ;; Make _ and : word chars
> +    (modify-syntax-entry ?\_ "w" st)
> +    (modify-syntax-entry ?\: "w" st)
>      st))

>  (defun LaTeX-expl3--set-TeX-exit-mark (_optional &optional pos)
> @@ -710,7 +710,10 @@ Pass OPTIONAL, PROMPT and NO-PARAM to `TeX-arg-expl3-macro', which see."

>                                  ("cs_undefine:N" "\\")
>                                  ("cs_undefine:c" "\\"))
> -                              'function)))
> +                              'function))
> +   ;; Also tell font-lock to update its internals
> +   (setq font-lock-major-mode nil)
> +   (font-lock-set-defaults))
>   TeX-dialect)

>  (defvar LaTeX-expl3-package-options-list
> which then gives:
> x
> Any comments?

Well, I don't notice any difference between the two images. I actually
applied the proposed patch and opened the sample .dtx file, but it still
doesn't exhibit differences to my eyes. Am I missing some subtle
changes?

> +   ;; Also tell font-lock to update its internals
> +   (setq font-lock-major-mode nil)
> +   (font-lock-set-defaults))

1. What is the point to reset `font-lock-major-mode'? Isn't this
   `font-lock-set-defaults'?
2. Why do we need to tell font-lock to update its internals? No other
   style files do that.

Please forgive me if I'm saying something stupid.

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
#Gaza #StopMassiveKilling #CeasefireNOW

Message #11 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Arash Esbati <arash <at> gnu.org>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Fri, 07 Mar 2025 09:02:33 +0100

[Message part 1 (text/plain, inline)]

Hi Keita,

Ikumi Keita <ikumi <at> ikumi.que.jp> writes:

> Well, I don't notice any difference between the two images. I actually
> applied the proposed patch and opened the sample .dtx file, but it still
> doesn't exhibit differences to my eyes. Am I missing some subtle
> changes?

Thanks for looking at this.  Let me magnify the picture for you ;-)
Compare the ':nn':

[before1.png (image/png, inline)]

[after1.png (image/png, inline)]

[Message part 4 (text/plain, inline)]

or the '\cs' in these two:

[before2.png (image/png, inline)]

[after2.png (image/png, inline)]

[Message part 7 (text/plain, inline)]

> 1. What is the point to reset `font-lock-major-mode'? Isn't this
>    `font-lock-set-defaults'?
> 2. Why do we need to tell font-lock to update its internals? No other
>    style files do that.

I think the issue is that `font-lock' doesn't recognize the syntax
change for : and _ while the style is loaded.  I admit I'm not familiar
enough with the internals of `font-lock' to tell, but from trying, I had
to look at the internals of `font-lock-debug-fontify' and found out that
setting `font-lock-major-mode' to nil was needed.

Do you get other results if you apply the patch and comment out that
form?

Best, Arash

Message #14 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Arash Esbati <arash <at> gnu.org>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Fri, 07 Mar 2025 20:09:20 +0900

Hi Arash,

>>>>> Arash Esbati <arash <at> gnu.org> writes:
> Thanks for looking at this.  Let me magnify the picture for you ;-)
> Compare the ':nn':

> x
> x
> or the '\cs' in these two:
> x
> x

Thank you, now I see the difference of the two images. Nowever, on my
emacs, "next" doesn't get blue; it is gray instead for both cases. I'll
take closer look later.

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
#Gaza #StopMassiveKilling #CeasefireNOW

Message #17 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Arash Esbati <arash <at> gnu.org>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Sat, 08 Mar 2025 21:52:53 +0100

Hi Keita,

Ikumi Keita <ikumi <at> ikumi.que.jp> writes:

> Thank you, now I see the difference of the two images. Nowever, on my
> emacs, "next" doesn't get blue; it is gray instead for both cases. I'll
> take closer look later.

Just to be sure: You need the latest version of expl3.el from the Git
repo which was updated with commit 8e292a77.

Best, Arash

Message #20 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Keita Ikumi <ikumikeita <at> jcom.home.ne.jp>
To: Arash Esbati <arash <at> gnu.org>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Tue, 11 Mar 2025 17:08:56 +0900 (JST)

[Message part 1 (text/plain, inline)]

>
>
> ------ Original Message ------
>  日曜日, 2025/3/9  05:52, Arash Esbati<arash <at> gnu.org> によって書かれました:
>
> Hi Keita,
>
> Ikumi Keita <ikumi <at> ikumi.que.jp> writes:
>
>> 
>>  Thank you, now I see the difference of the two images. Nowever, on my
>> 
>>  emacs, "next" doesn't get blue; it is gray instead for both cases. I'll
>>  take closer look later.
>
> Just to be sure: You need the latest version of expl3.el from the Git
> repo which was updated with commit 8e292a77.
>
> Best, Arash
>
Sorry, I'm currently away from Tokyo and have difficulty about working with remote git repository. So please feel free to commit your proposal. I can investigate it another day. 

[Message part 2 (text/html, inline)]

Message #23 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Arash Esbati <arash <at> gnu.org>
To: Keita Ikumi <ikumikeita <at> jcom.home.ne.jp>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Tue, 11 Mar 2025 21:53:36 +0100

close 76781 14.1.0
thanks

Hi Keita,

Keita Ikumi <ikumikeita <at> jcom.home.ne.jp> writes:

> Sorry, I'm currently away from Tokyo and have difficulty about working
> with remote git repository. So please feel free to commit your
> proposal. I can investigate it another day.

Thanks, and no worries.  I've installed the change with commit
db11f39d.  I'm closing this report, we can reopen once you have a closer
look and something is off.

Best, Arash

Message #28 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Arash Esbati <arash <at> gnu.org>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Sat, 22 Mar 2025 17:56:58 +0900

[Message part 1 (text/plain, inline)]

Hi Arash,

>>>>> Arash Esbati <arash <at> gnu.org> writes:
> Thanks, and no worries.  I've installed the change with commit
> db11f39d.  I'm closing this report, we can reopen once you have a closer
> look and something is off.

I found out the origin of this bug. The culprit is that font lock is
turned on too early.

1. Let me begin the discussion with quote of the comment in tex.el for
   `find-file-hook':
,----
|   ;; Let `TeX-master-file' be called after a new file was opened and
|   ;; call `TeX-update-style' on any file opened. [...]
|   ;;
|   ;; `TeX-update-style' has to be called before
|   ;; `global-font-lock-mode', which may also be specified in
|   ;; `find-file-hook', gets called.  Otherwise style-based
|   ;; fontification will break (in XEmacs).  That means, `add-hook'
|   ;; cannot be called with a non-nil value of the APPEND argument.
|   ;;
|   ;; `(TeX-master-file nil nil t)' has to be called *before*
|   ;; `TeX-update-style' as the latter will call `TeX-master-file'
|   ;; without the `ask' bit set.
`----
   As this rather old comment implies, AUCTeX assumes that
   `TeX-update-style' runs _before_ font lock is turned on and style files
   are written under that assumption.

   If this assumtion holds, `font-lock-syntax-table' would be
   initialized after "expl3" style hook sets the syntax table by
   (set-syntax-table LaTeX-expl3-syntax-table)
   and fontification would be done as expected.

   However, modern emacsens actually don't satisfy that assumption.
   Major mode functions run `global-font-lock-mode-enable-in-buffer'
   (registered in `after-change-major-mode-hook'), which eventually
   runs `font-lock-set-defaults' which set `font-lock-syntax-table'
   at that time, copied from `docTeX-mode-syntax-table', not
   `LaTeX-expl3-syntax-table'. Thus the fontification was broken.

2. I think that there are two approaches to this bug.
   A: To move the call to `TeX-master-file' and `TeX-update-style' from
      `find-file-hook' to somewhere run _before_
      `after-change-major-mode-hook' and _after_ major mode hook and
      local variables entries are applied.
   B: To abandon the assumption in question and make a new AUCTeX policy
      to write style files assuming that style hooks are run after font
      lock is enabled.

   In my view, A isn't satisfactory:
   o It wouldn't be robust. The detail of the interaction between major
     mode function and turning on of font lock can be subject to further
     changes in future.
   o There is no suitable place for such "somewhere". Looking at the
     definition of `run-mode-hooks', the only possibility is to register
     the calls as local hook of `after-change-major-mode-hook' but I'm
     afraid that such treatment can break in future as described in the
     previous item.
   (On the other hand, there is a good aspect that we can merge call to
   `TeX-update-sytle' for non-file buffer in `TeX-mode-cleanup'.)

   If we take approach B, I think the proper solution is to use
   `font-latex-add-to-syntax-alist' instead of re-calling
   `font-lock-set-defaults' in expl3.el. See the attached patch.
   Additionally, we would have to revise similarly existing style files
   which install their own syntax tables. I expect this doesn't require
   much work; Here is the output of "grep -E set-syntax-table *.el"
   under style/ subdirectory:
brazilian.el:   (set-syntax-table LaTeX-brazilian-mode-syntax-table)
bulgarian.el:   (set-syntax-table LaTeX-bulgarian-mode-syntax-table)
expl3.el:   (set-syntax-table LaTeX-expl3-syntax-table)
german.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
icelandic.el:   (set-syntax-table LaTeX-icelandic-mode-syntax-table)
ngerman.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
plfonts.el:   (set-syntax-table LaTeX-plfonts-mode-syntax-table)
plhb.el:   (set-syntax-table LaTeX-plhb-mode-syntax-table)
polish.el:   (set-syntax-table LaTeX-polish-mode-syntax-table)
portuguese.el:   (set-syntax-table LaTeX-portuguese-mode-syntax-table)

What do you think about this?

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
#Gaza #StopMassiveKilling #CeasefireNOW

[0001-temporal-commit.patch (text/x-diff, attachment)]

Message #31 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Arash Esbati <arash <at> gnu.org>
To: Ikumi Keita <ikumi <at> ikumi.que.jp>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Thu, 27 Mar 2025 09:01:24 +0100

Hi Keita,

Ikumi Keita <ikumi <at> ikumi.que.jp> writes:

>    If we take approach B, I think the proper solution is to use
>    `font-latex-add-to-syntax-alist' instead of re-calling
>    `font-lock-set-defaults' in expl3.el. See the attached patch.

Many thanks for looking at this and finding the root cause.  I installed
your patch locally and it works, so I will install it with the next
update to expl3.el.

>    Additionally, we would have to revise similarly existing style files
>    which install their own syntax tables. I expect this doesn't require
>    much work; Here is the output of "grep -E set-syntax-table *.el"
>    under style/ subdirectory:
> brazilian.el:   (set-syntax-table LaTeX-brazilian-mode-syntax-table)
> bulgarian.el:   (set-syntax-table LaTeX-bulgarian-mode-syntax-table)
> expl3.el:   (set-syntax-table LaTeX-expl3-syntax-table)
> german.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
> icelandic.el:   (set-syntax-table LaTeX-icelandic-mode-syntax-table)
> ngerman.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
> plfonts.el:   (set-syntax-table LaTeX-plfonts-mode-syntax-table)
> plhb.el:   (set-syntax-table LaTeX-plhb-mode-syntax-table)
> polish.el:   (set-syntax-table LaTeX-polish-mode-syntax-table)
> portuguese.el:   (set-syntax-table LaTeX-portuguese-mode-syntax-table)

I had a brief look at files above and I think they all DTRT, especially
n?german.el.  So I think it was only me missing this in the latest
incarnation of expl3.el.

Best, Arash

Message #34 received at 76781 <at> debbugs.gnu.org (full text, mbox):

From: Ikumi Keita <ikumi <at> ikumi.que.jp>
To: Arash Esbati <arash <at> gnu.org>
Cc: 76781 <at> debbugs.gnu.org
Subject: Re: bug#76781: Fontification of expl3 code
Date: Thu, 27 Mar 2025 22:21:33 +0900

[Message part 1 (text/plain, inline)]

>>>>> Ikumi Keita <ikumi <at> ikumi.que.jp> writes:
> 2. I think that there are two approaches to this bug.
>    A: To move the call to `TeX-master-file' and `TeX-update-style' from
>       `find-file-hook' to somewhere run _before_
>       `after-change-major-mode-hook' and _after_ major mode hook and
>       local variables entries are applied.
>    B: To abandon the assumption in question and make a new AUCTeX policy
>       to write style files assuming that style hooks are run after font
>       lock is enabled.

>    In my view, A isn't satisfactory:

I changed my mind. :-) It seems that we can use
`hack-local-variables-hook'. The attached patch looks working well for
me, although this affects tex-jp.el.

>    o It wouldn't be robust. The detail of the interaction between major
>      mode function and turning on of font lock can be subject to further
>      changes in future.
>    o There is no suitable place for such "somewhere". Looking at the
>      definition of `run-mode-hooks', the only possibility is to register
>      the calls as local hook of `after-change-major-mode-hook' but I'm
>      afraid that such treatment can break in future as described in the
>      previous item.
>    (On the other hand, there is a good aspect that we can merge call to
>    `TeX-update-sytle' for non-file buffer in `TeX-mode-cleanup'.)

This doesn't happen for the attached patch because emacs doesn't run
`hack-local-variables-hook' for non-file buffer. However, we can
simplify `TeX-normal-mode', which doesn't run `find-file-hook' whereas
does `hack-local-variables-hook' through `normal-mode'.

>    If we take approach B, I think the proper solution is to use
>    `font-latex-add-to-syntax-alist' instead of re-calling
>    `font-lock-set-defaults' in expl3.el. See the attached patch.
>    Additionally, we would have to revise similarly existing style files
>    which install their own syntax tables. I expect this doesn't require
>    much work; Here is the output of "grep -E set-syntax-table *.el"
>    under style/ subdirectory:
> brazilian.el:   (set-syntax-table LaTeX-brazilian-mode-syntax-table)
> bulgarian.el:   (set-syntax-table LaTeX-bulgarian-mode-syntax-table)
> expl3.el:   (set-syntax-table LaTeX-expl3-syntax-table)
> german.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
> icelandic.el:   (set-syntax-table LaTeX-icelandic-mode-syntax-table)
> ngerman.el:   (set-syntax-table LaTeX-german-mode-syntax-table)
> plfonts.el:   (set-syntax-table LaTeX-plfonts-mode-syntax-table)
> plhb.el:   (set-syntax-table LaTeX-plhb-mode-syntax-table)
> polish.el:   (set-syntax-table LaTeX-polish-mode-syntax-table)
> portuguese.el:   (set-syntax-table LaTeX-portuguese-mode-syntax-table)

Regards,
Ikumi Keita
#StandWithUkraine #StopWarInUkraine
#Gaza #StopMassiveKilling #CeasefireNOW

[0001-temporal-commit.patch (text/x-diff, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.