GNU bug report logs - #67677
30.0.50; ERC 5.6: Use templates for formatting chat messages

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 67677 in the body.
You can then email your comments to 67677 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.50; ERC 5.6: Use templates for formatting chat messages
Date: Wed, 06 Dec 2023 23:06:35 -0800

[Message part 1 (text/plain, inline)]

Tags: patch

The most valuable messages in a typical target buffer are those
featuring a leading display name, i.e., chat messages. While ERC's many
modules exert a good deal of influence over the fit and finish of an
inserted "speaker message," the fundamental layout is mostly set in
stone beforehand. This currently happens in a somewhat haphazard
fashion, with ERC cobbling together various ingredients that may or may
not factor into the finished article, which makes predicting and
influencing crucial aspects of the final look a somewhat shaky affair.

Longtime users are likely acquainted with ERC's notion of a "language
catalog." On the surface, these are just collections of variables
sharing a common "erc-message-<catalog>-<key>" naming scheme. Each
variable's value is a `format-spec' template befitting its key. Some
keys are just IRC commands, like `JOIN', while others describe the
context in which they're typically summoned.

The real benefit of these template catalogs is twofold:

  1. They're "declarative" in the sense that their form describes their
     purpose as a model preview of the result, in plain, quasi-literal
     terms (what you see is what you get).

  2. They're also an interface, with each collection of keys and their
     respective format strings and spec parameters forming a schema of
     sorts, making extensibility essentially ingrained.

The catch here is that ERC only currently uses these catalogs for select
informative messages but not actual chat content. The proposal is thus
to extend this catalog system to encompass speaker messages.

The approach I've chosen tries to strike a balance between code reuse
and extensibility and disregards seemingly vital distinctions, like
whether a message is being sent or received. Instead, it prefers to
organize things in terms of envelope semantics. Who is it from? How
should it be delivered? Etc. The answers become parameters that form the
various catalog keys, for example, "ctcp-action-statusmsg" or
"query-notice-input". Most of these combinations come prescribed by the
IRC protocol itself and by compatibility concerns. It's mostly just the
terminology and the ordering of the components that we're free to tinker
with.

So far, the most perceptible gains are maintenance oriented. But moving
toward user-facing extensibility is mainly a matter of nailing down the
core set of keys for the base speaker catalog and exposing custom
catalogs by promoting what's currently an internal variable to a user
option. At that point, the possibilities are many. For example, over the
years, folks have expressed a desire to have both "heading-style"
display names, which occupy their own line above a message body, and
"bracketless" display names. Both are easily doable with such a system
in place. (PoC available.)

I was originally hoping to tackle this feature in 5.7, but I think the
proposed approach is mature enough to refine and ship with this release.
A number of tangential and ancillary changes are included in this set as
soft dependencies, many as follow-ons to recently "closed" bugs. I will
address some of them separately, as needed, in their respective threads.

Thanks.


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.38, cairo version 1.17.6) of 2023-12-06 built on localhost
Repository revision: d8a00879309a3bf62f6ffcae103aa3bdba776ee9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Fedora Linux 37 (Workstation Edition)

Configured using:
 'configure --enable-check-lisp-object-type --enable-checking=yes,glyphs
 'CFLAGS=-O0 -g3'
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config
gnus-util time-date mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils compile text-property-search comint
ansi-osc ansi-color ring comp-run comp-common rx erc auth-source cl-seq
eieio eieio-core cl-macs password-cache json subr-x map format-spec
cl-loaddefs cl-lib erc-backend erc-networks byte-opt gv bytecomp
byte-compile erc-common erc-compat erc-loaddefs rmc iso-transl tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd touch-screen
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo gtk x-toolkit xinput2 x multi-tty move-toolbar
make-network-process native-compile emacs)

Memory information:
((conses 16 82762 11177) (symbols 48 9671 0) (strings 32 25958 4639)
 (string-bytes 1 784755) (vectors 16 19276)
 (vector-slots 8 328221 11347) (floats 8 24 28) (intervals 56 260 0)
 (buffers 984 12))

[0001-5.6-Define-ERC-message-formatting-templates-with-def.patch (text/x-patch, attachment)]

[0002-5.6-Sequester-some-special-variable-declarations-in-.patch (text/x-patch, attachment)]

[0003-5.6-Double-hyphenate-internal-ERC-5.6-text-props.patch (text/x-patch, attachment)]

[0004-5.6-Add-erc-spkr-text-property-to-chat-messages.patch (text/x-patch, attachment)]

[0005-5.6-Add-utility-for-iterating-over-arrays-in-ERC.patch (text/x-patch, attachment)]

[0006-5.6-Make-erc-input-s-refoldp-slot-conditionally-avai.patch (text/x-patch, attachment)]

[0007-5.6-Add-erc-server-396-response-handler.patch (text/x-patch, attachment)]

[0008-5.6-Clarify-warning-for-process-dependent-input-in-E.patch (text/x-patch, attachment)]

[0009-5.6-Make-erc-get-user-mode-prefix-more-flexible.patch (text/x-patch, attachment)]

[0010-5.6-Skip-erc-ignored-user-p-when-erc-ignore-list-is-.patch (text/x-patch, attachment)]

[0011-5.6-Use-templates-for-formatting-chat-messages-in-ER.patch (text/x-patch, attachment)]

Message #8 received at 67677 <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: 67677 <at> debbugs.gnu.org
Cc: emacs-erc <at> gnu.org
Subject: Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat
 messages
Date: Mon, 18 Dec 2023 06:50:25 -0800

[Message part 1 (text/plain, inline)]

A version of these changes has been installed as

  49bfea4386f * Use templates for formatting chat messages in ERC

Unfortunately, they included a somewhat glaring omission involving CTCP
replies that's hopefully addressed by the attached patch.

[0001-5.6-Populate-erc-msg-prop-overrides-for-CTCP-replies.patch (text/x-patch, attachment)]

Message #15 received at 67677-done <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: 67677-done <at> debbugs.gnu.org
Cc: emacs-erc <at> gnu.org
Subject: Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat
 messages
Date: Sun, 07 Jan 2024 21:46:40 -0800

"J.P." <jp <at> neverwas.me> writes:

> A version of these changes has been installed as
>
>   49bfea4386f * Use templates for formatting chat messages in ERC
>
> Unfortunately, they included a somewhat glaring omission involving CTCP
> replies that's hopefully addressed by the attached patch.

The fix has been installed as

  23d692ed014 * Populate erc--msg-prop-overrides for CTCP replies

Closing.

Message #18 received at 67677 <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: 67677 <at> debbugs.gnu.org
Cc: emacs-erc <at> gnu.org
Subject: Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat
 messages
Date: Fri, 12 Jan 2024 08:19:15 -0800

[Message part 1 (text/plain, inline)]

Earlier work for this bug extended ERC's existing template catalog
system with an internal framework for dictating (hopefully more
explicitly and resolutely) how messages materialize in chat buffers.
Modules can make use of this framework to shape most aspects of message
formatting, which opens the door to radically contrasting styles, such
as "multi-part" messages with header/body/footer sections and messages
with integrated time stamps (meaning you can finally ditch that blessed
`stamp' module for good).

However, this approach remains awkward when it comes to

  1. minor modifications

  2. modularity itself (keeping modules loosely coupled)

At present, making use of the framework involves defining an entire
format catalog (although inheritance helps a bit in this regard). But
the boilerplate issue really begins to compound when trying to integrate
with other modules because the process is somewhat dependent on defining
yet more catalogs to serve the various combinations, and more still if
trying to keep things abstract.

To alleviate some of this awkwardness and cut down on the verbosity, I'm
proposing we introduce a more practical extension to this framework.
It'll be reserved for internal use at first, but with an eye toward
eventual export (likely in 5.7). The basic idea is that we define a
single abnormal hook per speaker catalog that runs just prior to
insertion, and we allow its members to influence the parameters passed
to `format-spec'. And, we do so in a convenient and structured (and
reusable) way, so members don't have to twiddle plists in search of a
single ingredient to manually splice into format strings based on the
verdict of some flimsy heuristic.

I'm tentatively calling this hook system "msgfspc" (one word), short for
"message format-spec." It works by defining a struct to accompany each
hook, with slots based on the catalog's common set of template
specifiers. Subscribing code then has the freedom to modify the template
itself and add or subtract specifiers as needed by mutating the struct
instance for that particular formatting pass. The client API looks like
this:

  ;; Module activation body

  (add-hook 'erc-msgfspec-speaker-hook
            #my-maybe-transform-on-msgfspec-speaker nil t)
  (setq my-state "foo")
  
  ;; Top level of package

  (defun my-maybe-transform-on-msgfspec-speaker (spec)
    (pcase spec

      ;; Modify an outgoing message template.

      ((cl-struct erc-msgfspec-speaker
                  (key (or 'input-chan-privmsg 'input-query-privmsg)))
       (erc-msgfspec-insert-spec-after
        spec ?n ?i (propertize "%i" 'font-lock-face 'my-face))
       (push `(?i . ,my-state) (erc-msgfspec-alist spec)))
  
      ;; Modify an incoming message body.

      ((cl-struct erc-msgfspec-speaker
                  (key (or 'chan-privmsg 'query-privmsg))
                  (\?m msg))
       (setf (erc-msgfspec-speaker-?m spec)
             (decode-coding-string (my-transform-message msg)
                                   'utf-8)))))

  ;; Note that at present, all the "erc-foo" symbols above are actually
  ;; "erc--foo" (internal)

There's at least one unfortunate aspect to the API scheme above: the
buffer where the working version of a template resides isn't current
when hooks run. This happens because members still need access to local
state in the ERC buffer where the actual insertion takes place. I've
experimented a bit with using the virtual buffer facility (via
`buffer-swap-text') to get around this, and it appears to work great
(even seemingly shaving a second or two off runs of ERC's extended test
suite). However, I'm quite reticent to introduce something I've never
used before and almost never see in the wild. Thus, this approach will
have to wait pending further investigation.

The current version of the proposed implementation can be found in the
second of the attached patches. The first is from bug#68265 but included
here because the associated demo addressing a real-world use case
requires both. Please see that bug's recent posts for links and
instructions.

Thanks.

[0000-v1-v2.diff (text/x-patch, attachment)]

[0001-5.6-Add-replacement-text-field-to-erc-input-struct.patch (text/x-patch, attachment)]

[0002-5.6-Expose-catalog-specific-message-formatter-in-ERC.patch (text/x-patch, attachment)]

Message #25 received at 67677 <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: 67677 <at> debbugs.gnu.org
Cc: emacs-erc <at> gnu.org
Subject: Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat
 messages
Date: Thu, 18 Jan 2024 18:16:34 -0800

"J.P." <jp <at> neverwas.me> writes:

> Earlier work for this bug extended ERC's existing template catalog
> system with an internal framework for dictating (hopefully more
> explicitly and resolutely) how messages materialize in chat buffers.
> Modules can make use of this framework to shape most aspects of message
> formatting, which opens the door to radically contrasting styles, such
> as "multi-part" messages with header/body/footer sections and messages
> with integrated time stamps (meaning you can finally ditch that blessed
> `stamp' module for good).
>
> However, this approach remains awkward when it comes to
>
>   1. minor modifications
>
>   2. modularity itself (keeping modules loosely coupled)
>
> At present, making use of the framework involves defining an entire
> format catalog (although inheritance helps a bit in this regard). But
> the boilerplate issue really begins to compound when trying to integrate
> with other modules because the process is somewhat dependent on defining
> yet more catalogs to serve the various combinations, and more still if
> trying to keep things abstract.

Perhaps this bears some clarifying because a traditional reason for
defining "behavior protocols" is to promote modularity and code reuse.
That'd still be the case here, but not so much in the short term, where
catalog proliferation would be unavoidable. For example, let's say a new
module for spoofing bridge nicknames (call it `masquerade') provides an
option for showing select envelope info on its own line in a
"headline-style" message format:

  <{DisplayName}> {Handle{Affiliation}}\n{Body}

Now suppose another module, `away-indicator', normally displays
"<speaker>" tags in italics for nicks currently away. And suppose this
module wants to integrate with templates supplying headline-style
formatting because there's more room to show a dedicated "away" lighter.
It would first need to detect when such formatting was in play (or
arrange for doing so later) during module init, perhaps by checking
whether the `masquerade' option's variable is bound and true. It would
then replace the current `erc-message-speaker-catalog', possibly set to
something like `masqueraade-speaker', with its own `away-masq-speaker'
(or whatever). It could not, though, elect to detect such behavior on
the fly in a hook member (likely by searching for behavioral clues, like
a newline following a "<speaker>" label) because by that time, the
message has already been formatted.

A superior approach to such an explicit, cross-contaminating integration
(and in-hook heuristics-based detection, for that matter) would be for
ERC to provide a base protocol for headline-style catalogs. This would
likely mean a higher order "meta template" sporting text properties for
identifying intervals relevant to insertion-time formatting. Such
properties need not be exposed directly to users and would likely be
removed after running insertion hooks. Implementers of compatible
catalogs would then declare support by invoking a constructor, perhaps
at top level, that "pre-renders" their actual user-facing templates
(although a runtime version may also work via the function variant of
the symbol-name-based `erc-message-{catalog}-{key}' interface).

For example, a headline-style meta template might look something like

  {Pre}{Headline}{Sep}{Body} ; in reality #("%P%H%S%B" 0 2 (...) ...)

with predefined text properties turning each specifier's argument into a
quasi field of sorts. This would allow the rendered version to do things
like have "{Pre}" and "{Sep}" contain (even multiple) newlines, possibly
as `display' property replacement text. But the main point would be to
make it trivial for "consumers," like `fill', to isolate and operate on
various components and implement consistent provider-agnostic behavior
in hook members without being thrown off by specifics, like the length
or visibility of "{Sep}". Of course, the cost of all this is complexity
and verbosity in the form of additional catalogs, even if some are just
stubs consisting mostly of glue. (That is unless we add some kind of
hook-based solution as proposed by this bug.)

> To alleviate some of this awkwardness and cut down on the verbosity, I'm
> proposing we introduce a more practical extension to this framework.
> It'll be reserved for internal use at first, but with an eye toward
> eventual export (likely in 5.7). The basic idea is that we define a
> single abnormal hook per speaker catalog that runs just prior to
> insertion, and we allow its members to influence the parameters passed
> to `format-spec'. And, we do so in a convenient and structured (and
> reusable) way, so members don't have to twiddle plists in search of a
> single ingredient to manually splice into format strings based on the
> verdict of some flimsy heuristic.
>
> I'm tentatively calling this hook system "msgfspc" (one word), short for
> "message format-spec."

This name is pretty ugly, but we can't really drop the leading "msg" for
just "fspec" or "fmtspec" because ERC already uses `format-spec' in
differing contexts, such as for creating its prompt and its mode line.
IOW, it must be qualified as message-related somehow. Secondly, the name
probably shouldn't contain hyphens because that hampers readability once
a reader has been conditioned to expect it. For example, I'm guessing
most would agree that `erc--msg-fmt-spec-foo-bar' is initially
preferable to `erc--msgfspec-foo-bar'. However, after repeated exposure,
the latter starts to appear less cluttered (IMO), with the elision from
`msgfmtspec' to `msgfspec' perhaps helping in this regard (hand wave).

> It works by defining a struct to accompany each
> hook, with slots based on the catalog's common set of template
> specifiers. Subscribing code then has the freedom to modify the template
> itself and add or subtract specifiers as needed by mutating the struct
> instance for that particular formatting pass. The client API looks like
> this:
>
>   ;; Module activation body
>
>   (add-hook 'erc-msgfspec-speaker-hook
>             #my-maybe-transform-on-msgfspec-speaker nil t)
>   (setq my-state "foo")
>   
>   ;; Top level of package
>
>   (defun my-maybe-transform-on-msgfspec-speaker (spec)
>     (pcase spec
>
>       ;; Modify an outgoing message template.
>
>       ((cl-struct erc-msgfspec-speaker
>                   (key (or 'input-chan-privmsg 'input-query-privmsg)))
>        (erc-msgfspec-insert-spec-after
>         spec ?n ?i (propertize "%i" 'font-lock-face 'my-face))
>        (push `(?i . ,my-state) (erc-msgfspec-alist spec)))
>   
>       ;; Modify an incoming message body.
>
>       ((cl-struct erc-msgfspec-speaker
>                   (key (or 'chan-privmsg 'query-privmsg))
>                   (\?m msg))
>        (setf (erc-msgfspec-speaker-?m spec)
>              (decode-coding-string (my-transform-message msg)
>                                    'utf-8)))))
>
>   ;; Note that at present, all the "erc-foo" symbols above are actually
>   ;; "erc--foo" (internal)
>
> There's at least one unfortunate aspect to the API scheme above: the
> buffer where the working version of a template resides isn't current
> when hooks run. This happens because members still need access to local
> state in the ERC buffer where the actual insertion takes place. I've
> experimented a bit with using the virtual buffer facility (via
> `buffer-swap-text') to get around this, and it appears to work great
> (even seemingly shaving a second or two off runs of ERC's extended test
> suite). However, I'm quite reticent to introduce something I've never
> used before and almost never see in the wild. Thus, this approach will
> have to wait pending further investigation.

An alternative approach would be to use positional arguments of the
hook's member functions to provide access to the various parameters,
instead of passing around a struct instance. This trades the burden of
having to learn the struct's composition with learning the hook's
calling convention. Such an implementation somewhat resembles args- or
return-filtering advice, except it'd almost certainly be implemented via
`run-hook-wrapped'.

Such an approach might be friendlier to more casual users who aren't
comfortable manipulating cl-struct objects. However, one possible
downside is a slightly elevated maintenance cost. IMO, evolving the
schema of a struct is pretty painless and transparent to the user as
long as they're able to take sessions offline to recompile dependencies.
However, updating function signatures, as would be required by the
wrapped-hook method, likely involves adding a `condition-case' to the
wrapper, wherein we'd trap v1 calls and issue a warning before adapting
them to handle v2's format.

I may implement a demo version with a simulated version migration so we
have something concrete to compare.

> The current version of the proposed implementation can be found in the
> second of the attached patches. The first is from bug#68265 but included
> here because the associated demo addressing a real-world use case
> requires both. Please see that bug's recent posts for links and
> instructions.
>
> Thanks.

In the second patch,

> +                              (char (car spec)))
> +                    (cl-assert (stringp (cdr spec)))
> +                    (push `(,(intern (format "?%c" char)) ""
> +                            :type 'string :documentation ,(cdr spec))

                                     ^ this should not be quoted
> +                          out)
> +                    (push char chars))

> +
> +       (cl-defstruct (erc-msgfspec-foo (:include erc--msgfspec))
> +         "Shared object for `foo' catalog message-format hook."
> +         (\?a "" :type 'string :documentation "Ay.")
> +         (\?b "" :type 'string :documentation "Bee.")
> +         (\?c "" :type 'string :documentation "See.")
> +         (my-slot nil :type list :documentation "OK."))

And likewise in the test's assertion for the macro expansion.

Message #30 received at 67677-done <at> debbugs.gnu.org (full text, mbox):

From: "J.P." <jp <at> neverwas.me>
To: 67677-done <at> debbugs.gnu.org
Cc: emacs-erc <at> gnu.org
Subject: Re: bug#67677: 30.0.50; ERC 5.6: Use templates for formatting chat
 messages
Date: Wed, 31 Jan 2024 19:14:25 -0800

Further work on this will be handled by bug#68861:

  30.0.50; ERC 5.x: Introduce a modern message-insertion API

Closing (again).

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.