GNU bug report logs - #77872
[PATCH 1/2] ansi-term: ignore CSI commands with subparams

Previous Next

To reply to this bug, email your comments to 77872 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Johannes Altmanninger <aclopte <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Cc: Johannes Altmanninger <aclopte <at> gmail.com>
Subject: [PATCH 1/2] ansi-term: ignore CSI commands with subparams
Date: Thu, 17 Apr 2025 20:46:32 +0200

* lisp/term.el (term-handle-ansi-escape): Check if parameters contain
subparameters (values separated by colon) and ignore those commands.
This prevents incorrect interpretation of advanced terminal features
that aren't yet properly supported, such as curly underlines (\e[4:3m).

Consider ignoring whitespace changes when reviewing this patch.
The essence of this change is the addition at the beginning of
term-handle-ansi-escape, to skip all CSI commands that contain at
least two subparameters (e.g. the ones that contain at least one ":").

Some background / longer description:

If I run

	emacs --eval '(ansi-term "/bin/bash")'

and type

	printf '\e[4:3mHELLO\e[m\n'

then I get text with a straight underline.

This is not what I expect, because "\e[4:3m" is supposed to turn on
curly/wavy underlines, see https://sw.kovidgoyal.net/kitty/underlines/.

My program wants to use curly underlines to indicate errors. Rendering
those as straight underlines is noisy and confusing because it suggests
a different meaning (in my program).

I'd rather ansi-term ignore the curly underline command until it's
implemented.

FWIW Emacs itself already supports various underline styles, even in
a TTY -- see e.g. 9f589eb9240 (Add support for colored and styled
underlines on tty frames, 2023-04-20). It would be fairly easy to
add support for curly underlines to ansi-term.

In general, I wonder what's story on Emacs terminals. ansi-term
has some other compatibility issues, for example it cannot parse
OSC (Operating System Command) sequences, which causes friction
with the fish shell. We can fix this fairly easily; alternatively
maybe default to another terminal implementation such as vterm
(https://github.com/akermu/emacs-libvterm) that offers better
xterm-compatibility.
---
 lisp/term.el | 230 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 127 insertions(+), 103 deletions(-)

diff --git a/lisp/term.el b/lisp/term.el
index a971300c055..8270a580435 100644
--- a/lisp/term.el
+++ b/lisp/term.el
@@ -3224,11 +3224,15 @@ term-emulate-terminal
                                 (if private (substring ctl-params 1) ctl-params)))
                           (term-handle-ansi-escape
                            proc
-                           (mapcar ;; We don't distinguish empty params
-                            ;; from 0 (according to [ECMA-48] we
-                            ;; should, but all commands we support
-                            ;; default to 0 values anyway).
-                            #'string-to-number
+                           (mapcar
+                             (lambda (param)
+                               (mapcar
+                                 ;; We don't distinguish empty params
+                                 ;; from 0 (according to [ECMA-48] we
+                                 ;; should, but all commands we support
+                                 ;; default to 0 values anyway).
+                                 #'string-to-number
+                                 (split-string param ":")))
                             (split-string ctl-params ";"))
                            (aref str (1- ctl-end))
                            private))))
@@ -3589,117 +3593,137 @@ term--handle-colors-list
 ;; i.e. we have previously seen Escape followed by ?[.
 
 (defun term-handle-ansi-escape (proc params char &optional private)
-  (cond
-   ((and private (not (memq char '(?h ?l))))
-    ;; Recognize private capabilities only for mode entry and exit
-    nil)
-   ((or (eq char ?H)  ;; cursor motion (terminfo: cup,home)
+  (let
+    ((supported-params
+      (cl-remove-if
+       (lambda (subparams) (> (length subparams) 1))
+       params)
+      ))
+    ;; We'll pass on only parameters without colon, e.g. parameters with at
+    ;; most one sub-parameter.
+    ;; However if all parameters were unsupported, make sure to ignore the
+    ;; entire sequence.
+    (when
+      (not
+       (and
+         (null supported-params)
+         (not (null params))))
+      (let
+	((params
+	  (mapcar
+           (lambda (subparams) (if (null subparams) 0 (car subparams)))
+	   supported-params)))
+	(cond
+	((and private (not (memq char '(?h ?l))))
+	;; Recognize private capabilities only for mode entry and exit
+	nil)
+	((or (eq char ?H)  ;; cursor motion (terminfo: cup,home)
 	;; (eq char ?f) ;; xterm seems to handle this sequence too, not
 	;; needed for now
 	)
-    (term-goto
-     (1- (max 1 (min (or (nth 0 params) 0) term-height)))
-     (1- (max 1 (min (or (nth 1 params) 0) term-width)))))
-   ;; \E[A - cursor up (terminfo: cuu, cuu1)
-   ((eq char ?A)
-    (term-handle-deferred-scroll)
-    (let ((tcr (term-current-row))
-          (scroll-amount (car params)))
-      (term-down
-       (if (< (- tcr scroll-amount) term-scroll-start)
+	(term-goto
+	(1- (max 1 (min (or (nth 0 params) 0) term-height)))
+	(1- (max 1 (min (or (nth 1 params) 0) term-width)))))
+	;; \E[A - cursor up (terminfo: cuu, cuu1)
+	((eq char ?A)
+	(term-handle-deferred-scroll)
+	(let ((tcr (term-current-row))
+	     (scroll-amount (car params)))
+	 (term-down
+	  (if (< (- tcr scroll-amount) term-scroll-start)
 	   ;; If the amount to move is before scroll start, move
 	   ;; to scroll start.
 	   (- term-scroll-start tcr)
-         (if (>= scroll-amount tcr)
+	    (if (>= scroll-amount tcr)
 	     (- tcr)
-           (- (max 1 scroll-amount))))
-       t)))
-   ;; \E[B - cursor down (terminfo: cud)
-   ((eq char ?B)
-    (let ((tcr (term-current-row))
-          (scroll-amount (car params)))
-      (unless (>= tcr term-scroll-end)
+	      (- (max 1 scroll-amount))))
+	  t)))
+	;; \E[B - cursor down (terminfo: cud)
+	((eq char ?B)
+	(let ((tcr (term-current-row))
+	     (scroll-amount (car params)))
+	 (unless (>= tcr term-scroll-end)
 	(term-down
-         (min (- term-scroll-end tcr) (max 1 scroll-amount))
-         t))))
-   ;; \E[C - cursor right (terminfo: cuf, cuf1)
-   ((eq char ?C)
-    (term-move-columns
-     (max 1
-          (if (>= (+ (car params) (term-current-column)) term-width)
+	    (min (- term-scroll-end tcr) (max 1 scroll-amount))
+	    t))))
+	;; \E[C - cursor right (terminfo: cuf, cuf1)
+	((eq char ?C)
+	(term-move-columns
+	(max 1
+	     (if (>= (+ (car params) (term-current-column)) term-width)
 	      (- term-width (term-current-column)  1)
-            (car params)))))
-   ;; \E[D - cursor left (terminfo: cub)
-   ((eq char ?D)
-    (term-move-columns (- (max 1 (car params)))))
-   ;; \E[G - cursor motion to absolute column (terminfo: hpa)
-   ((eq char ?G)
-    (term-move-columns (- (max 0 (min term-width (car params)))
-                          (term-current-column))))
-   ;; \E[J - clear to end of screen (terminfo: ed, clear)
-   ((eq char ?J)
-    (term-erase-in-display (car params)))
-   ;; \E[K - clear to end of line (terminfo: el, el1)
-   ((eq char ?K)
-    (term-erase-in-line (car params)))
-   ;; \E[L - insert lines (terminfo: il, il1)
-   ((eq char ?L)
-    (term-insert-lines (max 1 (car params))))
-   ;; \E[M - delete lines (terminfo: dl, dl1)
-   ((eq char ?M)
-    (term-delete-lines (max 1 (car params))))
-   ;; \E[P - delete chars (terminfo: dch, dch1)
-   ((eq char ?P)
-    (term-delete-chars (max 1 (car params))))
-   ;; \E[@ - insert spaces (terminfo: ich)
-   ((eq char ?@)
-    (term-insert-spaces (max 1 (car params))))
-   ;; \E[?h - DEC Private Mode Set
-
-   ;; N.B. we previously had a bug in which we'd decode \e[?<NR>h or
-   ;; \e[?<NR>l as a command with zero in the params field and so
-   ;; didn't recognize DEC private escape sequences.  However, the
-   ;; termcap and terminfo files had the non-? (question mark means DEC
-   ;; private) versions, so things kind of worked anyway.  To preserve
-   ;; compatibility, we recognize both private- and non-private
-   ;; messages for capabilities we added before we fixed the bug but
-   ;; require the private flag for capabilities we added after.
-   ((eq char ?h)
-    (cond ((eq (car params) 4) ;; (terminfo: smir)
-           (setq term-insert-mode t))
-          ((and private (eq (car params) 7)) ;; (terminfo: smam)
-           (setq term-auto-margins t))
-          ((eq (car params) 47) ;; (terminfo: smcup)
-           (term-switch-to-alternate-sub-buffer t))))
-   ;; \E[?l - DEC Private Mode Reset
-   ((eq char ?l)
-    (cond ((eq (car params) 4) ;; (terminfo: rmir)
-           (setq term-insert-mode nil))
-          ((and private (eq (car params) 7)) ;; (terminfo: rmam)
-           (setq term-auto-margins nil))
-          ((eq (car params) 47) ;; (terminfo: rmcup)
-           (term-switch-to-alternate-sub-buffer nil))))
-
-   ;; Modified to allow ansi coloring -mm
-   ;; \E[m - Set/reset modes, set bg/fg
-   ;;(terminfo: smso,rmso,smul,rmul,rev,bold,dim,sitm,ritm,blink,sgr0,invis,op,setab,setaf)
-   ((eq char ?m)
-    (term--handle-colors-list params))
-
-   ;; \E[6n - Report cursor position (terminfo: u7)
-   ((eq char ?n)
-    (term-handle-deferred-scroll)
-    (process-send-string proc
+	       (car params)))))
+	;; \E[D - cursor left (terminfo: cub)
+	((eq char ?D)
+	(term-move-columns (- (max 1 (car params)))))
+	;; \E[G - cursor motion to absolute column (terminfo: hpa)
+	((eq char ?G)
+	(term-move-columns (- (max 0 (min term-width (car params)))
+			     (term-current-column))))
+	;; \E[J - clear to end of screen (terminfo: ed, clear)
+	((eq char ?J)
+	(term-erase-in-display (car params)))
+	;; \E[K - clear to end of line (terminfo: el, el1)
+	((eq char ?K)
+	(term-erase-in-line (car params)))
+	;; \E[L - insert lines (terminfo: il, il1)
+	((eq char ?L)
+	(term-insert-lines (max 1 (car params))))
+	;; \E[M - delete lines (terminfo: dl, dl1)
+	((eq char ?M)
+	(term-delete-lines (max 1 (car params))))
+	;; \E[P - delete chars (terminfo: dch, dch1)
+	((eq char ?P)
+	(term-delete-chars (max 1 (car params))))
+	;; \E[@ - insert spaces (terminfo: ich)
+	((eq char ?@)
+	(term-insert-spaces (max 1 (car params))))
+	;; \E[?h - DEC Private Mode Set
+
+	;; N.B. we previously had a bug in which we'd decode \e[?<NR>h or
+	;; \e[?<NR>l as a command with zero in the params field and so
+	;; didn't recognize DEC private escape sequences.  However, the
+	;; termcap and terminfo files had the non-? (question mark means DEC
+	;; private) versions, so things kind of worked anyway.  To preserve
+	;; compatibility, we recognize both private- and non-private
+	;; messages for capabilities we added before we fixed the bug but
+	;; require the private flag for capabilities we added after.
+	((eq char ?h)
+	(cond ((eq (car params) 4) ;; (terminfo: smir)
+	      (setq term-insert-mode t))
+	     ((and private (eq (car params) 7)) ;; (terminfo: smam)
+	      (setq term-auto-margins t))
+	     ((eq (car params) 47) ;; (terminfo: smcup)
+	      (term-switch-to-alternate-sub-buffer t))))
+	;; \E[?l - DEC Private Mode Reset
+	((eq char ?l)
+	(cond ((eq (car params) 4) ;; (terminfo: rmir)
+	      (setq term-insert-mode nil))
+	     ((and private (eq (car params) 7)) ;; (terminfo: rmam)
+	      (setq term-auto-margins nil))
+	     ((eq (car params) 47) ;; (terminfo: rmcup)
+	      (term-switch-to-alternate-sub-buffer nil))))
+
+	;; Modified to allow ansi coloring -mm
+	;; \E[m - Set/reset modes, set bg/fg
+	;;(terminfo: smso,rmso,smul,rmul,rev,bold,dim,sitm,ritm,blink,sgr0,invis,op,setab,setaf)
+	((eq char ?m)
+	(term--handle-colors-list params))
+
+	;; \E[6n - Report cursor position (terminfo: u7)
+	((eq char ?n)
+	(term-handle-deferred-scroll)
+	(process-send-string proc
 			 ;; (terminfo: u6)
 			 (format "\e[%s;%sR"
 				 (1+ (term-current-row))
 				 (1+ (term-horizontal-column)))))
-   ;; \E[r - Set scrolling region (terminfo: csr)
-   ((eq char ?r)
-    (term-set-scroll-region
-     (1- (or (nth 0 params) 0))
-     (1- (or (nth 1 params) 0))))
-   (t)))
+	;; \E[r - Set scrolling region (terminfo: csr)
+	((eq char ?r)
+	(term-set-scroll-region
+	(1- (or (nth 0 params) 0))
+	(1- (or (nth 1 params) 0))))
+	(t))))))
 
 (defun term--reset-scroll-region ()
   "Set the scroll region to the full height of the terminal."
-- 
2.49.0

Message #8 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Johannes Altmanninger <aclopte <at> gmail.com>, Jared Finder <jared <at> finder.org>
Cc: 77872 <at> debbugs.gnu.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Fri, 18 Apr 2025 09:05:33 +0300

> Cc: Johannes Altmanninger <aclopte <at> gmail.com>
> From: Johannes Altmanninger <aclopte <at> gmail.com>
> Date: Thu, 17 Apr 2025 20:46:32 +0200
> 
> * lisp/term.el (term-handle-ansi-escape): Check if parameters contain
> subparameters (values separated by colon) and ignore those commands.
> This prevents incorrect interpretation of advanced terminal features
> that aren't yet properly supported, such as curly underlines (\e[4:3m).
> 
> Consider ignoring whitespace changes when reviewing this patch.
> The essence of this change is the addition at the beginning of
> term-handle-ansi-escape, to skip all CSI commands that contain at
> least two subparameters (e.g. the ones that contain at least one ":").
> 
> Some background / longer description:
> 
> If I run
> 
> 	emacs --eval '(ansi-term "/bin/bash")'
> 
> and type
> 
> 	printf '\e[4:3mHELLO\e[m\n'
> 
> then I get text with a straight underline.
> 
> This is not what I expect, because "\e[4:3m" is supposed to turn on
> curly/wavy underlines, see https://sw.kovidgoyal.net/kitty/underlines/.
> 
> My program wants to use curly underlines to indicate errors. Rendering
> those as straight underlines is noisy and confusing because it suggests
> a different meaning (in my program).

Why cannot you solve this problem by a suitable configuration of faces
in Emacs?

> I'd rather ansi-term ignore the curly underline command until it's
> implemented.

We could have this behavior as an option, conditioned on some user
option, perhaps.

> FWIW Emacs itself already supports various underline styles, even in
> a TTY -- see e.g. 9f589eb9240 (Add support for colored and styled
> underlines on tty frames, 2023-04-20). It would be fairly easy to
> add support for curly underlines to ansi-term.
> 
> In general, I wonder what's story on Emacs terminals. ansi-term
> has some other compatibility issues, for example it cannot parse
> OSC (Operating System Command) sequences, which causes friction
> with the fish shell. We can fix this fairly easily; alternatively
> maybe default to another terminal implementation such as vterm
> (https://github.com/akermu/emacs-libvterm) that offers better
> xterm-compatibility.

You don't tell in which Emacs version you see these issues.  Please
tell, it might be important.

Jared, any comments to the above and/or to the patch?

In any case, we are unable to accept such a large contribution without
you assigning the copyright for your changes to the FSF.  If you are
willing to do that, I will send you the form to fill and the
instructions to go with it, to start your legal paperwork rolling.

Thanks.

> ---
>  lisp/term.el | 230 ++++++++++++++++++++++++++++-----------------------
>  1 file changed, 127 insertions(+), 103 deletions(-)
> 
> diff --git a/lisp/term.el b/lisp/term.el
> index a971300c055..8270a580435 100644
> --- a/lisp/term.el
> +++ b/lisp/term.el
> @@ -3224,11 +3224,15 @@ term-emulate-terminal
>                                  (if private (substring ctl-params 1) ctl-params)))
>                            (term-handle-ansi-escape
>                             proc
> -                           (mapcar ;; We don't distinguish empty params
> -                            ;; from 0 (according to [ECMA-48] we
> -                            ;; should, but all commands we support
> -                            ;; default to 0 values anyway).
> -                            #'string-to-number
> +                           (mapcar
> +                             (lambda (param)
> +                               (mapcar
> +                                 ;; We don't distinguish empty params
> +                                 ;; from 0 (according to [ECMA-48] we
> +                                 ;; should, but all commands we support
> +                                 ;; default to 0 values anyway).
> +                                 #'string-to-number
> +                                 (split-string param ":")))
>                              (split-string ctl-params ";"))
>                             (aref str (1- ctl-end))
>                             private))))
> @@ -3589,117 +3593,137 @@ term--handle-colors-list
>  ;; i.e. we have previously seen Escape followed by ?[.
>  
>  (defun term-handle-ansi-escape (proc params char &optional private)
> -  (cond
> -   ((and private (not (memq char '(?h ?l))))
> -    ;; Recognize private capabilities only for mode entry and exit
> -    nil)
> -   ((or (eq char ?H)  ;; cursor motion (terminfo: cup,home)
> +  (let
> +    ((supported-params
> +      (cl-remove-if
> +       (lambda (subparams) (> (length subparams) 1))
> +       params)
> +      ))
> +    ;; We'll pass on only parameters without colon, e.g. parameters with at
> +    ;; most one sub-parameter.
> +    ;; However if all parameters were unsupported, make sure to ignore the
> +    ;; entire sequence.
> +    (when
> +      (not
> +       (and
> +         (null supported-params)
> +         (not (null params))))
> +      (let
> +	((params
> +	  (mapcar
> +           (lambda (subparams) (if (null subparams) 0 (car subparams)))
> +	   supported-params)))
> +	(cond
> +	((and private (not (memq char '(?h ?l))))
> +	;; Recognize private capabilities only for mode entry and exit
> +	nil)
> +	((or (eq char ?H)  ;; cursor motion (terminfo: cup,home)
>  	;; (eq char ?f) ;; xterm seems to handle this sequence too, not
>  	;; needed for now
>  	)
> -    (term-goto
> -     (1- (max 1 (min (or (nth 0 params) 0) term-height)))
> -     (1- (max 1 (min (or (nth 1 params) 0) term-width)))))
> -   ;; \E[A - cursor up (terminfo: cuu, cuu1)
> -   ((eq char ?A)
> -    (term-handle-deferred-scroll)
> -    (let ((tcr (term-current-row))
> -          (scroll-amount (car params)))
> -      (term-down
> -       (if (< (- tcr scroll-amount) term-scroll-start)
> +	(term-goto
> +	(1- (max 1 (min (or (nth 0 params) 0) term-height)))
> +	(1- (max 1 (min (or (nth 1 params) 0) term-width)))))
> +	;; \E[A - cursor up (terminfo: cuu, cuu1)
> +	((eq char ?A)
> +	(term-handle-deferred-scroll)
> +	(let ((tcr (term-current-row))
> +	     (scroll-amount (car params)))
> +	 (term-down
> +	  (if (< (- tcr scroll-amount) term-scroll-start)
>  	   ;; If the amount to move is before scroll start, move
>  	   ;; to scroll start.
>  	   (- term-scroll-start tcr)
> -         (if (>= scroll-amount tcr)
> +	    (if (>= scroll-amount tcr)
>  	     (- tcr)
> -           (- (max 1 scroll-amount))))
> -       t)))
> -   ;; \E[B - cursor down (terminfo: cud)
> -   ((eq char ?B)
> -    (let ((tcr (term-current-row))
> -          (scroll-amount (car params)))
> -      (unless (>= tcr term-scroll-end)
> +	      (- (max 1 scroll-amount))))
> +	  t)))
> +	;; \E[B - cursor down (terminfo: cud)
> +	((eq char ?B)
> +	(let ((tcr (term-current-row))
> +	     (scroll-amount (car params)))
> +	 (unless (>= tcr term-scroll-end)
>  	(term-down
> -         (min (- term-scroll-end tcr) (max 1 scroll-amount))
> -         t))))
> -   ;; \E[C - cursor right (terminfo: cuf, cuf1)
> -   ((eq char ?C)
> -    (term-move-columns
> -     (max 1
> -          (if (>= (+ (car params) (term-current-column)) term-width)
> +	    (min (- term-scroll-end tcr) (max 1 scroll-amount))
> +	    t))))
> +	;; \E[C - cursor right (terminfo: cuf, cuf1)
> +	((eq char ?C)
> +	(term-move-columns
> +	(max 1
> +	     (if (>= (+ (car params) (term-current-column)) term-width)
>  	      (- term-width (term-current-column)  1)
> -            (car params)))))
> -   ;; \E[D - cursor left (terminfo: cub)
> -   ((eq char ?D)
> -    (term-move-columns (- (max 1 (car params)))))
> -   ;; \E[G - cursor motion to absolute column (terminfo: hpa)
> -   ((eq char ?G)
> -    (term-move-columns (- (max 0 (min term-width (car params)))
> -                          (term-current-column))))
> -   ;; \E[J - clear to end of screen (terminfo: ed, clear)
> -   ((eq char ?J)
> -    (term-erase-in-display (car params)))
> -   ;; \E[K - clear to end of line (terminfo: el, el1)
> -   ((eq char ?K)
> -    (term-erase-in-line (car params)))
> -   ;; \E[L - insert lines (terminfo: il, il1)
> -   ((eq char ?L)
> -    (term-insert-lines (max 1 (car params))))
> -   ;; \E[M - delete lines (terminfo: dl, dl1)
> -   ((eq char ?M)
> -    (term-delete-lines (max 1 (car params))))
> -   ;; \E[P - delete chars (terminfo: dch, dch1)
> -   ((eq char ?P)
> -    (term-delete-chars (max 1 (car params))))
> -   ;; \E[@ - insert spaces (terminfo: ich)
> -   ((eq char ?@)
> -    (term-insert-spaces (max 1 (car params))))
> -   ;; \E[?h - DEC Private Mode Set
> -
> -   ;; N.B. we previously had a bug in which we'd decode \e[?<NR>h or
> -   ;; \e[?<NR>l as a command with zero in the params field and so
> -   ;; didn't recognize DEC private escape sequences.  However, the
> -   ;; termcap and terminfo files had the non-? (question mark means DEC
> -   ;; private) versions, so things kind of worked anyway.  To preserve
> -   ;; compatibility, we recognize both private- and non-private
> -   ;; messages for capabilities we added before we fixed the bug but
> -   ;; require the private flag for capabilities we added after.
> -   ((eq char ?h)
> -    (cond ((eq (car params) 4) ;; (terminfo: smir)
> -           (setq term-insert-mode t))
> -          ((and private (eq (car params) 7)) ;; (terminfo: smam)
> -           (setq term-auto-margins t))
> -          ((eq (car params) 47) ;; (terminfo: smcup)
> -           (term-switch-to-alternate-sub-buffer t))))
> -   ;; \E[?l - DEC Private Mode Reset
> -   ((eq char ?l)
> -    (cond ((eq (car params) 4) ;; (terminfo: rmir)
> -           (setq term-insert-mode nil))
> -          ((and private (eq (car params) 7)) ;; (terminfo: rmam)
> -           (setq term-auto-margins nil))
> -          ((eq (car params) 47) ;; (terminfo: rmcup)
> -           (term-switch-to-alternate-sub-buffer nil))))
> -
> -   ;; Modified to allow ansi coloring -mm
> -   ;; \E[m - Set/reset modes, set bg/fg
> -   ;;(terminfo: smso,rmso,smul,rmul,rev,bold,dim,sitm,ritm,blink,sgr0,invis,op,setab,setaf)
> -   ((eq char ?m)
> -    (term--handle-colors-list params))
> -
> -   ;; \E[6n - Report cursor position (terminfo: u7)
> -   ((eq char ?n)
> -    (term-handle-deferred-scroll)
> -    (process-send-string proc
> +	       (car params)))))
> +	;; \E[D - cursor left (terminfo: cub)
> +	((eq char ?D)
> +	(term-move-columns (- (max 1 (car params)))))
> +	;; \E[G - cursor motion to absolute column (terminfo: hpa)
> +	((eq char ?G)
> +	(term-move-columns (- (max 0 (min term-width (car params)))
> +			     (term-current-column))))
> +	;; \E[J - clear to end of screen (terminfo: ed, clear)
> +	((eq char ?J)
> +	(term-erase-in-display (car params)))
> +	;; \E[K - clear to end of line (terminfo: el, el1)
> +	((eq char ?K)
> +	(term-erase-in-line (car params)))
> +	;; \E[L - insert lines (terminfo: il, il1)
> +	((eq char ?L)
> +	(term-insert-lines (max 1 (car params))))
> +	;; \E[M - delete lines (terminfo: dl, dl1)
> +	((eq char ?M)
> +	(term-delete-lines (max 1 (car params))))
> +	;; \E[P - delete chars (terminfo: dch, dch1)
> +	((eq char ?P)
> +	(term-delete-chars (max 1 (car params))))
> +	;; \E[@ - insert spaces (terminfo: ich)
> +	((eq char ?@)
> +	(term-insert-spaces (max 1 (car params))))
> +	;; \E[?h - DEC Private Mode Set
> +
> +	;; N.B. we previously had a bug in which we'd decode \e[?<NR>h or
> +	;; \e[?<NR>l as a command with zero in the params field and so
> +	;; didn't recognize DEC private escape sequences.  However, the
> +	;; termcap and terminfo files had the non-? (question mark means DEC
> +	;; private) versions, so things kind of worked anyway.  To preserve
> +	;; compatibility, we recognize both private- and non-private
> +	;; messages for capabilities we added before we fixed the bug but
> +	;; require the private flag for capabilities we added after.
> +	((eq char ?h)
> +	(cond ((eq (car params) 4) ;; (terminfo: smir)
> +	      (setq term-insert-mode t))
> +	     ((and private (eq (car params) 7)) ;; (terminfo: smam)
> +	      (setq term-auto-margins t))
> +	     ((eq (car params) 47) ;; (terminfo: smcup)
> +	      (term-switch-to-alternate-sub-buffer t))))
> +	;; \E[?l - DEC Private Mode Reset
> +	((eq char ?l)
> +	(cond ((eq (car params) 4) ;; (terminfo: rmir)
> +	      (setq term-insert-mode nil))
> +	     ((and private (eq (car params) 7)) ;; (terminfo: rmam)
> +	      (setq term-auto-margins nil))
> +	     ((eq (car params) 47) ;; (terminfo: rmcup)
> +	      (term-switch-to-alternate-sub-buffer nil))))
> +
> +	;; Modified to allow ansi coloring -mm
> +	;; \E[m - Set/reset modes, set bg/fg
> +	;;(terminfo: smso,rmso,smul,rmul,rev,bold,dim,sitm,ritm,blink,sgr0,invis,op,setab,setaf)
> +	((eq char ?m)
> +	(term--handle-colors-list params))
> +
> +	;; \E[6n - Report cursor position (terminfo: u7)
> +	((eq char ?n)
> +	(term-handle-deferred-scroll)
> +	(process-send-string proc
>  			 ;; (terminfo: u6)
>  			 (format "\e[%s;%sR"
>  				 (1+ (term-current-row))
>  				 (1+ (term-horizontal-column)))))
> -   ;; \E[r - Set scrolling region (terminfo: csr)
> -   ((eq char ?r)
> -    (term-set-scroll-region
> -     (1- (or (nth 0 params) 0))
> -     (1- (or (nth 1 params) 0))))
> -   (t)))
> +	;; \E[r - Set scrolling region (terminfo: csr)
> +	((eq char ?r)
> +	(term-set-scroll-region
> +	(1- (or (nth 0 params) 0))
> +	(1- (or (nth 1 params) 0))))
> +	(t))))))
>  
>  (defun term--reset-scroll-region ()
>    "Set the scroll region to the full height of the terminal."
> -- 
> 2.49.0
> 
> 
> 
> 
> 

Message #11 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Johannes Altmanninger <aclopte <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77872 <at> debbugs.gnu.org, Jared Finder <jared <at> finder.org>
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Fri, 18 Apr 2025 11:00:37 +0200

On Fri, Apr 18, 2025 at 09:05:33AM +0300, Eli Zaretskii wrote:
> > Cc: Johannes Altmanninger <aclopte <at> gmail.com>
> > From: Johannes Altmanninger <aclopte <at> gmail.com>
> > Date: Thu, 17 Apr 2025 20:46:32 +0200
> >
> > * lisp/term.el (term-handle-ansi-escape): Check if parameters contain
> > subparameters (values separated by colon) and ignore those commands.
> > This prevents incorrect interpretation of advanced terminal features
> > that aren't yet properly supported, such as curly underlines (\e[4:3m).
> >
> > Consider ignoring whitespace changes when reviewing this patch.
> > The essence of this change is the addition at the beginning of
> > term-handle-ansi-escape, to skip all CSI commands that contain at
> > least two subparameters (e.g. the ones that contain at least one ":").
> >
> > Some background / longer description:
> >
> > If I run
> >
> > 	emacs --eval '(ansi-term "/bin/bash")'
> >
> > and type
> >
> > 	printf '\e[4:3mHELLO\e[m\n'
> >
> > then I get text with a straight underline.
> >
> > This is not what I expect, because "\e[4:3m" is supposed to turn on
> > curly/wavy underlines, see https://sw.kovidgoyal.net/kitty/underlines/.
> >
> > My program wants to use curly underlines to indicate errors. Rendering
> > those as straight underlines is noisy and confusing because it suggests
> > a different meaning (in my program).
>
> Why cannot you solve this problem by a suitable configuration of faces
> in Emacs?

I'm implementing the default behavior for a shell; it doesn't behoove
the shell to touch the user's (Emacs) configuration.

Now of course I could theoretically achieve my goal with this logic:

    if the terminal supports underline styles (terminfo Smulx or Su):
        use curly underlines
    else:
        use no underline at all

where the condition could be checked by:
1. reading the terminfo database based on $TERM
2. querying the terminal's embedded terminfo database via XTGETTCAP

Option 1 is not great because terminfo frequently has false negatives,
and many of those cannot easily be fixed because they use a generic
$TERM (that's the problem with user-agents).  For example, GNOME
Terminal and KDE's Konsole both support styled underlines but their
TERM=xterm-256color says otherwise.

    # False negative.
    $ infocmp -x xterm-256color | grep -E 'Smulx|Su'

    # Another false negative (here the terminfo db is "fixable").
    $ echo $TERM
    foot
    $ infocmp -x $TERM | grep -E 'Smulx|Su'

    # True positive.
    $ infocmp -x kitty | grep -E 'Smulx|Su'
    	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, TS=\E]2;,
    $ infocmp -x xterm-kitty | grep -E 'Smulx|Su'
    	am, ccc, hs, km, mc5i, mir, msgr, npc, xenl, Su, Tc, XF, fullkbd,
    	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, Sync=\EP=%p1%ds\E\\,

Note that only few terminals are "fixable" (by virtue of using a
non-impersonating $TERM), That comes with the added burden that one
may need to copy terminfo files to remote servers etc., which is
probably part of why few terminals do this.

Option 2 (XTGETTCAP) always gives the right answer because we are
asking the terminal directly -- both the request and response are
control sequences written to the pty.  However it's only implemented
by xterm, kitty and foot so far, so I only use it if there is no
other good option.

But I think there *is* a better (long-term) option here: if we can
assume that a terminal ignores commands that it doesn't recognize,
we don't need to query the capability at all, thus making it much
simpler to use curly underlines in e.g. a bash script. In practice,
this is already the case for a lot of other sequences.

Now the case of «printf "\e[4:3m"» may not be as obvious.
One argument for ignoring the sequence it is that if a user really
wants ansi-term's current behavior of "use curly underlines if
supported, else use straight underlines", they can simply print both:
"\e[4m\e[4:3m", or equivalently "\e[4;4:3m".

XTerm ignores "\e[4:3m" and most the dozens of terminals I tested
follow suit.  However I give you that there are some (important)
terminals render curly underline as straight underline:

- emacs ansi-term
- emacs-vterm
- GNU screen
- terminology
- tmux
- Vim

I plan to propose patches for all of them (ansi-term happened to be
the first)  We'll see what other terminal developers think. I can
link the discussions here. I guess I'll try tmux next.

Notice that most of these are terminals that run inside another
terminal.  So I guess the reason for this automatic fallback could
theoretically be that "tmux/screen/'emacs -nw' etc. cannot accurately
detect whether the underlying terminal supports curly underlines,
so we make the decision for them and give them straight underlines
when in doubt".  But that would be somewhat odd (because it "blocks"
progress) and actually, I haven't found that to be the case for the
terminals where I already looked at the git-blame (ansi-term and
vterm -- for both of them, the missing XTerm compatibility seems like
an oversight).  Note that the curly underline feature is mostly used
by text editors, which may be why no one had encountered this problem
yet with Emacs terminals.

>
> > I'd rather ansi-term ignore the curly underline command until it's
> > implemented.
>
> We could have this behavior as an option, conditioned on some user
> option, perhaps.

I would not be interested in an off-by-default option because if a
user says "this doesn't work in ansi-term" then I can already tell
them "use a different terminal, such as M-x vterm", so it wouldn't
really reduce effort.

I would be surprised if better XTerm compatibility would make things
worse but let's see what the consensus is.

If it's not acceptable to ignore curly underline sequences, I can
probably also implement them in term.el (i.e. actually recognize
\e[4:3m).

That would give us curly underlines in graphical emacs, but it
would barely change the "emacs -nw" behavior in practice, because,
as described above, asking terminfo for Smulx/Su has a lot of false
negatives.  Patch 2/2 would change "emacs -nw" to not
emit anything if the capability is not advertised.
So my perspective is: if we want to implement \e[4:3m now, we should
also take the second patch to actually fix my use case for wrong
terminfo.

In future, to get out of this mess, we should consider not asking
for Smulx/Su and simply emit the styled sequences unconditionally.
Unless Emacs still needs to run on hardware terminals that misinterpret
those, that would be the sanest option going forward.  It will
essentially force all terminal emulators to be compatible with this,
which is fine I think?  From the terminals I have tested, emitting
\e[4:3m would only cause regressions on these terminals: abduco, dvtm,
JetBrains IDE terminals and urxvt.  It should be easy to fix them or
add workarounds; in a few years it should no longer be an issue.

>
> > FWIW Emacs itself already supports various underline styles, even in
> > a TTY -- see e.g. 9f589eb9240 (Add support for colored and styled
> > underlines on tty frames, 2023-04-20). It would be fairly easy to
> > add support for curly underlines to ansi-term.
> >
> > In general, I wonder what's story on Emacs terminals. ansi-term
> > has some other compatibility issues, for example it cannot parse
> > OSC (Operating System Command) sequences, which causes friction
> > with the fish shell. We can fix this fairly easily; alternatively
> > maybe default to another terminal implementation such as vterm
> > (https://github.com/akermu/emacs-libvterm) that offers better
> > xterm-compatibility.
>
> You don't tell in which Emacs version you see these issues.  Please
> tell, it might be important.

Ah it seems like "cannot parse OSC" was the wrong conclusion because
a bash command like

    printf "\x1b]133;A;special_key=1\x07"

is already correctly ignored (though there is the problem that the
prompt is not redrawn).

The actual issue I'm seeing is that running fish shell version 4.0.1,
which prints the above command at startup, causes this text to show
in the terminal:

    133;A;special_key=1

This is with both Emacs 30.1 and latest master (2808bef2522).  fish
assumes basic VT100 compatibility.  Perhaps this is a timing issue.
I'm in a good position to extract a minimal reproducer, I can follow
up with that (probably in a separate thread?).

>
> Jared, any comments to the above and/or to the patch?
>
> In any case, we are unable to accept such a large contribution without
> you assigning the copyright for your changes to the FSF.  If you are
> willing to do that, I will send you the form to fill and the
> instructions to go with it, to start your legal paperwork rolling.

sure.

>
> Thanks.
>
> > ---
> >  lisp/term.el | 230 ++++++++++++++++++++++++++++-----------------------

Message #14 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Johannes Altmanninger <aclopte <at> gmail.com>
Cc: 77872 <at> debbugs.gnu.org, jared <at> finder.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Fri, 18 Apr 2025 14:40:45 +0300

> Date: Fri, 18 Apr 2025 11:00:37 +0200
> From: Johannes Altmanninger <aclopte <at> gmail.com>
> Cc: Jared Finder <jared <at> finder.org>, 77872 <at> debbugs.gnu.org
> 
> > > I'd rather ansi-term ignore the curly underline command until it's
> > > implemented.
> >
> > We could have this behavior as an option, conditioned on some user
> > option, perhaps.
> 
> I would not be interested in an off-by-default option because if a
> user says "this doesn't work in ansi-term" then I can already tell
> them "use a different terminal, such as M-x vterm", so it wouldn't
> really reduce effort.

Your program seems to have unusual requirements, in that it doesn't
want the fallback to underline for some reason that is specific to the
program.  So I don't see any way except opt-in behavior, because
making this the default would be backward-incompatible, and in most
situations, where any kind of underline is fine, it will not make
sense.

> If it's not acceptable to ignore curly underline sequences, I can
> probably also implement them in term.el (i.e. actually recognize
> \e[4:3m).

That might be better, but won't it slow down ansi-term?

> In future, to get out of this mess, we should consider not asking
> for Smulx/Su and simply emit the styled sequences unconditionally.

I'm not sure this is a viable alternative, but I'm not an expert on
these matters.

Message #17 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Johannes Altmanninger <aclopte <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77872 <at> debbugs.gnu.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Fri, 18 Apr 2025 22:33:36 -0700

On 2025-04-18 02:00, Johannes Altmanninger wrote:
> On Fri, Apr 18, 2025 at 09:05:33AM +0300, Eli Zaretskii wrote:
>> > Cc: Johannes Altmanninger <aclopte <at> gmail.com>
>> > From: Johannes Altmanninger <aclopte <at> gmail.com>
>> > Date: Thu, 17 Apr 2025 20:46:32 +0200
>> >
>> > * lisp/term.el (term-handle-ansi-escape): Check if parameters contain
>> > subparameters (values separated by colon) and ignore those commands.
>> > This prevents incorrect interpretation of advanced terminal features
>> > that aren't yet properly supported, such as curly underlines (\e[4:3m).
>> >
>> > Consider ignoring whitespace changes when reviewing this patch.
>> > The essence of this change is the addition at the beginning of
>> > term-handle-ansi-escape, to skip all CSI commands that contain at
>> > least two subparameters (e.g. the ones that contain at least one ":").
>> >
>> > Some background / longer description:
>> >
>> > If I run
>> >
>> > 	emacs --eval '(ansi-term "/bin/bash")'
>> >
>> > and type
>> >
>> > 	printf '\e[4:3mHELLO\e[m\n'
>> >
>> > then I get text with a straight underline.
>> >
>> > This is not what I expect, because "\e[4:3m" is supposed to turn on
>> > curly/wavy underlines, see https://sw.kovidgoyal.net/kitty/underlines/.
>> >
>> > My program wants to use curly underlines to indicate errors. Rendering
>> > those as straight underlines is noisy and confusing because it suggests
>> > a different meaning (in my program).
>> 
>> Why cannot you solve this problem by a suitable configuration of faces
>> in Emacs?
> 
> I'm implementing the default behavior for a shell; it doesn't behoove
> the shell to touch the user's (Emacs) configuration.
> 
> Now of course I could theoretically achieve my goal with this logic:
> 
>     if the terminal supports underline styles (terminfo Smulx or Su):
>         use curly underlines
>     else:
>         use no underline at all
> 
> where the condition could be checked by:
> 1. reading the terminfo database based on $TERM
> 2. querying the terminal's embedded terminfo database via XTGETTCAP
> 
> Option 1 is not great because terminfo frequently has false negatives,
> and many of those cannot easily be fixed because they use a generic
> $TERM (that's the problem with user-agents).  For example, GNOME
> Terminal and KDE's Konsole both support styled underlines but their
> TERM=xterm-256color says otherwise.
> 
>     # False negative.
>     $ infocmp -x xterm-256color | grep -E 'Smulx|Su'
> 
>     # Another false negative (here the terminfo db is "fixable").
>     $ echo $TERM
>     foot
>     $ infocmp -x $TERM | grep -E 'Smulx|Su'
> 
>     # True positive.
>     $ infocmp -x kitty | grep -E 'Smulx|Su'
>     	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, TS=\E]2;,
>     $ infocmp -x xterm-kitty | grep -E 'Smulx|Su'
>     	am, ccc, hs, km, mc5i, mir, msgr, npc, xenl, Su, Tc, XF, fullkbd,
>     	Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, Sync=\EP=%p1%ds\E\\,
> 
> Note that only few terminals are "fixable" (by virtue of using a
> non-impersonating $TERM), That comes with the added burden that one
> may need to copy terminfo files to remote servers etc., which is
> probably part of why few terminals do this.
> 
> Option 2 (XTGETTCAP) always gives the right answer because we are
> asking the terminal directly -- both the request and response are
> control sequences written to the pty.  However it's only implemented
> by xterm, kitty and foot so far, so I only use it if there is no
> other good option.
> 
> But I think there *is* a better (long-term) option here: if we can
> assume that a terminal ignores commands that it doesn't recognize,
> we don't need to query the capability at all, thus making it much
> simpler to use curly underlines in e.g. a bash script. In practice,
> this is already the case for a lot of other sequences.
> 
> Now the case of «printf "\e[4:3m"» may not be as obvious.
> One argument for ignoring the sequence it is that if a user really
> wants ansi-term's current behavior of "use curly underlines if
> supported, else use straight underlines", they can simply print both:
> "\e[4m\e[4:3m", or equivalently "\e[4;4:3m".
> 
> XTerm ignores "\e[4:3m" and most the dozens of terminals I tested
> follow suit.  However I give you that there are some (important)
> terminals render curly underline as straight underline:
> 
> - emacs ansi-term
> - emacs-vterm
> - GNU screen
> - terminology
> - tmux
> - Vim
> 
> I plan to propose patches for all of them (ansi-term happened to be
> the first)  We'll see what other terminal developers think. I can
> link the discussions here. I guess I'll try tmux next.
> 
> Notice that most of these are terminals that run inside another
> terminal.  So I guess the reason for this automatic fallback could
> theoretically be that "tmux/screen/'emacs -nw' etc. cannot accurately
> detect whether the underlying terminal supports curly underlines,
> so we make the decision for them and give them straight underlines
> when in doubt".  But that would be somewhat odd (because it "blocks"
> progress) and actually, I haven't found that to be the case for the
> terminals where I already looked at the git-blame (ansi-term and
> vterm -- for both of them, the missing XTerm compatibility seems like
> an oversight).  Note that the curly underline feature is mostly used
> by text editors, which may be why no one had encountered this problem
> yet with Emacs terminals.
> 
>> 
>> > I'd rather ansi-term ignore the curly underline command until it's
>> > implemented.
>> 
>> We could have this behavior as an option, conditioned on some user
>> option, perhaps.
> 
> I would not be interested in an off-by-default option because if a
> user says "this doesn't work in ansi-term" then I can already tell
> them "use a different terminal, such as M-x vterm", so it wouldn't
> really reduce effort.
> 
> I would be surprised if better XTerm compatibility would make things
> worse but let's see what the consensus is.
> 
> If it's not acceptable to ignore curly underline sequences, I can
> probably also implement them in term.el (i.e. actually recognize
> \e[4:3m).

I think this is a much preferred option. Emacs 30 in a terminal already 
supports displaying wavy underlines. Adding support for wavy underlines 
to ansi-term looks like it wouldn't be harder than the proposed patch.

> That would give us curly underlines in graphical emacs, but it
> would barely change the "emacs -nw" behavior in practice, because,
> as described above, asking terminfo for Smulx/Su has a lot of false
> negatives.  Patch 2/2 would change "emacs -nw" to not
> emit anything if the capability is not advertised.
> So my perspective is: if we want to implement \e[4:3m now, we should
> also take the second patch to actually fix my use case for wrong
> terminfo.

(Note for other readers, patch 2/2 is in another bug: 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=77871)

> In future, to get out of this mess, we should consider not asking
> for Smulx/Su and simply emit the styled sequences unconditionally.
> Unless Emacs still needs to run on hardware terminals that misinterpret
> those, that would be the sanest option going forward.  It will
> essentially force all terminal emulators to be compatible with this,
> which is fine I think?  From the terminals I have tested, emitting
> \e[4:3m would only cause regressions on these terminals: abduco, dvtm,
> JetBrains IDE terminals and urxvt.  It should be easy to fix them or
> add workarounds; in a few years it should no longer be an issue.

If I'm understanding this right, you're suggesting to have Emacs always 
emit the wavy underline escape sequence for :underline (:style wave), 
even on terminals it believes does not support wavy underlines. This 
seems like a significant behavior change. While libraries packaged 
within Emacs look to be good about using a face supports spec when 
defining their faces, the same can't be said for external Emacs 
libraries.

I also don't understand how this makes things better. Are there 
terminals that you encounter that Emacs does not accurately report 
(display-supports-face-attributes-p '(:underline (:style wave))) for? 
Local testing on a Mac with iTerm2, kitty, Alacritty, Apple 
Terminal.app, Rio, and WezTerm correctly reported t vs nil.

  -- MJF

>> > FWIW Emacs itself already supports various underline styles, even in
>> > a TTY -- see e.g. 9f589eb9240 (Add support for colored and styled
>> > underlines on tty frames, 2023-04-20). It would be fairly easy to
>> > add support for curly underlines to ansi-term.
>> >
>> > In general, I wonder what's story on Emacs terminals. ansi-term
>> > has some other compatibility issues, for example it cannot parse
>> > OSC (Operating System Command) sequences, which causes friction
>> > with the fish shell. We can fix this fairly easily; alternatively
>> > maybe default to another terminal implementation such as vterm
>> > (https://github.com/akermu/emacs-libvterm) that offers better
>> > xterm-compatibility.
>> 
>> You don't tell in which Emacs version you see these issues.  Please
>> tell, it might be important.
> 
> Ah it seems like "cannot parse OSC" was the wrong conclusion because
> a bash command like
> 
>     printf "\x1b]133;A;special_key=1\x07"
> 
> is already correctly ignored (though there is the problem that the
> prompt is not redrawn).
> 
> The actual issue I'm seeing is that running fish shell version 4.0.1,
> which prints the above command at startup, causes this text to show
> in the terminal:
> 
>     133;A;special_key=1
> 
> This is with both Emacs 30.1 and latest master (2808bef2522).  fish
> assumes basic VT100 compatibility.  Perhaps this is a timing issue.
> I'm in a good position to extract a minimal reproducer, I can follow
> up with that (probably in a separate thread?).
> 
>> 
>> Jared, any comments to the above and/or to the patch?
>> 
>> In any case, we are unable to accept such a large contribution without
>> you assigning the copyright for your changes to the FSF.  If you are
>> willing to do that, I will send you the form to fill and the
>> instructions to go with it, to start your legal paperwork rolling.
> 
> sure.
> 
>> 
>> Thanks.
>> 
>> > ---
>> >  lisp/term.el | 230 ++++++++++++++++++++++++++++-----------------------

Message #20 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 77872 <at> debbugs.gnu.org, Johannes Altmanninger <aclopte <at> gmail.com>
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Sat, 19 Apr 2025 08:15:20 -0700

On 2025-04-18 04:40, Eli Zaretskii wrote:
>> Date: Fri, 18 Apr 2025 11:00:37 +0200
>> From: Johannes Altmanninger <aclopte <at> gmail.com>
>> Cc: Jared Finder <jared <at> finder.org>, 77872 <at> debbugs.gnu.org
>> 
>> If it's not acceptable to ignore curly underline sequences, I can
>> probably also implement them in term.el (i.e. actually recognize
>> \e[4:3m).
> 
> That might be better, but won't it slow down ansi-term?

I wouldn't be worried about this. The affected code 
(term-handle-ansi-escape) is only called for \e[ escape sequences and 
already parses the entire escape sequence string.

  -- MJF

Message #23 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Johannes Altmanninger <aclopte <at> gmail.com>
To: Jared Finder <jared <at> finder.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77872 <at> debbugs.gnu.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Sun, 20 Apr 2025 10:03:00 +0200

On Fri, Apr 18, 2025 at 10:33:36PM -0700, Jared Finder wrote:
> On 2025-04-18 02:00, Johannes Altmanninger wrote:
> > If it's not acceptable to ignore curly underline sequences, I can
> > probably also implement them in term.el (i.e. actually recognize
> > \e[4:3m).
> 
> I think this is a much preferred option. Emacs 30 in a terminal already
> supports displaying wavy underlines. Adding support for wavy underlines to
> ansi-term looks like it wouldn't be harder than the proposed patch.

It would be a step in the right direction, yeah
(though unrelated to my troubles with terminfo)

Given that we don't want to break this specific implicitly-supported command,
we should probably do the same for others, if any.
So the question is, is there any other command of the form
"CSI Ps : Ps X" where ansi-term's current interpretation as "CSI Ps X" is
reasonable. 

We'd need to do a bit of research, based on the commands that ansi-term
supports today.

Not sure when I can get to that (either way, I'll likely not be able
to enable curly underlines by default; there are various issues left
with other terminals).

> > That would give us curly underlines in graphical emacs, but it
> > would barely change the "emacs -nw" behavior in practice, because,
> > as described above, asking terminfo for Smulx/Su has a lot of false
> > negatives.  Patch 2/2 would change "emacs -nw" to not
> > emit anything if the capability is not advertised.
> > So my perspective is: if we want to implement \e[4:3m now, we should
> > also take the second patch to actually fix my use case for wrong
> > terminfo.
> 
> (Note for other readers, patch 2/2 is in another bug:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=77871)

Yeah sorry I missed the "we prefer the 'git format-patch' method with
attachment" part from the CONTRIBUTE file.

> > In future, to get out of this mess, we should consider not asking
> > for Smulx/Su and simply emit the styled sequences unconditionally.
> > Unless Emacs still needs to run on hardware terminals that misinterpret
> > those, that would be the sanest option going forward.  It will
> > essentially force all terminal emulators to be compatible with this,
> > which is fine I think?  From the terminals I have tested, emitting
> > \e[4:3m would only cause regressions on these terminals: abduco, dvtm,
> > JetBrains IDE terminals and urxvt.  It should be easy to fix them or
> > add workarounds; in a few years it should no longer be an issue.
> 
> If I'm understanding this right, you're suggesting to have Emacs always emit
> the wavy underline escape sequence for :underline (:style wave), even on
> terminals it believes does not support wavy underlines.
> This seems like a significant behavior change.

Yes, deferring to the underlying terminal here would be a noticeable
change. I'd argue it would be simpler overall to have the middle-man 
be less opinionated but of course that's subjective. It's not super
obvious how nested terminals should behave. I personally tend to
avoid them nowadays though they are very useful sometimes.

> While libraries packaged within Emacs look to
> be good about using a face supports spec when defining their faces, the same
> can't be said for external Emacs libraries.

Right. The fallback behavior is helpful to some users/libraries and
the opposite to others.

I realized today even if I queried those nestable terminals (like tmux and ansi-term),
and they always returned the correct answer as to whether Smulx/Su is supported,
it wouldn't really help me because they this doesn't tell me anything
about the parent terminal.
I guess I can work around it for tmux (which does tell me the parent
terminal's name and pty) but it sounds pretty difficult to standardize
an interface for this.

> I also don't understand how this makes things better. Are there terminals
> that you encounter that Emacs does not accurately report
> (display-supports-face-attributes-p '(:underline (:style wave))) for? Local
> testing on a Mac with iTerm2, kitty, Alacritty, Apple Terminal.app, Rio, and
> WezTerm correctly reported t vs nil.

Couple problems.

1.

I took a naïve approach and installed Emacs 29.4 as well as these
terminals via Homebrew on a (mostly vanilla) macOS. I've been running
"emacs -nw" with no config, evaluating that expression gave nil for
all of them.

This result is consistent with

	{ infocmp -x $TERM; infocmp -x $TERM_PROGRAM } | grep -E '\b(Su|Smulx)\b'

which gave no results on this.

I guess it's my bad to use macOS's system libraries that are probably not updated often.

After `brew install ncurses` version 6.5, my $(find /opt/homebrew -name infocmp) did manage
to find a terminfo entry for say kitty, but not others like iTerm.app.
Presumably "TERMINFO=/opt/homebrew/opt/ncurses/share/terminfo emacs -nw" would work too.

On my Linux system only kitty, alacritty and rio have it.

	$ for term in iterm2 kitty xterm-kitty alacritty Apple_Terminal rio wezterm; do
	    echo $term
	    infocmp -x $term | grep -E '\b(Su|Smulx)\b'
	done
	iterm2
	kitty
		Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, TS=\E]2;,
	xterm-kitty
		am, ccc, hs, km, mc5i, mir, msgr, npc, xenl, Su, Tc, XF, fullkbd,
		Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, Sync=\EP=%p1%ds\E\\,
	alacritty
		PE=\E[201~, PS=\E[200~, Se=\E[0 q, Smulx=\E[4:%p1%dm,
	Apple_Terminal
	rio
		PE=\E[201~, PS=\E[200~, Se=\E[0 q, Smulx=\E[4:%p1%dm,
	wezterm

Of course we can and should add Smulx/Su to the others.
But the bigger problems is:

2.

Out of the ones you mention,
alacritty, Apple Terminal.app, iTerm and wezterm
impersonate XTerm by setting `TERM=xterm256color`
(as do others like gnome-terminal and Konsole).

As far as I can tell this means that
/usr/share/terminfo/x/xterm+256color should be used, which does *not*
support curly and colored underlines.

I'd be very curious to hear what kind of magic Emacs and/or your
system uses to somehow still get Su/Smulx support for those terminals.

Especially for Terminal.app, which does not even support XTVERSION;
the only way to work around its quirks is by checking if $TERM_PROGRAM
is "Apple_Terminal", which I don't see Emacs doing.


3.

However way you got Emacs to think Terminal.app supports curly
underlines, it definitely does not support the de-facto standard
control sequence here. Even worse: it interprets that command as
"paint the background yellow". To reproduce:

	printf '\033[4:3mcurly\033[m'

Message #26 received at 77872 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Johannes Altmanninger <aclopte <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 77872 <at> debbugs.gnu.org
Subject: Re: bug#77872: [PATCH 1/2] ansi-term: ignore CSI commands with
 subparams
Date: Sun, 20 Apr 2025 08:25:01 -0700

On 2025-04-20 01:03, Johannes Altmanninger wrote:
> On Fri, Apr 18, 2025 at 10:33:36PM -0700, Jared Finder wrote:
>> On 2025-04-18 02:00, Johannes Altmanninger wrote:
>> > If it's not acceptable to ignore curly underline sequences, I can
>> > probably also implement them in term.el (i.e. actually recognize
>> > \e[4:3m).
>> 
>> I think this is a much preferred option. Emacs 30 in a terminal 
>> already
>> supports displaying wavy underlines. Adding support for wavy 
>> underlines to
>> ansi-term looks like it wouldn't be harder than the proposed patch.
> 
> It would be a step in the right direction, yeah
> (though unrelated to my troubles with terminfo)
> 
> Given that we don't want to break this specific implicitly-supported 
> command,
> we should probably do the same for others, if any.
> So the question is, is there any other command of the form
> "CSI Ps : Ps X" where ansi-term's current interpretation as "CSI Ps X" 
> is
> reasonable.
> 
> We'd need to do a bit of research, based on the commands that ansi-term
> supports today.
> 
> Not sure when I can get to that (either way, I'll likely not be able
> to enable curly underlines by default; there are various issues left
> with other terminals).

Sounds appropriate to me, though I think just adding support to term.el 
for wavy underlines would be a step in the right direction and useful on 
its own. No need to wait for perfection here.

>> > In future, to get out of this mess, we should consider not asking
>> > for Smulx/Su and simply emit the styled sequences unconditionally.
>> > Unless Emacs still needs to run on hardware terminals that misinterpret
>> > those, that would be the sanest option going forward.  It will
>> > essentially force all terminal emulators to be compatible with this,
>> > which is fine I think?  From the terminals I have tested, emitting
>> > \e[4:3m would only cause regressions on these terminals: abduco, dvtm,
>> > JetBrains IDE terminals and urxvt.  It should be easy to fix them or
>> > add workarounds; in a few years it should no longer be an issue.
>> 
>> If I'm understanding this right, you're suggesting to have Emacs 
>> always emit
>> the wavy underline escape sequence for :underline (:style wave), even 
>> on
>> terminals it believes does not support wavy underlines.
>> This seems like a significant behavior change.
> 
> Yes, deferring to the underlying terminal here would be a noticeable
> change. I'd argue it would be simpler overall to have the middle-man
> be less opinionated but of course that's subjective. It's not super
> obvious how nested terminals should behave. I personally tend to
> avoid them nowadays though they are very useful sometimes.

From my perspective, it goes against the usual Emacs pattern where 
unsupported face specs are ignored.  However, I do think it makes sense 
for term.el to properly report what it does or doesn't support, hence my 
comment about checking what Emacs believes the underlying terminal 
supports with display-supports-face-attributes-p.

>> While libraries packaged within Emacs look to
>> be good about using a face supports spec when defining their faces, 
>> the same
>> can't be said for external Emacs libraries.
> 
> Right. The fallback behavior is helpful to some users/libraries and
> the opposite to others.
> 
> I realized today even if I queried those nestable terminals (like tmux 
> and ansi-term),
> and they always returned the correct answer as to whether Smulx/Su is 
> supported,
> it wouldn't really help me because they this doesn't tell me anything
> about the parent terminal.
> I guess I can work around it for tmux (which does tell me the parent
> terminal's name and pty) but it sounds pretty difficult to standardize
> an interface for this.
> 
>> I also don't understand how this makes things better. Are there 
>> terminals
>> that you encounter that Emacs does not accurately report
>> (display-supports-face-attributes-p '(:underline (:style wave))) for? 
>> Local
>> testing on a Mac with iTerm2, kitty, Alacritty, Apple Terminal.app, 
>> Rio, and
>> WezTerm correctly reported t vs nil.
> 
> Couple problems.
> 
> 1.
> 
> I took a naïve approach and installed Emacs 29.4 as well as these
> terminals via Homebrew on a (mostly vanilla) macOS. I've been running
> "emacs -nw" with no config, evaluating that expression gave nil for
> all of them.

Emacs 30 appears to have changed how styling underlines is detected. 
(https://git.savannah.gnu.org/cgit/emacs.git/tree/etc/NEWS?h=emacs-30#n244) 
 Can you run the same test on Emacs 30.1 or later?  Emacs 30.1 is 
available via Homebrew as well, https://formulae.brew.sh/cask/emacs.  I 
tested with Emacs installed via brew install --cask.

> This result is consistent with
> 
> 	{ infocmp -x $TERM; infocmp -x $TERM_PROGRAM } | grep -E 
> '\b(Su|Smulx)\b'
> 
> which gave no results on this.
> 
> I guess it's my bad to use macOS's system libraries that are probably 
> not updated often.
> 
> After `brew install ncurses` version 6.5, my $(find /opt/homebrew -name 
> infocmp) did manage
> to find a terminfo entry for say kitty, but not others like iTerm.app.
> Presumably "TERMINFO=/opt/homebrew/opt/ncurses/share/terminfo emacs 
> -nw" would work too.
> 
> On my Linux system only kitty, alacritty and rio have it.
> 
> 	$ for term in iterm2 kitty xterm-kitty alacritty Apple_Terminal rio 
> wezterm; do
> 	    echo $term
> 	    infocmp -x $term | grep -E '\b(Su|Smulx)\b'
> 	done
> 	iterm2
> 	kitty
> 		Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, TS=\E]2;,
> 	xterm-kitty
> 		am, ccc, hs, km, mc5i, mir, msgr, npc, xenl, Su, Tc, XF, fullkbd,
> 		Smulx=\E[4:%p1%dm, Ss=\E[%p1%d q, Sync=\EP=%p1%ds\E\\,
> 	alacritty
> 		PE=\E[201~, PS=\E[200~, Se=\E[0 q, Smulx=\E[4:%p1%dm,
> 	Apple_Terminal
> 	rio
> 		PE=\E[201~, PS=\E[200~, Se=\E[0 q, Smulx=\E[4:%p1%dm,
> 	wezterm
> 
> Of course we can and should add Smulx/Su to the others.
> But the bigger problems is:
> 
> 2.
> 
> Out of the ones you mention,
> alacritty, Apple Terminal.app, iTerm and wezterm
> impersonate XTerm by setting `TERM=xterm256color`
> (as do others like gnome-terminal and Konsole).
> 
> As far as I can tell this means that
> /usr/share/terminfo/x/xterm+256color should be used, which does *not*
> support curly and colored underlines.
> 
> I'd be very curious to hear what kind of magic Emacs and/or your
> system uses to somehow still get Su/Smulx support for those terminals.
> 
> Especially for Terminal.app, which does not even support XTVERSION;
> the only way to work around its quirks is by checking if $TERM_PROGRAM
> is "Apple_Terminal", which I don't see Emacs doing.

I don't think I have done anything special here.  I have a mostly stock 
MacOS, version 15.3.2 with the following leaf packages installed from 
Homebrew with no customization:

$ brew leaves
aspell
autoconf
bash
bash-completion
binutils
ghostscript
most
nano
p7zip
pkgconf
popler
python <at> 3.13
texinfo

> 3.
> 
> However way you got Emacs to think Terminal.app supports curly
> underlines, it definitely does not support the de-facto standard
> control sequence here. Even worse: it interprets that command as
> "paint the background yellow". To reproduce:
> 
> 	printf '\033[4:3mcurly\033[m'

I did not get Emacs to think Terminal.app supports wavy underlines.  
Emacs 30.1 returns nil for (display-supports-face-attributes-p 
'(:underline (:style wave))) on Terminal.app.  I think one other 
terminal didn't support wavy underlines and Emacs returned nil for as 
well, though I do not recall which one.  All the other terminals 
supported wavy underlines and Emacs returned t when checked with 
display-supports-face-attributes-p.  I see the same "paint the 
background yellow" behavior in Terminal.app when I emit escape sequences 
directly.

  -- MJF

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.