GNU bug report logs - #78263
[PATCH 1/3] Ignore OSC sequences in term instead of printing them

Previous Next

To reply to this bug, email your comments to 78263 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Zermatten <szermatt <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: [PATCH 1/3] Ignore OSC sequences in term instead of printing them
Date: Mon, 05 May 2025 17:30:35 +0300

[Message part 1 (text/plain, inline)]

Tags: patch


This change detects OSC sequences and ignores them. This is what's
normally expected and avoids strange outputs.

Starting with version 4, the fish shell has started sending out OSC
sequences by default, which looks pretty bad on term. This change avoids
this particular problem.

This is PATCH 1/3, because I'd like to propose making it possible to
handle OSC sequences in term in follow-up changes. Emacs already has
handlers for OSC sequences in ansi-osc, why not make use of them?

In GNU Emacs 31.0.50 (build 5, x86_64-apple-darwin23.6.0, NS
 appkit-2487.70 Version 14.7.5 (Build 23H527)) of 2025-05-02 built on
 boomer.zia
Repository revision: 99ca41b6ef300653a0d15b73a0c0d446a9a9e059
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2487
System Description:  macOS 14.7.5

Configured using:
 'configure --program-suffix=-head --with-tree-sitter
 --with-native-compilation=aot'

[0001-Ignore-OSC-sequences-in-term-instead-of-printing-the.patch (text/patch, attachment)]

Message #8 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephane Zermatten <szermatt <at> gmail.com>
Cc: per <at> bothner.com, dann <at> ics.uci.edu, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Mon, 05 May 2025 18:55:06 +0300

> Cc: Dan Nicolaescu <dann <at> ics.uci.edu>, Per Bothner <per <at> bothner.com>
> From: Stephane Zermatten <szermatt <at> gmail.com>
> Date: Mon, 05 May 2025 17:30:35 +0300
> 
> This change detects OSC sequences and ignores them. This is what's
> normally expected and avoids strange outputs.
> 
> Starting with version 4, the fish shell has started sending out OSC
> sequences by default, which looks pretty bad on term. This change avoids
> this particular problem.
> 
> This is PATCH 1/3, because I'd like to propose making it possible to
> handle OSC sequences in term in follow-up changes. Emacs already has
> handlers for OSC sequences in ansi-osc, why not make use of them?

I indeed think that teaching Emacs about these sequences is better
than filtering them out.  Since you seem to think the same, and will
follow up with patches to that effect, why do we need this patch at
all?

Message #11 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Per Bothner <per <at> bothner.com>
To: Stephane Zermatten <szermatt <at> gmail.com>, 78263 <at> debbugs.gnu.org
Cc: Eli Zaretskii <eliz <at> gnu.org>, Dan Nicolaescu <dann <at> ics.uci.edu>
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Mon, 5 May 2025 09:36:45 -0700

The whole term-control-seq-regexp is fundamentally broken, and does not appear
in any version of term.el I have worked on or blessed.  It assumes all relevant sequences
are complete in a single call to term-emulate-terminal. That is not a safe assumption.

The correct way to parse terminal escape sequences is using a state machine, as in
my original implementation. Alternatively, if you really want to use pattern matching,
it is possible to recognize "incomplete escape sequence", and save that until the
next call.  However, I don't see any code to handle that; it is possible I'm missing something.
(That approach can in theory lead to O(n^2) behavior, which the state machine approach avoids,
but I don't think that is a major problem.)

I agree with Eli that term should when possible handle OSC sequences rather than ignoring them.
(Ones it can't handle should of course be ignored.)

On 5/5/25 7:30 AM, Stephane Zermatten wrote:
> Tags: patch
> 
> 
> This change detects OSC sequences and ignores them. This is what's
> normally expected and avoids strange outputs.
> 
> Starting with version 4, the fish shell has started sending out OSC
> sequences by default, which looks pretty bad on term. This change avoids
> this particular problem.
> 
> This is PATCH 1/3, because I'd like to propose making it possible to
> handle OSC sequences in term in follow-up changes. Emacs already has
> handlers for OSC sequences in ansi-osc, why not make use of them?
> 
> In GNU Emacs 31.0.50 (build 5, x86_64-apple-darwin23.6.0, NS
>   appkit-2487.70 Version 14.7.5 (Build 23H527)) of 2025-05-02 built on
>   boomer.zia
> Repository revision: 99ca41b6ef300653a0d15b73a0c0d446a9a9e059
> Repository branch: master
> Windowing system distributor 'Apple', version 10.3.2487
> System Description:  macOS 14.7.5
> 
> Configured using:
>   'configure --program-suffix=-head --with-tree-sitter
>   --with-native-compilation=aot'
> 

-- 
	--Per Bothner
per <at> bothner.com   http://per.bothner.com/

Message #14 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Per Bothner <per <at> bothner.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Dan Nicolaescu <dann <at> ics.uci.edu>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Mon, 5 May 2025 19:57:45 +0300

[Message part 1 (text/plain, inline)]

I attached a merged version of all 3 patches to this e-mail that handles OSC sequences, so they’re handled right away.  I’m happy to do it that way, if it’s more convenient. 

[0001-Handle-OSC-sequences-in-term.patch (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

> On 5 May 2025, at 19:36, Per Bothner <per <at> bothner.com> wrote:
> 
> The whole term-control-seq-regexp is fundamentally broken, and does not appear
> in any version of term.el I have worked on or blessed.  It assumes all relevant sequences
> are complete in a single call to term-emulate-terminal. That is not a safe assumption.
> 
> The correct way to parse terminal escape sequences is using a state machine, as in
> my original implementation. Alternatively, if you really want to use pattern matching,
> it is possible to recognize "incomplete escape sequence", and save that until the
> next call.  However, I don't see any code to handle that; it is possible I'm missing something.
> (That approach can in theory lead to O(n^2) behavior, which the state machine approach avoids,
> but I don't think that is a major problem.)

A state machine is indeed the right way of handling terminal escape sequences, and it would be a great idea to refactor term.el to do it that way. I don’t really want to use pattern matching. I’m using it here because it integrates with the current code. 

The current implementation does indeed recognize incomplete escape sequence, See term-control-seq-prefix-regexp and its use in term-emulate-terminal. It’s a bit, well, funny, but it seems to work. It keeps incomplete sequences in term-terminal-undecoded-bytes for the next call. 

In the patch attached to this e-mail, you’ll notice that I had to add some code that detects whether a sequence is terminated and buffers it for next time, in term-terminal-undecoded-bytes to be consistent with the current implementation. This is covered by the tests. 

If you guys think it’s time to refactor term.el to use a state machine and have time to spend on it, I’d be happy to help as much as I can. It’s a bit scary, because, unless I missed something, test coverage of the existing code isn’t very high, so adding more tests would be, I think, a good first step before engaging in larger changes. 
 
> 
> I agree with Eli that term should when possible handle OSC sequences rather than ignoring them.
> (Ones it can't handle should of course be ignored.)
> 
> On 5/5/25 7:30 AM, Stephane Zermatten wrote:
>> Tags: patch
>> This change detects OSC sequences and ignores them. This is what's
>> normally expected and avoids strange outputs.
>> Starting with version 4, the fish shell has started sending out OSC
>> sequences by default, which looks pretty bad on term. This change avoids
>> this particular problem.
>> This is PATCH 1/3, because I'd like to propose making it possible to
>> handle OSC sequences in term in follow-up changes. Emacs already has
>> handlers for OSC sequences in ansi-osc, why not make use of them?
>> In GNU Emacs 31.0.50 (build 5, x86_64-apple-darwin23.6.0, NS
>>  appkit-2487.70 Version 14.7.5 (Build 23H527)) of 2025-05-02 built on
>>  boomer.zia
>> Repository revision: 99ca41b6ef300653a0d15b73a0c0d446a9a9e059
>> Repository branch: master
>> Windowing system distributor 'Apple', version 10.3.2487
>> System Description:  macOS 14.7.5
>> Configured using:
>>  'configure --program-suffix=-head --with-tree-sitter
>>  --with-native-compilation=aot'
> 
> -- 
> 	--Per Bothner
> per <at> bothner.com   http://per.bothner.com/
> 

Message #17 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Per Bothner <per <at> bothner.com>
To: Stéphane Zermatten <szermatt <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Dan Nicolaescu <dann <at> ics.uci.edu>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Mon, 5 May 2025 11:46:24 -0700

[Message part 1 (text/plain, inline)]

On 5/5/25 9:57 AM, Stéphane Zermatten wrote:
> A state machine is indeed the right way of handling terminal escape sequences, and it would be a great idea to refactor term.el to do it that way. I don’t really want to use pattern matching. I’m using it here because it integrates with the current code.
> 
> The current implementation does indeed recognize incomplete escape sequence, See term-control-seq-prefix-regexp and its use in term-emulate-terminal. It’s a bit, well, funny, but it seems to work. It keeps incomplete sequences in term-terminal-undecoded-bytes for the next call.
> 
> In the patch attached to this e-mail, you’ll notice that I had to add some code that detects whether a sequence is terminated and buffers it for next time, in term-terminal-undecoded-bytes to be consistent with the current implementation. This is covered by the tests.
> 
> If you guys think it’s time to refactor term.el to use a state machine and have time to spend on it, I’d be happy to help as much as I can. It’s a bit scary, because, unless I missed something, test coverage of the existing code isn’t very high, so adding more tests would be, I think, a good first step before engaging in larger changes.

I don't feel stronly about state machines - I think it would be more robust, but I don't know
if that justifies the effort. In principle using the term-terminal-undecoded-bytes logic is fine;
however I'm not sure how "complete" the logic is in terms of unusual unhandled escape sequences or garbled inputs.

If somebody does re-write term to use a state machine, it might save a little time and thought
to re-use the code from my original term.el (attached).
-- 
	--Per Bothner
per <at> bothner.com   http://per.bothner.com/

[term.el (text/x-emacs-lisp, attachment)]

Message #20 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stéphane Zermatten <szermatt <at> gmail.com>,
 Jared Finder <jared <at> finder.org>
Cc: per <at> bothner.com, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Thu, 08 May 2025 13:03:26 +0300

> From: Stéphane Zermatten <szermatt <at> gmail.com>
> Date: Mon, 5 May 2025 19:57:45 +0300
> Cc: 78263 <at> debbugs.gnu.org,
>  Dan Nicolaescu <dann <at> ics.uci.edu>,
>  Eli Zaretskii <eliz <at> gnu.org>
> 
> I attached a merged version of all 3 patches to this e-mail that handles OSC sequences, so they’re handled right away.  I’m happy to do it that way, if it’s more convenient. 

Thanks, see some minor comments below.

Jared, any further comments about the issue or the patch?

> * lisp/term.el (term-emulate-terminal) handle OSC sequences

The log message should mention all the functions/variables where you
make changes, with a short description of each change.  Please also
mention the bug number.

> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
> +                               ("2" . ansi-osc-window-title-handler)
> +                               ("7" . ansi-osc-directory-tracker)
> +                               ("8" . ansi-osc-hyperlink-handler))
> +  "OSC sequence handler function alist.
> +
> +OSC (Operating System Command) is a category of ANSI escape sequence
> +used in terminal application to introduce custom commands. Terminals
> +ignore unknown OSC sequences by default. Handlers can be registered here
> +to add support for new OSC sequences to `term'.
> +
> +Functions in this alist are passed matching valid OSC sequences as
> +they're sent to the terminal.
> +
> +Valid OSC sequences are of the form
> +  ESC ] code ; text BEL
> +  ESC ] code ; text ESC \
> +
> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
> +appears before the semicolon.
> +
> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
> +active and its point and state active at the time the OSC sequence
> +appeared in the stream."
> +  :type '(alist :key-type string :value-type function)
> +  :group 'term)

New defcustoms should have a :version tag.

> +(defconst term-osc--max-bytes (* 32 1024 1024)
> +  "Limit the length of OSC sequences to keep in memory.")

This begs the question: what happens with longer sequences?  Can the
doc string answer that question?

> +                              (with-demoted-errors "term OSC error: %S"
> +                                (funcall
> +                                 func code
> +                                 (decode-coding-string
> +                                  (substring seq-str text-start end-mark)
> +                                  locale-coding-system t)))))

Why locale-coding-system?  Wouldn't the result of calling
process-coding-system with the process running in term.el be a better
candidate?

Last, but not least: you don't seem to have a copyright assignment on
file, without which we cannot accept such a large contribution.  Would
you like to start the paperwork of copyright assignment at this time?
If yes, I will send you the form to fill and the instructions.

Message #23 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Per Bothner <per <at> bothner.com>, Jared Finder <jared <at> finder.org>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Tue, 13 May 2025 17:24:29 +0300

[Message part 1 (text/plain, inline)]

Thank you for the review! I addressed the points you mentioned. See the updated patch attached to this e-mail. 

> On 8 May 2025, at 13:03, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: Stéphane Zermatten <szermatt <at> gmail.com>
>> Date: Mon, 5 May 2025 19:57:45 +0300
>> Cc: 78263 <at> debbugs.gnu.org,
>> Dan Nicolaescu <dann <at> ics.uci.edu>,
>> Eli Zaretskii <eliz <at> gnu.org>
>> 
>> I attached a merged version of all 3 patches to this e-mail that handles OSC sequences, so they’re handled right away.  I’m happy to do it that way, if it’s more convenient.
> 
> Thanks, see some minor comments below.
> 
> Jared, any further comments about the issue or the patch?
> 
>> * lisp/term.el (term-emulate-terminal) handle OSC sequences
> 
> The log message should mention all the functions/variables where you
> make changes, with a short description of each change.  Please also
> mention the bug number.

I updated the description, trying to follow the format of older commits.

> 
>> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
>> +                               ("2" . ansi-osc-window-title-handler)
>> +                               ("7" . ansi-osc-directory-tracker)
>> +                               ("8" . ansi-osc-hyperlink-handler))
>> +  "OSC sequence handler function alist.
>> +
>> +OSC (Operating System Command) is a category of ANSI escape sequence
>> +used in terminal application to introduce custom commands. Terminals
>> +ignore unknown OSC sequences by default. Handlers can be registered here
>> +to add support for new OSC sequences to `term'.
>> +
>> +Functions in this alist are passed matching valid OSC sequences as
>> +they're sent to the terminal.
>> +
>> +Valid OSC sequences are of the form
>> +  ESC ] code ; text BEL
>> +  ESC ] code ; text ESC \
>> +
>> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
>> +appears before the semicolon.
>> +
>> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
>> +active and its point and state active at the time the OSC sequence
>> +appeared in the stream."
>> +  :type '(alist :key-type string :value-type function)
>> +  :group 'term)
> 
> New defcustoms should have a :version tag.

I added one. I guess we’re at 31.1, now. 

> 
>> +(defconst term-osc--max-bytes (* 32 1024 1024)
>> +  "Limit the length of OSC sequences to keep in memory.")
> 
> This begs the question: what happens with longer sequences?  Can the
> doc string answer that question?

emulate-terminal goes back into its normal state, because presumably the end tag was lost.
I tried to describe that.  

> 
>> +                              (with-demoted-errors "term OSC error: %S"
>> +                                (funcall
>> +                                 func code
>> +                                 (decode-coding-string
>> +                                  (substring seq-str text-start end-mark)
>> +                                  locale-coding-system t)))))
> 
> Why locale-coding-system?  Wouldn't the result of calling
> process-coding-system with the process running in term.el be a better
> candidate?

process-coding-system is just 'binary because emulate-terminal want to process the escape sequences before the string is decoded. Using locale-coding-system is the way strings are decoded everywhere in emulate-terminal.

> 
> Last, but not least: you don't seem to have a copyright assignment on
> file, without which we cannot accept such a large contribution.  Would
> you like to start the paperwork of copyright assignment at this time?
> If yes, I will send you the form to fill and the instructions.

Yes, I still need to do that. I sent out a request to assign <at> gnu.org <mailto:assign <at> gnu.org> following the instructions in etc/copyright-assign.txt but have yet to hear back from them. 



[Message part 2 (text/html, inline)]

[0001-Handle-OSC-sequences-in-term-bug-78263.patch (application/octet-stream, attachment)]

[Message part 4 (text/html, inline)]

Message #26 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stéphane Zermatten <szermatt <at> gmail.com>
Cc: per <at> bothner.com, jared <at> finder.org, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Sat, 24 May 2025 12:06:29 +0300

> From: Stéphane Zermatten <szermatt <at> gmail.com>
> Date: Tue, 13 May 2025 17:24:29 +0300
> Cc: Jared Finder <jared <at> finder.org>,
>  Per Bothner <per <at> bothner.com>,
>  78263 <at> debbugs.gnu.org
> 
> > Last, but not least: you don't seem to have a copyright assignment on
> > file, without which we cannot accept such a large contribution.  Would
> > you like to start the paperwork of copyright assignment at this time?
> > If yes, I will send you the form to fill and the instructions.
> 
> Yes, I still need to do that. I sent out a request to assign <at> gnu.org <mailto:assign <at> gnu.org> following the instructions in etc/copyright-assign.txt but have yet to hear back from them. 

Any progress in this matter?  If you still haven't heard from
assign <at> gnu.org, please ping them and CC me.

Jared, would you please review the patch and comment on it?

Thanks.

Message #29 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Per Bothner <per <at> bothner.com>, Jared Finder <jared <at> finder.org>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Sat, 31 May 2025 11:37:23 +0300

There’s been some progress on the copyright assignment front: I got the document from assign <at> gnu.org and sent it back 3 days ago. Just waiting for the confirmation. I’ll ping this bug once I get it. 

> On 24 May 2025, at 12:06, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: Stéphane Zermatten <szermatt <at> gmail.com>
>> Date: Tue, 13 May 2025 17:24:29 +0300
>> Cc: Jared Finder <jared <at> finder.org>,
>> Per Bothner <per <at> bothner.com>,
>> 78263 <at> debbugs.gnu.org
>> 
>>> Last, but not least: you don't seem to have a copyright assignment on
>>> file, without which we cannot accept such a large contribution.  Would
>>> you like to start the paperwork of copyright assignment at this time?
>>> If yes, I will send you the form to fill and the instructions.
>> 
>> Yes, I still need to do that. I sent out a request to assign <at> gnu.org <mailto:assign <at> gnu.org> following the instructions in etc/copyright-assign.txt but have yet to hear back from them.
> 
> Any progress in this matter?  If you still haven't heard from
> assign <at> gnu.org, please ping them and CC me.
> 
> Jared, would you please review the patch and comment on it?
> 
> Thanks.

Message #32 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: jared <at> finder.org, Stéphane Zermatten <szermatt <at> gmail.com>
Cc: per <at> bothner.com, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Sat, 31 May 2025 12:18:09 +0300

Ping!  We are waiting for the copyright assignment, but meanwhile,
Jared, could you please review the patch and comment?

> From: Stéphane Zermatten <szermatt <at> gmail.com>
> Date: Tue, 13 May 2025 17:24:29 +0300
> Cc: Jared Finder <jared <at> finder.org>,
>  Per Bothner <per <at> bothner.com>,
>  78263 <at> debbugs.gnu.org
> 
> Thank you for the review! I addressed the points you mentioned. See the updated patch attached to this e-mail. 
> 
> > On 8 May 2025, at 13:03, Eli Zaretskii <eliz <at> gnu.org> wrote:
> > 
> >> From: Stéphane Zermatten <szermatt <at> gmail.com>
> >> Date: Mon, 5 May 2025 19:57:45 +0300
> >> Cc: 78263 <at> debbugs.gnu.org,
> >> Dan Nicolaescu <dann <at> ics.uci.edu>,
> >> Eli Zaretskii <eliz <at> gnu.org>
> >> 
> >> I attached a merged version of all 3 patches to this e-mail that handles OSC sequences, so they’re handled right away.  I’m happy to do it that way, if it’s more convenient.
> > 
> > Thanks, see some minor comments below.
> > 
> > Jared, any further comments about the issue or the patch?
> > 
> >> * lisp/term.el (term-emulate-terminal) handle OSC sequences
> > 
> > The log message should mention all the functions/variables where you
> > make changes, with a short description of each change.  Please also
> > mention the bug number.
> 
> I updated the description, trying to follow the format of older commits.
> 
> > 
> >> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
> >> +                               ("2" . ansi-osc-window-title-handler)
> >> +                               ("7" . ansi-osc-directory-tracker)
> >> +                               ("8" . ansi-osc-hyperlink-handler))
> >> +  "OSC sequence handler function alist.
> >> +
> >> +OSC (Operating System Command) is a category of ANSI escape sequence
> >> +used in terminal application to introduce custom commands. Terminals
> >> +ignore unknown OSC sequences by default. Handlers can be registered here
> >> +to add support for new OSC sequences to `term'.
> >> +
> >> +Functions in this alist are passed matching valid OSC sequences as
> >> +they're sent to the terminal.
> >> +
> >> +Valid OSC sequences are of the form
> >> +  ESC ] code ; text BEL
> >> +  ESC ] code ; text ESC \
> >> +
> >> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
> >> +appears before the semicolon.
> >> +
> >> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
> >> +active and its point and state active at the time the OSC sequence
> >> +appeared in the stream."
> >> +  :type '(alist :key-type string :value-type function)
> >> +  :group 'term)
> > 
> > New defcustoms should have a :version tag.
> 
> I added one. I guess we’re at 31.1, now. 
> 
> > 
> >> +(defconst term-osc--max-bytes (* 32 1024 1024)
> >> +  "Limit the length of OSC sequences to keep in memory.")
> > 
> > This begs the question: what happens with longer sequences?  Can the
> > doc string answer that question?
> 
> emulate-terminal goes back into its normal state, because presumably the end tag was lost.
> I tried to describe that.  
> 
> > 
> >> +                              (with-demoted-errors "term OSC error: %S"
> >> +                                (funcall
> >> +                                 func code
> >> +                                 (decode-coding-string
> >> +                                  (substring seq-str text-start end-mark)
> >> +                                  locale-coding-system t)))))
> > 
> > Why locale-coding-system?  Wouldn't the result of calling
> > process-coding-system with the process running in term.el be a better
> > candidate?
> 
> process-coding-system is just 'binary because emulate-terminal want to process the escape sequences before the string is decoded. Using locale-coding-system is the way strings are decoded everywhere in emulate-terminal.
> 
> > 
> > Last, but not least: you don't seem to have a copyright assignment on
> > file, without which we cannot accept such a large contribution.  Would
> > you like to start the paperwork of copyright assignment at this time?
> > If yes, I will send you the form to fill and the instructions.
> 
> Yes, I still need to do that. I sent out a request to assign <at> gnu.org <mailto:assign <at> gnu.org> following the instructions in etc/copyright-assign.txt but have yet to hear back from them. 

Message #35 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Stéphane Zermatten <szermatt <at> gmail.com>, per <at> bothner.com,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Sat, 31 May 2025 12:42:27 -0700

Sorry for being quiet. I was on work travel and am back now. I will take 
a look!

  -- MJF

On 2025-05-31 02:18, Eli Zaretskii wrote:
> Ping!  We are waiting for the copyright assignment, but meanwhile,
> Jared, could you please review the patch and comment?
> 
>> From: Stéphane Zermatten <szermatt <at> gmail.com>
>> Date: Tue, 13 May 2025 17:24:29 +0300
>> Cc: Jared Finder <jared <at> finder.org>,
>>  Per Bothner <per <at> bothner.com>,
>>  78263 <at> debbugs.gnu.org
>> 
>> Thank you for the review! I addressed the points you mentioned. See 
>> the updated patch attached to this e-mail.
>> 
>> > On 8 May 2025, at 13:03, Eli Zaretskii <eliz <at> gnu.org> wrote:
>> >
>> >> From: Stéphane Zermatten <szermatt <at> gmail.com>
>> >> Date: Mon, 5 May 2025 19:57:45 +0300
>> >> Cc: 78263 <at> debbugs.gnu.org,
>> >> Dan Nicolaescu <dann <at> ics.uci.edu>,
>> >> Eli Zaretskii <eliz <at> gnu.org>
>> >>
>> >> I attached a merged version of all 3 patches to this e-mail that handles OSC sequences, so they’re handled right away.  I’m happy to do it that way, if it’s more convenient.
>> >
>> > Thanks, see some minor comments below.
>> >
>> > Jared, any further comments about the issue or the patch?
>> >
>> >> * lisp/term.el (term-emulate-terminal) handle OSC sequences
>> >
>> > The log message should mention all the functions/variables where you
>> > make changes, with a short description of each change.  Please also
>> > mention the bug number.
>> 
>> I updated the description, trying to follow the format of older 
>> commits.
>> 
>> >
>> >> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
>> >> +                               ("2" . ansi-osc-window-title-handler)
>> >> +                               ("7" . ansi-osc-directory-tracker)
>> >> +                               ("8" . ansi-osc-hyperlink-handler))
>> >> +  "OSC sequence handler function alist.
>> >> +
>> >> +OSC (Operating System Command) is a category of ANSI escape sequence
>> >> +used in terminal application to introduce custom commands. Terminals
>> >> +ignore unknown OSC sequences by default. Handlers can be registered here
>> >> +to add support for new OSC sequences to `term'.
>> >> +
>> >> +Functions in this alist are passed matching valid OSC sequences as
>> >> +they're sent to the terminal.
>> >> +
>> >> +Valid OSC sequences are of the form
>> >> +  ESC ] code ; text BEL
>> >> +  ESC ] code ; text ESC \
>> >> +
>> >> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
>> >> +appears before the semicolon.
>> >> +
>> >> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
>> >> +active and its point and state active at the time the OSC sequence
>> >> +appeared in the stream."
>> >> +  :type '(alist :key-type string :value-type function)
>> >> +  :group 'term)
>> >
>> > New defcustoms should have a :version tag.
>> 
>> I added one. I guess we’re at 31.1, now.
>> 
>> >
>> >> +(defconst term-osc--max-bytes (* 32 1024 1024)
>> >> +  "Limit the length of OSC sequences to keep in memory.")
>> >
>> > This begs the question: what happens with longer sequences?  Can the
>> > doc string answer that question?
>> 
>> emulate-terminal goes back into its normal state, because presumably 
>> the end tag was lost.
>> I tried to describe that.
>> 
>> >
>> >> +                              (with-demoted-errors "term OSC error: %S"
>> >> +                                (funcall
>> >> +                                 func code
>> >> +                                 (decode-coding-string
>> >> +                                  (substring seq-str text-start end-mark)
>> >> +                                  locale-coding-system t)))))
>> >
>> > Why locale-coding-system?  Wouldn't the result of calling
>> > process-coding-system with the process running in term.el be a better
>> > candidate?
>> 
>> process-coding-system is just 'binary because emulate-terminal want to 
>> process the escape sequences before the string is decoded. Using 
>> locale-coding-system is the way strings are decoded everywhere in 
>> emulate-terminal.
>> 
>> >
>> > Last, but not least: you don't seem to have a copyright assignment on
>> > file, without which we cannot accept such a large contribution.  Would
>> > you like to start the paperwork of copyright assignment at this time?
>> > If yes, I will send you the form to fill and the instructions.
>> 
>> Yes, I still need to do that. I sent out a request to assign <at> gnu.org 
>> <mailto:assign <at> gnu.org> following the instructions in 
>> etc/copyright-assign.txt but have yet to hear back from them.

Message #38 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Stéphane Zermatten <szermatt <at> gmail.com>, per <at> bothner.com,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Sun, 01 Jun 2025 22:21:15 -0700

Only minor comments, otherwise the patch generally looks good.

> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
> +                               ("2" . ansi-osc-window-title-handler)
> +                               ("7" . ansi-osc-directory-tracker)
> +                               ("8" . ansi-osc-hyperlink-handler))

The variable ansi-osc-handlers in ansi-osc.el doesn't handle OSC 0 (set 
icon and title), but here it's handled.  I would think the two should be 
identical.  Also what's the rationale for duplicating ansi-osc-handlers 
instead of just adding a new custom variable that tells term to look at 
asci-osc-handlers?

> +(ert-deftest term-ignore-osc()

Style nit: there should be a space between the symbol term-ignore-osc 
and the ().  This should look like a defun of no arguments.


Apologies for taking so long to get to this.

  -- MJF

Message #41 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Jared Finder <jared <at> finder.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Per Bothner <per <at> bothner.com>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Mon, 2 Jun 2025 20:01:32 +0300

[Message part 1 (text/plain, inline)]

I updated the change, see the new version attached.

No news from the copyright assignment front yet.

> On 2 Jun 2025, at 08:21, Jared Finder <jared <at> finder.org> wrote:
> 
> Only minor comments, otherwise the patch generally looks good.
> 
>> +(defcustom term-osc-handlers '(("0" . ansi-osc-window-title-handler)
>> +                               ("2" . ansi-osc-window-title-handler)
>> +                               ("7" . ansi-osc-directory-tracker)
>> +                               ("8" . ansi-osc-hyperlink-handler))
> 
> The variable ansi-osc-handlers in ansi-osc.el doesn't handle OSC 0 (set icon and title), but here it's handled.  I

OSC 0 is the code I saw most commonly used in the wild and it works well with ansi-osc-window-title-handler, so I propose, in the new version of this patch, to add it to ansi-osc-handlers (which term now calls). 

Should this particular change be in another patch instead, with another set of reviewers? 

> would think the two should be identical.  Also what's the rationale for duplicating ansi-osc-handlers instead of just adding a new custom variable that tells term to look at asci-osc-handlers?

OSC handlers run in different environments when called by term than when called by ansi-osc. ansi-osc is typically called on a compilation buffer, during or even after compilation. 

That said, it’s true that, in many cases, OSC handlers, if they’re written correctly and especially if they’re just cosmetic, will work in both environments, so it might be convenient to be able to configure them once for both environments, as long as we can configure them differently if needed.

It’s not a straightforward situation. I thought that a solution with two separate configurations variable would be simpler and safer. 

I updated the patch to allow delegating to ansi-osc-handlers, but still allow overriding if necessary.
I find this delegation part a bit complicated. 

The root of the problem is that we can’t assume that handlers written for one environment work in the other environment, or even that they make sense.

When called by term:
 - they are run at the time they appear in the stream, at the position the cursor is in (not necessarily at the end of the text)
 - they have access to a process they can communicate with

When called by ansi-osc:
 - they are run at the position they appear in a buffer containing a mix of text and uninterpreted terminal sequences followig it 
 - they cannot assume that there is a process to communicate with

The first point means that a handler written for ansi-osc might very well be confused when run in a terminal, depending on what they need to do. The current handlers happen to all work in both environment, but that’s not a given. 

The second point means that a handler written for term might also be confused not having access to a real terminal and process to communicate with.  

For example, a handler for OSC 4;c;spec written for the compilation buffer mightimplement "Change Color Number c to the color specified by spec” but it would not implement "If a "?" is given rather than a name or RGB specification, xterm replies with a control sequence of the same form which can be used to set the corresponding color.” while a handler for term could implement both. 
(quotes from https://www.xfree86.org/current/ctlseqs.html)


> 
>> +(ert-deftest term-ignore-osc()
> 
> Style nit: there should be a space between the symbol term-ignore-osc and the ().  This should look like a defun of no arguments.

I fixed this.



[Message part 2 (text/html, inline)]

[0001-Handle-OSC-sequences-in-term-bug-78263.patch (application/octet-stream, attachment)]

[Message part 4 (text/html, inline)]

Message #44 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stéphane Zermatten <szermatt <at> gmail.com>
Cc: per <at> bothner.com, jared <at> finder.org, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Tue, 03 Jun 2025 14:09:17 +0300

> From: Stéphane Zermatten <szermatt <at> gmail.com>
> Date: Mon, 2 Jun 2025 20:01:32 +0300
> Cc: Eli Zaretskii <eliz <at> gnu.org>,
>  Per Bothner <per <at> bothner.com>,
>  78263 <at> debbugs.gnu.org
> 
> I updated the change, see the new version attached.

I have a few minor comments below.

> No news from the copyright assignment front yet.

It will take around two weeks.

> Handlers can be registered to term-osc-handler, which by defaults,
> supports OSC 0, 2, 7 and 8, using handlers defined in ansi-osc. Unknown
> OSC sequences are properly ignored.
> 
> * lisp/term.el:
>   (term-emulate-terminal): handle OSC sequences
>   term-osc-handler: new custom option
>   term-control-seq-regexp: extended regexp
>   term--osc-max-byte: new private constant
> * lisp/ansi-osc.el:
>   ansi-osc-handler: handle OSC 0 (like OSC 2)

Please reformat the log message according to our conventions: function
and variable names in parentheses, 2 spaces between sentences, etc.
For the details see CONTRIBUTE and also consult the many examples of
log messages in Git.

> +(defcustom term-osc-handlers nil
> +  "Terminal-specific OSC sequence handler function alist.
> +
> +OSC (Operating System Command) is a category of ANSI escape sequence
> +used in terminal application to introduce custom commands. Terminals
> +ignore unknown OSC sequences by default. Handlers can be registered here
> +to add support for new OSC sequences to `term'.

Two spaces between sentences, here and elsewhere.

> +Functions in this alist are passed matching valid OSC sequences as
> +they're sent to the terminal.
> +
> +Valid OSC sequences are of the form
> +  ESC ] code ; text BEL
> +  ESC ] code ; text ESC \
> +
> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
> +appears before the semicolon.
> +
> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
> +active and its point and state active at the time the OSC sequence
> +appeared in the stream.

This should clarify what is TEXT in this context.

Thanks.

Message #47 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Per Bothner <per <at> bothner.com>, Jared Finder <jared <at> finder.org>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Wed, 4 Jun 2025 21:04:58 +0300

[0001-Handle-OSC-sequences-in-term-bug-78263.patch (application/octet-stream, attachment)]

[Message part 2 (text/plain, inline)]

> On 3 Jun 2025, at 14:09, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
>> From: Stéphane Zermatten <szermatt <at> gmail.com>
>> Date: Mon, 2 Jun 2025 20:01:32 +0300
>> Cc: Eli Zaretskii <eliz <at> gnu.org>,
>> Per Bothner <per <at> bothner.com>,
>> 78263 <at> debbugs.gnu.org
>> 
>> I updated the change, see the new version attached.
> 
> I have a few minor comments below.
> 
>> No news from the copyright assignment front yet.
> 
> It will take around two weeks.
> 
>> Handlers can be registered to term-osc-handler, which by defaults,
>> supports OSC 0, 2, 7 and 8, using handlers defined in ansi-osc. Unknown
>> OSC sequences are properly ignored.
>> 
>> * lisp/term.el:
>>  (term-emulate-terminal): handle OSC sequences
>>  term-osc-handler: new custom option
>>  term-control-seq-regexp: extended regexp
>>  term--osc-max-byte: new private constant
>> * lisp/ansi-osc.el:
>>  ansi-osc-handler: handle OSC 0 (like OSC 2)
> 
> Please reformat the log message according to our conventions: function
> and variable names in parentheses, 2 spaces between sentences, etc.
> For the details see CONTRIBUTE and also consult the many examples of
> log messages in Git.

Done (Correctly, this time, hopefully)

> 
>> +(defcustom term-osc-handlers nil
>> +  "Terminal-specific OSC sequence handler function alist.
>> +
>> +OSC (Operating System Command) is a category of ANSI escape sequence
>> +used in terminal application to introduce custom commands. Terminals
>> +ignore unknown OSC sequences by default. Handlers can be registered here
>> +to add support for new OSC sequences to `term'.
> 
> Two spaces between sentences, here and elsewhere.

Done 

> 
>> +Functions in this alist are passed matching valid OSC sequences as
>> +they're sent to the terminal.
>> +
>> +Valid OSC sequences are of the form
>> +  ESC ] code ; text BEL
>> +  ESC ] code ; text ESC \
>> +
>> +Each entry has the form (CODE . FUNCTION), where CODE is the string that
>> +appears before the semicolon.
>> +
>> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
>> +active and its point and state active at the time the OSC sequence
>> +appeared in the stream.
> 
> This should clarify what is TEXT in this context.

A rephrased it a bit. Hopefully it is clearer.

> 
> Thanks.

Message #50 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stéphane Zermatten <szermatt <at> gmail.com>
Cc: per <at> bothner.com, jared <at> finder.org, 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Thu, 05 Jun 2025 08:29:25 +0300

> From: Stéphane Zermatten <szermatt <at> gmail.com>
> Date: Wed, 4 Jun 2025 21:04:58 +0300
> Cc: Jared Finder <jared <at> finder.org>,
>  Per Bothner <per <at> bothner.com>,
>  78263 <at> debbugs.gnu.org
> 
> > Please reformat the log message according to our conventions: function
> > and variable names in parentheses, 2 spaces between sentences, etc.
> > For the details see CONTRIBUTE and also consult the many examples of
> > log messages in Git.
> 
> Done (Correctly, this time, hopefully)

Yes, thanks.

> >> +(defcustom term-osc-handlers nil
> >> +  "Terminal-specific OSC sequence handler function alist.
> >> +
> >> +OSC (Operating System Command) is a category of ANSI escape sequence
> >> +used in terminal application to introduce custom commands. Terminals
> >> +ignore unknown OSC sequences by default. Handlers can be registered here
> >> +to add support for new OSC sequences to `term'.
> > 
> > Two spaces between sentences, here and elsewhere.
> 
> Done 

Not in all the cases.  Here's one:

> +(defcustom term-osc-handlers nil
> +  "Terminal-specific OSC sequence handler function alist.
> +
> +OSC (Operating System Command) is a category of ANSI escape sequence
> +used in terminal application to introduce custom commands. Terminals
                                                            ^^
Here's another:

> +   ;; OSC (see [ECMA-48] section 8.3.89 "Operation System Command") The
                                                                     ^^^
Here, a period is missing as well.

> >> +FUNCTION is called with two arguments CODE and TEXT with the term buffer
> >> +active and its point and state active at the time the OSC sequence
> >> +appeared in the stream.
> > 
> > This should clarify what is TEXT in this context.
> 
> A rephrased it a bit. Hopefully it is clearer.

It is, thanks.

Please fix those nits, and then we should wait for your
copyright-assignment paperwork to be completed, and for Jared to
respond to your previous message.

Message #53 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Stéphane Zermatten <szermatt <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Per Bothner <per <at> bothner.com>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Wed, 04 Jun 2025 22:44:20 -0700

On 2025-06-04 11:04, Stéphane Zermatten wrote:
>> On 3 Jun 2025, at 14:09, Eli Zaretskii <eliz <at> gnu.org> wrote:
>> 
>>> From: Stéphane Zermatten <szermatt <at> gmail.com>
>>> Date: Mon, 2 Jun 2025 20:01:32 +0300
>>> Cc: Eli Zaretskii <eliz <at> gnu.org>,
>>> Per Bothner <per <at> bothner.com>,
>>> 78263 <at> debbugs.gnu.org
>>> 
>>> I updated the change, see the new version attached.

One additional comment -- why is term-osc-handlers a defcustom at all?  
It seems to me like a variable that is used just to implement the OSC 
handlers and not something I'd expect users to customize.  Looking at 
ansi-osc-handlers, it's just a defvar.  I can't think of a case where I 
would expect a user to add or remove a custom handler.  I do think users 
may want to enable or disable certain features, but that would affect 
multiple handlers like if a user wanted to not allow changing dynamic 
colors (OSC 10 to 19).  That seems better served with separate defcustom 
variables for each feature, like a term-allow-color-ops defcustom.

I'm otherwise ok with the patch.

  -- MJF

Message #56 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Jared Finder <jared <at> finder.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Stéphane Zermatten <szermatt <at> gmail.com>, per <at> bothner.com,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Wed, 04 Jun 2025 22:49:07 -0700

On 2025-06-04 22:29, Eli Zaretskii wrote:
> Please fix those nits, and then we should wait for your
> copyright-assignment paperwork to be completed, and for Jared to
> respond to your previous message.

I think Stéphane's latest patch addressed my question about using 
ansi-osc-handlers in term.  I saw this new code in it:

+                            (when-let* ((func (cdr (or (assoc-string
+                                                        code 
term-osc-handlers)
+                                                       (assoc-string
+                                                        code 
ansi-osc-handlers)))))

Which looks right to me.  When we want to have different handlers for 
term vs *compilation* buffers, we can just add a different handler to 
each variable.

  -- MJF

Message #59 received at 78263 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Zermatten <szermatt <at> gmail.com>
To: Jared Finder <jared <at> finder.org>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Per Bothner <per <at> bothner.com>,
 78263 <at> debbugs.gnu.org
Subject: Re: bug#78263: [PATCH 1/3] Ignore OSC sequences in term instead of
 printing them
Date: Fri, 6 Jun 2025 11:28:33 +0300

[Message part 1 (text/plain, inline)]

I just got confirmation that the copyright assignent paperwork is complete.

Here’s an updated version of the patch, with: 
 - 2 sentence formatting fixes
 - term-osc-handlers just a defvar instead of a defcustom

[0001-Handle-OSC-sequences-in-term-bug-78263.patch (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

> On 5 Jun 2025, at 08:49, Jared Finder <jared <at> finder.org> wrote:
> 
> On 2025-06-04 22:29, Eli Zaretskii wrote:
>> Please fix those nits, and then we should wait for your
>> copyright-assignment paperwork to be completed, and for Jared to
>> respond to your previous message.
> 
> I think Stéphane's latest patch addressed my question about using ansi-osc-handlers in term.  I saw this new code in it:
> 
> +                            (when-let* ((func (cdr (or (assoc-string
> +                                                        code term-osc-handlers)
> +                                                       (assoc-string
> +                                                        code ansi-osc-handlers)))))
> 
> Which looks right to me.  When we want to have different handlers for term vs *compilation* buffers, we can just add a different handler to each variable.
> 
>  -- MJF

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.