GNU bug report logs - #26217
25.2; shell syntax does not know for i do

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26217 in the body.
You can then email your comments to 26217 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Martin Vath <martin <at> mvath.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.2; shell syntax does not know for i do
Date: Wed, 22 Mar 2017 10:10:44 +0100 (CET)

1. M-x shell-script-mode
2. Type: "
for i do echo 1; done
for i; do echo 1; done
"

Despite both lines are correct full loops according to POSIX
and in fact understood by all current shells (see below),
this does not correspond to the indentation of emacs and
coloring of "do" in the first line.

That the first line is indeed valid by POSIX and understood
by the shells was discussed recently in the German Usegroup
de.comp.os.unix.shell (OT under "Funktion aus find aufrufen"),
but you can also check the grammer for "for_clause" in
http://pubs.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html
noting that it is valid that "linebreak" expands to the empty
string. Another hint that this is intentional in POSIX is the
explicit mentioning of "do" in Section 2.4 of the above page
as the _third_ word in a for command.

Tested on GNU Emacs 25.2.1 and GNU Emacs 24.4.1

(Removing further build data, since the issue was reproduced
on different machines and distributions before reporting.)

Message #20 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Mauro Aranda <maurooaranda <at> gmail.com>
To: 26217 <at> debbugs.gnu.org
Cc: Martin Vath <martin <at> mvath.de>, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#2910: 23.0.60; Shell-script coloring bug
Date: Fri, 13 Oct 2023 09:07:50 -0300

Martin Vath <martin <at> mvath.de> writes:

> 1. M-x shell-script-mode
> 2. Type: "
> for i do echo 1; done
> for i; do echo 1; done
> "
>
> Despite both lines are correct full loops according to POSIX
> and in fact understood by all current shells (see below),
> this does not correspond to the indentation of emacs and
> coloring of "do" in the first line.

This Bug Report (and the other ones merged with it) have two parts:
- No highlight for "do" in:
for i do echo 1; done

This works if using bash-ts-mode, which claims in its docstring it can
support Bash or sh.

I don't know what's the decision (if there is one) about situations like
these.  If the bugs in lang-mode are fixed by lang-ts-mode, will these
bug reports be treated as: wontfix? fixed? open in case someone wants to
spend time in lang-mode?

- Wrong indentation for lines after:
for i do echo 1; done

I took a look at this, and ISTM that giving "do" a special treatment
like sh-smie--sh-keyword-p gives to "in" might fix this, perhaps by
reusing sh-smie--sh-keyword-in-p.

I know very little about SMIE, so I'm CCing Stefan M.

Message #23 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Mauro Aranda <maurooaranda <at> gmail.com>
Cc: Martin Vath <martin <at> mvath.de>, 26217 <at> debbugs.gnu.org
Subject: Re: bug#2910: 23.0.60; Shell-script coloring bug
Date: Fri, 13 Oct 2023 12:06:57 -0400

> I don't know what's the decision (if there is one) about situations like
> these.  If the bugs in lang-mode are fixed by lang-ts-mode, will these
> bug reports be treated as: wontfix? fixed? open in case someone wants to
> spend time in lang-mode?

AFAIK we don't consider the `foo-ts-mode` to obsolete the other modes.
Maybe we will, but we don't yet.  IMO I think we'd first need to have
a good long-term strategy about what we'll do when tree-sitter becomes
unmaintained/obsolete.  IOW I think we need to develop our own layer of
abstraction above tree-sitter so that we can accommodate other
parser backends.

FWIW, it's not clear at all what such a layer would look like, so we're
pretty far from it.  I'd welcome people start thinking about it, maybe
by looking at existing alternatives like our own `wisi` (in GNU ELPA),
SMIE, maybe LSP (assuming there are servers out there which can provide
that kind of functionality), etc...

> - Wrong indentation for lines after:
> for i do echo 1; done
>
> I took a look at this, and ISTM that giving "do" a special treatment
> like sh-smie--sh-keyword-p gives to "in" might fix this, perhaps by
> reusing sh-smie--sh-keyword-in-p.

Sounds about right.


        Stefan

Message #26 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Mauro Aranda <maurooaranda <at> gmail.com>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Martin Vath <martin <at> mvath.de>, 26217 <at> debbugs.gnu.org
Subject: Re: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 09:44:37 -0300

[Message part 1 (text/plain, inline)]

tags 26217 patch
quit


On 13/10/23 13:06, Stefan Monnier wrote:
>> I don't know what's the decision (if there is one) about situations like
>> these.  If the bugs in lang-mode are fixed by lang-ts-mode, will these
>> bug reports be treated as: wontfix? fixed? open in case someone wants to
>> spend time in lang-mode?
>
> AFAIK we don't consider the `foo-ts-mode` to obsolete the other modes.
> Maybe we will, but we don't yet.  IMO I think we'd first need to have
> a good long-term strategy about what we'll do when tree-sitter becomes
> unmaintained/obsolete.  IOW I think we need to develop our own layer of
> abstraction above tree-sitter so that we can accommodate other
> parser backends.
>
> FWIW, it's not clear at all what such a layer would look like, so we're
> pretty far from it.  I'd welcome people start thinking about it, maybe
> by looking at existing alternatives like our own `wisi` (in GNU ELPA),
> SMIE, maybe LSP (assuming there are servers out there which can provide
> that kind of functionality), etc...

Thank you for your response.  I don't know if this has been raised in
emacs-devel, but IMO it should be.

>> - Wrong indentation for lines after:
>> for i do echo 1; done
>>
>> I took a look at this, and ISTM that giving "do" a special treatment
>> like sh-smie--sh-keyword-p gives to "in" might fix this, perhaps by
>> reusing sh-smie--sh-keyword-in-p.
>
> Sounds about right.

I attach a patch that should also handle the fontification issue. It
also comes with some tests, and I've did some manual testing on my own.
I haven't found problems, but I'm suspicious because it seems too easy.

[0001-Fix-indentation-and-fontification-in-shell-script-Bu.patch (text/x-patch, attachment)]

Message #31 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Mauro Aranda <maurooaranda <at> gmail.com>
Cc: Martin Vath <martin <at> mvath.de>, 26217 <at> debbugs.gnu.org
Subject: Re: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 10:58:40 -0400

> I haven't found problems, but I'm suspicious because it seems too easy.

Welcome to the world of Emacs, where "if it's easy it must be right".


        Stefan

Message #36 received at 26217-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Mauro Aranda <maurooaranda <at> gmail.com>
Cc: Martin Vath <martin <at> mvath.de>, 2910-done <at> debbugs.gnu.org,
 26217-done <at> debbugs.gnu.org
Subject: Re: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 11:01:40 -0400

>> I haven't found problems, but I'm suspicious because it seems too easy.
> Welcome to the world of Emacs, where "if it's easy it must be right".

BTW, pushed to `master`, thank you.


        Stefan

Message #64 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Mauro Aranda <maurooaranda <at> gmail.com>
Cc: Martin Vath <martin <at> mvath.de>, 26217 <at> debbugs.gnu.org
Subject: Re: bug#26217: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 19:43:00 +0300

On 13/10/2023 19:06, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> I don't know what's the decision (if there is one) about situations like
>> these.  If the bugs in lang-mode are fixed by lang-ts-mode, will these
>> bug reports be treated as: wontfix? fixed? open in case someone wants to
>> spend time in lang-mode?
> 
> AFAIK we don't consider the `foo-ts-mode` to obsolete the other modes.
> Maybe we will, but we don't yet.  IMO I think we'd first need to have
> a good long-term strategy about what we'll do when tree-sitter becomes
> unmaintained/obsolete.  IOW I think we need to develop our own layer of
> abstraction above tree-sitter so that we can accommodate other
> parser backends.
> 
> FWIW, it's not clear at all what such a layer would look like, so we're
> pretty far from it.  I'd welcome people start thinking about it, maybe
> by looking at existing alternatives like our own `wisi` (in GNU ELPA),
> SMIE, maybe LSP (assuming there are servers out there which can provide
> that kind of functionality), etc...

I don't know how feasible that would be, given that the ts major modes 
we write have to reference fairly low level concerns (such as node 
names, different across all grammars).

Maybe porting Lezer (https://lezer.codemirror.net/) could become a 
replacement in such a scenario, but then we're back to maintaining our 
own grammars again, and with lower performance by an order of a magnitude.

Message #67 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: Martin Vath <martin <at> mvath.de>, Mauro Aranda <maurooaranda <at> gmail.com>,
 26217 <at> debbugs.gnu.org
Subject: Re: bug#26217: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 15:07:01 -0400

>> FWIW, it's not clear at all what such a layer would look like, so we're
>> pretty far from it.  I'd welcome people start thinking about it, maybe
>> by looking at existing alternatives like our own `wisi` (in GNU ELPA),
>> SMIE, maybe LSP (assuming there are servers out there which can provide
>> that kind of functionality), etc...
> I don't know how feasible that would be,

That's the wrong way to look at it.  There is no doubt that it
is feasible.  What is under question is what it would take.

> given that the ts major modes we write have to reference fairly low
> level concerns (such as node names, different across all grammars).

That just means that a given set of highlighting/indentation rules would
not necessarily work with all possible parser-backends.  But maybe we
could bridge the gap by allowing some intermediate layer that could do
things like translate node names (could be useful even within the
tree-sitter context to deal with evolving grammars).

I'm not saying this is the way to go, mind you.  I don't know how it
could/should work.

But I do think we could do worse than start thinking about it, because
tree-sitter is the kind of technology that's on the "treadmill", whereas
Emacs' evolution has a different pace: how well will Emacs-29's TS modes
work with 2028's tree-sitter grammars?

> Maybe porting Lezer (https://lezer.codemirror.net/) could become

Interesting, thanks.

> a replacement in such a scenario, but then we're back to maintaining
> our own grammars again, and with lower performance by an order of
> a magnitude.

We could look into Lezer support just to help us guide the design of an
intermediate layer API.  No need to maintain lots of our own grammars or
take the performance impact :-)


        Stefan

Message #70 received at 26217 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Martin Vath <martin <at> mvath.de>, Mauro Aranda <maurooaranda <at> gmail.com>,
 26217 <at> debbugs.gnu.org
Subject: Re: bug#26217: bug#2910: 23.0.60; Shell-script coloring bug
Date: Sat, 14 Oct 2023 22:49:18 +0300

On 14/10/2023 22:07, Stefan Monnier wrote:
>>> FWIW, it's not clear at all what such a layer would look like, so we're
>>> pretty far from it.  I'd welcome people start thinking about it, maybe
>>> by looking at existing alternatives like our own `wisi` (in GNU ELPA),
>>> SMIE, maybe LSP (assuming there are servers out there which can provide
>>> that kind of functionality), etc...
>> I don't know how feasible that would be,
> 
> That's the wrong way to look at it.  There is no doubt that it
> is feasible.  What is under question is what it would take.

And whether it would be in any way more economical than just waiting for 
the next comparable solution to come along and writing either 
compatibility shims inside each ts mode, or copying code and adapting 
into the next set of *-ts<N>-mode modes.

So for now we could be better off just coming up with an easier way to 
migrate user configurations across major modes for a given language than 
what we have now.

>> given that the ts major modes we write have to reference fairly low
>> level concerns (such as node names, different across all grammars).
> 
> That just means that a given set of highlighting/indentation rules would
> not necessarily work with all possible parser-backends.  But maybe we
> could bridge the gap by allowing some intermediate layer that could do
> things like translate node names (could be useful even within the
> tree-sitter context to deal with evolving grammars).

That's not impossible, but IME the sets of rules are fairly uniformly 
lower-level.

> I'm not saying this is the way to go, mind you.  I don't know how it
> could/should work.
> 
> But I do think we could do worse than start thinking about it, because
> tree-sitter is the kind of technology that's on the "treadmill", whereas
> Emacs' evolution has a different pace: how well will Emacs-29's TS modes
> work with 2028's tree-sitter grammars?

1. That's why I'm sure we'll start using some grammar pinning - either 
to the commit hash, or to the version range.

2. Overall it's a solid point: either tree-sitter "stabilizes" after a 
while, or goes off the map.

>> Maybe porting Lezer (https://lezer.codemirror.net/) could become
> 
> Interesting, thanks.
> 
>> a replacement in such a scenario, but then we're back to maintaining
>> our own grammars again, and with lower performance by an order of
>> a magnitude.
> 
> We could look into Lezer support just to help us guide the design of an
> intermediate layer API.  No need to maintain lots of our own grammars or
> take the performance impact :-)

It's as good an approach as any, though we have no guarantee that the 
"next tree-sitter" will have a similar enough shape to either of TS or 
Lezer, for an easy future migration.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.