GNU bug report logs - #76791
31.0.50; forward-list doesn't work in bash-ts-mode

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 76791 in the body.
You can then email your comments to 76791 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: the_wurfkreuz <the_wurfkreuz <at> proton.me>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: 31.0.50; forward-list doesn't work in bash-ts-mode
Date: Thu, 06 Mar 2025 21:03:15 +0000

[Message part 1 (text/plain, inline)]

I stumbled upon a situation where the 'forward-list' doesn't work, butusing `backward-list` works on the same bracket pair.

Steps to reproduce:

1. emacs -Q
2. Remove everything from the scratch buffer and paste:

#!/bin/sh

check_dirs_file() {
if [ "$(wc -l "$dirs_file")" -gt 10 ]; then
tmp_dirs_file="$(mktemp)"
sed '1d' "$dirs_file" > "$tmp_dirs_file"
cp "$tmp_dirs_file" "$dirs_file"
fi
}

3. M-x bash-ts-mode
4. Move the cursor to the opening bracket of '$(mktemp)' and execute
'forward-list'. It gives "No next group" message.

At the same time moving the cursor to the closing bracket and using
'backward-list' will work. Also, if i enable bash-mode instead of the
treesitter one, then it will work for the opening bracket too.

In GNU Emacs 31.0.50 (build 5, x86_64-pc-linux-gnu, GTK+ Version
3.24.43, cairo version 1.18.0) of 2025-03-06 built on ub-void
Repository revision: 9faa8d32a3db893db01debaaaa1db79ec1c1b33e
Repository branch: master
System Description: Void Linux

Configured using:
'configure --with-native-compilation=aot --with-tree-sitter --with-gif
--with-png --with-jpeg --with-rsvg --with-tiff --with-imagemagick
--with-pgtk --with-mailutils'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ IMAGEMAGICK JPEG
LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP
XIM GTK3 ZLIB

Important settings:
value of $LC_COLLATE: C
value of $LANG: en_US.UTF-8
locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
windmove-mode: t
savehist-mode: t
meow-magit-mode: t
meow-global-mode: t
meow-mode: t
meow-esc-mode: t
envrc-global-mode: t
envrc-mode: t
buffer-terminator-mode: t
global-git-commit-mode: t
magit-auto-revert-mode: t
marginalia-mode: t
vertico-mode: t
corfu-popupinfo-mode: t
corfu-history-mode: t
corfu-echo-mode: t
global-corfu-mode: t
corfu-mode: t
which-key-mode: t
undo-fu-session-global-mode: t
undo-fu-session-mode: t
global-treesit-auto-mode: t
global-auto-revert-mode: t
save-place-mode: t
electric-pair-mode: t
recentf-mode: t
global-display-line-numbers-mode: t
display-line-numbers-mode: t
override-global-mode: t
tooltip-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tab-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
minibuffer-regexp-mode: t
buffer-read-only: t
line-number-mode: t
indent-tabs-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t

Load-path shadows:
/home/wurfkreuz/.emacs.d/elpa/transient-20250108.1351/transient hides /usr/local/share/emacs/31.0.50/lisp/transient

Features:
(shadow sort mail-extr emacsbug vertico-directory wdired face-remap
cus-start shortdoc find-dired help-fns radix-tree tramp-cmds windmove
em-unix em-term term disp-table ehelp em-script em-prompt em-pred em-ls
em-hist em-glob em-extpipe em-cmpl em-dirs em-basic em-banner em-alias
em-elecslash esh-mode esh-var org-appear image-file image-converter
oc-basic ol-eww eww vtable mule-util url-queue mm-url ol-rmail ol-mhe
ol-irc ol-info ol-gnus nnselect gnus-art mm-uu mml2015 mm-view mml-smime
smime gnutls dig gnus-sum shr pixel-fill kinsoku url-file svg dom
gnus-group gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud nnimap
nnmail mail-source utf7 nnoo gnus-spec gnus-int gnus-range gnus-win gnus
nnheader range ol-docview doc-view image-mode exif ol-bibtex bibtex
ol-bbdb ol-w3m ol-doi org-link-doi dired-aux whitespace bug-reference
raku-detect python vc-git vc-dispatcher jka-compr tramp-cache time-stamp
tramp-sh savehist f eshell em-tramp esh-cmd esh-ext esh-proc esh-opt
esh-io esh-arg esh-module esh-module-loaddefs esh-util meow meow-tutor
meow-cheatsheet meow-cheatsheet-layout meow-core meow-shims meow-esc
meow-command array meow-thing meow-visual meow-keypad meow-beacon
meow-helpers meow-util meow-keymap meow-face meow-var delsel org-tempo
tempo org-drill persist org-agenda embark-org org-element org-persist
org-id org-refile org-element-ast inline avl-tree org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-src ob-comint org-pcomplete
org-list org-footnote org-faces org-entities org-version ob-emacs-lisp
ob-core ob-eval org-cycle org-table ol org-fold org-fold-core org-keys
oc org-loaddefs cal-menu calendar cal-loaddefs org-compat org-macs eglot
external-completion jsonrpc xref ert ewoc debug backtrace kdl-mode
generic markdown-mode color nix-mode nix-repl nix-shell nix-store
nix-log nix-instantiate nix-shebang nix-format nix yaml-mode
dockerfile-mode terraform-mode noutline outline hcl-mode lua-mode
perl-mode flymake envrc inheritenv buffer-terminator transpose-frame
magit-bookmark magit-submodule magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff git-commit log-edit message sendmail yank-media puny rfc822
mml mml-sec epa derived epg rfc6068 epg-config gnus-util mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log magit-core magit-autorevert magit-margin
magit-transient magit-process magit-mode benchmark magit-git magit-base
magit-section cursor-sensor crm smerge-mode diff-mode track-changes diff
hydra lv with-editor server embark-consult embark ffap thingatpt consult
bookmark marginalia vertico project cape corfu-popupinfo corfu-history
corfu-echo corfu orderless tempel-collection tempel wgrep grep compile
text-property-search which-key docker docker-context docker-volume
docker-network docker-image docker-container docker-faces docker-core
docker-compose docker-process docker-utils tablist advice tablist-filter
semantic/wisent/comp semantic/wisent semantic/wisent/wisent
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local find-func cedet dired dired-loaddefs tramp
trampver tramp-integration files-x tramp-message tramp-compat shell
pcomplete comint ansi-osc ring parse-time iso8601 time-date ansi-color
tramp-loaddefs docker-group transient format-spec edmacro kmacro compat
dash aio generator s goto-chg undo-fu-session undo-fu zig-ts-mode
clojure-ts-mode treesit-auto crontab-mode sh-script smie treesit
executable rose-pine-theme cus-edit pp cus-load midnight autorevert
filenotify saveplace desktop frameset elec-pair recentf tree-widget
wid-edit comp comp-cstr warnings comp-run comp-common
display-line-numbers cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
easy-mmode use-package-core finder-inf annalist-autoloads
buffer-terminator-autoloads cape-autoloads clojure-ts-mode-autoloads
corfu-autoloads crontab-mode-autoloads cursor-undo-autoloads
docker-autoloads aio-autoloads dockerfile-mode-autoloads
embark-consult-autoloads consult-autoloads embark-autoloads
envrc-autoloads f-autoloads go-mode-autoloads goto-chg-autoloads
haskell-mode-autoloads hydra-autoloads inheritenv-autoloads
lua-mode-autoloads lv-autoloads magit-autoloads pcase
marginalia-autoloads markdown-mode-autoloads meow-autoloads
nix-mode-autoloads magit-section-autoloads orderless-autoloads
org-appear-autoloads org-drill-autoloads persist-autoloads
queue-autoloads raku-mode-autoloads rx s-autoloads tablist-autoloads
tempel-collection-autoloads tempel-autoloads terraform-mode-autoloads
dash-autoloads hcl-mode-autoloads transient-autoloads
transpose-frame-autoloads treesit-auto-autoloads undo-fu-autoloads
undo-fu-session-autoloads vertico-autoloads wgrep-autoloads info
with-editor-autoloads yaml-mode-autoloads zig-ts-mode-autoloads package
browse-url xdg url url-proxy url-privacy url-expand url-methods
url-history url-cookie generate-lisp-file url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs icons
password-cache json subr-x map byte-opt gv bytecomp byte-compile
url-vars cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/pgtk-win pgtk-win term/common-win touch-screen pgtk-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty move-toolbar make-network-process tty-child-frames
native-compile emacs)

Memory information:
((conses 16 1038121 175970) (symbols 48 53127 3) (strings 32 259945 6376)
(string-bytes 1 8988021) (vectors 16 96301) (vector-slots 8 1284030 76940)
(floats 8 749 262) (intervals 56 32165 4053) (buffers 992 36))

[Message part 2 (text/html, inline)]

Message #8 received at 76791 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: the_wurfkreuz <the_wurfkreuz <at> proton.me>
Cc: 76791 <at> debbugs.gnu.org
Subject: Re: bug#76791: 31.0.50; forward-list doesn't work in bash-ts-mode
Date: Fri, 07 Mar 2025 09:10:13 +0200

> I stumbled upon a situation where the 'forward-list' doesn't work, but
> using `backward-list` works on the same bracket pair.
>
> Steps to reproduce:
>
> 1. emacs -Q
> 2. Remove everything from the scratch buffer and paste:
>
> #!/bin/sh
>
> check_dirs_file() {
>     if [ "$(wc -l "$dirs_file")" -gt 10 ]; then
> tmp_dirs_file="$(mktemp)"
> sed '1d' "$dirs_file" > "$tmp_dirs_file"
> cp "$tmp_dirs_file" "$dirs_file"
>     fi
> }
>
> 3. M-x bash-ts-mode
> 4. Move the cursor to the opening bracket of '$(mktemp)' and execute
> 'forward-list'. It gives "No next group" message.
>
> At the same time moving the cursor to the closing bracket and using
> 'backward-list' will work. Also, if i enable bash-mode instead of the
> treesitter one, then it will work for the opening bracket too.

Thanks for the bug report.  This is the curse of the boundary condition.
For this particular case it can be fixed by the following patch.
But it will break some other cases.  And it's hard as hell to figure out
when to use '<' and when to use '<='.

diff --git a/lisp/treesit.el b/lisp/treesit.el
index c5c08f47dc3..3aa554985e4 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -2961,8 +2961,8 @@ treesit--forward-list-with-default
                              (>= default-pos (treesit-node-end sibling))))
                        (or (null parent)
                            (if (> arg 0)
-                               (< default-pos (treesit-node-end parent))
-                             (> default-pos (treesit-node-start parent)))))
+                               (<= default-pos (treesit-node-end parent))
+                             (>= default-pos (treesit-node-start parent)))))
               (goto-char default-pos))
             (when sibling
               (goto-char (if (> arg 0)

Message #11 received at 76791 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Yuan Fu <casouri <at> gmail.com>
Cc: the_wurfkreuz <the_wurfkreuz <at> proton.me>, 76791 <at> debbugs.gnu.org
Subject: Re: bug#76791: 31.0.50; forward-list doesn't work in bash-ts-mode
Date: Sat, 15 Mar 2025 21:15:52 +0200

[Message part 1 (text/plain, inline)]

> check_dirs_file() {
>     if [ "$(wc -l "$dirs_file")" -gt 10 ]; then
> tmp_dirs_file="$(mktemp)"
> sed '1d' "$dirs_file" > "$tmp_dirs_file"
> cp "$tmp_dirs_file" "$dirs_file"
>     fi
> }
>
> 3. M-x bash-ts-mode
> 4. Move the cursor to the opening bracket of '$(mktemp)' and execute
> 'forward-list'. It gives "No next group" message.

The problem is that many tree-sitter grammars are so imperfect that even
the current heuristics in 'treesit-forward-list' can't help to handle
many cases.  Here are a few examples:

1. bash-ts-mode

  $(a)
=>
  (command_substitution "$("
   (command
    name: (command_name (word)))
   ")")

Here the open paren is inside the node "$(".

2. ruby-ts-mode

  "#{a}"
=>
  (string " (interpolation "#{" (identifier) "}") ")

Here the open curly brace is inside the node "#{".

3. elixir-ts-mode

  &(a)
=>
  (unary_operator operator: "&" operand: "(" (identifier) ")")

Here the open paren is the second node "(" after "&".

4. go-ts-mode

  switch a { }
=>
  (expression_switch_statement "switch" value: (identifier) "{" "}")

Here the open curly brace is far from the sibling "switch" node.

(A similar case in 'for_statement' is already handled
surprisingly well by the current heuristics in 'treesit-forward-list'.)

I see only 2 variants how to allow 'forward-list' to handle all cases:

1. Improve the current heuristics to decide when to fall back
   to the default syntax-based navigation;

2. Explicitly specify the positions that should fall back
   to the default syntax-based navigation.

I can't find a better heuristics, so probably we need to have
another thing 'sexp-list' that specifies when to use
'forward-sexp-default-function' and 'forward-list-default-function'.

This patch solves all the above problems:

[treesit-sexp-list.patch (text/x-diff, inline)]

diff --git a/lisp/treesit.el b/lisp/treesit.el
index 46332cb1e4b..060686e8dde 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -3015,6 +3015,12 @@ treesit--forward-list-with-default
         ;; Use the default function only if it doesn't go
         ;; over the sibling and doesn't go out of the current group.
         (or (when (and default-pos
+                       (treesit-node-match-p
+                        (treesit-node-at (if (> arg 0) (point)
+                                           (max (1- (point)) (point-min))))
+                        'sexp-list t))
+              (goto-char default-pos))
+            (when (and default-pos
                        (or (null sibling)
                            (if (> arg 0)
                                (<= default-pos (treesit-node-start sibling))
diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index c4b7d0837a4..0c493e3cd77 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -1662,6 +1662,11 @@ bash-ts-mode
                                  "command_substitution"
                                  "process_substitution")
                          eos))
+                   (sexp-list
+                    ("$(" .
+                     ,(lambda (node)
+                        (equal (treesit-node-type (treesit-node-parent node))
+                               "command_substitution"))))
                    (sentence
                     ,(rx bos (or "redirected_statement"
                                  "declaration_command"
diff --git a/lisp/progmodes/ruby-ts-mode.el b/lisp/progmodes/ruby-ts-mode.el
index 1594f301641..84a3842d9ab 100644
--- a/lisp/progmodes/ruby-ts-mode.el
+++ b/lisp/progmodes/ruby-ts-mode.el
@@ -1253,6 +1253,12 @@ ruby-ts-mode
                                 "hash")
                                eos)
                               #'ruby-ts--list-p))
+                 (sexp-list
+                  ("#{" . ,(lambda (node)
+                             ;; for C-M-f in "abc #{ghi} def"
+                             (and (eq (char-after (point)) ?{)
+                                  (equal (treesit-node-type (treesit-node-parent node))
+                                         "interpolation")))))
                  (sentence ,(rx bos (or "return"
                                         "body_statement"
                                         "call"
@@ -1260,19 +1273,8 @@ ruby-ts-mode
                                 eos))
                  (text ,(lambda (node)
                           (or (member (treesit-node-type node)
-                                      '("comment" "string_content" "heredoc_content"))
-                              ;; for C-M-f in hash[:key] and hash['key']
-                              (and (member (treesit-node-text node)
-                                           '("[" "]"))
-                                   (equal (treesit-node-type
-                                           (treesit-node-parent node))
-                                          "element_reference"))
-                              ;; for C-M-f in "abc #{ghi} def"
-                              (and (member (treesit-node-text node)
-                                           '("#{" "}"))
-                                   (equal (treesit-node-type
-                                           (treesit-node-parent node))
-                                          "interpolation"))))))))
+                                      '("comment" "string_content"
+                                        "heredoc_content"))))))))
 
   ;; Imenu.
   (setq-local imenu-create-index-function #'ruby-ts--imenu)
diff --git a/lisp/progmodes/elixir-ts-mode.el b/lisp/progmodes/elixir-ts-mode.el
index d50692d87c0..3d50e85f41b 100644
--- a/lisp/progmodes/elixir-ts-mode.el
+++ b/lisp/progmodes/elixir-ts-mode.el
@@ -722,6 +712,13 @@ elixir-ts-mode
     (setq-local treesit-simple-indent-rules elixir-ts--indent-rules)
 
     ;; Navigation.
+    (setq-local treesit-thing-settings
+                `((elixir
+                   (sexp-list
+                    ("(" . ,(lambda (node)
+                              ;; for C-M-f in "&(&1)"
+                              (equal (treesit-node-type (treesit-node-parent node))
+                                     "unary_operator"))))
     (setq-local treesit-defun-type-regexp
                 '("call" . elixir-ts--defun-p))
 
diff --git a/lisp/progmodes/go-ts-mode.el b/lisp/progmodes/go-ts-mode.el
index d117fe21590..1932ee69606 100644
--- a/lisp/progmodes/go-ts-mode.el
+++ b/lisp/progmodes/go-ts-mode.el
@@ -306,6 +306,10 @@ go-ts-mode
                                  "argument_list"
                                  "literal_value")
                          eos))
+                   (sexp-list
+                    (lambda (node)
+                      (equal (treesit-node-type (treesit-node-parent node))
+                             "expression_switch_statement")))
                    (sentence
                     (or "declaration" "statement")))))
 

Message #14 received at 76791 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Yuan Fu <casouri <at> gmail.com>
Cc: the_wurfkreuz <the_wurfkreuz <at> proton.me>, 76791 <at> debbugs.gnu.org
Subject: Re: bug#76791: 31.0.50; forward-list doesn't work in bash-ts-mode
Date: Thu, 10 Apr 2025 19:21:10 +0300

close 76791 31.0.50
thanks

>> check_dirs_file() {
>>     if [ "$(wc -l "$dirs_file")" -gt 10 ]; then
>> tmp_dirs_file="$(mktemp)"
>> sed '1d' "$dirs_file" > "$tmp_dirs_file"
>> cp "$tmp_dirs_file" "$dirs_file"
>>     fi
>> }
>>
>> 3. M-x bash-ts-mode
>> 4. Move the cursor to the opening bracket of '$(mktemp)' and execute
>> 'forward-list'. It gives "No next group" message.
>
> The problem is that many tree-sitter grammars are so imperfect that even
> the current heuristics in 'treesit-forward-list' can't help to handle
> many cases.  Here are a few examples:
>
> 1. bash-ts-mode
>
>   $(a)
> =>
>   (command_substitution "$("
>    (command
>     name: (command_name (word)))
>    ")")
>
> Here the open paren is inside the node "$(".
>
> 2. ruby-ts-mode
>
>   "#{a}"
> =>
>   (string " (interpolation "#{" (identifier) "}") ")
>
> Here the open curly brace is inside the node "#{".
>
> 3. elixir-ts-mode
>
>   &(a)
> =>
>   (unary_operator operator: "&" operand: "(" (identifier) ")")
>
> Here the open paren is the second node "(" after "&".
>
> 4. go-ts-mode
>
>   switch a { }
> =>
>   (expression_switch_statement "switch" value: (identifier) "{" "}")
>
> Here the open curly brace is far from the sibling "switch" node.
>
> (A similar case in 'for_statement' is already handled
> surprisingly well by the current heuristics in 'treesit-forward-list'.)
>
> I see only 2 variants how to allow 'forward-list' to handle all cases:
>
> 1. Improve the current heuristics to decide when to fall back
>    to the default syntax-based navigation;
>
> 2. Explicitly specify the positions that should fall back
>    to the default syntax-based navigation.
>
> I can't find a better heuristics, so probably we need to have
> another thing 'sexp-list' that specifies when to use
> 'forward-sexp-default-function' and 'forward-list-default-function'.
>
> This patch solves all the above problems:

Now pushed and closed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.