GNU bug report logs - #61849
29.0.60; Unable to use treesit-install-language-grammar because repo doesn't have parser.c

Previous Next

To reply to this bug, email your comments to 61849 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Misha Zharov <mishazharov1 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.60; Unable to use treesit-install-language-grammar because repo
 doesn't have parser.c
Date: Sun, 26 Feb 2023 23:34:57 -0800

Add (sql "https://github.com/m-novikov/tree-sitter-sql") to
treesit-language-source-alist. Then use
treesit-install-language-grammar. The following error occurs:

    Error encountered when installing language grammar: (treesit-error
    Command: cc -fPIC -c -I. parser.c Error output: cc1: fatal error:
    parser.c: No such file or directory compilation terminated.)

This error occurs because we need to run a configuration step on the
repo to generate parser.c. I believe this is because we need to either
run `npm run generate` or `tree-sitter generate` to generate the
`src/parser.c` file. It would be nice if we could implement a patch to:

  1. Allow users to specify a configuration step to configure the repo
     before searching for parser.c (like passing a lambda into
     `treesit-language-source-alist`)
  2. Allow users to specify a git hash that should be checked out before
     the configuration step is run. I know currently different branches
     are supported, but not all projects have release branches.
  3. Alternatively this function can be split into 2 function, the first one
      would clone and configure the repository, and well as find the
      required files. The second function would simply compile the required
      files into the shared libraries, and move them to the appropriate location

I can have a go at implementing some of the above if those changes are
welcome. The reason this is important is because it seems like more
repos in the future will move away from providing the autogenerated
files in their repos, so this function might be on borrowed time in its
current incarnation.


In GNU Emacs 29.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.33, cairo version 1.16.0) of 2023-02-24 built on misha-N552VX
Repository revision: 5cf50d60041c82deccc4b32a8ecdb1a28b6e8f91
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: Ubuntu 22.04.2 LTS

Configured using:
 'configure --with-json --with-cairo --with-xwidgets
 --prefix=/opt/emacs/ --with-x-toolkit=gtk3 --with-tree-sitter
 --with-native-compilation'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11
XDBE XIM XINPUT2 XPM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LANG: en_CA.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  server-mode: t
  windmove-mode: t
  marginalia-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  override-global-mode: t
  global-company-mode: t
  company-mode: t
  savehist-mode: t
  vertico-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  isearch-fold-quotes-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/misha/.emacs.d/straight/build/transient/transient hides
/opt/emacs/share/emacs/29.0.60/lisp/transient
/home/misha/.emacs.d/straight/build/xref/xref hides
/opt/emacs/share/emacs/29.0.60/lisp/progmodes/xref
/home/misha/.emacs.d/straight/build/project/project hides
/opt/emacs/share/emacs/29.0.60/lisp/progmodes/project
/home/misha/.emacs.d/straight/build/let-alist/let-alist hides
/opt/emacs/share/emacs/29.0.60/lisp/emacs-lisp/let-alist

Features:
(shadow sort mail-extr emacsbug treesit pulse jka-compr consult-xref
cl-print debug backtrace cus-start cus-load misearch multi-isearch
wid-edit lee-ho-fook shortdoc help-fns radix-tree vc-hg vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs log-view vc bug-reference face-remap
magit-bookmark magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit package browse-url url-handlers magit-repos
magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode
diff git-commit log-edit message sendmail yank-media puny dired
dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config
gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils
mailheader pcvs-util add-log magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process with-editor shell
magit-mode transient magit-git magit-base magit-section crm
embark-consult consult-vertico consult bookmark embark-org embark ffap
vc-git diff-mode vc-dispatcher mule-util pp comp comp-cstr server
checkdoc lisp-mnt flymake-proc flymake warnings init windmove
rustic-spellcheck rustic-expand rustic-lsp rustic-playground
rustic-rustfix rustic-racer rustic-babel rustic-rustfmt org-element
org-persist org-id org-refile avl-tree org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-src ob-comint org-pcomplete pcomplete
org-list org-footnote org-faces org-entities time-date ob-emacs-lisp
ob-core ob-eval org-cycle org-table ol org-fold org-fold-core org-keys
oc org-loaddefs find-func cal-menu calendar cal-loaddefs org-version
org-compat org-macs format-spec rustic-comint rustic-clippy rustic-doc
xdg f f-shortdoc url url-proxy url-privacy url-expand url-methods
url-history url-cookie generate-lisp-file url-domsuf url-util mailcap
rustic-popup rustic-cargo rustic-compile spinner compile
text-property-search comint ansi-osc ansi-color s xterm-color
markdown-mode color url-parse auth-source eieio eieio-core
password-cache json map url-vars noutline outline icons
rustic-interaction rustic rust-utils thingatpt rust-mode rx dash
rustic-autoloads xterm-color-autoloads spinner-autoloads
project-autoloads xref-autoloads let-alist-autoloads f-autoloads
s-autoloads rust-mode-autoloads embark-consult-autoloads
embark-autoloads consult-autoloads marginalia marginalia-autoloads
orderless orderless-autoloads flyspell ispell display-line-numbers
edmacro kmacro use-package-bind-key bind-key easy-mmode company-oddmuse
company-keywords company-etags etags fileloop generator xref project
byte-opt ring company-gtags company-dabbrev-code company-dabbrev
company-files company-clang company-capf company-cmake company-semantic
company-template company-bbdb company pcase company-autoloads savehist
vertico compat vertico-autoloads exec-path-from-shell
exec-path-from-shell-autoloads use-package-core magit-autoloads
magit-section-autoloads git-commit-autoloads with-editor-autoloads
transient-autoloads dash-autoloads compat-autoloads info finder-inf
markdown-mode-autoloads straight-autoloads cl-seq cl-extra help-mode
straight subr-x cl-macs gv cl-loaddefs cl-lib bytecomp byte-compile
wombat-theme rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads xwidget-internal dbusbind inotify dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
xinput2 x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 401016 69034)
 (symbols 48 29231 0)
 (strings 32 116731 2681)
 (string-bytes 1 5343184)
 (vectors 16 68498)
 (vector-slots 8 1875280 195634)
 (floats 8 416 486)
 (intervals 56 2213 217)
 (buffers 984 30))

Message #8 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Misha Zharov <mishazharov1 <at> gmail.com>
Cc: 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60;
 Unable to use treesit-install-language-grammar because repo doesn't
 have parser.c
Date: Mon, 27 Feb 2023 22:00:48 +0200

> From: Misha Zharov <mishazharov1 <at> gmail.com>
> Date: Sun, 26 Feb 2023 23:34:57 -0800
> 
> Add (sql "https://github.com/m-novikov/tree-sitter-sql") to
> treesit-language-source-alist. Then use
> treesit-install-language-grammar. The following error occurs:
> 
>     Error encountered when installing language grammar: (treesit-error
>     Command: cc -fPIC -c -I. parser.c Error output: cc1: fatal error:
>     parser.c: No such file or directory compilation terminated.)
> 
> This error occurs because we need to run a configuration step on the
> repo to generate parser.c. I believe this is because we need to either
> run `npm run generate` or `tree-sitter generate` to generate the
> `src/parser.c` file. It would be nice if we could implement a patch to:
> 
>   1. Allow users to specify a configuration step to configure the repo
>      before searching for parser.c (like passing a lambda into
>      `treesit-language-source-alist`)
>   2. Allow users to specify a git hash that should be checked out before
>      the configuration step is run. I know currently different branches
>      are supported, but not all projects have release branches.
>   3. Alternatively this function can be split into 2 function, the first one
>       would clone and configure the repository, and well as find the
>       required files. The second function would simply compile the required
>       files into the shared libraries, and move them to the appropriate location

I'm not sure we should incorporate in Emacs so much of this
specialized stuff.  treesit-install-language-grammar is meant for
doing the simple steps of compiling C/C++ sources in a boilerplate
repository into a shared library.  Anything significantly more complex
should IMO be left to manual procedures by people who know what they
are doing, especially if that requires to have specialized tools
installed.

(Btw, why not use https://github.com/DerekStride/tree-sitter-sql
instead?)

Message #11 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Misha Zharov <mishazharov1 <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Mon, 27 Feb 2023 19:13:59 -0800

Thanks for getting back to me

> (Btw, why not use https://github.com/DerekStride/tree-sitter-sql
> instead?)

I just landed on the other one first, no particular reason.  Initially it
worked until they removed the autogenerated code that was in
the repo.  As a workaround I can switch to the sql grammar that you
have recommended, but the problem might occur again in other
repos.

> I'm not sure we should incorporate in Emacs so much of this
> specialized stuff.

That's fair, but I fear that more repos will remove the autogenerated
parser code, which will make treesit-install-language-grammar
much less useful.  Perhaps this won't occur often, but I wanted to
consult on a possible solution.  However, since this appears to be
working as intended, then that is okay as well.

Thanks for taking the time to look into this.

On Mon, Feb 27, 2023 at 12:00 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Misha Zharov <mishazharov1 <at> gmail.com>
> > Date: Sun, 26 Feb 2023 23:34:57 -0800
> >
> > Add (sql "https://github.com/m-novikov/tree-sitter-sql") to
> > treesit-language-source-alist. Then use
> > treesit-install-language-grammar. The following error occurs:
> >
> >     Error encountered when installing language grammar: (treesit-error
> >     Command: cc -fPIC -c -I. parser.c Error output: cc1: fatal error:
> >     parser.c: No such file or directory compilation terminated.)
> >
> > This error occurs because we need to run a configuration step on the
> > repo to generate parser.c. I believe this is because we need to either
> > run `npm run generate` or `tree-sitter generate` to generate the
> > `src/parser.c` file. It would be nice if we could implement a patch to:
> >
> >   1. Allow users to specify a configuration step to configure the repo
> >      before searching for parser.c (like passing a lambda into
> >      `treesit-language-source-alist`)
> >   2. Allow users to specify a git hash that should be checked out before
> >      the configuration step is run. I know currently different branches
> >      are supported, but not all projects have release branches.
> >   3. Alternatively this function can be split into 2 function, the first one
> >       would clone and configure the repository, and well as find the
> >       required files. The second function would simply compile the required
> >       files into the shared libraries, and move them to the appropriate location
>
> I'm not sure we should incorporate in Emacs so much of this
> specialized stuff.  treesit-install-language-grammar is meant for
> doing the simple steps of compiling C/C++ sources in a boilerplate
> repository into a shared library.  Anything significantly more complex
> should IMO be left to manual procedures by people who know what they
> are doing, especially if that requires to have specialized tools
> installed.
>
> (Btw, why not use https://github.com/DerekStride/tree-sitter-sql
> instead?)

Message #14 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Misha Zharov <mishazharov1 <at> gmail.com>
Cc: 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Tue, 28 Feb 2023 14:04:51 +0200

> From: Misha Zharov <mishazharov1 <at> gmail.com>
> Date: Mon, 27 Feb 2023 19:13:59 -0800
> Cc: 61849 <at> debbugs.gnu.org
> 
> Thanks for getting back to me
> 
> > (Btw, why not use https://github.com/DerekStride/tree-sitter-sql
> > instead?)
> 
> I just landed on the other one first, no particular reason.  Initially it
> worked until they removed the autogenerated code that was in
> the repo.  As a workaround I can switch to the sql grammar that you
> have recommended, but the problem might occur again in other
> repos.
> 
> > I'm not sure we should incorporate in Emacs so much of this
> > specialized stuff.
> 
> That's fair, but I fear that more repos will remove the autogenerated
> parser code, which will make treesit-install-language-grammar
> much less useful.

If that starts to happen too much, we'll need to rethink this feature.
It is supposed to be extremely reliable, and this should not rely on
tools that might not be installed, because giving clueless users a
tool that fails unless one "tinkers" is not a good idea, IMO.

I'll leave this bug open for now.

Message #17 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Yuan Fu <casouri <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mishazharov1 <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use 
 treesit-install-language-grammar because repo doesn't have parser.c
Date: Fri, 3 Mar 2023 14:27:05 -0800

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Misha Zharov <mishazharov1 <at> gmail.com>
>> Date: Mon, 27 Feb 2023 19:13:59 -0800
>> Cc: 61849 <at> debbugs.gnu.org
>> 
>> Thanks for getting back to me
>> 
>> > (Btw, why not use https://github.com/DerekStride/tree-sitter-sql
>> > instead?)
>> 
>> I just landed on the other one first, no particular reason.  Initially it
>> worked until they removed the autogenerated code that was in
>> the repo.  As a workaround I can switch to the sql grammar that you
>> have recommended, but the problem might occur again in other
>> repos.
>> 
>> > I'm not sure we should incorporate in Emacs so much of this
>> > specialized stuff.
>> 
>> That's fair, but I fear that more repos will remove the autogenerated
>> parser code, which will make treesit-install-language-grammar
>> much less useful.
>
> If that starts to happen too much, we'll need to rethink this feature.
> It is supposed to be extremely reliable, and this should not rely on
> tools that might not be installed, because giving clueless users a
> tool that fails unless one "tinkers" is not a good idea, IMO.
>
> I'll leave this bug open for now.

Not sure why that project removed the grammar.c files, it seems
ubiquitous for tree-sitter language grammar projects to include the
generated grammar.c file in the repo.

Yuan

Message #20 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Yuan Fu <casouri <at> gmail.com>
Cc: mishazharov1 <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use 
 treesit-install-language-grammar because repo doesn't have parser.c
Date: Sat, 04 Mar 2023 09:06:37 +0200

> From: Yuan Fu <casouri <at> gmail.com>
> Date: Fri, 3 Mar 2023 14:27:05 -0800
> Cc: mishazharov1 <at> gmail.com,
>  61849 <at> debbugs.gnu.org
> 
> Not sure why that project removed the grammar.c files, it seems
> ubiquitous for tree-sitter language grammar projects to include the
> generated grammar.c file in the repo.

Maybe we should file an issue in their issue tracker, asking them to
bring that file back.

Message #23 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: casouri <at> gmail.com, mishazharov1 <at> gmail.com
Cc: 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60;
 Unable to use treesit-install-language-grammar because repo doesn't
 have parser.c
Date: Sat, 04 Mar 2023 13:06:50 +0200

> Cc: mishazharov1 <at> gmail.com, 61849 <at> debbugs.gnu.org
> Date: Sat, 04 Mar 2023 09:06:37 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Yuan Fu <casouri <at> gmail.com>
> > Date: Fri, 3 Mar 2023 14:27:05 -0800
> > Cc: mishazharov1 <at> gmail.com,
> >  61849 <at> debbugs.gnu.org
> > 
> > Not sure why that project removed the grammar.c files, it seems
> > ubiquitous for tree-sitter language grammar projects to include the
> > generated grammar.c file in the repo.
> 
> Maybe we should file an issue in their issue tracker, asking them to
> bring that file back.

Now done, see https://github.com/m-novikov/tree-sitter-sql/issues/72

Message #26 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Misha Zharov <mishazharov1 <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Sat, 4 Mar 2023 12:52:23 -0800

> Now done, see https://github.com/m-novikov/tree-sitter-sql/issues/72

Great, thanks for following up and filing the ticket.  I subscribed to
it and will watch it for any further developments

Message #29 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: casouri <at> gmail.com, mishazharov1 <at> gmail.com
Cc: 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60;
 Unable to use treesit-install-language-grammar because repo doesn't
 have parser.c
Date: Sun, 19 Mar 2023 09:38:23 +0200

> Cc: 61849 <at> debbugs.gnu.org
> Date: Sat, 04 Mar 2023 13:06:50 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > Cc: mishazharov1 <at> gmail.com, 61849 <at> debbugs.gnu.org
> > Date: Sat, 04 Mar 2023 09:06:37 +0200
> > From: Eli Zaretskii <eliz <at> gnu.org>
> > 
> > > From: Yuan Fu <casouri <at> gmail.com>
> > > Date: Fri, 3 Mar 2023 14:27:05 -0800
> > > Cc: mishazharov1 <at> gmail.com,
> > >  61849 <at> debbugs.gnu.org
> > > 
> > > Not sure why that project removed the grammar.c files, it seems
> > > ubiquitous for tree-sitter language grammar projects to include the
> > > generated grammar.c file in the repo.
> > 
> > Maybe we should file an issue in their issue tracker, asking them to
> > bring that file back.
> 
> Now done, see https://github.com/m-novikov/tree-sitter-sql/issues/72

The other grammar for SQL followed suit, unfortunately, see

  https://github.com/DerekStride/tree-sitter-sql/issues/120

Please chime in to try to convince them to go back to including the
generated parser files.

Message #32 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Misha Zharov <mishazharov1 <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Sun, 19 Mar 2023 22:14:44 -0700

> Please chime in to try to convince them to go back to including the
> generated parser files.

Thanks for the update.  I've now done this, but I think that it will be
difficult to make sure that community sticks to this standard of keeping
the autogenerated files in the repo.  Their arguments do have some
merit regarding the difficulty of resolving merge conflicts and overall
maintenance burden.  At the same time, having the autogenerated file
present is a nice QOL feature for users because it's easily reproducible.

I'm not sure what the ultimate solution to this will be though.  It might be
necessary to come up with a new paradigm of installing treesitter
grammars that addresses some of the current issues

Message #35 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Misha Zharov <mishazharov1 <at> gmail.com>
Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Mon, 20 Mar 2023 21:28:20 +0200

> From: Misha Zharov <mishazharov1 <at> gmail.com>
> Date: Sun, 19 Mar 2023 22:14:44 -0700
> Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
> 
> > Please chime in to try to convince them to go back to including the
> > generated parser files.
> 
> Thanks for the update.  I've now done this, but I think that it will be
> difficult to make sure that community sticks to this standard of keeping
> the autogenerated files in the repo.  Their arguments do have some
> merit regarding the difficulty of resolving merge conflicts and overall
> maintenance burden.  At the same time, having the autogenerated file
> present is a nice QOL feature for users because it's easily reproducible.

Bummer:

  https://github.com/DerekStride/tree-sitter-sql/issues/120#issuecomment-1476609242

> I'm not sure what the ultimate solution to this will be though.  It might be
> necessary to come up with a new paradigm of installing treesitter
> grammars that addresses some of the current issues

Maybe.  better yet, the various distros should hopefully pick up the
gauntlet and startr providing prebuilt binaries of these grammar
libraries for users to install using standard tools.

Message #38 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Eli Zaretskii <eliz <at> gnu.org>, Misha Zharov <mishazharov1 <at> gmail.com>
Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use
 treesit-install-language-grammar because repo doesn't have parser.c
Date: Mon, 20 Mar 2023 22:04:41 +0200

On 20/03/2023 21:28, Eli Zaretskii wrote:
>> From: Misha Zharov<mishazharov1 <at> gmail.com>
>> Date: Sun, 19 Mar 2023 22:14:44 -0700
>> Cc:casouri <at> gmail.com,61849 <at> debbugs.gnu.org
>>
>>> Please chime in to try to convince them to go back to including the
>>> generated parser files.
>> Thanks for the update.  I've now done this, but I think that it will be
>> difficult to make sure that community sticks to this standard of keeping
>> the autogenerated files in the repo.  Their arguments do have some
>> merit regarding the difficulty of resolving merge conflicts and overall
>> maintenance burden.  At the same time, having the autogenerated file
>> present is a nice QOL feature for users because it's easily reproducible.
> Bummer:
> 
>    https://github.com/DerekStride/tree-sitter-sql/issues/120#issuecomment-1476609242

Suppose the repositories remove the generated grammar files. What would 
be sufficient for us to regenerate them?

tree-sitter-sql apparently uses the tree-sitter-cli program. Would 
having it on the user's system suffice?

For a lot of developers NPM will already be installed. If the only 
remaining step will be 'npm install -g tree-sitter-cli', or dropping one 
of the pre-built binaries from 
https://github.com/tree-sitter/tree-sitter/releases into a directory on 
PATH, I think they could manage to do it once, to be used in all 
grammars which don't keep generated files in the repo.

Message #41 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: mishazharov1 <at> gmail.com, casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
Subject: Re: bug#61849: 29.0.60; Unable to use
 treesit-install-language-grammar because repo doesn't have parser.c
Date: Mon, 20 Mar 2023 22:20:44 +0200

> Date: Mon, 20 Mar 2023 22:04:41 +0200
> Cc: casouri <at> gmail.com, 61849 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov <at> yandex.ru>
> 
> Suppose the repositories remove the generated grammar files. What would 
> be sufficient for us to regenerate them?

I hope someone will write a generation tool in Emacs Lisp.

Message #44 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mishazharov1 <at> gmail.com, casouri <at> gmail.com, 61849 <at> debbugs.gnu.org,
 Dmitry Gutov <dgutov <at> yandex.ru>
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Mon, 20 Mar 2023 22:01:15 +0000

>> Suppose the repositories remove the generated grammar files. What would 
>> be sufficient for us to regenerate them?
>
> I hope someone will write a generation tool in Emacs Lisp.
>

That can't possibly happen, IMO.  Generating the grammar.json files from 
the grammar.js files with an Emacs Lisp program would amount to implement 
a JavaScript interpreter in Emacs Lisp.  And generating the parser.c files 
from the grammar.json files would amount to reimplement the generator, 
which is about 13500 lines of non-trivial Rust code, in Emacs Lisp.

Message #47 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gregory Heytings <gregory <at> heytings.org>
Cc: mishazharov1 <at> gmail.com, casouri <at> gmail.com, 61849 <at> debbugs.gnu.org,
 dgutov <at> yandex.ru
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Tue, 21 Mar 2023 05:27:50 +0200

> Date: Mon, 20 Mar 2023 22:01:15 +0000
> From: Gregory Heytings <gregory <at> heytings.org>
> cc: Dmitry Gutov <dgutov <at> yandex.ru>, mishazharov1 <at> gmail.com, casouri <at> gmail.com, 
>     61849 <at> debbugs.gnu.org
> 
> 
> >> Suppose the repositories remove the generated grammar files. What would 
> >> be sufficient for us to regenerate them?
> >
> > I hope someone will write a generation tool in Emacs Lisp.
> >
> 
> That can't possibly happen, IMO.  Generating the grammar.json files from 
> the grammar.js files with an Emacs Lisp program would amount to implement 
> a JavaScript interpreter in Emacs Lisp.

So you are saying that generating a parser would need a JavaScript
interpreter as part of the generation?  Does tree-sitter-cli tool
invoke it, then (I didn't study its sources)?

> And generating the parser.c files from the grammar.json files would
> amount to reimplement the generator, which is about 13500 lines of
> non-trivial Rust code, in Emacs Lisp.

That's what I hoped someone will do, yes.  It's a non-trivial job, but
surely isn't impossible.  Reimplementing that in some other widely
available language, like Python or Perl, would also do.  Or maybe the
Rust front-end to GCC will become available soon enough.  Or something.

Message #50 received at 61849 <at> debbugs.gnu.org (full text, mbox):

From: Gregory Heytings <gregory <at> heytings.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mishazharov1 <at> gmail.com, casouri <at> gmail.com, 61849 <at> debbugs.gnu.org,
 dgutov <at> yandex.ru
Subject: Re: bug#61849: 29.0.60; Unable to use treesit-install-language-grammar
 because repo doesn't have parser.c
Date: Tue, 21 Mar 2023 09:39:31 +0000

>> Generating the grammar.json files from the grammar.js files with an 
>> Emacs Lisp program would amount to implement a JavaScript interpreter 
>> in Emacs Lisp.
>
> So you are saying that generating a parser would need a JavaScript 
> interpreter as part of the generation?
>

Yes.

>
> Does tree-sitter-cli tool invoke it, then (I didn't study its sources)?
>

Yes, here: https://github.com/tree-sitter/tree-sitter/blob/master/cli/src/generate/mod.rs#L169.

>> And generating the parser.c files from the grammar.json files would 
>> amount to reimplement the generator, which is about 13500 lines of 
>> non-trivial Rust code, in Emacs Lisp.
>
> That's what I hoped someone will do, yes.  It's a non-trivial job, but 
> surely isn't impossible.  Reimplementing that in some other widely 
> available language, like Python or Perl, would also do.  Or maybe the 
> Rust front-end to GCC will become available soon enough.  Or something.
>

It's not impossible, but what would be the benefit (and/or incentive) of 
doing that?  Rust is already widely available, under a liberal licence, 
and it is also not necessary to create the parser.c file, which is 
architecture-independent, on the computer on which it is compiled.  Also 
note that generating the parser.c file can use _a lot_ of resources, e.g. 
generating the parser.c file of tree-sitter-c-sharp from the grammar.json 
file uses 40 (fourty!) GB of memory and takes several minutes.  I'll let 
you imagine what these numbers would be with an interpreted language like 
Elisp, Python or Perl.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.