GNU bug report logs - #55396
29.0.50; project-find-file don't work on a project with many submodules

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 55396 in the body.
You can then email your comments to 55396 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eason Huang <aqua0210 <at> foxmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.50; project-find-file don't work on a project with many
 submodules
Date: Fri, 13 May 2022 20:56:12 +0800

I try to use `M-x project-find-file` in my `.emacs.d` config project. It
takes a long time(about 1 minutes), and finally get a error as bellow:

```
process-file: Variable binding depth exceeds max-specpdl-size
```

The .emacs.d project include 95 submudules of Git, is this too huge for
project.el?
You can visit my config: https://github.com/Eason0210/emacs.d

On other git project with a few submodules(about 14), it works well.
For exmaple this one: https://github.com/emacscollective/emacs.g


And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
Then reproduce the issue, will get the following debug error:

Debugger entered--Lisp error: (excessive-variable-binding)
  call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
  process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
  vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
  project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")

  .... there are 1100 lines more here ...

  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
  #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
  project--vc-list-files("~/.emacs.d/" Git nil)
  #f(compiled-function (dir) #<bytecode -0x1a2f966d7f3875dc>)("~/.emacs.d/")
  mapcan(#f(compiled-function (dir) #<bytecode -0x1a2f966d7f3875dc>) ("~/.emacs.d/"))
  #f(compiled-function (project &optional dirs) #<bytecode -0x13f4e1776db2687e>)((vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  apply(#f(compiled-function (project &optional dirs) #<bytecode -0x13f4e1776db2687e>) (vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  project-files((vc Git "~/.emacs.d/") ("~/.emacs.d/"))
  project-find-file-in(#("init.el" 0 7 (fontified t help-echo "mouse-2: visit this file in other window" mouse-face highlight dired-filename t)) ("~/.emacs.d/") (vc Git "~/.emacs.d/") nil)
  project-find-file(nil)
  funcall-interactively(project-find-file nil)
  command-execute(project-find-file record)
  execute-extended-command(nil "project-find-file" "project-find-fi")
  funcall-interactively(execute-extended-command nil "project-find-file" "project-find-fi")
  command-execute(execute-extended-command)

My platfrom informations:

In GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin21.4.0, NS appkit-2113.40 Version 12.3.1 (Build 21E258))
 of 2022-05-09 built on macbook
Repository revision: 4f1e748df208ced08c7cda8f96e6a5638ad14240
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.3.1

Configured using:
 'configure --with-ns --with-modules
 '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp:/usr/local/share/emacs/site-lisp'
 --with-xwidgets --with-native-compilation
 'CFLAGS=-I/usr/local/opt/gcc/include -I/usr/local/opt/libgccjit/include
 -O2' 'LDFLAGS=-L/usr/local/opt/gcc/lib/gcc/11
 -L/usr/local/opt/gcc/lib/gcc/11/gcc/x86_64-apple-darwin21/11
 -L/usr/local/opt/libgccjit/lib/gcc/11 -I/usr/local/opt/gcc/include
 -I/usr/local/opt/libgccjit/include -Wl,-headerpad_max_install_names''

Configured features:
ACL DBUS GIF GLIB GMP GNUTLS JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XIM XWIDGETS ZLIB

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/d

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media rmc puny
rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config
gnus-util text-property-search time-date mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils help-fns radix-tree
cl-print debug backtrace help-mode find-func thingatpt vc-mtn vc-hg
vc-git diff-mode vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view
easy-mmode pcvs-util vc vc-dispatcher project seq gv subr-x byte-opt
bytecomp byte-compile cconv dired-aux cl-loaddefs cl-lib dired
dired-loaddefs iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice simple cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads
xwidget-internal dbusbind kqueue cocoa ns lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 765142 20103)
 (symbols 48 9014 0)
 (strings 32 215443 3200)
 (string-bytes 1 8854733)
 (vectors 16 21867)
 (vector-slots 8 482129 32483)
 (floats 8 30 306)
 (intervals 56 31551 79)
 (buffers 992 17))


-- 
Eason Huang

Message #8 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Eason Huang <aqua0210 <at> foxmail.com>, 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Sun, 15 May 2022 04:48:00 +0300

On 13.05.2022 15:56, Eason Huang wrote:
> I try to use `M-x project-find-file` in my `.emacs.d` config project. It
> takes a long time(about 1 minutes), and finally get a error as bellow:
> 
> ```
> process-file: Variable binding depth exceeds max-specpdl-size
> ```
> 
> The .emacs.d project include 95 submudules of Git, is this too huge for
> project.el?

Hi! That's a lot, but the above error indicates excess recursion. Do the 
submodules in your repo in turn have checked out submodules inside, and 
so on?

As a workaround, you can set project-vc-merge-submodules to nil (at 
least temporarily, until we get a better fix).

Message #11 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Eason Huang <aqua0210 <at> foxmail.com>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Sun, 15 May 2022 12:09:36 +0800

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 13.05.2022 15:56, Eason Huang wrote:
>> I try to use `M-x project-find-file` in my `.emacs.d` config project. It
>> takes a long time(about 1 minutes), and finally get a error as bellow:
>> ```
>> process-file: Variable binding depth exceeds max-specpdl-size
>> ```
>> The .emacs.d project include 95 submudules of Git, is this too huge
>> for
>> project.el?
>
> Hi! That's a lot, but the above error indicates excess recursion. Do
> the submodules in your repo in turn have checked out submodules
> inside, and so on?

I try another project with 42 submodules, when I use `Borg` to install
 `Corfu` extensions, I need to set two load-path on `.gitmodules`. And this
 setting will trigger the same issue.

```
[submodule "corfu"]
	load-path = .
	load-path = extensions
	path = lib/corfu
	url = git <at> github.com:minad/corfu.git
```
if I remove the load-path, just leave it as below, project.el wokrs
well.
```
[submodule "corfu"]
	path = lib/corfu
	url = git <at> github.com:minad/corfu.git
```

May be the above infomation can help you.

> As a workaround, you can set project-vc-merge-submodules to nil (at
> least temporarily, until we get a better fix).
>

Thanks, I try to set project-vc-merge-submodules to nil, and It works.

By the way, I also tried `projectile` on the huge .emacs.d, It works but
can feel the latency.

-- 
Eason Huang

Message #14 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Eason Huang <aqua0210 <at> foxmail.com>, 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Mon, 13 Jun 2022 01:17:28 +0300

Hi again, sorry for the delay.

On 13.05.2022 15:56, Eason Huang wrote:
> And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
> Then reproduce the issue, will get the following debug error:
> 
> Debugger entered--Lisp error: (excessive-variable-binding)
>    call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
>    vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
>    project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)

Looking at the backtrace again, it seems the problem is not related to 
the number of submodules. It's a plain infloop.

Could you try to help with debugging? Alternatively, you could provide a 
simple repo with this problem that doesn't require Borg to initialize. 
Though it probably doesn't (shouldn't) require Borg anyway, given how 
the problem looks.

What does project--vc-list-files do in your case? It calls 'git 
ls-files' to fetch the list of files in the parent repo, then parses the 
list of submodules in it, and repeats the same call inside each 
submodule (using (concat default-directory module) as target).

Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over and 
over again. So it seems like (project--git-submodules) returns a list 
which has "." as one of its elements.

How does that happen? Do you have a submodule entry which points to "."?

Message #17 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Eason Huang <aqua0210 <at> foxmail.com>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Mon, 13 Jun 2022 23:00:25 +0800

Dmitry Gutov <dgutov <at> yandex.ru> writes:

Hi Dmitry,

Thanks for reply.

> Hi again, sorry for the delay.
>
> On 13.05.2022 15:56, Eason Huang wrote:
>> And I try to start emacs with `emacs -q`, set `(setq debug-on-errort)`.
>> Then reproduce the issue, will get the following debug error:
>> Debugger entered--Lisp error: (excessive-variable-binding)
>>    call-process("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    process-file("git" nil (t nil) nil "--no-pager" "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--call((t nil) "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--out-ok("ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    vc-git--run-command-string(nil "ls-files" "-z" "-c" "-o" "--exclude-standard")
>>    project--vc-list-files("/Users/eason/.emacs.d/lib/aggressive-indent" Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)("lib/aggressive-indent")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>>    #f(compiled-function (module) #<bytecode 0xfc6c01609385f70>)(".")
>>    project--vc-list-files("/Users/eason/.emacs.d/." Git nil)
>
> Looking at the backtrace again, it seems the problem is not related to
> the number of submodules. It's a plain infloop.
>
> Could you try to help with debugging? Alternatively, you could provide
> a simple repo with this problem that doesn't require Borg to
> initialize. Though it probably doesn't (shouldn't) require Borg
> anyway, given how the problem looks.

I create an simple which only include a submodule, I take vertico for
example. you can get it here:

https://github.com/Eason0210/sample-project.git

Steps to reproduce the issue:

1. Create an empty folder call `.emacs.d` (The name is not matter, you
can use any name)

2. cd .emacs.d, and let's say i want to add vertico as an submodule

```
git init
git submodule add --name vertico git <at> github.com:minad/vertico.git lib/vertico
git add .
git commit -m "add vertico"
```
3. so far so good.
4. Modify the .gitmodules file, add load-path for extensions directory
of vertico repo

```
[submodule "vertico"]
	path = lib/vertico
	url = git <at> github.com:minad/vertico.git
	load-path = .
	load-path = extensions
   #+end_src
```
5. Now perform ~M-x, project-find-file~ in the .emacs.d directory will
cause the issue.



> What does project--vc-list-files do in your case? It calls 'git
> ls-files' to fetch the list of files in the parent repo, then parses
> the list of submodules in it, and repeats the same call inside each
> submodule (using (concat default-directory module) as target).

I try 'git ls-files .' in vertico submodule in iterm and get this:

```
➜  vertico git:(main) git ls-files .
LICENSE
README.org
extensions/vertico-buffer.el
extensions/vertico-directory.el
extensions/vertico-flat.el
extensions/vertico-grid.el
extensions/vertico-indexed.el
extensions/vertico-mouse.el
extensions/vertico-multiform.el
extensions/vertico-quick.el
extensions/vertico-repeat.el
extensions/vertico-reverse.el
extensions/vertico-unobtrusive.el
vertico.el
```

```
➜  vertico git:(main) git ls-files extensions
extensions/vertico-buffer.el
extensions/vertico-directory.el
extensions/vertico-flat.el
extensions/vertico-grid.el
extensions/vertico-indexed.el
extensions/vertico-mouse.el
extensions/vertico-multiform.el
extensions/vertico-quick.el
extensions/vertico-repeat.el
extensions/vertico-reverse.el
extensions/vertico-unobtrusive.el
```

> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
> and over again. So it seems like (project--git-submodules) returns a
> list which has "." as one of its elements.
>
> How does that happen? Do you have a submodule entry which points to "."?
>

I found that it's this line `load-path = .` cause the issue. Borg need
this line to add the vertico directory to load-path when user add a
subDirectory to load-path. May be it is an issue of Borg?


-- 
Eason Huang

Message #20 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Eason Huang <aqua0210 <at> foxmail.com>
Cc: 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Tue, 14 Jun 2022 04:03:59 +0300

On 13.06.2022 18:00, Eason Huang wrote:
>> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
>> and over again. So it seems like (project--git-submodules) returns a
>> list which has "." as one of its elements.
>>
>> How does that happen? Do you have a submodule entry which points to "."?
>>
> I found that it's this line `load-path = .` cause the issue. Borg need
> this line to add the vertico directory to load-path when user add a
> subDirectory to load-path. May be it is an issue of Borg?

Now that I've tried adding that locally, it indeed what triggered the 
problem. It was a bug in 'project--git-submodules', which I've just 
fixed on master in commit 915b34d280.

I think file listing together with submodules should work fine now in 
your case. Not sure about the performance though: we use one process 
call per submodule, so the overhead might get noticeable with 42 of 
them. But please give it a try.

Message #23 received at 55396 <at> debbugs.gnu.org (full text, mbox):

From: Eason Huang <aqua0210 <at> foxmail.com>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 55396 <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Tue, 14 Jun 2022 19:22:15 +0800

Dmitry Gutov <dgutov <at> yandex.ru> writes:

> On 13.06.2022 18:00, Eason Huang wrote:
>>> Looking at the backtrace, it mentions "/Users/eason/.emacs.d/." over
>>> and over again. So it seems like (project--git-submodules) returns a
>>> list which has "." as one of its elements.
>>>
>>> How does that happen? Do you have a submodule entry which points to "."?
>>>
>> I found that it's this line `load-path = .` cause the issue. Borg need
>> this line to add the vertico directory to load-path when user add a
>> subDirectory to load-path. May be it is an issue of Borg?
>
> Now that I've tried adding that locally, it indeed what triggered the
> problem. It was a bug in 'project--git-submodules', which I've just
> fixed on master in commit 915b34d280.
>
> I think file listing together with submodules should work fine now in
> your case. Not sure about the performance though: we use one process
> call per submodule, so the overhead might get noticeable with 42 of
> them. But please give it a try.
>

Great, I tried on my huge .emacs.d project, it works well and fast.

Tested on macOS with latest commit.

You can close this bug now. Thanks for your great work on it.

-- 
Eason Huang

Message #28 received at 55396-done <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Eason Huang <aqua0210 <at> foxmail.com>
Cc: 55396-done <at> debbugs.gnu.org
Subject: Re: bug#55396: 29.0.50; project-find-file don't work on a project
 with many submodules
Date: Tue, 14 Jun 2022 15:47:00 +0300

On 14.06.2022 14:22, Eason Huang wrote:
> Great, I tried on my huge .emacs.d project, it works well and fast.
> 
> Tested on macOS with latest commit.
> 
> You can close this bug now. Thanks for your great work on it.

Thanks for testing! Closing.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.