GNU logs - #69188, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#69188: 30.0.50; project-files + project-find-file is slow in large repositories
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 18 Feb 2024 18:22:02 +0000
Resent-Message-ID: <handler.69188.B.170828047510191 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 69188 <at> debbugs.gnu.org
Cc: Dmitry Gutov <dmitry@HIDDEN>
From: Spencer Baugh <sbaugh@HIDDEN>
Date: Thu, 15 Feb 2024 17:55:46 -0500
Message-ID: <iera5o11gnh.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=64.215.233.18; envelope-from=sbaugh@HIDDEN;
 helo=mxout5.mail.janestreet.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049,
 DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -3.2 (---)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.2 (----)


(project-files (project-current)) takes around 1 second in Linux (80k
files) and 7 seconds in my larger (500k file) repository.

With this patch:
diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index c7c07c3d34c..037beaa835a 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -667,12 +667,15 @@
                                               (setq i (concat i "**"))))
                                         i)))
                                    extra-ignores)))))
-       (setq files
-             (mapcar
-              (lambda (file) (concat default-directory file))
-              (split-string
-               (apply #'vc-git--run-command-string nil "ls-files" args)
-               "\0" t)))
+       (with-temp-buffer
+         (let ((ok (apply #'vc-git--out-ok "ls-files" args))
+               (pt (point-min)))
+           (unless ok
+             (error "File listing failed: %s" (buffer-string)))
+           (goto-char pt)
+           (while (search-forward "\0" nil t)
+             (push (concat default-directory (buffer-substring-no-properties pt (1- (point)))) files)
+             (setq pt (point)))))
        (when (project--vc-merge-submodules-p default-directory)
          ;; Unfortunately, 'ls-files --recurse-submodules' conflicts with '-o'.
          (let* ((submodules (project--git-submodules))

project-files in Linux takes around .75 seconds.

If I further remove the (concat default-directory ...) around each file,
it speeds up to .5 seconds.

(Note that git ls-files itself takes only around 20 milliseconds)

My large repository (which uses Mercurial) has a custom project-files
which is basically:

(with-temp-buffer
  (unless (zerop (apply #'call-process "rhg" nil t nil "files"))
    (error "File listing failed: %s" (buffer-string)))
  (goto-char (point-min))
  (let ((pt (point))
        res)
    (while (search-forward "\n" nil t)
      (push (file-name-concat default-directory (buffer-substring-no-properties pt (1- (point)))) res)
      (setq pt (point)))
    res))

Likewise, removing the (concat default-directory ...) speeds my
project-files up from 7 seconds to 4.5 seconds.

This is especially silly because project-find-file then just removes
this default-directory again from all the files, which has yet more
overhead.

My proposal: Could we find a way to make the default-directory not
necessary for the files returned from project-files?

Perhaps project-files could be allowed to return relative file paths
which are relative to the project root.  Then in the common case where
all the files are within the project root, project-find-file would be
way faster.  Happy to implement this, if it makes sense.

Another optimization I've considered: We could run the process
asynchronously so project-files parsing can be parallel with the
process; but the process is usually very fast anyway, that's not most of
the overhead, so that won't be a big win.

However, that would make it easy for project-files as a whole to be
asynchronous.  Then that would allow project-find-file to start the
listing in the background, and then we'd write a completion table which
completes only over whatever files we've already read into Emacs.  I
think this would be a lot nicer for most use-cases, and I'd again be
happy to implement this.

Also happy to implement any other optimizations you think might make
sense.


In GNU Emacs 30.0.50 (build 37, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.15.12, Xaw scroll bars) of 2024-02-13 built on
 igm-qws-u22796a
Repository revision: a24a2b1ceb12f11c9d345190fbf554f27c4ec186
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.9 (Green Obsidian)

Configured using:
 'configure -C --with-x-toolkit=lucid 'CFLAGS=-O0 -g3'
 --without-native-compilation --without-gif'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty move-toolbar make-network-process emacs)

Memory information:
((conses 16 65052 9318) (symbols 48 9539 0) (strings 32 22452 1449)
 (string-bytes 1 659675) (vectors 16 9245)
 (vector-slots 8 111110 9295) (floats 8 40 17) (intervals 56 262 0)
 (buffers 976 10))

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Spencer Baugh <sbaugh@HIDDEN>
Subject: bug#69188: Acknowledgement (30.0.50; project-files +
 project-find-file is slow in large repositories)
Message-ID: <handler.69188.B.170828047510191.ack <at> debbugs.gnu.org>
References: <iera5o11gnh.fsf@HIDDEN>
X-Gnu-PR-Message: ack 69188
X-Gnu-PR-Package: emacs
Reply-To: 69188 <at> debbugs.gnu.org
Date: Sun, 18 Feb 2024 18:22:03 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 69188 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
69188: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D69188
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 18 Feb 2024 18:57:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Feb 18 13:57:17 2024
Received: from localhost ([127.0.0.1]:38276 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1rbmLl-0007je-Hk
	for submit <at> debbugs.gnu.org; Sun, 18 Feb 2024 13:57:17 -0500
Received: from eggs.gnu.org ([209.51.188.92]:45462)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1rbmLg-0007jJ-Vn; Sun, 18 Feb 2024 13:57:15 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1rbmLG-00014z-ER; Sun, 18 Feb 2024 13:56:46 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=5GXb3EjeBGvqcIFHrQIKYajPL8gOklMVatE6vzWcp0g=; b=V1RKgXfk7p3h
 qdmMwrS/ZzdFtuTpqvvnhXWCYPem8dxxWZRJ573GhnwztDglweA6biy/Gy6g2R4B3X6jVHdqLDk/a
 VU5Cam2RRP0O0GOXa5udJ9nMmpZpTx7FHM8xQUNdoIcg5FRMosB6q2wyf7zsBXs0qTHBZlzTlOyge
 TrNw6aN5N0+dUyS/kZXi2QGcxrslZVabqQlZC7nadOkuSpLtthuvOjuZdsGhLvHbIn27Lh4d5+rij
 hqWIwPsv5Ki7cTYSblSAawfFsjmAFUhpTXQ39tpm5mcjnrsuVTEBbqtSB56T/mXjXDkiULPzNxjwq
 KY+B4EznI7QBnpS2tvlyYg==;
Date: Sun, 18 Feb 2024 20:56:43 +0200
Message-Id: <86y1bhr47o.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <iera5o11gnh.fsf@HIDDEN>
 (message from Spencer Baugh on Thu, 15 Feb 2024 17:55:46 -0500)
Subject: Re: bug#69233: 30.0.50;
 project-files + project-find-file is slow in large repositories
References: <iera5o11gnh.fsf@HIDDEN>
X-Spam-Score: -4.2 (----)
X-Debbugs-Envelope-To: control
Cc: dmitry@HIDDEN, 69233 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.2 (-----)

merge 69233 69188
thanks

> Cc: Dmitry Gutov <dmitry@HIDDEN>
> From: Spencer Baugh <sbaugh@HIDDEN>
> Date: Thu, 15 Feb 2024 17:55:46 -0500
> 
> 
> (project-files (project-current)) takes around 1 second in Linux (80k
> files) and 7 seconds in my larger (500k file) repository.

This is a duplicate of another bug report you submitted not long ago.

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#69188: 30.0.50; project-files + project-find-file is slow in large repositories
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 23 Feb 2024 21:56:03 +0000
Resent-Message-ID: <handler.69188.B69188.170872532911665 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Eli Zaretskii <eliz@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 69233 <at> debbugs.gnu.org, 69188 <at> debbugs.gnu.org
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <86bk8dr0g1.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 18 Feb
 2024 22:18:06 +0200")
References: <iera5o11gnh.fsf@HIDDEN>
 <86y1bhr47o.fsf@HIDDEN>
 <f95199e0-c585-4dea-beb1-305b9cac83f5@HIDDEN>
 <86frxpr1yl.fsf@HIDDEN>
 <391ea08d-9d52-4f03-a602-045b76ac862c@HIDDEN>
 <86bk8dr0g1.fsf@HIDDEN>
Date: Fri, 23 Feb 2024 16:34:38 -0500
Message-ID: <ierh6hy6f0x.fsf_-_@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Eli Zaretskii <eliz@HIDDEN> writes:
>> Date: Sun, 18 Feb 2024 22:11:43 +0200
>> Cc: sbaugh@HIDDEN, 69233 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>> 
>> On 18/02/2024 21:45, Eli Zaretskii wrote:
>> >> Date: Sun, 18 Feb 2024 21:42:37 +0200
>> >> Cc:69233 <at> debbugs.gnu.org
>> >> From: Dmitry Gutov<dmitry@HIDDEN>
>> >>
>> >> On 18/02/2024 20:56, Eli Zaretskii wrote:
>> >>> This is a duplicate of another bug report you submitted not long ago.
>> >> Any reason I didn't receive the first one to my inbox?
>> > I don't have the foggiest, sorry.
>> 
>> It seems Spencer didn't get the confirmation email either, or he 
>> wouldn't resubmit.
>
> One can know if debbugs received a report via the Web interface.

Yes, it seems that all my email was backed up for a day or so, for
whatever reason.  Sorry for the noise.

(Or maybe I just think this is such an important bug that I submitted it
twice :) )

Last modified: Fri, 23 Feb 2024 22:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.