GNU logs - #62837, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 14 Apr 2023 15:38:01 +0000
Resent-Message-ID: <handler.62837.B.168148667128051 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 62837 <at> debbugs.gnu.org
Cc: Dmitry Gutov <dgutov@HIDDEN>
From: Spencer Baugh <sbaugh@HIDDEN>
Date: Fri, 14 Apr 2023 11:37:43 -0400
Message-ID: <ierwn2ecw54.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Received-SPF: pass client-ip=64.215.233.18; envelope-from=sbaugh@HIDDEN;
 helo=mxout5.mail.janestreet.com
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

--=-=-=
Content-Type: text/plain

Tags: patch


When project-files is available, this is a much more efficient
fallback than the current grep fallback.  Ultimately, this is
motivated by making xref-find-references faster by default even in the
absence of an index.

* lisp/cedet/semantic/symref/project.el:
Add.
* lisp/cedet/semantic/symref.el (semantic-symref-tool-alist):
Add project tool

In GNU Emacs 29.0.60 (build 3, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.15.12, Xaw scroll bars) of 2023-03-13 built on
 igm-qws-u22796a
Repository revision: e759905d2e0828eac4c8164b09113b40f6899656
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: CentOS Linux 7 (Core)

Configured using:
 'configure --with-x-toolkit=lucid --with-modules
 --with-gif=ifavailable'


--=-=-=
Content-Type: text/patch
Content-Disposition: attachment;
 filename=0001-Add-a-semantic-symref-backend-which-uses-xref-matche.patch

From 2241f428f0d4809d00f397aafd97270272e966e0 Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@HIDDEN>
Date: Fri, 14 Apr 2023 11:26:49 -0400
Subject: [PATCH] Add a semantic-symref backend which uses
 xref-matches-in-files

When project-files is available, this is a much more efficient
fallback than the current grep fallback.  Ultimately, this is
motivated by making xref-find-references faster by default even in the
absence of an index.

* lisp/cedet/semantic/symref/project.el:
Add.
* lisp/cedet/semantic/symref.el (semantic-symref-tool-alist):
Add project tool
---
 lisp/cedet/semantic/symref.el         |  2 +
 lisp/cedet/semantic/symref/project.el | 73 +++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
 create mode 100644 lisp/cedet/semantic/symref/project.el

diff --git a/lisp/cedet/semantic/symref.el b/lisp/cedet/semantic/symref.el
index 1ebd7ea154b..7dfe892b7e8 100644
--- a/lisp/cedet/semantic/symref.el
+++ b/lisp/cedet/semantic/symref.el
@@ -93,6 +93,8 @@ semantic-symref-tool-alist
        idutils)
      ( (lambda (rootdir) (file-exists-p (expand-file-name "cscope.out" rootdir))) .
        cscope )
+     ( (lambda (rootdir) (project-current nil rootdir)) .
+       project)
     )
   "Alist of tools usable by `semantic-symref'.
 Each entry is of the form:
diff --git a/lisp/cedet/semantic/symref/project.el b/lisp/cedet/semantic/symref/project.el
new file mode 100644
index 00000000000..e822e7a2ba3
--- /dev/null
+++ b/lisp/cedet/semantic/symref/project.el
@@ -0,0 +1,73 @@
+;;; semantic/symref/project.el --- Symref implementation using project and xref  -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2008-2023 Free Software Foundation, Inc.
+
+;; Author: Spencer Baugh <sbaugh@HIDDEN>
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+;; Implement the symref tool API using project-files and
+;; xref-matches-in-files, which in turn use grep or more efficient
+;; tools if available.
+;;
+;; This is basically a replacement for the symref GREP tool, as a new
+;; lowest-common-denominator which works without indices or
+;; project-specific configuration.  It has better performance than the
+;; GREP tool because project-files provides a narrower set of files to
+;; search, and xref-matches-in-files is highly efficient.
+
+(require 'semantic/symref)
+(require 'project)
+(require 'xref)
+
+;;; Code:
+
+;;;###autoload
+(defclass semantic-symref-tool-project (semantic-symref-tool-baseclass) ()
+  "A symref tool implementation using project.el.
+This uses `xref-matches-in-files' over `project-files'")
+
+(cl-defmethod semantic-symref-perform-search ((tool semantic-symref-tool-project))
+  (pcase-let
+      (((eieio
+         searchfor
+         (searchtype (or 'symbol 'regexp))
+         ;; for now, we only really support being called by
+         ;; xref-backend-references, and this is what it passes.
+         (resulttype 'line-and-text))
+        tool))
+    (mapcar
+     (pcase-lambda
+       ((cl-struct xref-match-item
+                  summary (location
+                           (cl-struct xref-file-location file line))))
+       (list line file summary))
+     (xref-matches-in-files searchfor (project-files (project-current))))))
+
+(add-to-list 'semantic-symref-tool-alist
+             '((lambda (rootdir) (project-current nil rootdir))
+               . project))
+
+(provide 'semantic/symref/project)
+
+;; Local variables:
+;; generated-autoload-file: "../loaddefs.el"
+;; generated-autoload-load-name: "semantic/symref/project"
+;; End:
+
+;;; semantic/symref/project.el ends here
-- 
2.30.2


--=-=-=--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Spencer Baugh <sbaugh@HIDDEN>
Subject: bug#62837: Acknowledgement ([PATCH] Add a semantic-symref backend
 which uses xref-matches-in-files)
Message-ID: <handler.62837.B.168148667128051.ack <at> debbugs.gnu.org>
References: <ierwn2ecw54.fsf@HIDDEN>
X-Gnu-PR-Message: ack 62837
X-Gnu-PR-Package: emacs
X-Gnu-PR-Keywords: patch
Reply-To: 62837 <at> debbugs.gnu.org
Date: Fri, 14 Apr 2023 15:38:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 62837 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
62837: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D62837
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 14 Apr 2023 22:39:02 +0000
Resent-Message-ID: <handler.62837.B62837.168151190921565 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Spencer Baugh <sbaugh@HIDDEN>, 62837 <at> debbugs.gnu.org
Feedback-ID: ib1d9465d:Fastmail
Message-ID: <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN>
Date: Sat, 15 Apr 2023 01:38:18 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
References: <ierwn2ecw54.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <ierwn2ecw54.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi!

On 14/04/2023 18:37, Spencer Baugh wrote:
> When project-files is available, this is a much more efficient
> fallback than the current grep fallback.  Ultimately, this is
> motivated by making xref-find-references faster by default even in the
> absence of an index.

It's a clever enough idea, but unfortunately it doesn't look like the 
performance is always improved by this change.

E.g. I have this checkout of gecko-dev (a big project, just for testing: 
https://github.com/mozilla/gecko-dev) which contains different types of 
files: cpp, js, py.

If I do an xref-find-references search with the current code, it 
finishes in around ~0.8s. 'find' is not that slow, actually:

   time find . -type f -name "*.cpp" >/dev/null

reports just 400 ms here.

Whereas with your patch the search, depending on the language (cpp -- 
more files, py -- less files) can take 3 seconds and more.

Why? First of all, project-files returns all files (which are then all 
searched), whereas semantic-symref-filepattern-alist contains a mapping 
from modes to file globs, limiting both the scan and subsequent search 
to those.

Second -- using project-files means we're forced to round-trip the list 
of files names from the first project's stdout, to buffer, then to a 
list of Lisp strings, and then back to another buffer, to use as stdin. 
I have a couple of things planner in the medium term to improve that, 
but some overhead is probably unavoidable (unless we get some new 
primitive that would allow "piping" between process buffers).

Perhaps you could describe your case where you *did* see a significant 
improvement from this patch, and we can discuss the best steps to 
address that.

BTW, at first I figured you're using MacOS (which historically has 
bundled outdated versions of find and grep, with worse performance). But 
apparently not?

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 15 Apr 2023 06:51:01 +0000
Resent-Message-ID: <handler.62837.B62837.168154145231597 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: sbaugh@HIDDEN, 62837 <at> debbugs.gnu.org
Date: Sat, 15 Apr 2023 09:50:36 +0300
Message-Id: <83o7npej0j.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN> (message from
 Dmitry Gutov on Sat, 15 Apr 2023 01:38:18 +0300)
References: <ierwn2ecw54.fsf@HIDDEN>
 <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> Date: Sat, 15 Apr 2023 01:38:18 +0300
> From: Dmitry Gutov <dgutov@HIDDEN>
> 
> On 14/04/2023 18:37, Spencer Baugh wrote:
> > When project-files is available, this is a much more efficient
> > fallback than the current grep fallback.  Ultimately, this is
> > motivated by making xref-find-references faster by default even in the
> > absence of an index.
> 
> It's a clever enough idea, but unfortunately it doesn't look like the 
> performance is always improved by this change.

Maybe we could offer that as optional behavior, turned on by some user
option?  Then people who do experience performance boost could use it.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 15 Apr 2023 12:38:01 +0000
Resent-Message-ID: <handler.62837.B62837.168156225226641 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 62837
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, 62837 <at> debbugs.gnu.org
Received: via spool by 62837-submit <at> debbugs.gnu.org id=B62837.168156225226641
          (code B ref 62837); Sat, 15 Apr 2023 12:38:01 +0000
Received: (at 62837) by debbugs.gnu.org; 15 Apr 2023 12:37:32 +0000
Received: from localhost ([127.0.0.1]:48484 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pnf9o-0006vb-7N
	for submit <at> debbugs.gnu.org; Sat, 15 Apr 2023 08:37:32 -0400
Received: from forward502a.mail.yandex.net ([178.154.239.82]:54468)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dgutov@HIDDEN>) id 1pnf9l-0006vO-FP
 for 62837 <at> debbugs.gnu.org; Sat, 15 Apr 2023 08:37:31 -0400
Received: from mail-nwsmtp-smtp-production-main-84.vla.yp-c.yandex.net
 (mail-nwsmtp-smtp-production-main-84.vla.yp-c.yandex.net
 [IPv6:2a02:6b8:c0f:26a7:0:640:a2d5:0])
 by forward502a.mail.yandex.net (Yandex) with ESMTP id 272035EA52;
 Sat, 15 Apr 2023 15:37:27 +0300 (MSK)
Received: by mail-nwsmtp-smtp-production-main-84.vla.yp-c.yandex.net
 (smtp/Yandex) with ESMTPSA id NbYJ8nUDUOs0-mokcr5nz; 
 Sat, 15 Apr 2023 15:37:26 +0300
X-Yandex-Fwd: 1
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail;
 t=1681562246; bh=5kwzW/WJHfYLOaA7Zcc5o83+io2+k1yX1nQbC1SG+CU=;
 h=In-Reply-To:From:Subject:Message-ID:Cc:References:Date:To;
 b=YUr3OQntP+aGP08c4tpfe8bao/dFuvP3iOFVqkvRxsBz/RpyJ7HrnC51bhXf6mrVA
 V955cHPsAcOawELOfpUv8kpHTpikZwAeghxm/2m+kRsj9aZ/vY9uoYfHqDPfbq0DOY
 T04YprvCM5I5EdLjWdcJPGc5Q4QSZDv1m/PeEK8U=
Authentication-Results: mail-nwsmtp-smtp-production-main-84.vla.yp-c.yandex.net;
 dkim=pass header.i=@yandex.ru
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailauth.nyi.internal (Postfix) with ESMTP id 09F3C27C0054;
 Sat, 15 Apr 2023 08:37:22 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Sat, 15 Apr 2023 08:37:23 -0400
X-ME-Sender: <xms:gpo6ZGLbj-X6x2xSDRLRQ-Du6tKpivn5j-XCOJLIj8Epu34r4x1Syg>
 <xme:gpo6ZOKQqH-je0SBOhYONlp2X4Ovm3-SggxDA7LTf93hH1fPJKpKDYBcZfmVJLV2d
 yUcQ1OOhZB99CBvWE8>
X-ME-Received: <xmr:gpo6ZGsUqVDLy_Um46OboinIfCnUZ-EIEFYQcX7U4DxEutPc-Exg_kpWnvTf-ukm>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdelvddghedvucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegughhuthhovheshigrnhguvgigrdhruheqnecuggftrfgrth
 htvghrnheptdffgeegkeelteevtdekleethfeftdduvdegkedtkedujefhfedtveeftdff
 udevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hguhhtohhvodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqddufeeffeelleeh
 hedvqddvleegjeejjeejiedqughguhhtohhvpeephigrnhguvgigrdhruhesfhgrshhtmh
 grihhlrdgtohhm
X-ME-Proxy: <xmx:gpo6ZLY_3SHjAtg2hNlAzuaNzuRhL1vsXkp-LHzzIBrfaKS4yPi8HA>
 <xmx:gpo6ZNZoff7EX8RhVeY8mTdldKUbyuv5pddZ1WAzq3F5lowmknOllQ>
 <xmx:gpo6ZHCDxMHwSCYK6rlpBiiXNvRPKwkUZJblNWcOqIW9E_kNqvhwTQ>
 <xmx:gpo6ZMxLvCLWB0683rluHBR-cZqCMEDoW9DTtpPJwMpicAKRo0Gue1axk9o>
Feedback-ID: ib1d9465d:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat,
 15 Apr 2023 08:37:21 -0400 (EDT)
Message-ID: <60f84f7d-ce20-f51e-5219-c845ccc7c6bc@HIDDEN>
Date: Sat, 15 Apr 2023 15:37:20 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Content-Language: en-US
References: <ierwn2ecw54.fsf@HIDDEN>
 <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN> <83o7npej0j.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <83o7npej0j.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 15/04/2023 09:50, Eli Zaretskii wrote:
>> Date: Sat, 15 Apr 2023 01:38:18 +0300
>> From: Dmitry Gutov<dgutov@HIDDEN>
>>
>> On 14/04/2023 18:37, Spencer Baugh wrote:
>>> When project-files is available, this is a much more efficient
>>> fallback than the current grep fallback.  Ultimately, this is
>>> motivated by making xref-find-references faster by default even in the
>>> absence of an index.
>> It's a clever enough idea, but unfortunately it doesn't look like the
>> performance is always improved by this change.
> Maybe we could offer that as optional behavior, turned on by some user
> option?  Then people who do experience performance boost could use it.

Sure. That's also possible. But I'd like more info anyway, for example, 
to be able to make the choice about which value of said option should be 
the default.

Or if the scenario with the improvement turns out to be a rare one, 
concentrate on what project.el needs to provide to make it better.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 15 Apr 2023 21:57:02 +0000
Resent-Message-ID: <handler.62837.B62837.168159579326863 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 62837
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 62837 <at> debbugs.gnu.org
Received: via spool by 62837-submit <at> debbugs.gnu.org id=B62837.168159579326863
          (code B ref 62837); Sat, 15 Apr 2023 21:57:02 +0000
Received: (at 62837) by debbugs.gnu.org; 15 Apr 2023 21:56:33 +0000
Received: from localhost ([127.0.0.1]:50239 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1pnnsn-0006zD-4E
	for submit <at> debbugs.gnu.org; Sat, 15 Apr 2023 17:56:33 -0400
Received: from s.wrqvtbkv.outbound-mail.sendgrid.net ([149.72.123.24]:55360)
 by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from
 <bounces+21787432-a32f-62837=debbugs.gnu.org@HIDDEN>)
 id 1pnnsj-0006yz-Un
 for 62837 <at> debbugs.gnu.org; Sat, 15 Apr 2023 17:56:31 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=catern.com;
 h=from:subject:in-reply-to:references:mime-version:to:cc:content-type:
 content-transfer-encoding:cc:content-type:from:subject:to;
 s=s1; bh=07tQshVT/WWFlAUv7JEh1t6vvanp1KE3ek56GSf3BbU=;
 b=AmCmlyhF27OhLEkc0JohpNvp/KVM0NyPiV+SENg/O3scTDPALE4L4MIe79wzUbRn0EUi
 M+P0F3lczoIEAfu2Z6Oohv4kdJW/8ZhLndX3pN3Xtn+CeQdkv2GPFf3w77mBOs69SS08Ch
 h8XjaH88ORM88lquW1leBltLdU6MJEouV4Ll41kDUSF7T3EI+gueiIEbw0JYc/GPI1mVhQ
 AnzeVtbkbHcjpINle1Ov2wVh7mVomI4+020nWPR7dqegdm/O842UNifmLxjvVQ0M14i54k
 k51hFEwpSuGbUv3YS4NeKW+Q+WUlMoh2TCqKea+m8AVHJkf04chvoVuB/3IoCA7g==
Received: by filterdrecv-59cb65cf6d-sfxbf with SMTP id
 filterdrecv-59cb65cf6d-sfxbf-1-643B1D88-6
 2023-04-15 21:56:24.218152664 +0000 UTC m=+4181133.501188792
Received: from earth.catern.com (unknown) by geopod-ismtpd-21 (SG) with ESMTP
 id AaY7TWcbTA-kXfcQ9GzfCA Sat, 15 Apr 2023 21:56:24.128 +0000 (UTC)
X-Comment: SPF check N/A for local connections - client-ip=::1; helo=localhost;
 envelope-from=sbaugh@HIDDEN; receiver=<UNKNOWN> 
Received: from localhost (localhost [IPv6:::1])
 by earth.catern.com (Postfix) with ESMTPSA id AD3A96009C;
 Sat, 15 Apr 2023 17:56:23 -0400 (EDT)
From: sbaugh@HIDDEN
In-Reply-To: <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN> (Dmitry Gutov's
 message of "Sat, 15 Apr 2023 01:38:18 +0300")
References: <ierwn2ecw54.fsf@HIDDEN>
 <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN>
Date: Sat, 15 Apr 2023 21:56:24 +0000 (UTC)
Message-ID: <871qkkn720.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
X-SG-EID: ZgbRq7gjGrt0q/Pjvxk7wM0yQFRdOkTJAtEbkjCkHbL6zzdHd5XYdx9oPpLbhEXktScYfJrqNWiEAtEymSC9fQvqHY0ZRop0Pb3VVdemI5TGx9DgntAE0AF3iwcAKTy6eVh4m88tD9QL0USO1xE0mA1/xyfZz/5RYgsp+xIHzH9wvQ+va+9HKBPlAt+2aXHetn1Erl2A1fhUGrVcgPSJSw==
X-Entity-ID: d/0VcHixlS0t7iB1YKCv4Q==
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Score: 1.2 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: Dmitry Gutov <dgutov@HIDDEN> writes: > Hi! > > On
 14/04/2023
 18:37, Spencer Baugh wrote: >> When project-files is available, this is a
 much more efficient >> fallback than the current grep fallback [...] 
 Content analysis details:   (1.2 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in
 bl.spamcop.net
 [Blocked - see <https://www.spamcop.net/bl.shtml?149.72.123.24>]
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
 [149.72.123.24 listed in wl.mailspike.net]
 0.0 UNPARSEABLE_RELAY      Informational: message has unparseable relay
 lines
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

Dmitry Gutov <dgutov@HIDDEN> writes:
> Hi!
>
> On 14/04/2023 18:37, Spencer Baugh wrote:
>> When project-files is available, this is a much more efficient
>> fallback than the current grep fallback.  Ultimately, this is
>> motivated by making xref-find-references faster by default even in the
>> absence of an index.
>
> It's a clever enough idea, but unfortunately it doesn't look like the
> performance is always improved by this change.
>
> E.g. I have this checkout of gecko-dev (a big project, just for
> testing: https://github.com/mozilla/gecko-dev) which contains
> different types of files: cpp, js, py.
>
> If I do an xref-find-references search with the current code, it
> finishes in around ~0.8s. 'find' is not that slow, actually:
>
>   time find . -type f -name "*.cpp" >/dev/null
>
> reports just 400 ms here.
>
> Whereas with your patch the search, depending on the language (cpp --
> more files, py -- less files) can take 3 seconds and more.
>
> Why? First of all, project-files returns all files (which are then all
> searched), whereas semantic-symref-filepattern-alist contains a
> mapping from modes to file globs, limiting both the scan and
> subsequent search to those.
>
> Second -- using project-files means we're forced to round-trip the
> list of files names from the first project's stdout, to buffer, then
> to a list of Lisp strings, and then back to another buffer, to use as
> stdin. I have a couple of things planner in the medium term to improve
> that, but some overhead is probably unavoidable (unless we get some
> new primitive that would allow "piping" between process buffers).

Yes, this is a very good point.

> Perhaps you could describe your case where you *did* see a significant
> improvement from this patch, and we can discuss the best steps to
> address that.

In short: I have a project.el backend for a large monorepo which has a
project-files backend which returns only the subset of files which are
relevant to work happening in a given clone.  (Generally a user will
have many clones and be doing different work in each one.)  The
relevant-files subset is determined by integration with the build
system.

So running find returns a vast number of files and then searches over
those, whereas running a search over project-files searches a much
smaller number of files.

Regarding your medium-term plans to improve project-files performance -
wildly guessing, but perhaps you have in mind a way to run a subprocess
that outputs the project-files list?  Let's call it
"project-files-process".  And then project-files-process could be piped
to grep instead, for maximum efficiency?  If that was the idea, then my
own backend could certainly have a project-files-process implementation
too, for maximum efficiency.

> BTW, at first I figured you're using MacOS (which historically has
> bundled outdated versions of find and grep, with worse
> performance). But apparently not?

Nope, Linux.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 19 Apr 2023 01:11:01 +0000
Resent-Message-ID: <handler.62837.B62837.168186663825116 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 62837
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: sbaugh@HIDDEN
Cc: Spencer Baugh <sbaugh@HIDDEN>, 62837 <at> debbugs.gnu.org
Received: via spool by 62837-submit <at> debbugs.gnu.org id=B62837.168186663825116
          (code B ref 62837); Wed, 19 Apr 2023 01:11:01 +0000
Received: (at 62837) by debbugs.gnu.org; 19 Apr 2023 01:10:38 +0000
Received: from localhost ([127.0.0.1]:60727 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1powLF-0006X2-Le
	for submit <at> debbugs.gnu.org; Tue, 18 Apr 2023 21:10:38 -0400
Received: from forward501c.mail.yandex.net ([178.154.239.209]:59462)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dgutov@HIDDEN>) id 1powLA-0006Wl-EC
 for 62837 <at> debbugs.gnu.org; Tue, 18 Apr 2023 21:10:35 -0400
Received: from mail-nwsmtp-smtp-production-canary-88.sas.yp-c.yandex.net
 (mail-nwsmtp-smtp-production-canary-88.sas.yp-c.yandex.net
 [IPv6:2a02:6b8:c08:1d21:0:640:c9e4:0])
 by forward501c.mail.yandex.net (Yandex) with ESMTP id 409745F1B4;
 Wed, 19 Apr 2023 04:10:30 +0300 (MSK)
Received: by mail-nwsmtp-smtp-production-canary-88.sas.yp-c.yandex.net
 (smtp/Yandex) with ESMTPSA id RAR4N3HDdqM0-yBW1moyX; 
 Wed, 19 Apr 2023 04:10:29 +0300
X-Yandex-Fwd: 1
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail;
 t=1681866629; bh=uw0KeozqTZ3gtcmDDKO3ttxfDy8f+sDeQtgmPNgA8XM=;
 h=In-Reply-To:From:Subject:Message-ID:Cc:References:Date:To;
 b=hCjHsYPonpqkHcoS/sdwJf8Q6DExncQcCaAsBi7mx38M8EXz3pKGFh0Ao/JBSj2ot
 Q3nYjahvjEeq/lc9Gcy58h3FWa6OLJ04UJ1D6oE6xjdwziZXz3gU9axHE41sgLpimE
 0l8e/Ote3wH+DoMYu4/sgrQJKOKI/jdefnM0Kn4M=
Authentication-Results: mail-nwsmtp-smtp-production-canary-88.sas.yp-c.yandex.net;
 dkim=pass header.i=@yandex.ru
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailauth.nyi.internal (Postfix) with ESMTP id 6333027C005B;
 Tue, 18 Apr 2023 21:10:27 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute3.internal (MEProxy); Tue, 18 Apr 2023 21:10:27 -0400
X-ME-Sender: <xms:gj8_ZO9rIXbdXRrL17cjw28iwd-OKftQcqfml6HOAG4q6GS78dBiPQ>
 <xme:gj8_ZOvKfV2KOTjIRgMWpVPCjhcG0hTmu690d7UaB2Nffrtx4EQ3-vi3do6NO5RqI
 3BIfdIXkK0mTVZWFMU>
X-ME-Received: <xmr:gj8_ZEDg4qS4jGyR2LBh4b5pcpix-RBNdD1rHttPM4l_YhvU7xKX-sI6b1OGJzc>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdelledggeegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegughhuthhovheshigrnhguvgigrdhruheqnecuggftrfgrth
 htvghrnheptdffgeegkeelteevtdekleethfeftdduvdegkedtkedujefhfedtveeftdff
 udevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hguhhtohhvodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqddufeeffeelleeh
 hedvqddvleegjeejjeejiedqughguhhtohhvpeephigrnhguvgigrdhruhesfhgrshhtmh
 grihhlrdgtohhm
X-ME-Proxy: <xmx:gj8_ZGdn9oly2nInZYUmKWJOPpCyD0UESuK1iLhAyEEOQb_G2xdpMA>
 <xmx:gj8_ZDNVCAAma6cksITmrA8UsrNvGnwKo9hUl-D_IU73UUuBnjlPCQ>
 <xmx:gj8_ZAm1AsbBO7va9bZRCdfQn-i_2qJRvmWXf2nAkD8CYFSRK4XcIw>
 <xmx:gz8_ZJ1YNtmx43UyAhJ5RsbqpyHWH5m-ijqN9aLPtVNs4eFQVB2Cc1ABovs>
Feedback-ID: ib1d9465d:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 18 Apr 2023 21:10:26 -0400 (EDT)
Message-ID: <37ee089e-5c42-15c6-c8c1-48642bf4e180@HIDDEN>
Date: Wed, 19 Apr 2023 04:10:24 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Content-Language: en-US
References: <ierwn2ecw54.fsf@HIDDEN>
 <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN> <871qkkn720.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
In-Reply-To: <871qkkn720.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -1.1 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.1 (--)

On 16/04/2023 00:56, sbaugh@HIDDEN wrote:

>> Perhaps you could describe your case where you *did* see a significant
>> improvement from this patch, and we can discuss the best steps to
>> address that.
> 
> In short: I have a project.el backend for a large monorepo which has a
> project-files backend which returns only the subset of files which are
> relevant to work happening in a given clone.  (Generally a user will
> have many clones and be doing different work in each one.)  The
> relevant-files subset is determined by integration with the build
> system.
> 
> So running find returns a vast number of files and then searches over
> those, whereas running a search over project-files searches a much
> smaller number of files.

Neat.

> Regarding your medium-term plans to improve project-files performance -
> wildly guessing, but perhaps you have in mind a way to run a subprocess
> that outputs the project-files list?  Let's call it
> "project-files-process".  And then project-files-process could be piped
> to grep instead, for maximum efficiency?  If that was the idea, then my
> own backend could certainly have a project-files-process implementation
> too, for maximum efficiency.

That might be step number 3, although I'm not sure yet which kind of 
code will be required for the piping to be done efficiently enough.

The other two things I was looking at are:

- Use relative file names (less text to parse, memory to allocate, GC to 
thrash). The awkward part is how to merge that with the idea that 
project-files can include files from directories ("external roots"). 
Split those off into a different method? Treat them as separate projects 
to flat-map the lists of files at?

- Add arguments to allow filtering the files using the underlying tool. 
That can also result is much fewer files to parse in the output under 
suitable circumstances (e.g. we'd be able to pass a list of globs here).

There is one implementation of the second item in the branch 
scratch/etags-regen.

And both items need to be done carefully enough to maintain some 
backward compatibility.

So unless you're in a hurry, give me a few weeks to get around to this.

Further suggestions and patches are welcome, of course.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#62837: [PATCH] Add a semantic-symref backend which uses xref-matches-in-files
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 19 Apr 2023 01:27:01 +0000
Resent-Message-ID: <handler.62837.B62837.168186758826793 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 62837
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: patch
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: sbaugh@HIDDEN, 62837 <at> debbugs.gnu.org
Received: via spool by 62837-submit <at> debbugs.gnu.org id=B62837.168186758826793
          (code B ref 62837); Wed, 19 Apr 2023 01:27:01 +0000
Received: (at 62837) by debbugs.gnu.org; 19 Apr 2023 01:26:28 +0000
Received: from localhost ([127.0.0.1]:60748 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1powaY-0006y4-BG
	for submit <at> debbugs.gnu.org; Tue, 18 Apr 2023 21:26:28 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:35583)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1powaT-0006xl-8N
 for 62837 <at> debbugs.gnu.org; Tue, 18 Apr 2023 21:26:25 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <37ee089e-5c42-15c6-c8c1-48642bf4e180@HIDDEN> (Dmitry Gutov's
 message of "Wed, 19 Apr 2023 04:10:24 +0300")
References: <ierwn2ecw54.fsf@HIDDEN>
 <5e6eddd5-4b38-5765-05f3-dd6c1927edd3@HIDDEN>
 <871qkkn720.fsf@HIDDEN>
 <37ee089e-5c42-15c6-c8c1-48642bf4e180@HIDDEN>
Date: Tue, 18 Apr 2023 21:26:13 -0400
Message-ID: <ierbkjky85m.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dgutov@HIDDEN> writes:
> On 16/04/2023 00:56, sbaugh@HIDDEN wrote:
>
>>> Perhaps you could describe your case where you *did* see a significant
>>> improvement from this patch, and we can discuss the best steps to
>>> address that.
>> In short: I have a project.el backend for a large monorepo which has
>> a
>> project-files backend which returns only the subset of files which are
>> relevant to work happening in a given clone.  (Generally a user will
>> have many clones and be doing different work in each one.)  The
>> relevant-files subset is determined by integration with the build
>> system.
>> So running find returns a vast number of files and then searches
>> over
>> those, whereas running a search over project-files searches a much
>> smaller number of files.
>
> Neat.
>
>> Regarding your medium-term plans to improve project-files performance -
>> wildly guessing, but perhaps you have in mind a way to run a subprocess
>> that outputs the project-files list?  Let's call it
>> "project-files-process".  And then project-files-process could be piped
>> to grep instead, for maximum efficiency?  If that was the idea, then my
>> own backend could certainly have a project-files-process implementation
>> too, for maximum efficiency.
>
> That might be step number 3, although I'm not sure yet which kind of
> code will be required for the piping to be done efficiently enough.
>
> The other two things I was looking at are:
>
> - Use relative file names (less text to parse, memory to allocate, GC
>   to thrash). The awkward part is how to merge that with the idea that
>   project-files can include files from directories ("external
>   roots"). Split those off into a different method? Treat them as
>   separate projects to flat-map the lists of files at?
>
> - Add arguments to allow filtering the files using the underlying
>   tool. That can also result is much fewer files to parse in the
>   output under suitable circumstances (e.g. we'd be able to pass a
>  list of globs here).
>
> There is one implementation of the second item in the branch
> scratch/etags-regen.
>
> And both items need to be done carefully enough to maintain some
> backward compatibility.
>
> So unless you're in a hurry, give me a few weeks to get around to this.
>
> Further suggestions and patches are welcome, of course.

I'm in no hurry.  I will probably add this backend locally at my site in
the meantime.  We have no existing (non-trivial) xref-find-references
backend, so speeding this one up isn't too urgent (it's not competing
with anything), but definitely I am interested in project-files (and
project.el in general) speed improvements and will try to help out as it
becomes relevant.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 11 Sep 2023 23:24:10 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Sep 11 19:24:10 2023
Received: from localhost ([127.0.0.1]:55141 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfqGI-0005f9-0e
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 19:24:10 -0400
Received: from mail-lj1-x22b.google.com ([2a00:1450:4864:20::22b]:44374)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <stefankangas@HIDDEN>) id 1qfqGC-0005eF-Uu
 for control <at> debbugs.gnu.org; Mon, 11 Sep 2023 19:24:08 -0400
Received: by mail-lj1-x22b.google.com with SMTP id
 38308e7fff4ca-2bf78950354so41236741fa.1
 for <control <at> debbugs.gnu.org>; Mon, 11 Sep 2023 16:24:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20221208; t=1694474635; x=1695079435; darn=debbugs.gnu.org;
 h=to:subject:message-id:date:mime-version:from:from:to:cc:subject
 :date:message-id:reply-to;
 bh=xHVLgS0gzvADaq9tyk2g47L0bcT5Ug4bpL/675tqMu0=;
 b=JP4nVcl0ssRaIpI3udjm3ZKXxIBxPK7bJ9OyoPfNepfn5Anqd4KtkEZkbB1GVyfgKX
 aCSXbBKZyWbltSrTN/Uy3Dvtm6XVuKAkhUB0gFF77t2Ayx+zR1lOaDt+Ctd9r7GZplvc
 gjFgpInPrus0PnlR4RKXysa9ne+AQPY8xBEO/IdUrNnCW3zPPAl0j8xaXAiV3H9xnENx
 UXUlj5ydc5bPooaxq4FawxBghX5EzhjCL5iXTXfoy/V956FxivLe5QsFqyuehDtY+83w
 DmvvJKAt7Dzn/GH5Y2F4LjB8RRn2m5MTB8ggma0H0IE4/b5u3wvWUIXENP6/Ks9/a6Sf
 Gkcw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1694474635; x=1695079435;
 h=to:subject:message-id:date:mime-version:from:x-gm-message-state
 :from:to:cc:subject:date:message-id:reply-to;
 bh=xHVLgS0gzvADaq9tyk2g47L0bcT5Ug4bpL/675tqMu0=;
 b=cmr3dC71/Zxgp6A/ARgy4JVaemWbbOFq09P5haZUloBARR+oq/lvIsymUrrV+cyASr
 Fn5AH4flrNBTSh81+fIqWi3nfP8t24b8pGZC5ebDP4zAReI6+hHHaGyAsRj8Ohue8gKi
 t31M9qi1zbg3qfFEVkY0VSze7jhyRTWobctY3hSsQU9EL6skOJqo1Z5YTMObL5MDS1w/
 0VqZgaTEa5OcmBLSW47PC9AJaX9BAs+2EZZh7wnHcMhfLPfxQDAKYlQ5yVWFFrU2kBU3
 RGT7Yar7BSk+KLF6yutwlqQxdBq8N2/kne89CYqPV7P5V/1fo1no3t8h7nKAtv+uw/9h
 xtvg==
X-Gm-Message-State: AOJu0Yw3suFWk7F3Qhcxzw+16yBCSMjly88CMowkYe3krGx0SpQ12cSm
 hvu0YGz0/yeqqO50b2goyTb1Y3k0C6qi/Mq3RWzMR8/u
X-Google-Smtp-Source: AGHT+IEd9ooe5KLDE8WX/AKlHRcWX2rBxfMk/SC40xZv6s3Kr8n7ldv3tHO4IJgTqC4sExdF/UrpPGgT5YvHXxv0JxM=
X-Received: by 2002:a2e:9012:0:b0:2bc:db5a:9540 with SMTP id
 h18-20020a2e9012000000b002bcdb5a9540mr9112497ljg.42.1694474634630; Mon, 11
 Sep 2023 16:23:54 -0700 (PDT)
Received: from 753933720722 named unknown by gmailapi.google.com with
 HTTPREST; Mon, 11 Sep 2023 16:23:54 -0700
From: Stefan Kangas <stefankangas@HIDDEN>
MIME-Version: 1.0
Date: Mon, 11 Sep 2023 16:23:54 -0700
Message-ID: <CADwFkmknv2672Uu=kyCmw8hpPFac6us7P4bPwG7nexp3sJRj3Q@HIDDEN>
Subject: control message for bug #62837
To: control <at> debbugs.gnu.org
Content-Type: text/plain; charset="UTF-8"
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

severity 62837 wishlist
quit

Last modified: Mon, 11 Sep 2023 23:30:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.