GNU logs - #50733, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 22 Sep 2021 09:31:02 +0000
Resent-Message-ID: <handler.50733.B.163230301223794 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 50733 <at> debbugs.gnu.org
From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Date: Wed, 22 Sep 2021 11:27:47 +0200
Message-ID: <m1h7edq7sc.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
References: <m1h7edq7sc.fsf.ref@HIDDEN>
Content-Length: 958
Received-SPF: pass client-ip=77.238.177.58; envelope-from=mardani29@HIDDEN;
 helo=sonic310-37.consmr.mail.ir2.yahoo.com
X-Spam_score_int: 1
X-Spam_score: 0.1
X-Spam_bar: /
X-Spam_report: (0.1 / 5.0 requ) BAYES_20=-0.001, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

When trying to search in a big project with thousands of files using
`project-find-regexp`, Emacs can synchronously block for a long time
until the results are computed for display in an *xref* buffer.

In contrast, the rgrep command and friends create a `grep-mode` buffer
and asynchronously fill it with results as they are computed by the grep
tool.  Emacs is not blocked and the user can start exploring matches
right away.

Is there a plan to eventually replace `grep-mode` with `xref-mode` in
Emacs?  If so, I think that computing results asynchronously is the most
important feature we'll lose.  When using `xref-mode` with an index
(etags, for example), the blocking is not noticeable; but when using a
grep tool the delay can be annoying if the repository is huge.

Another idea could be to use `grep-mode` for `project-find-regexp`, just
like `grep`, `lgrep`, `rgrep` and so one, but there's probably a reason
why it's using `xref-mode`.

Thanks.

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Subject: bug#50733: Acknowledgement (28.0.1; project-find-regexp can block
 Emacs for a long time)
Message-ID: <handler.50733.B.163230301223794.ack <at> debbugs.gnu.org>
References: <m1h7edq7sc.fsf@HIDDEN>
X-Gnu-PR-Message: ack 50733
X-Gnu-PR-Package: emacs
Reply-To: 50733 <at> debbugs.gnu.org
Date: Wed, 22 Sep 2021 09:31:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 50733 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
50733: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D50733
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 22 Sep 2021 19:10:01 +0000
Resent-Message-ID: <handler.50733.B50733.163233779417711 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>, 50733 <at> debbugs.gnu.org
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
Date: Wed, 22 Sep 2021 22:09:43 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <m1h7edq7sc.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Daniel,

On 22.09.2021 12:27, Daniel Martín via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> 
> When trying to search in a big project with thousands of files using
> `project-find-regexp`, Emacs can synchronously block for a long time
> until the results are computed for display in an *xref* buffer.
> 
> In contrast, the rgrep command and friends create a `grep-mode` buffer
> and asynchronously fill it with results as they are computed by the grep
> tool.  Emacs is not blocked and the user can start exploring matches
> right away.
> 
> Is there a plan to eventually replace `grep-mode` with `xref-mode` in
> Emacs?

No solid plan to replace everything (a lot of people will object, for 
various reasons), but it's the direction I want to see. At least in a 
core set of IDE-ish commands.

> If so, I think that computing results asynchronously is the most
> important feature we'll lose.  When using `xref-mode` with an index
> (etags, for example), the blocking is not noticeable; but when using a
> grep tool the delay can be annoying if the repository is huge.

Non-blocking UI for Xref is possible, but it requires a fair amount of 
work, possibly some low-level one. Not in the cards for Emacs 28, at least.

But maybe we could improve things another way first.

IIRC you are using macOS. I received another report recently that 
find/grep based tooling, and project-find-regexp in particular, are 
pretty slow on that OS.

When you say "block for a long time", how long are we talking about?

For reference, when a do a search across Emacs sources checkout, and 
there are few matches, it returns in ~100ms (with warm cache and SSD 
disk, to be clear).

To try it, evaluate

   (benchmark 1 '(project-find-regexp "new-collection"))

And a similar search across the Linux kernel source tree takes ~1s.

The first thing to try is to either install the latest GNU Grep and make 
sure Emacs uses it, or install ripgrep and

   (setq xref-search-program 'ripgrep)

Please report if that helps and how much.

Another benchmark to try is

   (benchmark 1 '(project-files (project-current)))

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 22 Sep 2021 22:01:01 +0000
Resent-Message-ID: <handler.50733.B50733.163234805917201 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
Date: Wed, 22 Sep 2021 23:58:42 +0200
In-Reply-To: <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> (Dmitry Gutov's
 message of "Wed, 22 Sep 2021 22:09:43 +0300")
Message-ID: <m1fstwl1bh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (darwin)
MIME-Version: 1.0
Content-Type: text/plain
Content-Length: 2918
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Dmitry Gutov <dgutov@HIDDEN> writes:
>
> IIRC you are using macOS. I received another report recently that
> find/grep based tooling, and project-find-regexp in particular, are 
> pretty slow on that OS.

Yes, this is on macOS.

>
> When you say "block for a long time", how long are we talking about?
>>
> To try it, evaluate
>
>   (benchmark 1 '(project-find-regexp "new-collection"))

I usually work on a monorepo with ~67000 tracked files (many of them big
binary files).  Here's what I get when using ripgrep as the xref search
program:

Elapsed time: 36.087181s (8.067474s in 22 GCs)

Running the same search with ripgrep from the command line takes around
6 seconds.

>
> Another benchmark to try is
>
>   (benchmark 1 '(project-files (project-current)))

Elapsed time: 1.590223s (0.432372s in 1 GCs)

Here's an ELisp profile of the first benchmark:

        8696  78% - command-execute
        8696  78%  - call-interactively
        8493  76%   - funcall-interactively
        8480  76%    - eval-expression
        8479  76%     - eval
        8479  76%      - project-find-regexp
        8227  74%       - xref--show-xrefs
        8227  74%        - xref--show-xref-buffer
        5584  50%         - #<compiled 0x140b5a40100bafc6>
        5584  50%          - apply
        5584  50%           - project--find-regexp-in-files
        5574  50%            - xref-matches-in-files
        3016  27%             - xref--convert-hits
        3000  27%              - mapcan
        2992  27%               - #<compiled -0x6cdcd56218925c3>
        2734  24%                - xref--collect-matches
        2094  18%                 - xref--collect-matches-1
         800   7%                  + xref-make-match
         774   7%                  + xref-make-file-location
         104   0%                   xref--find-file-buffer
          80   0%                   file-remote-p
          51   0%                   xref--regexp-syntax-dependent-p
         906   8%             + xref--process-file-region
         331   2%               sort
        1413  12%         + xref--analyze
        1230  11%         + xref--show-common-initialize
         249   2%       + project-files
           3   0%       + project-current
           9   0%    + minibuffer-complete
           4   0%    + execute-extended-command
         203   1%   + byte-code
        2314  20% - ...
        2314  20%    Automatic GC
          27   0% + timer-event-handler

The search time is reduced when I use a more specific search term,
presumably because the number of results is lower and the Elisp
post-processing takes less time.  Here's what I got, for example, when I
search for something with results from only one file:

Elapsed time: 6.859815s (0.864738s in 2 GCs)

Compared to the time taken by the same query from the command line
(6.5s) shows that the Elisp post-processing time is probably negligible
in this scenario.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 22 Sep 2021 23:10:02 +0000
Resent-Message-ID: <handler.50733.B50733.163235216613817 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163235216613817
          (code B ref 50733); Wed, 22 Sep 2021 23:10:02 +0000
Received: (at 50733) by debbugs.gnu.org; 22 Sep 2021 23:09:26 +0000
Received: from localhost ([127.0.0.1]:52134 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTBMk-0003am-1c
	for submit <at> debbugs.gnu.org; Wed, 22 Sep 2021 19:09:26 -0400
Received: from mail-wr1-f51.google.com ([209.85.221.51]:43694)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTBMi-0003aa-6k
 for 50733 <at> debbugs.gnu.org; Wed, 22 Sep 2021 19:09:25 -0400
Received: by mail-wr1-f51.google.com with SMTP id w17so11507142wrv.10
 for <50733 <at> debbugs.gnu.org>; Wed, 22 Sep 2021 16:09:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=pSfkgWcZdSX3yJlvAs0FLQTzea83j3Gy8sBJoFSXYno=;
 b=fmLk16HBhsSQWkWqE1PpCd9feYZFYmRsBS8t+DTGUXgCQwIriGrF+5X9lQXjIBpPgl
 jZv35eVbJ7sCk/Bjm4zIZY0v+R6QUzqYRX+v76TlV/0F0+nwG5u6vr46gqua0zplpkJF
 yOEhPLggJA95Xj/BqcBi4H8qWVybs5qhd139uE9fybODgKNzIvgIhoR4sr540SuaTOeG
 z02D9b+3GVCXjexh5uHV2p0lv1c8RZFqhEOOrvctIWLexQN9f6TdfX+60VgoVU+s2Jvq
 eXlHwimF35utUWI7JfCVv+GNv2mge6lEgJyPC9BkZnpEPFAd947op6IPwzs0OZIHhY1j
 Z0YQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=pSfkgWcZdSX3yJlvAs0FLQTzea83j3Gy8sBJoFSXYno=;
 b=08cEXLdh0zsa54D45idgPT+4MPmqYuhMz7xJuS+3E5LM96MH9qKmovclsnB0d0O6/F
 IwMFXM+1lwefH0s5kd+BTNppyuS0Odh5IeK1m1WQtEo2njTisSy6P2hRhEo5weY3zx9V
 2H3gWnz2vfCV4QsAxjoahFIetOXFVJOar8SnGfY1Tkto57yag00yeKY2jf5Mir9K4dhH
 IUm0OE6JimVWGiFMR8M24WP63JG2Rd1pP6+HyVYA9Aqh7RwJ+dP6XQK4AJnHBQHENPM2
 ftMRQxAEh7poDKPwgX1aY1CTd8OmQi4tcwMHWyrHAgctHM2EJbEpPAWK1sidJ3EHXrwT
 /zbw==
X-Gm-Message-State: AOAM531/bLA2J8Mpijkie2gph8c1UkUwwQ7+Us0x8CY4ZmZ37/p+fAOe
 GLhvmB+6sV85PGxlL66GF2Z5JdJaOio=
X-Google-Smtp-Source: ABdhPJz2WT05CLNkiBKASL1uEkmjDpnP93Pnc38FFEyi8i/gKSuAa2CvKPQpvycle7VdtJccafskuA==
X-Received: by 2002:a5d:4bc8:: with SMTP id l8mr1585917wrt.105.1632352158490; 
 Wed, 22 Sep 2021 16:09:18 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id y9sm8488402wmj.36.2021.09.22.16.09.17
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Wed, 22 Sep 2021 16:09:17 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
Date: Thu, 23 Sep 2021 02:09:16 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <m1fstwl1bh.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 23.09.2021 00:58, Daniel Martín wrote:
> Dmitry Gutov <dgutov@HIDDEN> writes:
>>
>> IIRC you are using macOS. I received another report recently that
>> find/grep based tooling, and project-find-regexp in particular, are
>> pretty slow on that OS.
> 
> Yes, this is on macOS.
> 
>>
>> When you say "block for a long time", how long are we talking about?
>>>
>> To try it, evaluate
>>
>>    (benchmark 1 '(project-find-regexp "new-collection"))
> 
> I usually work on a monorepo with ~67000 tracked files (many of them big
> binary files).  Here's what I get when using ripgrep as the xref search
> program:
> 
> Elapsed time: 36.087181s (8.067474s in 22 GCs)

Thanks for testing. Did the switch to ripgrep help much?

I wonder if we should advertise this setting and recommendation more 
prominently, at least until we get auto-detection.

> Running the same search with ripgrep from the command line takes around
> 6 seconds.

Is that with an SSD?

Your project sounds respectable. The torvalds-linux repo I have checked 
out here is also 70000 files, but I guess your files are bigger.

>> Another benchmark to try is
>>
>>    (benchmark 1 '(project-files (project-current)))
> 
> Elapsed time: 1.590223s (0.432372s in 1 GCs)

That's a while (I wonder if you find 'project-find-file' usable with 
this kind of performance), but still better than I might have expected.

> Here's an ELisp profile of the first benchmark:
> 
>          8696  78% - command-execute
>          8696  78%  - call-interactively
>          8493  76%   - funcall-interactively
>          8480  76%    - eval-expression
>          8479  76%     - eval
>          8479  76%      - project-find-regexp
>          8227  74%       - xref--show-xrefs
>          8227  74%        - xref--show-xref-buffer
>          5584  50%         - #<compiled 0x140b5a40100bafc6>
>          5584  50%          - apply
>          5584  50%           - project--find-regexp-in-files
>          5574  50%            - xref-matches-in-files
>          3016  27%             - xref--convert-hits
>          3000  27%              - mapcan
>          2992  27%               - #<compiled -0x6cdcd56218925c3>
>          2734  24%                - xref--collect-matches
>          2094  18%                 - xref--collect-matches-1
>           800   7%                  + xref-make-match
>           774   7%                  + xref-make-file-location
>           104   0%                   xref--find-file-buffer
>            80   0%                   file-remote-p
>            51   0%                   xref--regexp-syntax-dependent-p
>           906   8%             + xref--process-file-region
>           331   2%               sort
>          1413  12%         + xref--analyze
>          1230  11%         + xref--show-common-initialize
>           249   2%       + project-files
>             3   0%       + project-current
>             9   0%    + minibuffer-complete
>             4   0%    + execute-extended-command
>           203   1%   + byte-code
>          2314  20% - ...
>          2314  20%    Automatic GC
>            27   0% + timer-event-handler

When you have a lot of matches, at some point Lisp overhead is going to 
show up. E.g., the searches seem almost instantaneous with up to several 
thousand matches here, but 10000s and 100000s - yeah, I have to wait.

Help with optimizations in that area (around/in xref-matches-in-files 
and xref--convert-hits) is welcome, but I'm not sure how much more we 
can squeeze.

> The search time is reduced when I use a more specific search term,
> presumably because the number of results is lower and the Elisp
> post-processing takes less time.  Here's what I got, for example, when I
> search for something with results from only one file:
> 
> Elapsed time: 6.859815s (0.864738s in 2 GCs)
> 
> Compared to the time taken by the same query from the command line
> (6.5s) shows that the Elisp post-processing time is probably negligible
> in this scenario.

It's a good result. A little suspicious, though: given that 
project-find-regexp calls project-files first, and the latter takes 
1.5s, the difference should ~ that time. But I guess rg also needs to 
traverse the directory tree, and spends some time on doing that too.

What else can be done -- again, if someone wants to investigate an 
asynchronous/nonblocking API for Xref (or using threads) -- welcome. The 
case when most of the time is spent in the subprocess is a good match. 
But I don't think we'll manage this for the upcoming release.

Another thing you can do is set up the additional ignores for the 
project. If those big binary files are not something you are interested 
in searching and touching, you could add ignore entries for them. When 
the vc project backend is in use (default), it is currently done via 
.dir-locals.el: the variable is project-vc-ignores, it's a list of 
strings that should be globs. See its docstring and the explanation in 
project-ignores's docstring.

Note that ignores also affect project-find-file.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 06:14:01 +0000
Resent-Message-ID: <handler.50733.B50733.163237762227109 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163237762227109
          (code B ref 50733); Thu, 23 Sep 2021 06:14:01 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 06:13:42 +0000
Received: from localhost ([127.0.0.1]:52481 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTHzK-00073B-D6
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 02:13:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47974)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTHzI-00072x-F1
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 02:13:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34504)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTHzD-0003XG-40; Thu, 23 Sep 2021 02:13:35 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4812
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTHzB-00085K-Tx; Thu, 23 Sep 2021 02:13:35 -0400
Date: Thu, 23 Sep 2021 09:13:33 +0300
Message-Id: <83wnn7vmya.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <m1fstwl1bh.fsf@HIDDEN> (bug-gnu-emacs@HIDDEN)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 50733 <at> debbugs.gnu.org
> Date: Wed, 22 Sep 2021 23:58:42 +0200
> From:  Daniel Mart�n via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN>
> 
> > When you say "block for a long time", how long are we talking about?
> >>
> > To try it, evaluate
> >
> >   (benchmark 1 '(project-find-regexp "new-collection"))
> 
> I usually work on a monorepo with ~67000 tracked files (many of them big
> binary files).  Here's what I get when using ripgrep as the xref search
> program:
> 
> Elapsed time: 36.087181s (8.067474s in 22 GCs)
> 
> Running the same search with ripgrep from the command line takes around
> 6 seconds.

How many matches does that report?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 17:42:02 +0000
Resent-Message-ID: <handler.50733.B50733.1632418891904 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.1632418891904
          (code B ref 50733); Thu, 23 Sep 2021 17:42:02 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 17:41:31 +0000
Received: from localhost ([127.0.0.1]:55787 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTSiw-0000EW-VZ
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 13:41:31 -0400
Received: from mail-wr1-f45.google.com ([209.85.221.45]:35598)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTSiv-0000EH-6d
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 13:41:29 -0400
Received: by mail-wr1-f45.google.com with SMTP id i23so19546249wrb.2
 for <50733 <at> debbugs.gnu.org>; Thu, 23 Sep 2021 10:41:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:from:to:cc:references:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=fip419H7bM7NyaIUa+CiKSnGptwJ5z+QemFjEQr4pEc=;
 b=B7cYhHgdW4lcg/DwJPkqS4ziInRGYQw4hff5beWVcCxJ1xxejWFY3PQqkdjJzQftDr
 TxhHDMWVG7MEUqLyFXl+GiP9U8nNO9YvQCUnHkt9e+aQwwFKHyeRrgIhWL8JLPnopf1W
 xjVN7CESsNW1QN2pMN2VXo/o2x2INUhEb/tgtrRaD0ioCGeseigoFVVTRaHYM0KKAubX
 s9hRhUL055iv6U6AqhTLPrclw5bov0fLDsgaMNy8epQBqDkLCcfX7t2+b9ElJ4JSMiEw
 iVTmIBRFtaW5Y4ehbxFy6uuy2Nxfzd9LofEZBsfnvBcFsbSZmafqCeiTLTXHrZfAfJLK
 aQnw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:from:to:cc:references:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=fip419H7bM7NyaIUa+CiKSnGptwJ5z+QemFjEQr4pEc=;
 b=lHpeOgddY4kIGETyzxd6eXoQ/4dl6hwRWt3mipIK5u3ABGxT39gUBG9Xps4AaN/82L
 YDCl0lXU8/LMWeVtvFF28D0s3GB1Zg99U+NR2Et+VhOZ/pbW66Bju+Pm+X+KfcIOnK4R
 enMD8iXJkYQl7R0jEHAh4PC1/C/1s1AvK+g19aUugo3wsWjsAX5zj5Zcsx8UNwfppgVd
 lv+tDrs+dJUbfRUJEiEh1L/CTwJqAKR32R1RQS0Dpk+94D0GTDfCiYd6mLkE9XLKDdB8
 sxUaOJaNtB41rD9eUah+SxG8UOVE5NhB+Yv8LXWT9jgE5gwi2QnImSYGb18PuK9NlFGs
 7EjA==
X-Gm-Message-State: AOAM530Y2xfTlPQOKNekJsxqhuIg0u9cji72uUxfypyhWumwd66nbnZY
 yognWfMbCKhExEcrFaJzG7fh3gQLV/Q=
X-Google-Smtp-Source: ABdhPJwDzRAP95UF++FixF7ySWtpNHHTRm5G0xdQrnLvJsP9lNB9SE97fPbDOOQdsglNQ+8FCW4CxA==
X-Received: by 2002:a1c:f21a:: with SMTP id s26mr5978758wmc.117.1632418883288; 
 Thu, 23 Sep 2021 10:41:23 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id z12sm5927724wro.75.2021.09.23.10.41.22
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Thu, 23 Sep 2021 10:41:22 -0700 (PDT)
From: Dmitry Gutov <dgutov@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
Message-ID: <042972ce-31d7-7bac-0eef-287bec89a560@HIDDEN>
Date: Thu, 23 Sep 2021 20:41:21 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 23.09.2021 02:09, Dmitry Gutov wrote:
> Help with optimizations in that area (around/in xref-matches-in-files 
> and xref--convert-hits) is welcome, but I'm not sure how much more we 
> can squeeze.

Meanwhile, I've pushed a couple of tweaks in a different place (the 
printing code) that should reduce the overhead somewhat.

Maybe turning those 36 seconds into 30 or even 20, depending on the 
average number of matches per file.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 20:43:02 +0000
Resent-Message-ID: <handler.50733.B50733.163242976628942 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163242976628942
          (code B ref 50733); Thu, 23 Sep 2021 20:43:02 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 20:42:46 +0000
Received: from localhost ([127.0.0.1]:55968 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTVYM-0007Wk-Dj
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 16:42:46 -0400
Received: from sonic302-21.consmr.mail.ir2.yahoo.com ([87.248.110.84]:42349)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mardani29@HIDDEN>) id 1mTVYK-0007WU-Rc
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 16:42:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s2048;
 t=1632429759; bh=xjYZcmSxyZndyanPLEmX4N4ICcsxZyHkGOOqTBrQRMs=;
 h=From:To:Cc:Subject:References:Date:In-Reply-To:From:Subject:Reply-To;
 b=S9gGsjsAge9jlpIvtSWCjt7e8gymPRF0fwN+pnxFHCabJLbr29zxzYiPqBuF0FDbcVH2vQ1y4keDQAWBFYH2lRV+H5oMx09IhqNF+GkeJ9rA3X3b9BYUZflWJ1b4gfNWMtdeLhMieXDqkxrxMiNVydbdKCxJfpsaNxRojbySJN3X8xQiHiwdUFiuZoZ1eag/AXm+NOqlE7ojYGPkuaH4p2j0Il11VZGGZCCobOR9PrdHOnnvhvh/UGkhRf40EImPcmrzBv0iOaMuoLv/z5wY+Mt7dbSFdiXKyTZcDw0HHddaPYVGwni6EdS5ZGbGuoU80Pzn7oCa85ahVZtAVl71Ww==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1632429759; bh=KLg/MXRbnQuxvGgXMZyZrHqNVziIij7Kd9RA36VYflB=;
 h=X-Sonic-MF:From:To:Subject:Date:From:Subject;
 b=leMfASpdJybw8G+yAuvjoxAj9YGfkYvtoJg4ff3eFpGihcCdJAWLZJCHH8O6GCYO5hcmZ2lzzQmlHnXqza2/ZGfzpRhImdfk+mLlzG7aAzLKOxVnvxV945/FoiBIzUPtiVlLsWkvUwCYAxia6sP+9d8nYS5SUaVPKJXLgHLVocU/ARsm1aOQZTsSUyTz2wDQCADKnwkojgBLBxBaEPTb8Xw82OqZu/wSjpjuczuZ38ATiuwk4IIX2YNiD8LZtY2dAAcVTw1Ey+nTZek1U8s9Bkn2hzWbIWd+P0k0XKvgTIUfFc5atPdhDbUvCEEIQH3FwfytCkd+gf5sEy7USt4LBQ==
X-YMail-OSG: pI7MHOMVM1kAck7ww4o9oDNuU.8T3hPRNHIioRyVNdb_Fkbvy9sjgO0OafKZYz0
 m5tJ3JnJvaxd0Yv1bbj35MwxCC8_pn5uHje4HMge.6h042lAw4G4gTklQGc39NeS9AnIp3QPmxZu
 1ZGCuUJ3fc1RRLzj5cL_V75HNXHM_Mr2TlnTMKPhiAF89KdLrvjIHjgv7RfDpBwTTpUrhSTwYW1o
 ln4RTw1z3IYbpxPv6YY7DXhCmeSpQNIgXdB50QOz9REXOHLaUqwD5Rzvb0qCj1wkY74NVpAfGwQ8
 K.82XjtUYjWBJ48OWGkwxYhuEr6BE9LxXnOrbo4i8PmFWMOd4bS2k_7KYDDZajwEuW8rBaXww.r2
 jk_gnKvLsNnJ37wObFuMraIMABL6MRn0YtlJGmabY0KrZ1dN1bmAjvw.rkUvajvxHYVTuL6NaUl8
 SbgMJY.6b4sl4lkBScrUayYKmP3PdTy6r7wn8oc5OJ_yFQk_sb6BsAipVFL0XI9GgCboZWZF36OV
 swuMN9Lv_TXi94HPts.x0aCihyktc91OrAoPVbILb41Yukrie5k09e1JfoagSQZC4pOWCe_FwoWd
 tygzjNkitGpv_C88Fg5n6WgaA0urg73olG0I2fX47yDr86Eh5C6GQBJbM6YnuqEwE83aECOWsCHC
 tuOMBGEUDrO3SBnucwRHF2rkQkI6rN7_9fs1qXOUodGpSuG56jzjL82ijpBjIXpxsz1cruW8D8wv
 AhaQcmrEQWZOlrWofy8arZ0gfOp9N11NKpXnIcZO2FHAdXCqPHlipRBPWZnHwoODBTrt_f5JwK8S
 LWje_1vYX3s6_PnFo.6qYV1Xs1ifG3ydwzrihKRzlGsVNTZIHHh2PEnWTXnaNxnpgtjdjN9IjJd5
 WuLl.vAYaUc1ouCrsEYBsJH.j6XRF148HR0yvP0nmQYhD9rdZwsGkI1BC1TQFE85_a7PHSvLK6Zt
 vogDaiSQbAGfznAfs0NTJ2x7KSIWTytWI9sYvNTqJYQTYEixAYtTCSd6l4fsasUUxJbPSplZoKwa
 RCViLdNANOPFWAnZ2JLxUf8y3Ej9kQaGI3VgQnfCTtD6pLSHFcrFk71DQDwr4LyObT7mzrQW7zmh
 jhhBr_W4.SQELacOYoyRn9y5FqExkSYkQ_CY_.GSbxQ9ITIvxnWNr7M1AufyqKt8r0cK6ujAdGD6
 2YXZsCmp_8SXkykFvFXlr_gR1YmLQrAjNAw2u3K2V.3P52AWzqJHavZUqVJ3O9YoZ6sPyZlaYk_v
 L_YYWf5tinDju4cJ3_bU2HCRyMse4a_H6HjKvibG7_QIzNcppnczeYWrMqEJ.pb_dfN81ReSPw94
 G3XxfZEQ1w.Lm4r.Gk1lmk82wl_1PosYtaGRsit7Syba9F8sebMkHeCc31kFvt.8UCgIs2xsVPlb
 LVRmOWn9Moo9gDtWm_gRqrgTgfdCFXMILXadxphrDAIA_nZCGppbOSsz18Yx_k310dUsUzz948Pj
 IiinVTVeOpYFCoyzA.OzlnlYfigwyd1MvD9pmRiuj.sBl1qEPSy35m00B6f4SxgIbvP9tBsLOUhs
 S_hfmFJFFKOgnIq5x9ajH4_TLlKauXaO7Ug3V7EW2cH_esKXDPwjm2pC5kB5AwOLowYdv1JdrENO
 MCPIKJG0ajit87sfcMFGeWJJt6.HHbYIBs3AweiW.00VXXEktYosMs1rZSAM4_QWxDwXc4OJzCxa
 ZHootTF415ftY_zkvrbcYEvsGW3GzYR2xGX.9AycEdcYxukpWl6pNLaVaIGeJcIe9SQX_AhwXGDH
 hZYBJikkMSGBeTHPh8cCb14L7hv_ASAH0wPLNQX6YvH5tetqSkq.7sdXeR2FMEdotl6nfvN1fMkd
 IXajathhA5y._95bSPAtaEPGXtkzIKtXSIaY_lUjkSJLYtYbzS1zj5frxMEjWe4fWfeZYbh7zoF9
 UKIiEVHXGZ8PQJCMt7Naf_7jzihwlo5KWUySe0dBVFmBmztsnVPuqxRKjfXZrQUb2F9gtU9vIDUr
 W_7YufJXkACSK.GWe29j6dOqhthm31Oc1.1hQ115BhLx8VYIkkermOBRIlVlJHTBmV.d3BuYamPi
 Fe9kKgbHLTLhiTrXerfTwV11hMR2is5LQ7YRT5ovrbOtTR4uoPTj61bLEagU0zMk5UVR.Bgo9Fbi
 JL3D5Wh5a7Li.0Hcw0GbPCaltHJl7bLkkEuh4ExEnFwZPLM.1F_TCtw0zRpb63NARnqxavtwlFkH
 QP002UKgcsSzRPZtdXTmO_aIAznVPiDUE34MPXLns0EeuVnzqWDgusYLaKpvY47qb2g--
X-Sonic-MF: <mardani29@HIDDEN>
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic302.consmr.mail.ir2.yahoo.com with HTTP; Thu, 23 Sep 2021 20:42:39 +0000
Received: by kubenode505.mail-prod1.omega.ir2.yahoo.com (VZM Hermes SMTP
 Server) with ESMTPA ID 6de23168455f45fd9d3a5e82e5ee70e6; 
 Thu, 23 Sep 2021 20:42:33 +0000 (UTC)
From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN> <83wnn7vmya.fsf@HIDDEN>
Date: Thu, 23 Sep 2021 22:42:32 +0200
In-Reply-To: <83wnn7vmya.fsf@HIDDEN> (Eli Zaretskii's message of "Thu, 23 Sep
 2021 09:13:33 +0300")
Message-ID: <m11r5fdnwn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (darwin)
MIME-Version: 1.0
Content-Type: text/plain
X-Mailer: WebService/1.1.19043
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo
Content-Length: 800
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

>> 
>> I usually work on a monorepo with ~67000 tracked files (many of them big
>> binary files).  Here's what I get when using ripgrep as the xref search
>> program:
>> 
>> Elapsed time: 36.087181s (8.067474s in 22 GCs)
>> 
>> Running the same search with ripgrep from the command line takes around
>> 6 seconds.
>
> How many matches does that report?

~70400 matches spread across ~3700 files.  Perhaps it's an extreme case
(I used a generic search term, "color").  I tried a more specific search
term ("clang analyzer") which returned 30 matches across 14 files.
Here's the benchmark:

Elapsed time: 10.176629s (0.946311s in 2 GCs)

I guess the command could signal ongoing progress somehow, to assure the
 user that Emacs is actually working and not frozen.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 21:18:01 +0000
Resent-Message-ID: <handler.50733.B50733.163243185016829 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163243185016829
          (code B ref 50733); Thu, 23 Sep 2021 21:18:01 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 21:17:30 +0000
Received: from localhost ([127.0.0.1]:56025 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTW5y-0004NN-3s
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 17:17:30 -0400
Received: from sonic306-21.consmr.mail.ir2.yahoo.com ([77.238.176.207]:41373)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mardani29@HIDDEN>) id 1mTW5v-0004N6-Mb
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 17:17:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s2048;
 t=1632431841; bh=AY3cqMPSKg5LrD29RJ8dNl31U2kJsiGMJL316GtitiU=;
 h=From:To:Cc:Subject:References:Date:In-Reply-To:From:Subject:Reply-To;
 b=NqwCm0KYYYHADqGY15FJj03pFJqm4RNuHW4aU+h2WpbCh8Abn+bMn+0yryoasBPSxKqvzCoA8knGsosY9ThSaZVTrFLMZz3S7naMKkfxOJj0RuYRXljwbIv2cmLX9zWi9VzwCW2V6i6DmmSZK06h7/VsiuQVmpBQqqfmeKC2ZvoZOiHTAZiR8Qfe4VG1cQb5MK9gRIxEyYlE6MINijjZwI9amKaC4tdwRgMD4HMq7d3goGH3TdWofUr6lYnJdeRyKz3vUSnIclgI3OxxM/pCIYJ1QtNj4EzOjY2FJm5yob8mP8J8cmaP4ou3n588MRwoU2v8PJxquGqdAP3SeCAo6w==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1632431841; bh=Zud8a4WvaJLwWjg7JUfgYVJS42Sr54vs38lnHSiAO3v=;
 h=X-Sonic-MF:From:To:Subject:Date:From:Subject;
 b=tNS2ESPp8lZm9QpUWWqkYtloyHJf2kEk0MdQoqRsI96FKwlLfQHGYhpnlLqjxTQaDqRNBCcB2Od+o1f+j9p5KgSqXVqWnUdjtJpruz7ffnWTAjRiBdoi/+F28sZJfjPPp9OpkbC3nSbV7t0V6oY5XxKJK2eQHi91v5YGBSs+FAsh3zCPb928552flqv4qjFHrG4iItvfYjpSANWWypRMufTuIOjXyWJ65O1faAT7Nh6HlYeCcdQGsfbRlO34a8tPzkfRS8c2r/8LvzqPQSHxOP/W4ab6tth6XPzqhi1kRJJPxX98dGaTU9OKRGMp4prWxpHzSAwCbf7sZN7HMxQ9mg==
X-YMail-OSG: _3S_ZaUVM1nn1alvY_gP6DNz0V5yRQfuut80.uccHrZEU9L4zcC.CE_wraNPAxh
 Yh8UHQ5ystR_QNIiQrycEjOjDkdJ3L2LMeoHMOkG0uyrrTPofTStFzSQ6Jn2kBwCEUyG.lD8QzDB
 5ePexT9YpjMb_z8n2QpgWH_t7Q0VhCouRWYFCLhi_nweUqZ62HGb7ll3nRXZCmavP.9KU62ePCvi
 2c4GYkKZMDjVUJHHVDqgLIInJDxx0Zo7vAMg5qDvJjFd.iH8DVvJMPtnfuaDbS4hSThETMogON7T
 XvjXajqzrR3FNasgVV80bxYEa6O6vfKpyUt0u6Kn65a0OKEv17BwyHPGqUbDFdsxF2PaqsMjz1Qc
 zCqthKIeKvYTEzSlCSbJdcnPeE6NzfkLJ3eAiPd4h1QIG3RDruNIccCvBysNe.Tx1KATvYE4SSGX
 t1jPTgAD3GRl60Xhf.e0AuFAHg3LsHV0.6jWh4ntk74iVPBsXFpzO3qPaOnmBVygyqZ_uX6qVBKb
 aazjvDPxIyNMNbhMt4xhFrJNmb2PllicqEERJpzirv.nV.djpirV9e.miVIaWVusUlYZFc3sHi7b
 XKgQaKPsi4nfBteuIh2k_MiDuqtBRdyk_blA5ezFmDPj2P0ZPUrnMQZ2kiN_ZWTuxCe8EqYM9r3m
 UD24_ZetGK7TajtrDc4VGvEqHiIoC9oYP7ci3EY2qUdi93CJrPbGz3fSCQ082hjWMcear5X_VARa
 hejkRxqO9D5HaLs3itMcBGrgKO2fiHXPXGMkYmI9GTE_MTeSV22EBYCcWpMo4wpPHa9TgfiYwNwN
 H2Adaymr2_RYZrwXxYIN3GGUNhOZWeUN4kyat2MguESODK0me.O0XMzv6wiO1vmWFEM3w0fmd75F
 wPlRR5s0o8Sz186xOsMhM7DLvh4A5UAAOkE1pIupOg86XvHtMS7HAMRvmqx9VPty5NuqKSPvCRum
 0wNbfMeeM1VvbXTW6E6hVYTK.OGThea41Ot9F4Yc97lBy.W_xYBc9Yq1o.9b09J.JWXuoc.eTjws
 aJSmG8VE67SDWoY8Vjn3oFW5CW85N_2gMlOIuNZVKQQwYMI9C0WuL8tye58Nn2tn4VxmiwFR9Jsc
 yxGG6g8920p9Cp6yqo6Vj3zerfUILBiI_3hEJkd_Ynmdg4hFlk5QwFTScvyU5S_42MBo7BbKBnN5
 SV3uwcvfqovRNThXBEBXU.mmyJIed2xdWmoLP63o2PHFFJn26kPOgwMmbI9r2lxWzStPE9jZ7hRI
 kbUZiwjxgk1ZSANnUBgN5fQnMPJWF5JDdwFJS3CV2daGB_gW12xnf3.SfK5IKbCSm8KOB_SxU_TG
 JtvGJVu0KG8HSqU1n79IQDwg_Ka8cikeF6wK0V.Yiru8Y3IoNtfU1bQhtNXJkVlEcI2k0yVqJI6k
 3ELZph4hgKMQMsXAEG5YsmzAoLZrD2QAQnrwJhrU1ImnC5nbW1SjCNP6tG.ZRX06jJTxlpuXPO0O
 qZnnWmRMRP6m0qYPXKxQsC4H04M6rqQ37p1AmOKQO_huU1MYawJN2EyKK434oOs2_IUc.bbHnyPQ
 ObdV8yNvdlagGh.HpJn.m6DP8RDnaLv6ra2bmk8o8sIWNt4xwVs9GvgNpg0gjAwz9eW8r0ICouli
 KS5Qgo4kM3vzTtEa.bUVxofOO_.rjHjRup5z1BnEEA.gEAFLYKXwajKwVri0x4ggJJ1TmfhSBxWn
 DwA9U7aISt.gOA0BkOW7ODmMkSueXsqLymdMJvB5NMCTcp6LQxzxbA.PAo9eQ2qhx0HpSmFu0Bg4
 K112eO5Ov16P2Rl1t7xraGvyh2EJLMpEmJOZ0jjTWmA5ITL5KpaWR0j8L_IeBRKe2gp6kdwGIvwJ
 6I0iusNVA.Ig9E0qgqj9cjoe9l0NxPDhg_houvRFDAZkNqkCocac6pWzrU54DvmjWWyw.c3RToEt
 QfRRyCCjp3hT6NzqWg5bB7v6xit52ieWpYt2wODo_cqqY68VGwpp6H7nwgxP7CzjJsagZnPpOr16
 hXmYKhDGgT7QHELt85ii.fgVAqNDC1PjC0IvS25m5H_f0UPDSU_d13BoCaxZdn11MOXYC.glHGOm
 .MuT_ELrZPg3fIZVCQdkiIYSup67Ro6GqEooD3ckfjkEZN42WuQ6YghLFmdmNVmD4IimvZaIpq8f
 PhraEMY.kxJW9EePfyDxPKJf6LW.WCN1ZdeqH9vVOOD8XvJBZQmlPwwzAgLHmXrg.zzusxw4Un01
 aMDSCKl96DLiGyEYI3Wb5ZLIg6pAoSY2NE.P8NC3PB9Ck0xTQ73HJ3KJRuyLbOE4rYMw-
X-Sonic-MF: <mardani29@HIDDEN>
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic306.consmr.mail.ir2.yahoo.com with HTTP; Thu, 23 Sep 2021 21:17:21 +0000
Received: by kubenode508.mail-prod1.omega.ir2.yahoo.com (VZM Hermes SMTP
 Server) with ESMTPA ID a650228e3f091b9bcd71fbbe00b65a63; 
 Thu, 23 Sep 2021 21:17:15 +0000 (UTC)
From: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
Date: Thu, 23 Sep 2021 23:17:14 +0200
In-Reply-To: <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> (Dmitry Gutov's
 message of "Thu, 23 Sep 2021 02:09:16 +0300")
Message-ID: <m1wnn7c7qd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (darwin)
MIME-Version: 1.0
Content-Type: text/plain
X-Mailer: WebService/1.1.19043
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo
Content-Length: 1147
X-Spam-Score: 0.2 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.8 (/)

Dmitry Gutov <dgutov@HIDDEN> writes:

>
> Thanks for testing. Did the switch to ripgrep help much?
>
> I wonder if we should advertise this setting and recommendation more
> prominently, at least until we get auto-detection.
>

When I set xref-search-program to 'grep, the search is almost 10x slower.

>> Running the same search with ripgrep from the command line takes around
>> 6 seconds.
>
> Is that with an SSD?

Yes, it's a fast MacBook Pro from 2017.

>
> Another thing you can do is set up the additional ignores for the
> project. If those big binary files are not something you are
> interested in searching and touching, you could add ignore entries for
> them. When the vc project backend is in use (default), it is currently
> done via .dir-locals.el: the variable is project-vc-ignores, it's a
> list of strings that should be globs. See its docstring and the
> explanation in project-ignores's docstring.
>

I sometimes do C-u C-x p g so that I can limit the search to certain
file extensions.  That helps a little bit.

So far, the command that offers me the best performance for this kind of
project grepping is `vc-git-grep`.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 22:19:02 +0000
Resent-Message-ID: <handler.50733.B50733.163243551115963 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163243551115963
          (code B ref 50733); Thu, 23 Sep 2021 22:19:02 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 22:18:31 +0000
Received: from localhost ([127.0.0.1]:56173 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTX30-00049O-R7
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 18:18:31 -0400
Received: from mail-wr1-f44.google.com ([209.85.221.44]:43970)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTX2z-00049A-1u
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 18:18:29 -0400
Received: by mail-wr1-f44.google.com with SMTP id w17so21336328wrv.10
 for <50733 <at> debbugs.gnu.org>; Thu, 23 Sep 2021 15:18:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=GfVHs5FvcgyeOdssr52zCxgBNWbEf5dEKef+mmC/oEk=;
 b=Z+zDQuiHdXr3w9dd2oK3nqIygRbwqV/bFrMYNmSwXRwfPPVLv3PYt0Z3P/co7/L24c
 DLF81hT8CcImk217DuKnRTCNpcDiE09XZzxdW5CXFhgHUJ1k1DA/OkcWxzF43wWcjstE
 46HCHgAKJVuofnOzyzJR0Jri2n0/pjAvhsEFW8qFP7ZAWOldQIimgMP2Z2iBnkHQuL8x
 Vg+kWSP/Ttvmf/01nIldSXRerdPq/Tw1sUSfblFC80NTjp2f9rskRKq+utmoaZ1dh08G
 MZ8XUoFjI1NPxkGxTtMAfBzb31+itPqUSGlrTpJOg4+MA4KwErYPj0oicPvz79mo/ftI
 nB0w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=GfVHs5FvcgyeOdssr52zCxgBNWbEf5dEKef+mmC/oEk=;
 b=BqLLFGoGuNTxvnKkwZ0XBPcAi46ks9ur2ru0mri2ics08tvUw1jkjQkdB2qDDf66MZ
 AymEKFgH5xb8DtI+ZTp45E0rwF7upoDaqNAi9P5g7cNQ1kHghMUNJtsQNGhtYlvc+2kW
 Zty7wED9I8/07PBKG69a9RcKIpCl0BwRq/VPVp7zmoFw90nnp2vWmoKFKvwMKI7VgRAF
 YnHVsUrhO6cvnNDC06GQYwo6Y6tYkZ+iIF3Ip5vYYJbD/s8rKzkmoMPMpf/NB89IbB9Y
 jcgOlXpHEtVc8d7WRvFLI02l1daDt/EZhZbbgAi3r4A7YRsQE9vaVWZc0Rij+Y4rwfkn
 VYrQ==
X-Gm-Message-State: AOAM5337TrZrrBE8hiqL1mkJ418nR9wZj6ZVqm0bT+vU/bhSlaljvJOd
 XFt/Ve1ni4B33F5PzqAVGN48uLHs/qw=
X-Google-Smtp-Source: ABdhPJz44VVYNKIVhotsg3sDhm1I2/z71mohwE0/oajNfFgtS/WHdR+Kdgd6vVwGCqPk010FdJwqdQ==
X-Received: by 2002:a5d:4803:: with SMTP id l3mr7856124wrq.61.1632435503192;
 Thu, 23 Sep 2021 15:18:23 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id o1sm5820226wmq.26.2021.09.23.15.18.21
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Thu, 23 Sep 2021 15:18:22 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <83wnn7vmya.fsf@HIDDEN> <m11r5fdnwn.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <d1850012-1776-2d41-f9ca-0501333dff9c@HIDDEN>
Date: Fri, 24 Sep 2021 01:18:18 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <m11r5fdnwn.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 23.09.2021 23:42, Daniel Martín wrote:
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
>>>
>>> I usually work on a monorepo with ~67000 tracked files (many of them big
>>> binary files).  Here's what I get when using ripgrep as the xref search
>>> program:
>>>
>>> Elapsed time: 36.087181s (8.067474s in 22 GCs)
>>>
>>> Running the same search with ripgrep from the command line takes around
>>> 6 seconds.
>>
>> How many matches does that report?
> 
> ~70400 matches spread across ~3700 files.  Perhaps it's an extreme case
> (I used a generic search term, "color").  I tried a more specific search
> term ("clang analyzer") which returned 30 matches across 14 files.
> Here's the benchmark:
> 
> Elapsed time: 10.176629s (0.946311s in 2 GCs)

30 matches means parsing and printing the matches won't be a bottleneck. 
Just fetching the file names, sending it to the search process, and the 
search itself.

> I guess the command could signal ongoing progress somehow, to assure the
>   user that Emacs is actually working and not frozen.

I suppose we could add some messages like in the patch below.

It's a simple approach where "done" is printed after the search is done 
and the results are parsed, but before they have been printed.

So there can still be a perceptible delay when your search gets 10s of 
1000s of hits or more.

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 68beedad3d..2b69fb7d57 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -851,9 +851,11 @@ project-or-external-find-regexp
  (defun project--find-regexp-in-files (regexp files)
    (unless files
      (user-error "Empty file list"))
+  (message "Searching...")
    (let ((xrefs (xref-matches-in-files regexp files)))
      (unless xrefs
        (user-error "No matches for: %s" regexp))
+    (message "Searching... done")
      xrefs))

  (defvar project-regexp-history-variable 'grep-regexp-history)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 23 Sep 2021 22:42:01 +0000
Resent-Message-ID: <handler.50733.B50733.163243687118378 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163243687118378
          (code B ref 50733); Thu, 23 Sep 2021 22:42:01 +0000
Received: (at 50733) by debbugs.gnu.org; 23 Sep 2021 22:41:11 +0000
Received: from localhost ([127.0.0.1]:56203 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTXOx-0004mL-8R
	for submit <at> debbugs.gnu.org; Thu, 23 Sep 2021 18:41:11 -0400
Received: from mail-wr1-f51.google.com ([209.85.221.51]:42725)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTXOv-0004m7-7U
 for 50733 <at> debbugs.gnu.org; Thu, 23 Sep 2021 18:41:10 -0400
Received: by mail-wr1-f51.google.com with SMTP id i24so5652601wrc.9
 for <50733 <at> debbugs.gnu.org>; Thu, 23 Sep 2021 15:41:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=eSs9nsiXuJDZQndAhmDUMwIEy41raMYOH4dwmHi4Mh4=;
 b=J9seVlJTds7cNi0d6M0BXwQZEV8yPmrEV/oAPAB0MfmuirEYitD4nMDbb068fuE9/T
 OOp2/xkCbWVkpyqaKGHGbFeAMShQD3Sj/GoxwLkYaRZMuR1KCbafveDhN4CNSlDEf6VB
 I+4Wgz3hqfxknNQO0e05dYKbiQIVUOkIIagHnUUQCORngUcilqw9JNgvFnPlQnmn+G51
 tzn8E+UHfePeusqvBNfGgWdIVbBwXSCoI3Tg3pOV23au4kpdZtHMCVUSKSfOCHI6KiyG
 xR/Ss1bNItSIppyMNAiaLACeNszQOxgzHKvmFNYk7Ih3DCmz7gIkjIqL/9zWn3Gj4hzM
 lYeQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=eSs9nsiXuJDZQndAhmDUMwIEy41raMYOH4dwmHi4Mh4=;
 b=mRlxy3YPM1vLk26motpIcYINaIugktOapPoYHQHcAm9Zo5EP5HTg14yyM+w7AL9lqq
 BmvaGu3P4rXsGOfmk/kewK2S2KTYgqdkx6CFi2lJNq0PUrZAt7CfwDDt0TFF4PtdpTRz
 LL9I2YCDZ7IcbBOD1orHRdVduF+gVPZUaR9OdKIsewAVh3HeQ7b8+gEqdiBgrgez5DKn
 iqFg22HnpJ+TJ5TECH86S4LKD/rguchD1tmkV43DrimUAF+YooaZTqyVl79prOJnkjyE
 viMeD6dqOHTqxV438Bs/SmZi0nXEfPaC+SKj4RNrUB6Vs3CH0fPWwg24i2EKY8KHkM3S
 zmwA==
X-Gm-Message-State: AOAM533ejO7XAyQqAOKGKkAzPN1myZG4rtbboP56aJPHdQWV6wRyHlk6
 HaXXoyOBwi8TQYPFWntQWAWCPlXSBbM=
X-Google-Smtp-Source: ABdhPJwE27mzDWNqo2xhVILI5TVqth+dMLYHfeblkYDa1+t0VanLyQ4OfzTEVaicbW1OvyeB4yilzQ==
X-Received: by 2002:a1c:ed13:: with SMTP id l19mr18507250wmh.48.1632436863591; 
 Thu, 23 Sep 2021 15:41:03 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id 198sm6724289wma.16.2021.09.23.15.41.01
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Thu, 23 Sep 2021 15:41:03 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
Date: Fri, 24 Sep 2021 01:40:58 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <m1wnn7c7qd.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 00:17, Daniel Martín wrote:
> Dmitry Gutov <dgutov@HIDDEN> writes:
> 
>>
>> Thanks for testing. Did the switch to ripgrep help much?
>>
>> I wonder if we should advertise this setting and recommendation more
>> prominently, at least until we get auto-detection.
>>
> 
> When I set xref-search-program to 'grep, the search is almost 10x slower.

I wonder if we could do more about this problem.

Perhaps on a platform like macOS we should consider bundling some 
up-to-date search program, be it GNU Grep or Ripgrep.

Eli, have there ever been a similar proposal under discussion? IIRC 
having to install external tools has been an issue on MS Windows as well 
for a long time. We give recommendations, but Grep has never been 
distributed together with Emacs. Any particular reason?

For comparison, VS Code bundles ripgrep. Or at least its macOS releases do.

>> Another thing you can do is set up the additional ignores for the
>> project. If those big binary files are not something you are
>> interested in searching and touching, you could add ignore entries for
>> them. When the vc project backend is in use (default), it is currently
>> done via .dir-locals.el: the variable is project-vc-ignores, it's a
>> list of strings that should be globs. See its docstring and the
>> explanation in project-ignores's docstring.
>>
> 
> I sometimes do C-u C-x p g so that I can limit the search to certain
> file extensions.  That helps a little bit.

Ignore entries are a bit more flexible: you don't need to repeat the 
preference every time. But they don't really work when you need to 
ignore only some files some of the time.

> So far, the command that offers me the best performance for this kind of
> project grepping is `vc-git-grep`.

'ripgrep' might be even faster, and there are Emacs interfaces for it 
like deadgrep (https://github.com/Wilfred/deadgrep).

But with Xref we're trying to create a consistent abstract UI that 
honors user's settings and can be used across commands, with extra 
features that can act on search results. There is no reason why it can't 
be snappy, but it's definitely more work.

(I haven't forgotten about bug#49731, BTW.)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 06:05:02 +0000
Resent-Message-ID: <handler.50733.B50733.163246344632738 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163246344632738
          (code B ref 50733); Fri, 24 Sep 2021 06:05:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 06:04:06 +0000
Received: from localhost ([127.0.0.1]:56480 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTeJZ-0008Vy-U8
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 02:04:06 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38086)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTeJY-0008VQ-7L
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 02:04:04 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44332)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTeJS-0005rZ-PL; Fri, 24 Sep 2021 02:03:58 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4988
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTeJS-0001BL-CG; Fri, 24 Sep 2021 02:03:58 -0400
Date: Fri, 24 Sep 2021 09:03:51 +0300
Message-Id: <83o88itsqg.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <m11r5fdnwn.fsf@HIDDEN> (message from Daniel
 =?UTF-8?Q?Mart=C3=ADn?= on Thu, 23 Sep 2021 22:42:32 +0200)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN> <83wnn7vmya.fsf@HIDDEN> <m11r5fdnwn.fsf@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Daniel Mart�n <mardani29@HIDDEN>
> Cc: 50733 <at> debbugs.gnu.org,  dgutov@HIDDEN
> Date: Thu, 23 Sep 2021 22:42:32 +0200
> 
> >> Elapsed time: 36.087181s (8.067474s in 22 GCs)
> >> 
> >> Running the same search with ripgrep from the command line takes around
> >> 6 seconds.
> >
> > How many matches does that report?
> 
> ~70400 matches spread across ~3700 files.  Perhaps it's an extreme case
> (I used a generic search term, "color").  I tried a more specific search
> term ("clang analyzer") which returned 30 matches across 14 files.
> Here's the benchmark:
> 
> Elapsed time: 10.176629s (0.946311s in 2 GCs)

This means processing 70K lines of output takes 28 - 9 = 19 seconds,
which is the lion's share of the run time in those situations.  9 sec
to scan the files and 8 sec to GC.

> I guess the command could signal ongoing progress somehow, to assure the
>  user that Emacs is actually working and not frozen.

Currently, that's impossible, because we never return to the main loop
while we wait for the process to terminate, so no Lisp can run during
that time.  That'd require to add a new hook, which the call-process
machinery would call while we wait, or maybe even a C-level function
to report the progress directly.  Shouldn't be too hard, I think, see
wait_for_termination.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 06:36:02 +0000
Resent-Message-ID: <handler.50733.B50733.16324653153728 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16324653153728
          (code B ref 50733); Fri, 24 Sep 2021 06:36:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 06:35:15 +0000
Received: from localhost ([127.0.0.1]:56509 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTenZ-0000xl-CT
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 02:35:15 -0400
Received: from eggs.gnu.org ([209.51.188.92]:41696)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTenW-0000xE-UG
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 02:35:04 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45126)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTenR-000773-C4; Fri, 24 Sep 2021 02:34:57 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2901
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTenQ-0005km-AS; Fri, 24 Sep 2021 02:34:57 -0400
Date: Fri, 24 Sep 2021 09:34:49 +0300
Message-Id: <83k0j6trau.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> (message from
 Dmitry Gutov on Fri, 24 Sep 2021 01:40:58 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

> Cc: 50733 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 24 Sep 2021 01:40:58 +0300
> 
> Perhaps on a platform like macOS we should consider bundling some 
> up-to-date search program, be it GNU Grep or Ripgrep.
> 
> Eli, have there ever been a similar proposal under discussion? IIRC 
> having to install external tools has been an issue on MS Windows as well 
> for a long time. We give recommendations, but Grep has never been 
> distributed together with Emacs. Any particular reason?
> 
> For comparison, VS Code bundles ripgrep. Or at least its macOS releases do.

Bundling an external program is problematic, but it can be done, of
course, provided that the license is right.  The problem is, we don't
provide macOS binaries, and we cannot rely on the fact that the
Windows binaries distribution will have its maintainer forever: we
already had periods of time without such a person.

We can recommend installing such tools, of course.

Btw, I don't understand why we focus on general-purpose text-searching
tools for these features.  Why not focus on packages like ID Utils
instead, they are so much faster.  Daniel, could you time the same
search in that large tree when xref-search-program is 'gid'?  (You'd
need to run 'mkid' first, to create the ID database, but that is
one-time, and is very fast.)  As I told many times, I think this is
the future: program language sensitive tools that use a precomputed
DB.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Juri Linkov <juri@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 07:04:02 +0000
Resent-Message-ID: <handler.50733.B50733.163246701215374 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, Dmitry Gutov <dgutov@HIDDEN>
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163246701215374
          (code B ref 50733); Fri, 24 Sep 2021 07:04:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 07:03:32 +0000
Received: from localhost ([127.0.0.1]:56569 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTfF5-0003zu-R6
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 03:03:31 -0400
Received: from relay11.mail.gandi.net ([217.70.178.231]:42085)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <juri@HIDDEN>) id 1mTfF5-0003zg-2E
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 03:03:31 -0400
Received: (Authenticated sender: juri@HIDDEN)
 by relay11.mail.gandi.net (Postfix) with ESMTPSA id 5185210000E;
 Fri, 24 Sep 2021 07:03:22 +0000 (UTC)
From: Juri Linkov <juri@HIDDEN>
Organization: LINKOV.NET
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
 <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN>
Date: Fri, 24 Sep 2021 10:02:36 +0300
In-Reply-To: <83k0j6trau.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 24 Sep
 2021 09:34:49 +0300")
Message-ID: <87lf3mmp6b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

> Btw, I don't understand why we focus on general-purpose text-searching
> tools for these features.  Why not focus on packages like ID Utils
> instead, they are so much faster.  Daniel, could you time the same
> search in that large tree when xref-search-program is 'gid'?  (You'd
> need to run 'mkid' first, to create the ID database, but that is
> one-time, and is very fast.)  As I told many times, I think this is
> the future: program language sensitive tools that use a precomputed
> DB.

While we have the make target `make tags` that generates the tags database
in the Emacs source tree, for trying to use ID Utils I can't find
a similar target that would create the ID database.  Is its command line
more complex than etags?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Gregory Heytings <gregory@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 09:01:02 +0000
Resent-Message-ID: <handler.50733.B50733.163247401827947 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, Dmitry Gutov <dgutov@HIDDEN>
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163247401827947
          (code B ref 50733); Fri, 24 Sep 2021 09:01:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 09:00:18 +0000
Received: from localhost ([127.0.0.1]:56697 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTh46-0007Gg-6c
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 05:00:18 -0400
Received: from heytings.org ([95.142.160.155]:45708)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gregory@HIDDEN>) id 1mTh43-0007GU-Jl
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 05:00:16 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heytings.org;
 s=20210101; t=1632474014;
 bh=OrFZpJ/c1rQCb5696SIQzrfQluv+vNJYidvvW4DrMPQ=;
 h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:From;
 b=gFX0bKymHF2jz/pagTga/a0Ha0KRsOqN4tlJH70hd+zOvlAA9dgZetUiGV2fpRIET
 I9xd7+Z4syQAskdZ0/Zqbes+B/NnSESytgfZdqIwOrJZGRH3+6QYoBpr/7Sh8oecOA
 A+QkUKSM2qb3BWfB8V16OY4+Bs4Xj+vZDEx+ZviWjfUapydGwDPfRfeRrEX1HQOVZc
 ME2Ghvt6UYOShmzbSFHSpR0tyeV15dvFNG+xdsz0zeVONeutqrlSqoTteNYl4S0nvo
 Mr0J0IPzj4G6M/T6FrwnAC/BK8l7zXn99r9g4oTp51qLxtnkilkvWnaukQh6XOcZeX
 OHBYNEGVpXE3w==
Date: Fri, 24 Sep 2021 09:00:13 +0000
From: Gregory Heytings <gregory@HIDDEN>
In-Reply-To: <83k0j6trau.fsf@HIDDEN>
Message-ID: <d70d4a91f222dc75d66e@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=us-ascii
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


>
> Btw, I don't understand why we focus on general-purpose text-searching 
> tools for these features.  Why not focus on packages like ID Utils 
> instead, they are so much faster.  Daniel, could you time the same 
> search in that large tree when xref-search-program is 'gid'?  (You'd 
> need to run 'mkid' first, to create the ID database, but that is 
> one-time, and is very fast.)  As I told many times, I think this is the 
> future: program language sensitive tools that use a precomputed DB.
>

id-tools are useful, but they do not provide the same functionality as 
grep/ripgrep.  On Emacs' trunk, mkid takes a few seconds to create a 12 MB 
database, and gid Eli returns only 4 occurrences.

This can be improved by using mkid --default-lang=text, but then mkid 
takes no less than 30 minutes (!) to create a 205 MB database, and gid Eli 
now returns 25 occurrences in ~1 second.

In comparison, rg Eli returns 6474 occurrences in < 0.1 seconds.

Moreover, incremental updates are not implemented in mkid.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 10:45:02 +0000
Resent-Message-ID: <handler.50733.B50733.16324802456727 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Juri Linkov <juri@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16324802456727
          (code B ref 50733); Fri, 24 Sep 2021 10:45:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 10:44:05 +0000
Received: from localhost ([127.0.0.1]:56746 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTigX-0001kR-F1
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 06:44:05 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36996)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTigV-0001jl-C1
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 06:44:03 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:35034)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTigP-0002hN-5X; Fri, 24 Sep 2021 06:43:57 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2174
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTigO-0004CE-On; Fri, 24 Sep 2021 06:43:57 -0400
Date: Fri, 24 Sep 2021 13:43:49 +0300
Message-Id: <83bl4itfru.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87lf3mmp6b.fsf@HIDDEN> (message from Juri Linkov on
 Fri, 24 Sep 2021 10:02:36 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
 <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <87lf3mmp6b.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Juri Linkov <juri@HIDDEN>
> Cc: Dmitry Gutov <dgutov@HIDDEN>,  50733 <at> debbugs.gnu.org,
>   mardani29@HIDDEN
> Date: Fri, 24 Sep 2021 10:02:36 +0300
> 
> > Btw, I don't understand why we focus on general-purpose text-searching
> > tools for these features.  Why not focus on packages like ID Utils
> > instead, they are so much faster.  Daniel, could you time the same
> > search in that large tree when xref-search-program is 'gid'?  (You'd
> > need to run 'mkid' first, to create the ID database, but that is
> > one-time, and is very fast.)  As I told many times, I think this is
> > the future: program language sensitive tools that use a precomputed
> > DB.
> 
> While we have the make target `make tags` that generates the tags database
> in the Emacs source tree, for trying to use ID Utils I can't find
> a similar target that would create the ID database.

We can easily add it.

> Is its command line more complex than etags?

No, it's much more simple.  Assuming you are in the top-level
directory of the Emacs source tree, the command is just:

     mkid

That produces a file named 'ID' in that directory.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 11:01:02 +0000
Resent-Message-ID: <handler.50733.B50733.16324812518663 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16324812518663
          (code B ref 50733); Fri, 24 Sep 2021 11:01:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 11:00:51 +0000
Received: from localhost ([127.0.0.1]:56759 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTiwl-0002Ff-Ho
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:00:51 -0400
Received: from eggs.gnu.org ([209.51.188.92]:40830)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTiwj-0002FS-IQ
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:00:50 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:37204)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTiwd-0000gK-2d; Fri, 24 Sep 2021 07:00:43 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3195
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTiwc-0006RG-Ks; Fri, 24 Sep 2021 07:00:42 -0400
Date: Fri, 24 Sep 2021 14:00:31 +0300
Message-Id: <838rzmtf00.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <d70d4a91f222dc75d66e@HIDDEN> (message from Gregory
 Heytings on Fri, 24 Sep 2021 09:00:13 +0000)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 24 Sep 2021 09:00:13 +0000
> From: Gregory Heytings <gregory@HIDDEN>
> cc: Dmitry Gutov <dgutov@HIDDEN>, 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
> 
> > Btw, I don't understand why we focus on general-purpose text-searching 
> > tools for these features.  Why not focus on packages like ID Utils 
> > instead, they are so much faster.  Daniel, could you time the same 
> > search in that large tree when xref-search-program is 'gid'?  (You'd 
> > need to run 'mkid' first, to create the ID database, but that is 
> > one-time, and is very fast.)  As I told many times, I think this is the 
> > future: program language sensitive tools that use a precomputed DB.
> 
> id-tools are useful, but they do not provide the same functionality as 
> grep/ripgrep.  On Emacs' trunk, mkid takes a few seconds to create a 12 MB 
> database, and gid Eli returns only 4 occurrences.
> 
> This can be improved by using mkid --default-lang=text, but then mkid 
> takes no less than 30 minutes (!) to create a 205 MB database, and gid Eli 
> now returns 25 occurrences in ~1 second.
> 
> In comparison, rg Eli returns 6474 occurrences in < 0.1 seconds.

We are talking about project.el, so the focus should be on looking for
strings that are meaningful in the programming-language context,
because that is what we need when programming.  Not looking for
arbitrary strings.  If someone wants to look for arbitrary strings,
they want "M-x grep" (with or without ripgrep), not
project-find-regexp.  The meaning of "word" or "symbol" is different
in different PLs; a tool like Grep can only approximate that, whereas
ID Utils uses the correct definition for each language.

> Moreover, incremental updates are not implemented in mkid.

It doesn't matter, as it's fast enough, and putting it on some kind of
cron job is more than enough to allow forgetting that this step
exists.

TANSTAAFL, so please let's not insist on looking for one.  We won't
find it.  Every alternative will let you pay something, so the issue
at hand is to find the alternative that is the best on balance, for
this particular feature, not one which is free of any price.

That said, you are free to use GP tools if you think it's best for
you.  And I will always prefer ID Utils.  My point is that we
shouldn't ignore solutions such as ID Utils, we should include them in
our consideration of what is "fast enough".  We shouldn't consider
only Grep, ripgrep, etc.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 11:42:02 +0000
Resent-Message-ID: <handler.50733.B50733.163248366321124 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163248366321124
          (code B ref 50733); Fri, 24 Sep 2021 11:42:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 11:41:03 +0000
Received: from localhost ([127.0.0.1]:56789 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTjZe-0005Ud-N5
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:41:03 -0400
Received: from mail-wr1-f43.google.com ([209.85.221.43]:43612)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTjZd-0005Tz-8q
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:41:02 -0400
Received: by mail-wr1-f43.google.com with SMTP id w17so26466598wrv.10
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 04:41:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=tDYHDv9o6MxnY6JaWu3+Zj5Irb6slADJBOuYq3LlbnA=;
 b=Wgwp7lHR7ehjAsoYwVChGbMsmVXLUBh1P+1Cacl3T1I6oD/dghbokprlyTRIXWe305
 QqaA/kPFgEVKuTb7Lri8SxX3wU1pa1EI6ggxqEXvvPZz5NZlAgHQYcBsE4FmjxdGyjE9
 1QgnvXW4FaTU82NQCs7x4fSH0j8LJEEX1wL2yqHTAOxjF5qDJeWx+R3efE6TwiPOzj+z
 7qtb2/JRPzUZJfDrz1S4CDnphxIWBGuYCK+g+409O0FwsFbynOW2RZeeyM8zGLsQuBwe
 OHdxoGSUffsdsjaLovHytLpXxwUWBNS8nifGsMcuJwR8E0Pu7WPvElmZCGNvifpwKuu4
 mj4Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=tDYHDv9o6MxnY6JaWu3+Zj5Irb6slADJBOuYq3LlbnA=;
 b=pKvaBfUZ9nKikcCSFfiQTm2jaOZo+pk+gZZfcFO9iGS6e3mIho8fPZY2S8d0c1Qeud
 GoWnQzKWePO0XHPIOfrr3bOAotFKNPdeT1ey86xJxUwpDIltG/fmxPHcjKiaKlWsXfXa
 0CgAEa3IsILVEjKssqbPC8puKA1Icowu6YMQ32o540icGGgBBdAcHc/AIrgm0fElh41b
 HqluKjUozMqZwXgQ6PuDJXhU88vADjAFkCGLgKSwvmEs25qt7F2tzCCEkTyz+4UDT1Ly
 RYhAgcOlT4UFej62SqI/Ie1yUDodp4E8vSX/EO2LzG38bph03efXT+JaKx7PdwTJdKiT
 NNbg==
X-Gm-Message-State: AOAM531HOEevpq6+Ev6EHy3IRrxRMv7JuAknAx1jp98L3uSp2i6Z3mh/
 HUQIbGJ0h/wjh1tQMyZE8j50DOukXEE=
X-Google-Smtp-Source: ABdhPJzXBw7+Tba+iM9QPB9ZOvM2Fa4OH1et/fbjRmaEbweOfC9KUogsr85QiH8sQBes7lsjRNSuSQ==
X-Received: by 2002:adf:f08d:: with SMTP id n13mr11015515wro.339.1632483654167; 
 Fri, 24 Sep 2021 04:40:54 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id j23sm8741072wmo.14.2021.09.24.04.40.52
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 04:40:53 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN>
Date: Fri, 24 Sep 2021 14:40:51 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <83k0j6trau.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 09:34, Eli Zaretskii wrote:
>> Cc: 50733 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@HIDDEN>
>> Date: Fri, 24 Sep 2021 01:40:58 +0300
>>
>> Perhaps on a platform like macOS we should consider bundling some
>> up-to-date search program, be it GNU Grep or Ripgrep.
>>
>> Eli, have there ever been a similar proposal under discussion? IIRC
>> having to install external tools has been an issue on MS Windows as well
>> for a long time. We give recommendations, but Grep has never been
>> distributed together with Emacs. Any particular reason?
>>
>> For comparison, VS Code bundles ripgrep. Or at least its macOS releases do.
> 
> Bundling an external program is problematic, but it can be done, of
> course, provided that the license is right.  The problem is, we don't
> provide macOS binaries, and we cannot rely on the fact that the
> Windows binaries distribution will have its maintainer forever: we
> already had periods of time without such a person.

If we just bundled GNU find and GNU grep in both distributions (or, for 
macOS, tell the distributors to do that), that would improve OOTB 
behavior for many users. It's not like there's a real chance that the 
user would prefer some different version of Grep, for example.

What is the concern about the presence of the current maintainer? Would 
we do something different if he leaves? We'd probably need to find 
another maintainer, no?

> We can recommend installing such tools, of course.

Maybe we could do better?

> Btw, I don't understand why we focus on general-purpose text-searching
> tools for these features.  Why not focus on packages like ID Utils
> instead, they are so much faster.  Daniel, could you time the same
> search in that large tree when xref-search-program is 'gid'?  (You'd
> need to run 'mkid' first, to create the ID database, but that is
> one-time, and is very fast.) 

There is no such option in xref-search-program. If you can suggest an 
appropriate invocation for xref-search-program-alist, I can try and add it.

But I thought id-tools are for scanning for identifiers, not for 
arbitrary regexp searches?

 >  As I told many times, I think this is
 > the future: program language sensitive tools that use a precomputed
 > DB.

xref-find-references implies some language-awareness. 
project-find-regexp does not.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 11:44:02 +0000
Resent-Message-ID: <handler.50733.B50733.163248382821409 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>, Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163248382821409
          (code B ref 50733); Fri, 24 Sep 2021 11:44:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 11:43:48 +0000
Received: from localhost ([127.0.0.1]:56797 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTjcK-0005ZE-GJ
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:43:48 -0400
Received: from mail-wr1-f54.google.com ([209.85.221.54]:36571)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTjcI-0005Z2-RV
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 07:43:47 -0400
Received: by mail-wr1-f54.google.com with SMTP id g16so26613188wrb.3
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 04:43:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=zBJUWatBn0Frk9S6k4DaeqSM1tXp8+Zd3BhncAjew+0=;
 b=lOZ/7TMjfe1TiO9hUnJ3PmSmyGP8U1f9VCCrM1yRNfBfXswg7AFHbFYfzSaPJnJEM3
 qsh26dwXxdugwfMGKGUn/yvLZcwVlL1m5zhts4NB1CkC/m1lLqLjunDxh31UJ/3o5anJ
 B0Rc3/Ep5nlCfOtqU3xIuWb1uiAijbgYOVTeAw6MmzdEhxFBVTJ3BKdZkY2J87FBpbE7
 tWsFJBe4n/EAU9LdnhCusVzIKGasAI1kDjlfMpXfqZ/rcffwqaswRSC3l3hXbtVJazdO
 l5CcobwGO3quQStmqXOPnPHcMc5Eie86Lr7Rk3a2eufIK5dypBvcNYT/ZgM8n5ChTUje
 mQCA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=zBJUWatBn0Frk9S6k4DaeqSM1tXp8+Zd3BhncAjew+0=;
 b=3JaXhF2wxSXAED/LO+TK402G1GK55f61ibhT8q9ychQmwOJUMKmnYyu/CAFbiUJbsu
 +oghf8SmrFOstSWs+RAj+hjeb003UDCwgM8e610HbXBk0tct/U/iJ2i74bjKXfMlJylo
 FddobydqEARB2suZ//AjP1Xvbxp296sJ6eUvtfLVesKu7NlfRSDup+xZ48DjOmNtBVGi
 1xe/wQj7Kg77GdwSQH81nX9kMLJb2hAZXekuMdeJ3/PS1ITNa2zcrOYu0hM/19rLGnld
 yF/wacEbuI6ugxAkDjamkws3MQB5sHt7WeBttwSRE3T5AxIET2YDLZYgDA3mFfESd9Ki
 23pg==
X-Gm-Message-State: AOAM532A6YIy8/AOKCjysTYOLx+9oRLoyVGNlKK2erYVB1U5FDWtJpWH
 CjN9Quth6/mlPOArqZsfWaI=
X-Google-Smtp-Source: ABdhPJzNLnLowNPduEW3WI5TELnZATRwyCgXpkmp1E9ZvLx3an6k4QBppamKB2GP7Se7nrMwlkUVzA==
X-Received: by 2002:adf:f949:: with SMTP id q9mr1499820wrr.331.1632483821231; 
 Fri, 24 Sep 2021 04:43:41 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id s24sm7761341wmh.34.2021.09.24.04.43.40
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 04:43:40 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <d1f33097-f1b5-25c3-8f21-06169d4e6c6d@HIDDEN>
Date: Fri, 24 Sep 2021 14:43:39 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <838rzmtf00.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 14:00, Eli Zaretskii wrote:
> If someone wants to look for arbitrary strings,
> they want "M-x grep" (with or without ripgrep), not
> project-find-regexp.

That's not true, sorry. project-find-regexp is for (basically) grepping 
across the project, the way it has been configured by the user (with 
ignore entries and everything). We don't make any effort to limit to 
program source code files, for example, and we want to support arbitrary 
regexps.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 12:03:02 +0000
Resent-Message-ID: <handler.50733.B50733.163248495531943 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163248495531943
          (code B ref 50733); Fri, 24 Sep 2021 12:03:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 12:02:35 +0000
Received: from localhost ([127.0.0.1]:56834 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTjuV-0008J7-9c
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 08:02:35 -0400
Received: from eggs.gnu.org ([209.51.188.92]:53364)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTjuT-0008E5-7A
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 08:02:34 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:45418)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTjuN-0004Vs-J0; Fri, 24 Sep 2021 08:02:27 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3440
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTjuM-0000oj-Gw; Fri, 24 Sep 2021 08:02:27 -0400
Date: Fri, 24 Sep 2021 15:02:20 +0300
Message-Id: <8335putc4z.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN> (message from
 Dmitry Gutov on Fri, 24 Sep 2021 14:40:51 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: mardani29@HIDDEN, 50733 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 24 Sep 2021 14:40:51 +0300
> 
> > Bundling an external program is problematic, but it can be done, of
> > course, provided that the license is right.  The problem is, we don't
> > provide macOS binaries, and we cannot rely on the fact that the
> > Windows binaries distribution will have its maintainer forever: we
> > already had periods of time without such a person.
> 
> If we just bundled GNU find and GNU grep in both distributions (or, for 
> macOS, tell the distributors to do that), that would improve OOTB 
> behavior for many users. It's not like there's a real chance that the 
> user would prefer some different version of Grep, for example.

You don't need to convince me, I have all of those installed, and
couldn't do without them.  The difficulties are practical, not
principal.

> What is the concern about the presence of the current maintainer? Would 
> we do something different if he leaves? We'd probably need to find 
> another maintainer, no?

Yes, and the question is: will we succeed, and how quickly?  We
already had a period of time without one, which means having no one is
not a theoretical danger.

> > Btw, I don't understand why we focus on general-purpose text-searching
> > tools for these features.  Why not focus on packages like ID Utils
> > instead, they are so much faster.  Daniel, could you time the same
> > search in that large tree when xref-search-program is 'gid'?  (You'd
> > need to run 'mkid' first, to create the ID database, but that is
> > one-time, and is very fast.) 
> 
> There is no such option in xref-search-program.

Hmm... I'm sure this worked in the past, at least for
xref-find-references?  Or does that use a different variable?

> If you can suggest an 
> appropriate invocation for xref-search-program-alist, I can try and add it.

Just "gid -r <R>".  But if this could run in an arbitrary directory,
there should also be a "-f .../ID" argument, telling it where to find
the ID database (usually, in the project's root).

> But I thought id-tools are for scanning for identifiers, not for 
> arbitrary regexp searches?

They can scan for identifiers that match regular expressions, where
"identifier" is defined by each file's PL.

>  >  As I told many times, I think this is
>  > the future: program language sensitive tools that use a precomputed
>  > DB.
> 
> xref-find-references implies some language-awareness. 
> project-find-regexp does not.

Are you sure people indeed use project-find-regexp _only_ for
language-agnostic searches?

Anyway, maybe we need a separate command, or a different (but similar)
tool, one that indexes the tokens in the project files instead of
scanning all of the files each time anew.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 12:23:02 +0000
Resent-Message-ID: <handler.50733.B50733.16324861712132 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16324861712132
          (code B ref 50733); Fri, 24 Sep 2021 12:23:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 12:22:51 +0000
Received: from localhost ([127.0.0.1]:56901 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTkE6-0000YK-QQ
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 08:22:51 -0400
Received: from mail-wr1-f46.google.com ([209.85.221.46]:33666)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTkE5-0000Y2-HU
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 08:22:49 -0400
Received: by mail-wr1-f46.google.com with SMTP id t18so27035169wrb.0
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 05:22:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=kpskrbAadRsxUY8i0fy8g7pyixeQZ4FzsKZGDiLXTjc=;
 b=ohd/VvU3Zg6sC9jMKgp+YOagaDjwazmE2X/XsWlq++Hm8lqw7wJoV/DuZYge9kkRtW
 qcyvtnXKWgpQc1ZacB44BZExOKQgEq0273rBU+Qh8h0ZeE9Y5NRZToZpFg8xb2zBtOh5
 WyIhanhmw6WFWSIraV0d4RmV6xSjyA2kyE1fZ/MBfgWNhlxOZk+zuYL/phYcgSpF41YG
 GAWKArgDPQXXfKXS8YywW1VJTxhFZksx507XQBuPq+cuEm1a5R7zQGJ6J3Rr/v+R9vme
 77l7XtjRMAA9t4kQoJ2eh0ilYw2C24ulENGYH/RtI79s5CYv5qNdHmTorqI5bFRmbDx4
 Hqfg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=kpskrbAadRsxUY8i0fy8g7pyixeQZ4FzsKZGDiLXTjc=;
 b=MgKGWHDrKlgBE2AG61wAhylsUuTb3qfpUILsUL5d95HFeSdf5bOBOIXn0Usq7JjoXn
 RcTBthckBxC7kBJvXWqg1mudA+5KKhuAZsIEir3B2uexXpAy1PJayMXg24aUnSr5FBnE
 AAAY1y+wannHKYXHJ+FDJ4puI26fc34RPfwAKuwX2a3D83uJG0pohDm7z93n2zTr4YvO
 DSBDhaeqYQf/5plVEnNRBkJywWadxT2PShi9wnpVfDTMy+bd3QboVQG/C8o4F2Cm7Usq
 bXQrFDibBH7eDr39h0xFiE8s1RL/R8aghKsv/X6kVuue7ybisUnThNH6PXrP1gLChRzh
 2abA==
X-Gm-Message-State: AOAM5309ZQ4jQ8t/SoPTaKcnEaPnPhE1h3ZNdzfSfFdUkXKI+imRR+4P
 MMNENX5uu0oQahrO6V7yLThLbJTvunA=
X-Google-Smtp-Source: ABdhPJywN0oFcIykHs8ezbcaCROg1SIHhhkW71U0ALqA3B/1oKdho+NXnEwOXfiyWg725zFHRrjM4w==
X-Received: by 2002:a5d:4f06:: with SMTP id c6mr11133635wru.384.1632486163669; 
 Fri, 24 Sep 2021 05:22:43 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id c77sm8264938wme.46.2021.09.24.05.22.42
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 05:22:42 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN> <8335putc4z.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <4c2be3c4-52f5-00c2-9c31-eafa0df5bfd9@HIDDEN>
Date: Fri, 24 Sep 2021 15:22:39 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <8335putc4z.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 15:02, Eli Zaretskii wrote:

> You don't need to convince me, I have all of those installed, and
> couldn't do without them.  The difficulties are practical, not
> principal.

Bundling extra tools affects the size of the distribution, for example. 
I figured that's something you might have an opinion on.

>> What is the concern about the presence of the current maintainer? Would
>> we do something different if he leaves? We'd probably need to find
>> another maintainer, no?
> 
> Yes, and the question is: will we succeed, and how quickly?  We
> already had a period of time without one, which means having no one is
> not a theoretical danger.

I'm not sure how that affects our opinion on bundling, though.

>>> Btw, I don't understand why we focus on general-purpose text-searching
>>> tools for these features.  Why not focus on packages like ID Utils
>>> instead, they are so much faster.  Daniel, could you time the same
>>> search in that large tree when xref-search-program is 'gid'?  (You'd
>>> need to run 'mkid' first, to create the ID database, but that is
>>> one-time, and is very fast.)
>>
>> There is no such option in xref-search-program.
> 
> Hmm... I'm sure this worked in the past, at least for
> xref-find-references?  Or does that use a different variable?

Different variable, yes: semantic-symref-tool.

It affects xref-find-references.

>> If you can suggest an
>> appropriate invocation for xref-search-program-alist, I can try and add it.
> 
> Just "gid -r <R>".  But if this could run in an arbitrary directory,
> there should also be a "-f .../ID" argument, telling it where to find
> the ID database (usually, in the project's root).

Anyway, it seems the most pressing problem is not the performance of the 
external tool (ripgrep is quite fast, for example). But our 
post-processing of the results, when there are a lot of them.

id-tools (or alternatives) could help avoid the "fetch all project 
files" step, though. At the cost of extra manual management, which I 
personally don't like much.

>> But I thought id-tools are for scanning for identifiers, not for
>> arbitrary regexp searches?
> 
> They can scan for identifiers that match regular expressions, where
> "identifier" is defined by each file's PL.
> 
>>   >  As I told many times, I think this is
>>   > the future: program language sensitive tools that use a precomputed
>>   > DB.
>>
>> xref-find-references implies some language-awareness.
>> project-find-regexp does not.
> 
> Are you sure people indeed use project-find-regexp _only_ for
> language-agnostic searches?

Oh, I use it for all kinds of searches. The point is not being limited 
in the search input or the choice of files to search.

Instead, we can and will add UI for specifying extra filters for the 
results.

> Anyway, maybe we need a separate command, or a different (but similar)
> tool, one that indexes the tokens in the project files instead of
> scanning all of the files each time anew.

"Tokens" implies searching only for identifiers, no?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 13:07:02 +0000
Resent-Message-ID: <handler.50733.B50733.163248878915765 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163248878915765
          (code B ref 50733); Fri, 24 Sep 2021 13:07:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 13:06:29 +0000
Received: from localhost ([127.0.0.1]:56924 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTkuL-00046C-5S
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 09:06:29 -0400
Received: from eggs.gnu.org ([209.51.188.92]:36518)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTkuJ-00045s-G3
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 09:06:27 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51736)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTkuE-0006ZE-5S; Fri, 24 Sep 2021 09:06:22 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3390
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTkuD-0004Fj-PL; Fri, 24 Sep 2021 09:06:22 -0400
Date: Fri, 24 Sep 2021 16:06:16 +0300
Message-Id: <83zgs2rulz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <4c2be3c4-52f5-00c2-9c31-eafa0df5bfd9@HIDDEN> (message from
 Dmitry Gutov on Fri, 24 Sep 2021 15:22:39 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN> <8335putc4z.fsf@HIDDEN>
 <4c2be3c4-52f5-00c2-9c31-eafa0df5bfd9@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: mardani29@HIDDEN, 50733 <at> debbugs.gnu.org
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 24 Sep 2021 15:22:39 +0300
> 
> On 24.09.2021 15:02, Eli Zaretskii wrote:
> 
> > You don't need to convince me, I have all of those installed, and
> > couldn't do without them.  The difficulties are practical, not
> > principal.
> 
> Bundling extra tools affects the size of the distribution, for example. 
> I figured that's something you might have an opinion on.

I doubt that the size of the distribution matters much these days.
Especially since tools like Grep usually have very small sizes of the
binary distribution.

> >> What is the concern about the presence of the current maintainer? Would
> >> we do something different if he leaves? We'd probably need to find
> >> another maintainer, no?
> > 
> > Yes, and the question is: will we succeed, and how quickly?  We
> > already had a period of time without one, which means having no one is
> > not a theoretical danger.
> 
> I'm not sure how that affects our opinion on bundling, though.

It might mean that bundling won't happen for mundane reasons.

> > Just "gid -r <R>".  But if this could run in an arbitrary directory,
> > there should also be a "-f .../ID" argument, telling it where to find
> > the ID database (usually, in the project's root).
> 
> Anyway, it seems the most pressing problem is not the performance of the 
> external tool (ripgrep is quite fast, for example). But our 
> post-processing of the results, when there are a lot of them.

Yes.

> id-tools (or alternatives) could help avoid the "fetch all project 
> files" step, though. At the cost of extra manual management, which I 
> personally don't like much.

Why is that a problem?  We have midnight.el which could be part of the
solution.

> > Anyway, maybe we need a separate command, or a different (but similar)
> > tool, one that indexes the tokens in the project files instead of
> > scanning all of the files each time anew.
> 
> "Tokens" implies searching only for identifiers, no?

For ID Utils, yes (although in text files those are words).  For some
other similar tool it could be something else.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Gregory Heytings <gregory@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 13:19:02 +0000
Resent-Message-ID: <handler.50733.B50733.163248950617366 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163248950617366
          (code B ref 50733); Fri, 24 Sep 2021 13:19:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 13:18:26 +0000
Received: from localhost ([127.0.0.1]:56929 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTl5u-0004W0-Ci
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 09:18:26 -0400
Received: from heytings.org ([95.142.160.155]:45976)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gregory@HIDDEN>) id 1mTl5t-0004Vs-7x
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 09:18:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heytings.org;
 s=20210101; t=1632489503;
 bh=hliP6yocegdMtgQTaX/ogGIiG8U9HDefrLGqlcKvc5Q=;
 h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:From;
 b=D/mIX9lhXTX2V9dwDzbekfyLlIpN0b9foe1UtEET+4HD5LruKn0QZ2PIl/Q67vDea
 WB8ktVppz78tDRgpJQpywXn8/xV2wFEBzZoccfiLWhasWdjOx67Y1Avb8WANcIHueN
 NbR9cRgbaiTtGEKgfu9hIK/eYa5fOaBWReqH9EbMh/HxgaQ30QzVgcKfy76Vzak1LC
 HHOT9TUELLd+DESQbQduQheaz9sQMgUrV2gQXwhhnxcpO94Jce01ApjoE7auI19Pk7
 hfojepcMDB3lgx4kRiVrsEHyXx8hndYS0rb0HQHiuVyokodJWii/hy5ZNY8nLGNZFi
 Rn8s1u/n1rWaQ==
Date: Fri, 24 Sep 2021 13:18:22 +0000
From: Gregory Heytings <gregory@HIDDEN>
In-Reply-To: <838rzmtf00.fsf@HIDDEN>
Message-ID: <d70d4a91f2b91f140d52@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=us-ascii
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


>
> We are talking about project.el, so the focus should be on looking for 
> strings that are meaningful in the programming-language context, because 
> that is what we need when programming.  Not looking for arbitrary 
> strings.
>

As Dmitry said, this isn't true.  But let's assume that it is.  The latter 
(arbitrary strings) is a superset of the former (identifiers).  In 
practice if you search for some identifier, it's not a real problem to see 
a few more matches in, say, the README or a ChangeLog file.

IMO, the one and only case where a specialized tool beats ripgrep (or just 
plain grep) is when you just want the place(s) where the identifier is 
defined.

>
> If someone wants to look for arbitrary strings, they want "M-x grep" 
> (with or without ripgrep), not project-find-regexp.  The meaning of 
> "word" or "symbol" is different in different PLs; a tool like Grep can 
> only approximate that, whereas ID Utils uses the correct definition for 
> each language.
>

That's not correct, mkid only supports a limited number of programming 
languages.  And it's not even precise: rg O_CREAT on the Emacs trunk for 
example returns 45 matches, gid O_CREAT returns 33 matches.

>> Moreover, incremental updates are not implemented in mkid.
>
> It doesn't matter, as it's fast enough, and putting it on some kind of 
> cron job is more than enough to allow forgetting that this step exists.
>

Five seconds to scan the whole Emacs trunk is IMO not fast enough (ripgrep 
does it in < 0.2 seconds).  And without incremental updates, updating the 
database would be necessary before each invocation of gid, because what 
users expect when they search for something are accurate results 
corresponding to the current state of the project, not results from, say, 
an hour ago.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 14:06:02 +0000
Resent-Message-ID: <handler.50733.B50733.163249232025657 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Daniel =?UTF-8?Q?Mart=C3=ADn?= <mardani29@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163249232025657
          (code B ref 50733); Fri, 24 Sep 2021 14:06:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 14:05:20 +0000
Received: from localhost ([127.0.0.1]:58957 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTlpI-0006fl-Ki
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 10:05:20 -0400
Received: from mail-wr1-f47.google.com ([209.85.221.47]:36671)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTlpG-0006fU-CK
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 10:05:18 -0400
Received: by mail-wr1-f47.google.com with SMTP id g16so27904578wrb.3
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 07:05:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=ecQ3W8flyXlm4J+LWOPaFOkTyKPq7jVMQHIuKtkBL6E=;
 b=chpbX6ykF1lGnScu8j7lTL/EUSz4+ARJ92nVQlo+OaTre0yyMQIrt+nf/sulhpupiT
 Aqde2clH4C5+f8QGv3XlYGOEL0be3+OdBi14JqxcYIYo8Ih6SSimdngdtBihG8wmmi1p
 kyDB9o//x7shLiYoReQjM3c5nQoI2HGNIaieOJSrEJ8gXXXNqiHIiflVm+zHYdsX3F/D
 FdQfz4eq7Ryx9lQKIOU1kUWANzJgS+IFdR+G3IXXYqFcKmo+qaFEHv2FMtwmwWsXNpsg
 qgFiB8lTtoOSP03h+0HE1sDw7AKcbxU32e9Zz1xP79NuWyZOR5ttv4F9Cf4KMl3tRcxF
 JSuw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=ecQ3W8flyXlm4J+LWOPaFOkTyKPq7jVMQHIuKtkBL6E=;
 b=nMuto80iEOEZSEI/yoT4ydrPHT5mHVhVBiLoDpv6Sqo3QV2zWD/VtORMkrLEvpvIEh
 GaP5B8V6JMKpxh6nnMtSwXJX+WRCzOxRTFEs9YbuASA+6bVBFYMur1ChDT7PPiukn+r2
 gryfANUzHakQmnFOUz355S/WCapUZ0MsRT8vBmnhWXR1AmrUdvH+bKzPv3s+BwRl6Z+6
 L0YCgK6xyJcKHbNSp4WysAxm0fqpY+oyN79VZwWT0dY3oL/CjEDYDDVPhsqetl8CCWFJ
 sPvmJIf9tIKY+KC5gyzi5hoeLuQ4Hdifr1xzKVWteL9AHt9yPdfyCS2nFTvTuqinvhIp
 Fm3A==
X-Gm-Message-State: AOAM531/UHkLcHrWI/TrowO5wnfSChejK7UmMuQKTWnxaa15nkkdMkTj
 NikaByOKUK/TmZLVnAog+lI=
X-Google-Smtp-Source: ABdhPJwRsvd9tC9oIBtGfryQOmXO1OVYIpn951E5MFRJewv6zX9i/P0M8wGtCmI/M4pQgRCco8Bc6g==
X-Received: by 2002:adf:e643:: with SMTP id b3mr12074393wrn.67.1632492312558; 
 Fri, 24 Sep 2021 07:05:12 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id i6sm2730820wrx.61.2021.09.24.07.05.10
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 07:05:11 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <5679c01b-51d0-903d-039a-8102f64fdb37@HIDDEN> <8335putc4z.fsf@HIDDEN>
 <4c2be3c4-52f5-00c2-9c31-eafa0df5bfd9@HIDDEN> <83zgs2rulz.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <1d42a902-b490-56f2-2257-eb5d834900c9@HIDDEN>
Date: Fri, 24 Sep 2021 17:05:08 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <83zgs2rulz.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 16:06, Eli Zaretskii wrote:

>>>> What is the concern about the presence of the current maintainer? Would
>>>> we do something different if he leaves? We'd probably need to find
>>>> another maintainer, no?
>>>
>>> Yes, and the question is: will we succeed, and how quickly?  We
>>> already had a period of time without one, which means having no one is
>>> not a theoretical danger.
>>
>> I'm not sure how that affects our opinion on bundling, though.
> 
> It might mean that bundling won't happen for mundane reasons.

All right. But we can ask.

Before we do that,

Daniel, could you do a couple of benchmarks specifically with GNU Grep?

There are instructions for installing it here: 
https://apple.stackexchange.com/a/193300

Ripgrep might be faster (at least on some inputs), but even if we 
supported it for all kinds of searches (which we currently don't), we 
really won't get auto-detection into Emacs 28, so if GNU Grep's 
performance is within ~20%, we should focus on it first.

To improve OOTB performance across the board.

>> id-tools (or alternatives) could help avoid the "fetch all project
>> files" step, though. At the cost of extra manual management, which I
>> personally don't like much.
> 
> Why is that a problem?  We have midnight.el which could be part of the
> solution.

As Gregory mentioned, users often expect the results to be up-to-date, 
especially when it concerns a command as common as "find regexp". Doing 
it on cron is not ideal.

But of course we're free to add extra knobs for particular power users.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 14:21:02 +0000
Resent-Message-ID: <handler.50733.B50733.16324932523327 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16324932523327
          (code B ref 50733); Fri, 24 Sep 2021 14:21:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 14:20:52 +0000
Received: from localhost ([127.0.0.1]:58980 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTm4G-0000rX-St
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 10:20:52 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51150)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTm4E-0000rG-Tr
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 10:20:47 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:54864)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTm48-0004jp-E2; Fri, 24 Sep 2021 10:20:40 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3970
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTm47-0005AZ-17; Fri, 24 Sep 2021 10:20:39 -0400
Date: Fri, 24 Sep 2021 17:20:32 +0300
Message-Id: <83sfxurr67.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <d70d4a91f2b91f140d52@HIDDEN> (message from Gregory
 Heytings on Fri, 24 Sep 2021 13:18:22 +0000)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 24 Sep 2021 13:18:22 +0000
> From: Gregory Heytings <gregory@HIDDEN>
> cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN, mardani29@HIDDEN
> 
> IMO, the one and only case where a specialized tool beats ripgrep (or just 
> plain grep) is when you just want the place(s) where the identifier is 
> defined.

No, a specialized tool that uses a DB will scale much better than any
tool which searches the filesystem.  _And_ it will be more accurate
(if used correctly).

> That's not correct, mkid only supports a limited number of programming 
> languages. And it's not even precise: rg O_CREAT on the Emacs trunk for 
> example returns 45 matches, gid O_CREAT returns 33 matches.

I'm sorry, but this has NIH written all over it.  Am I right guessing
that you aren't an active user of ID Utils, and perhaps didn't even
know about it before I mentioned it?

More to the point: are you saying that a tool that returns more
matches is necessarily better?  That'd fly in the face of our
switching to the Xref package, whose explicit goal was to provide
_fewer_ matches, the ones that matter.  Look closer at those matches
which gid "missed", and you will see why it didn't show them to you.

Oh, and if ripgrep finds only 45 matches, then something is wrong with
it, because there are actually no less than 119 literal matches for
O_CREAT in the tree (not counting many binary files that also match).
So by this measure, ripgrep is also not the right tool for the job.

> Five seconds to scan the whole Emacs trunk is IMO not fast enough (ripgrep 
> does it in < 0.2 seconds).

Those 5 sec are invested only when needed, while the time it takes
Grep/ripgrep to scan the files is invested every search.  Do this
enough times, and you paid too much time.

> And without incremental updates, updating the database would be
> necessary before each invocation of gid, because what users expect
> when they search for something are accurate results corresponding to
> the current state of the project, not results from, say, an hour
> ago.

It is very easy to trigger a new mkid run from a file-notification
that watches the project tree, so that it runs in the background
without the user noticing when needed.  Puff! the problem's gone.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Juri Linkov <juri@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 15:47:02 +0000
Resent-Message-ID: <handler.50733.B50733.163249837413759 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163249837413759
          (code B ref 50733); Fri, 24 Sep 2021 15:47:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 15:46:14 +0000
Received: from localhost ([127.0.0.1]:59082 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTnOw-0003Zr-Eq
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 11:46:14 -0400
Received: from relay1-d.mail.gandi.net ([217.70.183.193]:54423)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <juri@HIDDEN>) id 1mTnOu-0003ZZ-GV
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 11:46:12 -0400
Received: (Authenticated sender: juri@HIDDEN)
 by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 562A024000B;
 Fri, 24 Sep 2021 15:46:03 +0000 (UTC)
From: Juri Linkov <juri@HIDDEN>
Organization: LINKOV.NET
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
 <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <87lf3mmp6b.fsf@HIDDEN>
 <83bl4itfru.fsf@HIDDEN>
Date: Fri, 24 Sep 2021 18:30:56 +0300
In-Reply-To: <83bl4itfru.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 24 Sep
 2021 13:43:49 +0300")
Message-ID: <875yuqp0sn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

>> Is its command line more complex than etags?
>
> No, it's much more simple.  Assuming you are in the top-level
> directory of the Emacs source tree, the command is just:
>
>      mkid
>
> That produces a file named 'ID' in that directory.

I tried to generate 'ID' with 'mkid', and then xref-find-references
works fine with it.  The only problem is that it finds occurrences
in .elc files.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 15:51:02 +0000
Resent-Message-ID: <handler.50733.B50733.163249863214404 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>, Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163249863214404
          (code B ref 50733); Fri, 24 Sep 2021 15:51:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 15:50:32 +0000
Received: from localhost ([127.0.0.1]:59098 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTnT6-0003kG-H5
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 11:50:32 -0400
Received: from mail-wr1-f42.google.com ([209.85.221.42]:35829)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTnT4-0003jz-IS
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 11:50:31 -0400
Received: by mail-wr1-f42.google.com with SMTP id i23so28931111wrb.2
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 08:50:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=77ZvwczZiERf9UMGSimRisNOVwHR/o+qPvTGgePiHqM=;
 b=kglPZHlk2u68YrupPItjZgIgkTv25NjG1/AHpLIMeRfVDu5ovShVMmwjE5rNX7XaLP
 YdlLHxWRqmnBAOUuL67wqR+piGvzr4JI2bfRWbJO1KmGS4xzIs/tKDYSX8LKHJcdlY8K
 IFT12ZKw8pXKBEAcXMZ4tKb2ODhzSA6d29dHQu4UxqyGEg94fZtK5oX7NOtnVLuYQfSV
 +zAVQ1w3fw3QSrtiVNq1o1ecuEOyzGaOsXmQnQzzRLoJPAS2F1VR/PNiLKb79jilunyY
 Z4H+3ujhP7U56uNDJnzgOtgAAIMt5Hs3PykTLc+Hg34oKrOqPsKBI+pOtT4LFE7j9YuA
 YWhA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=77ZvwczZiERf9UMGSimRisNOVwHR/o+qPvTGgePiHqM=;
 b=RvafCk2yEU/vqzAw35iNF7rkxvxKKgkk/6a+PO8no2wHpfuAONz1FwZ21IpMJ5lcpr
 Y727++CC8t9fVgYBiXnr1LW2ow22c3usC2FJgyGbF+D3J+CY20D5UrhU+SL/TDWajLYn
 I0Mo1iO2f/d9LsBS8IQ5EBUz0PX12rUs9Rlaj55cGYeDwbmH31KPFUMIRjHks33cCG5d
 iVpohwmZWum5y6ROGRR3ssjs6pwEgovJsPJJpJ8Wk6TNUNKhoRfP41aEx60vV0NHw73G
 WJRz8Nz047TgtPsIIzXXd/RNE79LIPJgTvZU7dGeAd3m6GkeC84dhwZtKPk4XqMPNrYB
 M4IQ==
X-Gm-Message-State: AOAM5334FjHx8tHBLFaF2buOs6IYmomKnd+RaDMPCvwjxT1phECKtIZF
 psifE+VdWR8mhAmVgSnDgYQ=
X-Google-Smtp-Source: ABdhPJz+FPZayPwPx/PPA2yDNEE4322L3Q3HncYZOF6uknSsUxPMZ0KyMtcQgsIxH0JCMULZpAjn3w==
X-Received: by 2002:adf:d209:: with SMTP id j9mr12414205wrh.97.1632498624628; 
 Fri, 24 Sep 2021 08:50:24 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id c15sm9120839wrc.83.2021.09.24.08.50.22
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 08:50:24 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
 <d70d4a91f2b91f140d52@HIDDEN> <83sfxurr67.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <c203e7c9-722c-2ea5-f784-12a7f632c68c@HIDDEN>
Date: Fri, 24 Sep 2021 18:50:20 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <83sfxurr67.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 17:20, Eli Zaretskii wrote:

> More to the point: are you saying that a tool that returns more
> matches is necessarily better?  That'd fly in the face of our
> switching to the Xref package, whose explicit goal was to provide
> _fewer_ matches, the ones that matter.  Look closer at those matches
> which gid "missed", and you will see why it didn't show them to you.

Xref's find-definition and find-references indeed try to be more 
precise. But that's only for identifier searches.

> Oh, and if ripgrep finds only 45 matches, then something is wrong with
> it, because there are actually no less than 119 literal matches for
> O_CREAT in the tree (not counting many binary files that also match).
> So by this measure, ripgrep is also not the right tool for the job.

I'm guessing the remaining entries are somewhere in gitignore-d files?

>> Five seconds to scan the whole Emacs trunk is IMO not fast enough (ripgrep
>> does it in < 0.2 seconds).
> 
> Those 5 sec are invested only when needed, while the time it takes
> Grep/ripgrep to scan the files is invested every search.  Do this
> enough times, and you paid too much time.

FWIW, (project-find-regexp "O_CREAT") takes about 130ms here.

That's close to the perceptual "instant" time.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 16:09:02 +0000
Resent-Message-ID: <handler.50733.B50733.163249973416615 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Juri Linkov <juri@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163249973416615
          (code B ref 50733); Fri, 24 Sep 2021 16:09:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 16:08:54 +0000
Received: from localhost ([127.0.0.1]:59114 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTnks-0004Jv-Gz
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:08:54 -0400
Received: from eggs.gnu.org ([209.51.188.92]:34140)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTnkq-0004Je-Sg
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:08:53 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:57984)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTnkk-0003Tw-PC; Fri, 24 Sep 2021 12:08:46 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2599
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTnkk-0005Ww-6T; Fri, 24 Sep 2021 12:08:46 -0400
Date: Fri, 24 Sep 2021 19:08:40 +0300
Message-Id: <83pmsyrm5z.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <875yuqp0sn.fsf@HIDDEN> (message from Juri Linkov on
 Fri, 24 Sep 2021 18:30:56 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN>
 <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN>
 <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <87lf3mmp6b.fsf@HIDDEN>
 <83bl4itfru.fsf@HIDDEN> <875yuqp0sn.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Juri Linkov <juri@HIDDEN>
> Cc: dgutov@HIDDEN,  50733 <at> debbugs.gnu.org,  mardani29@HIDDEN
> Date: Fri, 24 Sep 2021 18:30:56 +0300
> 
> I tried to generate 'ID' with 'mkid', and then xref-find-references
> works fine with it.  The only problem is that it finds occurrences
> in .elc files.

You can edit the id-lang.map file to ignore the *.elc files while
scanning.  Or use the -p option to mkid to exclude them.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 16:20:02 +0000
Resent-Message-ID: <handler.50733.B50733.163250034217826 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: gregory@HIDDEN, 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250034217826
          (code B ref 50733); Fri, 24 Sep 2021 16:20:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 16:19:02 +0000
Received: from localhost ([127.0.0.1]:59123 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTnuf-0004dN-PT
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:19:02 -0400
Received: from eggs.gnu.org ([209.51.188.92]:35880)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTnue-0004d6-2w
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:19:00 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:58798)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTnuY-00023j-IO; Fri, 24 Sep 2021 12:18:54 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3220
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTnuY-0006eK-49; Fri, 24 Sep 2021 12:18:54 -0400
Date: Fri, 24 Sep 2021 19:18:51 +0300
Message-Id: <83k0j6rlp0.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <c203e7c9-722c-2ea5-f784-12a7f632c68c@HIDDEN> (message from
 Dmitry Gutov on Fri, 24 Sep 2021 18:50:20 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
 <d70d4a91f2b91f140d52@HIDDEN> <83sfxurr67.fsf@HIDDEN>
 <c203e7c9-722c-2ea5-f784-12a7f632c68c@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 24 Sep 2021 18:50:20 +0300
> 
> FWIW, (project-find-regexp "O_CREAT") takes about 130ms here.

"gid O_CREAT" takes 46 msec, and that's on MS-Windows.

Do we really need to keep this silly argument?  Every tool has its
advantages and disadvantages.  When used correctly, you will have only
the advantages and none of the disadvantages.  Can we agree on this
and move on, please?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 16:23:01 +0000
Resent-Message-ID: <handler.50733.B50733.163250053418231 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: gregory@HIDDEN, 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250053418231
          (code B ref 50733); Fri, 24 Sep 2021 16:23:01 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 16:22:14 +0000
Received: from localhost ([127.0.0.1]:59133 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTnxm-0004jy-Em
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:22:14 -0400
Received: from mail-wr1-f50.google.com ([209.85.221.50]:36695)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTnxi-0004jP-11
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:22:12 -0400
Received: by mail-wr1-f50.google.com with SMTP id g16so29137287wrb.3
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 09:22:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=/uVHaE4yiZqrqRkUC2w/ibk7/gMFuQH+IdnvhjIMyS4=;
 b=muXIc5A5IMoboIOmmdtwTPET6XxR+il6XPT9fpIJiSWtIkMMBJJaEQOtuViGQShxgC
 f+y8K1V9aP6btcX7ghNxO+xhjxPi96pw0d1ZH9v/hqf4XXgs59HnfNHNRjS1LYJQfqyV
 foLJH6GkYOYXlNbMHRerWZL135LuPfc9lN/V+hICvpMiWMjayrNP0ajxWuvxlaKIBvcQ
 zNKAQA15m31olDX47MaRxwEP/PYFftHWoAAh3rgyHcVvb/e1hDKRePPKRzmE9RCuLYaF
 DumNx1baFV0ekOpYEdFxNuye/7kBBg+H7700oNewNGmKveTbfglJuY7Cijzhl3aPhorA
 Vijg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=/uVHaE4yiZqrqRkUC2w/ibk7/gMFuQH+IdnvhjIMyS4=;
 b=M2g8S6yRAu8qIjJJYrJkymmG4n/l1fxNwaGR9QhC+gjfQzA1D1F1QRky0OfqnxB8LM
 xWAzDWxAzn3YtL3eLqe9i2AhmR4aRASy0nzPq75WR5ikvweaoNKM9cY1yUNbkPfNZZUo
 ZeL7AzE0GGyVmQ+/OtbyOaLoKYl7/GymO2IV8N2a3IZiQrTDNj/EnykFm0i3PYhyv8b2
 FDt7MYWqhitk1fLzvkwRXjuGsaWnH3cB/Hq7JKyG8lvJ/sqrNoZ6UmyPYBOpoqo6mgEo
 GUlcTLXAE2fuX7H5K5u4QaNGWDxr0zu1XHkyuUgLvgCA7lnC+a1KUNJeCFdc8OF0MxUR
 6nYA==
X-Gm-Message-State: AOAM532iJ3Oxw1aShto0GiZ+/aV4PMXCyMSjSsV/1PtMCbiFJwxOi4SP
 X08S1o0WQh8Gd9scnbTw3Yc=
X-Google-Smtp-Source: ABdhPJwire4XIpI9yG35rUyHy+fnUge1e/I0wwWJWkAY7yAFUZ3eQl+F47t7nhPuYfi428e1YLOsag==
X-Received: by 2002:a1c:3845:: with SMTP id f66mr2984708wma.63.1632500524076; 
 Fri, 24 Sep 2021 09:22:04 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id z17sm8439710wml.24.2021.09.24.09.22.02
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 09:22:03 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
 <d70d4a91f2b91f140d52@HIDDEN> <83sfxurr67.fsf@HIDDEN>
 <c203e7c9-722c-2ea5-f784-12a7f632c68c@HIDDEN> <83k0j6rlp0.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <da960745-2d3f-1f7d-d0b5-1bbd7e715deb@HIDDEN>
Date: Fri, 24 Sep 2021 19:22:01 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <83k0j6rlp0.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 1.4 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  On 24.09.2021 19:18,
 Eli Zaretskii wrote: >> Cc:50733 <at> debbugs.gnu.org, mardani29@HIDDEN
 >> From: Dmitry Gutov >> Date: Fri, 24 Sep 2021 18:50:20 +0300 >> >> FWIW,
 (project-find-regexp "O_CREAT") tak [...] 
 Content analysis details:   (1.4 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.0 FREEMAIL_FROM          Sender email is commonly abused enduser mail
 provider (raaahh[at]gmail.com)
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
 mail domains are different
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
 [209.85.221.50 listed in wl.mailspike.net]
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at https://www.dnswl.org/,
 no trust [209.85.221.50 listed in list.dnswl.org]
 0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and
 EnvelopeFrom freemail headers are different
 -0.1 NICE_REPLY_A           Looks like a legit reply (A)
 1.0 FREEMAIL_REPLY         From and body contain different freemails
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 19:18, Eli Zaretskii wrote:
>> Cc:50733 <at> debbugs.gnu.org,mardani29@HIDDEN
>> From: Dmitry Gutov<dgutov@HIDDEN>
>> Date: Fri, 24 Sep 2021 18:50:20 +0300
>>
>> FWIW, (project-find-regexp "O_CREAT") takes about 130ms here.
> "gid O_CREAT" takes 46 msec, and that's on MS-Windows.
> 
> Do we really need to keep this silly argument?  Every tool has its
> advantages and disadvantages.  When used correctly, you will have only
> the advantages and none of the disadvantages.

I'm not sure we're arguing.

That was my point: it's good to be able to 'M-x project-find-regexp' 
without a pre-generated index. But make use of it in 
'xref-find-references' when the user has made such an index.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Gregory Heytings <gregory@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 16:25:02 +0000
Resent-Message-ID: <handler.50733.B50733.163250069418550 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250069418550
          (code B ref 50733); Fri, 24 Sep 2021 16:25:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 16:24:54 +0000
Received: from localhost ([127.0.0.1]:59139 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTo0M-0004p7-1I
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:24:54 -0400
Received: from heytings.org ([95.142.160.155]:46230)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gregory@HIDDEN>) id 1mTo0J-0004ow-RT
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 12:24:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heytings.org;
 s=20210101; t=1632500690;
 bh=d+cTtaxQGlNv0JbKvE9qkG7Hu1v7arotiQrJAHFpsv8=;
 h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:From;
 b=H3RO9l7lmzsl+05Q1nCwd7SL8nWldSbW3pu31VMjiYR5AVxGpflpliiBYMizX9Zd0
 cZJXbsHHsABRhGb5NvsQNdUCa9tlsSL2vMiX4zxp4ReKF/D3LNqoDXwKAHIy47wL8l
 SjqQH9uhN6xcPFkNraETsxpTjS5aPRgCsq9WHDRTdHsv4RK/GNsPHcrwi1Ko3Neyn1
 0WbyAa0SG+g0OcWEOCs/bg2UhPSItAYWJVSl5wfji3PWd6tViM6Gpo9RmvSBrpclZ7
 QeIqYpXhPdvFBMiU39WZEbKMQYdlu9I/yHDyg91R97ehOtVx4eE7MFQgeHyiNw2DNa
 hee+VCx24M7mw==
Date: Fri, 24 Sep 2021 16:24:50 +0000
From: Gregory Heytings <gregory@HIDDEN>
In-Reply-To: <83sfxurr67.fsf@HIDDEN>
Message-ID: <d70d4a91f2070500e1a8@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
 <83sfxurr67.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=us-ascii
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


>> IMO, the one and only case where a specialized tool beats ripgrep (or 
>> just plain grep) is when you just want the place(s) where the 
>> identifier is defined.
>
> No, a specialized tool that uses a DB will scale much better than any 
> tool which searches the filesystem.  _And_ it will be more accurate (if 
> used correctly).
>

Sorry, but I simply don't believe this.  At least not for general regex 
searches.  I'd be interested to see some numbers to support your 
viewpoint.

>> That's not correct, mkid only supports a limited number of programming 
>> languages. And it's not even precise: rg O_CREAT on the Emacs trunk for 
>> example returns 45 matches, gid O_CREAT returns 33 matches.
>
> I'm sorry, but this has NIH written all over it.  Am I right guessing 
> that you aren't an active user of ID Utils, and perhaps didn't even know 
> about it before I mentioned it?
>

You are wrong; of course I knew about ID Utils.  I tried and compared it 
with ripgrep a few years ago, and concluded that it's a (far) less useful 
tool, at least for my purposes, for the reasons I mentioned: it works with 
a database that must be updated, which is slow, and it is not faster than 
ripgrep.

>
> More to the point: are you saying that a tool that returns more matches 
> is necessarily better?
>

The purpose of project-find-regexp is to find all matches.

>
> Look closer at those matches which gid "missed", and you will see why it 
> didn't show them to you.
>

I looked close (before you asked), and no, I don't see why some matches 
are not included.  For example it returns

lib/tempname.c:212: __GT_FILE: create the file using open(O_CREAT|O_EXCL)

but not

lib/tempname.h:47: GT_FILE: create a large file using open(O_CREAT|O_EXCL)

and it returns

lib/open.c:99: /* Fail if one of O_CREAT, O_WRONLY, O_RDWR is specified and the filename

but not

lisp/gnus/nnmaildir.el:387: ;; If Emacs had O_CREAT|O_EXCL, we could return number-open here.

>
> Oh, and if ripgrep finds only 45 matches, then something is wrong with 
> it, because there are actually no less than 119 literal matches for 
> O_CREAT in the tree (not counting many binary files that also match). So 
> by this measure, ripgrep is also not the right tool for the job.
>

No, there are exactly 45 matches of "O_CREAT" on a fresh clone of the 
trunk.

>> Five seconds to scan the whole Emacs trunk is IMO not fast enough 
>> (ripgrep does it in < 0.2 seconds).
>
> Those 5 sec are invested only when needed, while the time it takes 
> Grep/ripgrep to scan the files is invested every search.  Do this enough 
> times, and you paid too much time.
>

Did you read the numbers I mentioned earlier?  rg O_CREAT is as fast as 
gid O_CREAT.  And this is without regexes; rg O.*CREAT is three times 
faster than gid O.*CREAT.

>> And without incremental updates, updating the database would be 
>> necessary before each invocation of gid, because what users expect when 
>> they search for something are accurate results corresponding to the 
>> current state of the project, not results from, say, an hour ago.
>
> It is very easy to trigger a new mkid run from a file-notification that 
> watches the project tree, so that it runs in the background without the 
> user noticing when needed.  Puff! the problem's gone.
>

Your "very easy" solution is still, IMO, an unnecessary complexity, with 
little (if any) benefit.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 17:50:01 +0000
Resent-Message-ID: <handler.50733.B50733.163250576328644 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250576328644
          (code B ref 50733); Fri, 24 Sep 2021 17:50:01 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 17:49:23 +0000
Received: from localhost ([127.0.0.1]:59215 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTpK6-0007Rw-NO
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 13:49:22 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51338)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTpK2-0007Rc-DF
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 13:49:20 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:35124)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTpJw-0006Mc-En; Fri, 24 Sep 2021 13:49:12 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4768
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTpJw-0004A7-2D; Fri, 24 Sep 2021 13:49:12 -0400
Date: Fri, 24 Sep 2021 20:49:06 +0300
Message-Id: <83ilypsw31.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <d70d4a91f2070500e1a8@HIDDEN> (message from Gregory
 Heytings on Fri, 24 Sep 2021 16:24:50 +0000)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
 <83sfxurr67.fsf@HIDDEN> <d70d4a91f2070500e1a8@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 24 Sep 2021 16:24:50 +0000
> From: Gregory Heytings <gregory@HIDDEN>
> cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN, mardani29@HIDDEN
> 
> > Look closer at those matches which gid "missed", and you will see why it 
> > didn't show them to you.
> 
> I looked close (before you asked), and no, I don't see why some matches 
> are not included.  For example it returns
> 
> lib/tempname.c:212: __GT_FILE: create the file using open(O_CREAT|O_EXCL)
> 
> but not
> 
> lib/tempname.h:47: GT_FILE: create a large file using open(O_CREAT|O_EXCL)
> 
> and it returns
> 
> lib/open.c:99: /* Fail if one of O_CREAT, O_WRONLY, O_RDWR is specified and the filename
> 
> but not
> 
> lisp/gnus/nnmaildir.el:387: ;; If Emacs had O_CREAT|O_EXCL, we could return number-open here.

Hint: they are all in comments.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Gregory Heytings <gregory@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 17:53:02 +0000
Resent-Message-ID: <handler.50733.B50733.163250593128979 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250593128979
          (code B ref 50733); Fri, 24 Sep 2021 17:53:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 17:52:11 +0000
Received: from localhost ([127.0.0.1]:59219 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTpMp-0007XL-6D
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 13:52:11 -0400
Received: from heytings.org ([95.142.160.155]:46334)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gregory@HIDDEN>) id 1mTpMm-0007XC-Mn
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 13:52:09 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heytings.org;
 s=20210101; t=1632505927;
 bh=3Ga6BBpRmdD6Mku/tjhSUz0UjMImy9RrKkP98OV8bik=;
 h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:From;
 b=AiJ9bbWAUlNUp+5Y4+oVRFrCpHwt5FGvKjMODG6WD0/+pBurhmH5iPQ2XtWWz1RG1
 NNFmnz19nrfJQOxtDUZRz5X8rEEPMTz2T/A6pPPlYxWGwuKL0csIJ243LUJnglHZdl
 ehE862gclZnrxKDJ/DMTmmV1/ScVjQwGKRMzvmfxsdqSZYOF+tM2vidoALphhbo9y8
 PvtzfT0pc143OMIZxponpTlBNdNmq583wE/ovhlKSAMGzWsx7EP7PXrjUzdM379ATE
 D0a1XrgZFFm2G+8GBF8wcjvf4o8gAEjQWl57Iv8no2tq0ezteW743MhPjzMSzXV6eq
 qEYGy/wrbBvYQ==
Date: Fri, 24 Sep 2021 17:52:06 +0000
From: Gregory Heytings <gregory@HIDDEN>
In-Reply-To: <83ilypsw31.fsf@HIDDEN>
Message-ID: <d70d4a91f26e22ddb7d7@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
 <83sfxurr67.fsf@HIDDEN> <d70d4a91f2070500e1a8@HIDDEN>
 <83ilypsw31.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


>>> Look closer at those matches which gid "missed", and you will see why 
>>> it didn't show them to you.
>>
>> I looked close (before you asked), and no, I don't see why some matches 
>> are not included.  For example it returns
>>
>> lib/tempname.c:212: __GT_FILE: create the file using open(O_CREAT|O_EXCL)
>>
>> but not
>>
>> lib/tempname.h:47: GT_FILE: create a large file using open(O_CREAT|O_EXCL)
>>
>> and it returns
>>
>> lib/open.c:99: /* Fail if one of O_CREAT, O_WRONLY, O_RDWR is specified and the filename
>>
>> but not
>>
>> lisp/gnus/nnmaildir.el:387: ;; If Emacs had O_CREAT|O_EXCL, we could return number-open here.
>
> Hint: they are all in comments.
>

Hint: read again, and look closer.  gid returns some matches in comments, 
some not.  For no apparent reason.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Dmitry Gutov <dgutov@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 18:18:01 +0000
Resent-Message-ID: <handler.50733.B50733.163250745631899 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>, Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163250745631899
          (code B ref 50733); Fri, 24 Sep 2021 18:18:01 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 18:17:36 +0000
Received: from localhost ([127.0.0.1]:59246 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTplQ-0008IR-0w
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:17:36 -0400
Received: from mail-wr1-f42.google.com ([209.85.221.42]:37439)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1mTplN-0008IB-HC
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:17:34 -0400
Received: by mail-wr1-f42.google.com with SMTP id t8so30077230wrq.4
 for <50733 <at> debbugs.gnu.org>; Fri, 24 Sep 2021 11:17:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=sender:subject:to:cc:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=o1BuriHlpVWWNkKR4xgRtkIOH5nMJctpPqgnEuFXUCk=;
 b=SVdM/G1Bl5tLvcB5Aj2dgVTY9F67beanTWDterKjR8NQvpB6zA8D133Xzz/VGX4oi7
 /EsvwB5yWeX0D2jusH6d1BmUxWcIixyTXK+i8uQOFfuPCZwAkn2RnvmUATEYqRwciMDt
 ThDZMc4k2TmuJ5OmSpW01bnfNp44GKJsNfvsIy0LNg3bt1ATHgAVV+Zgjt0l3PcUhpNI
 wAejVBlytuZXsNMXeNoBWtlzITmTpZoKJVQTuA+0c76/MlVhAc77lKDGXOtmPGddXIGg
 Ap0a/O+22uIYbZ7u9yAw2YANuzMZyOjQr6E4897pFddwJA8stTY0EfMuP9wHue/4tMTc
 JzHw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:sender:subject:to:cc:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=o1BuriHlpVWWNkKR4xgRtkIOH5nMJctpPqgnEuFXUCk=;
 b=zQHHw1xjHhheJfOH84soLRnSj22TITYnXTLqejLMzaXs8xsq2lFR/WU7FEZeBUVJP/
 KXcbG5Uv4mZQoPYs36YlKXSzrtSMEeGFSwEi7VxH6snjJs24+2yKbSVFSvDGeKB3iRms
 /kaEJC4ux97a4y4Tqa4UwBs5ZxNdO/qe8sESVGTsmE8DRCEaURgP9rvmU/sEo/4S6U75
 Sd0iJCyrFPPA3Wnyve7cScFYW0FghpOw0gYf8Af27B9RS8s/l+UTDvl5IBkcBTZWieuR
 irNChwl6M52GB0YecqowSoNF/EbeW7rQ5eBXq2BB1XuP6dNR6Z58vip2d5xYoLBlAWeA
 +NJA==
X-Gm-Message-State: AOAM533v6O0g7MNAEtaTnorUc+tsno/cPF+n7tNB9tSS3PPGGPZOdzLU
 lMdAks/2d2LNMOYairpvDik=
X-Google-Smtp-Source: ABdhPJymqDn61QF7VUIcNiUIzwGO2MHQhfc9xIwXX7itUE9lKTxZq28KVoQUBbzBsPqUHBpzWVQy6Q==
X-Received: by 2002:a1c:2289:: with SMTP id i131mr3513362wmi.179.1632507447751; 
 Fri, 24 Sep 2021 11:17:27 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id g22sm12108131wmp.39.2021.09.24.11.17.25
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Fri, 24 Sep 2021 11:17:27 -0700 (PDT)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
 <d70d4a91f2b91f140d52@HIDDEN> <83sfxurr67.fsf@HIDDEN>
 <d70d4a91f2070500e1a8@HIDDEN> <83ilypsw31.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <843b49c3-08e1-239f-d6bc-850cc2d78ca1@HIDDEN>
Date: Fri, 24 Sep 2021 21:17:24 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
MIME-Version: 1.0
In-Reply-To: <83ilypsw31.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.4 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.6 (/)

On 24.09.2021 20:49, Eli Zaretskii wrote:
> Hint: they are all in comments.

It's not an obvious choice, BTW: whether to show references to the 
identifier in comments.

Personally, I think I prefer the current behavior of 
xref-find-references when using Grep for searching.

Because, when you are renaming a function, you don't want to forget the 
references to its name in comments across the project.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 18:49:02 +0000
Resent-Message-ID: <handler.50733.B50733.16325093413208 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Gregory Heytings <gregory@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16325093413208
          (code B ref 50733); Fri, 24 Sep 2021 18:49:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 18:49:01 +0000
Received: from localhost ([127.0.0.1]:59266 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTqFo-0000pg-W2
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:49:01 -0400
Received: from eggs.gnu.org ([209.51.188.92]:60808)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTqFl-0000pG-P4
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:49:00 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36570)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTqFf-0006Gz-K8; Fri, 24 Sep 2021 14:48:51 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4420
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTqFf-0005Y0-6t; Fri, 24 Sep 2021 14:48:51 -0400
Date: Fri, 24 Sep 2021 21:48:45 +0300
Message-Id: <83h7e9stbm.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <d70d4a91f26e22ddb7d7@HIDDEN> (message from Gregory
 Heytings on Fri, 24 Sep 2021 17:52:06 +0000)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
 <83sfxurr67.fsf@HIDDEN> <d70d4a91f2070500e1a8@HIDDEN>
 <83ilypsw31.fsf@HIDDEN> <d70d4a91f26e22ddb7d7@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 24 Sep 2021 17:52:06 +0000
> From: Gregory Heytings <gregory@HIDDEN>
> cc: 50733 <at> debbugs.gnu.org, dgutov@HIDDEN, mardani29@HIDDEN
> 
> > Hint: they are all in comments.
> >
> 
> Hint: read again, and look closer.  gid returns some matches in comments, 
> some not.  For no apparent reason.

Of course, there is a reason, you just don't understand it (and
generally think about gid as if it were a general-purpose text-search
utility).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 18:53:02 +0000
Resent-Message-ID: <handler.50733.B50733.16325095283794 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dgutov@HIDDEN>
Cc: gregory@HIDDEN, 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.16325095283794
          (code B ref 50733); Fri, 24 Sep 2021 18:53:02 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 18:52:08 +0000
Received: from localhost ([127.0.0.1]:59280 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTqIq-0000z8-ET
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:52:08 -0400
Received: from eggs.gnu.org ([209.51.188.92]:33050)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1mTqIo-0000yV-FU
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 14:52:07 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36616)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1mTqIj-0000WI-5H; Fri, 24 Sep 2021 14:52:01 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4615
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1mTqIi-0005rc-PF; Fri, 24 Sep 2021 14:52:01 -0400
Date: Fri, 24 Sep 2021 21:51:58 +0300
Message-Id: <83fsttst69.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <843b49c3-08e1-239f-d6bc-850cc2d78ca1@HIDDEN> (message from
 Dmitry Gutov on Fri, 24 Sep 2021 21:17:24 +0300)
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN> <83k0j6trau.fsf@HIDDEN>
 <d70d4a91f222dc75d66e@HIDDEN> <838rzmtf00.fsf@HIDDEN>
 <d70d4a91f2b91f140d52@HIDDEN> <83sfxurr67.fsf@HIDDEN>
 <d70d4a91f2070500e1a8@HIDDEN> <83ilypsw31.fsf@HIDDEN>
 <843b49c3-08e1-239f-d6bc-850cc2d78ca1@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN
> From: Dmitry Gutov <dgutov@HIDDEN>
> Date: Fri, 24 Sep 2021 21:17:24 +0300
> 
> On 24.09.2021 20:49, Eli Zaretskii wrote:
> > Hint: they are all in comments.
> 
> It's not an obvious choice, BTW: whether to show references to the 
> identifier in comments.
> 
> Personally, I think I prefer the current behavior of 
> xref-find-references when using Grep for searching.

I won't argue about personal preferences.  IMO, it isn't a matter of
preference, it's a matter of matching between a job you have to do and
the best tool for that job.

> Because, when you are renaming a function, you don't want to forget the 
> references to its name in comments across the project.

Renaming identifiers is not the only job that needs these outputs.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time
Resent-From: Gregory Heytings <gregory@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 24 Sep 2021 21:50:03 +0000
Resent-Message-ID: <handler.50733.B50733.163252018227597 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 50733
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 50733 <at> debbugs.gnu.org, mardani29@HIDDEN, dgutov@HIDDEN
Received: via spool by 50733-submit <at> debbugs.gnu.org id=B50733.163252018227597
          (code B ref 50733); Fri, 24 Sep 2021 21:50:03 +0000
Received: (at 50733) by debbugs.gnu.org; 24 Sep 2021 21:49:42 +0000
Received: from localhost ([127.0.0.1]:59384 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1mTt4g-0007B3-22
	for submit <at> debbugs.gnu.org; Fri, 24 Sep 2021 17:49:42 -0400
Received: from heytings.org ([95.142.160.155]:46584)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <gregory@HIDDEN>) id 1mTt4b-0007Ao-1X
 for 50733 <at> debbugs.gnu.org; Fri, 24 Sep 2021 17:49:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heytings.org;
 s=20210101; t=1632520175;
 bh=BxeEnxgeVl54unkmomP916WJGygGDmBc+YTXj1LRn+c=;
 h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:From;
 b=wy5Tcb8A6q5VBrp/r+PmBN+nHWhZsL2FeL6pO0KGfu0xfGPy8mZMkO/6ECaGG/EIQ
 elGkU2EoXclW8TBzi9ofK0Hkapvrk1FTYbOR28vY6N5kb2OqJ+VPNwJfUEsWXos3m+
 RU9M9pdRY3F5RtaSwXyQvMbjrwg1F6wlME7EgOCMf7L+4bsz5VHfeYUPv7Y7yuJ26H
 SrkKMlopHJpCWbgNNrBeG4NihYBdKbf+rIPZNIRsPFiBWpZx6i57KGjQ6lD+d7hMqX
 URXkA8qoKNCx3n8kpOqpPCByniRyRUrq6VD0r/XEblEV3bo9kOLXPULGkMV2TJC7wB
 75SxkkWqyuY/Q==
Date: Fri, 24 Sep 2021 21:49:35 +0000
From: Gregory Heytings <gregory@HIDDEN>
In-Reply-To: <83h7e9stbm.fsf@HIDDEN>
Message-ID: <d70d4a91f2631367bb6d@HIDDEN>
References: <m1h7edq7sc.fsf.ref@HIDDEN> <m1h7edq7sc.fsf@HIDDEN>
 <03aa81b5-6077-c35c-1a5f-ec4d867b59ac@HIDDEN> <m1fstwl1bh.fsf@HIDDEN>
 <df9e2547-dc4c-a0ea-e619-ee08a017601d@HIDDEN> <m1wnn7c7qd.fsf@HIDDEN>
 <63300a34-e487-02d1-c182-2b84438654d7@HIDDEN>
 <83k0j6trau.fsf@HIDDEN> <d70d4a91f222dc75d66e@HIDDEN>
 <838rzmtf00.fsf@HIDDEN> <d70d4a91f2b91f140d52@HIDDEN>
 <83sfxurr67.fsf@HIDDEN> <d70d4a91f2070500e1a8@HIDDEN>
 <83ilypsw31.fsf@HIDDEN> <d70d4a91f26e22ddb7d7@HIDDEN>
 <83h7e9stbm.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=us-ascii
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)


>> Hint: read again, and look closer.  gid returns some matches in 
>> comments, some not.  For no apparent reason.
>
> Of course, there is a reason, you just don't understand it
>

Who told you that I'm talking about myself?  FWIW, I understand the 
reason, but it's a completely non-apparent one, that isn't documented 
anywhere.  And it's not one that would make sense to most users.

>
> (and generally think about gid as if it were a general-purpose 
> text-search utility).
>

Who told you that I'm talking about myself?  I do not think of gid as a 
general-purpose text-search utility, but what most users want (and what 
project-find-regexp needs) is an efficient, easy to use and understand, 
predictable general-purpose text-search utility.  ripgrep is one such 
utility, idutils is not.  And, once again, ripgrep is, from a user point 
of view, as fast as or faster than idutils, and does not require any 
plumbing behind the scenes.

Out of curiosity, because of your "it doesn't scale" remark, I just 
compared the efficiency of ripgrep and idutils on the latest Linux kernel 
tarball (1.4 GB in 78464 files):

mkid takes 31 seconds

rg O_CREAT takes 0.18 seconds
gid O_CREAT takes 0.02 seconds
rg O.?CREAT takes 0.18 seconds
gid O.?CREAT takes 0.93 seconds
rg O.*CREAT takes 0.19 seconds
gid O.*CREAT takes 1.73 seconds

Isn't idutils the one that doesn't scale?

The only case in which idutils is faster (if one does not take the time 
that was spent to build the database into account, and if one considers 
that it's okay to ignore some matches in comments) is a plain identifier; 
from a user viewpoint getting an answer in 0.2 seconds on such a big code 
base is as good as getting an answer in 0.02 seconds.  It's slower, much 
slower in all other cases, whenever a regexp is used --- which is what 
project-find-regexp is all about.

Last modified: Fri, 24 Sep 2021 22:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.