GNU logs - #64735, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 19 Jul 2023 21:17:02 +0000
Resent-Message-ID: <handler.64735.B.168980140715907 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 64735 <at> debbugs.gnu.org
From: Spencer Baugh <sbaugh@HIDDEN>
Date: Wed, 19 Jul 2023 17:16:31 -0400
Message-ID: <iermszrwqj4.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Received-SPF: pass client-ip=64.215.233.18; envelope-from=sbaugh@HIDDEN;
 helo=mxout5.mail.janestreet.com
X-Spam_score_int: -18
X-Spam_score: -1.9
X-Spam_bar: -
X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_MSPIKE_H5=0.001,
 RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001,
 T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.9 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.1 (/)


Several important commands and functions invoke find; for example rgrep
and project-find-regexp.

Most of these add some set of ignores to the find command, pulling from
grep-find-ignored-files in the former case.  So the find command looks
like:

find -H . \( -path \*/SCCS/\* -o -path \*/RCS/\* [...more ignores...] \)
-prune -o -type f -print0

Alas, on my system, using GNU find, these ignores slow down find by
about 15x on a large directory tree, taking it from around .5 seconds to
7.8 seconds.

This is very noticeable overhead; removing the ignores makes rgrep and
other find-invoking commands substantially faster for me.

The overhead is linear in the number of ignores - that is, each
additional ignore adds a small fixed cost.  This suggests that find is
linearly scanning the list of ignores and checking each one, rather than
optimizing them to a single regexp and checking that regexp.

Obviously, GNU find should be optimizing this.  However they have
previously said they will not optimize this; I commented on this bug
https://savannah.gnu.org/bugs/index.php?58197 to request they rethink
that.  Hopefully as a fellow GNU project they will be interested in
helping us...

In Emacs alone, there are a few things we could do:
- we could mitigate the find bug by optimizing the regexp before we pass
it to find; this should basically remove all the overhead but makes the
find command uglier and harder to edit
- we could remove rare and likely irrelevant things from
completion-ignored-extensions and vc-ignore-dir-regexp (which are used
to build these lists of ignores)
- we could use our own recursive directory-tree walking implementation
(directory-files-recursively), if we found a nice way to pipe its output
directly to grep etc without going through Lisp.  (This could be nice
for project-files, at least)

Incidentally, I tried a find alternative, "bfs",
https://github.com/tavianator/bfs and it doesn't optimize this either,
sadly, so it also has the 15x slowdown.



In GNU Emacs 29.0.92 (build 5, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.15.12, Xaw scroll bars) of 2023-07-10 built on

Repository revision: dd15432ffacbeff0291381c0109f5b1245060b1d
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.8 (Green Obsidian)

Configured using:
 'configure --config-cache --with-x-toolkit=lucid
 --with-gif=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM LUCID
ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Memory information:
((conses 16 1939322 193013)
 (symbols 48 76940 49)
 (strings 32 337371 45355)
 (string-bytes 1 12322013)
 (vectors 16 148305)
 (vector-slots 8 3180429 187121)
 (floats 8 889 751)
 (intervals 56 152845 1238)
 (buffers 976 235)
 (heap 1024 978725 465480))

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Spencer Baugh <sbaugh@HIDDEN>
Subject: bug#64735: Acknowledgement (29.0.92; find invocations are ~15x
 slower because of ignores)
Message-ID: <handler.64735.B.168980140715907.ack <at> debbugs.gnu.org>
References: <iermszrwqj4.fsf@HIDDEN>
X-Gnu-PR-Message: ack 64735
X-Gnu-PR-Package: emacs
Reply-To: 64735 <at> debbugs.gnu.org
Date: Wed, 19 Jul 2023 21:17:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 64735 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
64735: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D64735
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 05:01:02 +0000
Resent-Message-ID: <handler.64735.B64735.16898292352447 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: 64735 <at> debbugs.gnu.org
Date: Thu, 20 Jul 2023 08:00:52 +0300
Message-Id: <837cqv41ob.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <iermszrwqj4.fsf@HIDDEN> (message from Spencer Baugh on
 Wed, 19 Jul 2023 17:16:31 -0400)
References: <iermszrwqj4.fsf@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: Spencer Baugh <sbaugh@HIDDEN>
> Date: Wed, 19 Jul 2023 17:16:31 -0400
> 
> 
> Several important commands and functions invoke find; for example rgrep
> and project-find-regexp.
> 
> Most of these add some set of ignores to the find command, pulling from
> grep-find-ignored-files in the former case.  So the find command looks
> like:
> 
> find -H . \( -path \*/SCCS/\* -o -path \*/RCS/\* [...more ignores...] \)
> -prune -o -type f -print0
> 
> Alas, on my system, using GNU find, these ignores slow down find by
> about 15x on a large directory tree, taking it from around .5 seconds to
> 7.8 seconds.
> 
> This is very noticeable overhead; removing the ignores makes rgrep and
> other find-invoking commands substantially faster for me.

grep-find-ignored-files is a customizable user option, so if this
slowdown bothers you, just customize it to avoid that.  And if there
are patterns there that are no longer pertinent or rare, we could
remove them from the default value.

I'm not sure we should bother more than these two simple measures.

> The overhead is linear in the number of ignores - that is, each
> additional ignore adds a small fixed cost.  This suggests that find is
> linearly scanning the list of ignores and checking each one, rather than
> optimizing them to a single regexp and checking that regexp.

If it uses fnmatch, it cannot do it any other way, I think

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 12:23:02 +0000
Resent-Message-ID: <handler.64735.B64735.168985574930394 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Eli Zaretskii <eliz@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
From: sbaugh@HIDDEN
In-Reply-To: <837cqv41ob.fsf@HIDDEN> (Eli Zaretskii's message of "Thu, 20 Jul
 2023 08:00:52 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <837cqv41ob.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 12:22:19 +0000 (UTC)
Message-ID: <87mszqixhh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Eli Zaretskii <eliz@HIDDEN> writes:
>> From: Spencer Baugh <sbaugh@HIDDEN>
>> Date: Wed, 19 Jul 2023 17:16:31 -0400
>> 
>> 
>> Several important commands and functions invoke find; for example rgrep
>> and project-find-regexp.
>> 
>> Most of these add some set of ignores to the find command, pulling from
>> grep-find-ignored-files in the former case.  So the find command looks
>> like:
>> 
>> find -H . \( -path \*/SCCS/\* -o -path \*/RCS/\* [...more ignores...] \)
>> -prune -o -type f -print0
>> 
>> Alas, on my system, using GNU find, these ignores slow down find by
>> about 15x on a large directory tree, taking it from around .5 seconds to
>> 7.8 seconds.
>> 
>> This is very noticeable overhead; removing the ignores makes rgrep and
>> other find-invoking commands substantially faster for me.
>
> grep-find-ignored-files is a customizable user option, so if this
> slowdown bothers you, just customize it to avoid that.

I think the fact that the default behavior is very slow, is bad.

> And if there are patterns there that are no longer pertinent or rare,
> we could remove them from the default value.

Sure!

So the thing to narrow down would be completion-ignored-extensions,
which is what populates grep-find-ignored-files.  Most things in that
list are irrelevant to most users, but all of them are relevant to some
users.

Most of these are language-specific things - e.g. there's a bunch of
Common Lisp compiled object (or something) extensions.

Perhaps we could modularize this, so that individual packages add things
to completion-ignored-extensions at load time.  Then
completion-ignored-extensions would only include things which are
relevant to a given user, as determined by what packages they load.

> I'm not sure we should bother more than these two simple measures.

Unfortunately those two simple measures help rgrep but they don't help
project-find-regexp (and others project.el commands using
project--files-in-directory such as project-find-file), since those
project commands pull their ignores from the version control system
through vc (not grep-find-ignored-files), and then pass them to find.

>> The overhead is linear in the number of ignores - that is, each
>> additional ignore adds a small fixed cost.  This suggests that find is
>> linearly scanning the list of ignores and checking each one, rather than
>> optimizing them to a single regexp and checking that regexp.
>
> If it uses fnmatch, it cannot do it any other way, I think

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 12:39:02 +0000
Resent-Message-ID: <handler.64735.B64735.168985670732323 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168985670732323
          (code B ref 64735); Thu, 20 Jul 2023 12:39:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 12:38:27 +0000
Received: from localhost ([127.0.0.1]:57936 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMSvJ-0008PF-TR
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 08:38:27 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:45303)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMSv9-0008Ot-NQ
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 08:38:24 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 7A7CC5C0159;
 Thu, 20 Jul 2023 08:38:08 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Thu, 20 Jul 2023 08:38:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :content-transfer-encoding:content-type:content-type:date:date
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689856688; x=1689943088; bh=GbBbzLKPBKtUO8BYZ1LApyV+j398P3T87SF
 co2FbRdQ=; b=dAt2IVxYL9WcotUybjgTGMPWWC6O09AHEZhLR6aOcv/Dr+qva7f
 pW/0retKKGFK0JbLj1/FadESjLG7VyYhAU8H9RXvfJPGrzpsKsg7eQfi24uGwfzr
 NKSLSYjS4vlHmmq2CwW6/8opH5yjXD5a+BbLMv7Y9RKSOBdCLHTjnJBypg6x1I48
 nisHenuubAEHB03N2F1l8ugrZefhY6tNSfjglsdRRdPiHnvaoqmgpVm2q2RQJQE0
 1F+izvBLkAJi8q+qoggbcQS3MBixIEz/EwiI05hOzZdNbeze5aRWS3bPg7rBw+E9
 7PrQH81jjHpJpI1WhKmmlNk+4jBR89c4cmg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :content-type:date:date:feedback-id:feedback-id:from:from
 :in-reply-to:in-reply-to:message-id:mime-version:references
 :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy
 :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1689856688; x=
 1689943088; bh=GbBbzLKPBKtUO8BYZ1LApyV+j398P3T87SFco2FbRdQ=; b=c
 jbpX4JTp2f8VGcgJDGaNg4DaYFJYs5ulvWQgiJw8jg2jankMJWL2tPolB12NJVla
 mGy+OfUwFEEhLIOStyji78w01uNS/j6y33YFEqC07V0F4O4CvaLAK7pRcN7z6xnM
 Yd9vvy/6QSuEorY09sVoaIZi6Sp03eqOOuO/jZ1mkGYBzb1jpDTFiSMoVbu0TCJY
 zzAX9bwWXkq+Ncekni72Olg7Nh8n1wh3sR4zGNWVgHsoKruBBsieLNktrBt3bQy3
 fBaqe+cirFzqbHWEfyjXP+095P5nsdrIIMMY0ntyFNmPCaXI/A9Jt5bZZoV61Rfq
 Icjpcycy/4GC74IsIEUsA==
X-ME-Sender: <xms:ryq5ZOIg1uaziGx2U17YbyarpLsIzwtPW6z9Otv5EoPgNsV8PLa3Qw>
 <xme:ryq5ZGIoBT_gFu9xijve9C9-eK7DUfSd1U6hwFXN4eKu2TTxUz_0Db1UQUL4WeXWv
 EHd4tO-3_k9L5idhSM>
X-ME-Received: <xmr:ryq5ZOuVMqXuW3W2z9rvss2-FCYOPXh5ULchoCgF64QKQR77qgsVXaUah6j-BtA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgdehgecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfhfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhithhr
 hicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrthhtvg
 hrnhepgeehtdehudejieetuedttdejkeehjeetueekheffuddtjeeuvdefveeuveeiuddu
 necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughmih
 htrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:ryq5ZDZjwgy55FwHBcqvrdNHskelja29lMAngqc2Y9VOk8MWI4SpJw>
 <xmx:ryq5ZFaG8pmBahNlovbgUlsbLjOsIbtBJzt75e8pUQhqwi6na42aJg>
 <xmx:ryq5ZPASI10-9PmfdJ8VHW4qi6hzCPHMM2kUEKpFjrRE2EomVDhaFA>
 <xmx:sCq5ZGBl5n5a4FNXIsbsInITSiywmmfFDLbGz7MzDoPetaknpM_UuA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 08:38:07 -0400 (EDT)
Message-ID: <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
Date: Thu, 20 Jul 2023 15:38:04 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <iermszrwqj4.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.1 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 00:16, Spencer Baugh wrote:
> In Emacs alone, there are a few things we could do:
> - we could mitigate the find bug by optimizing the regexp before we pass
> it to find; this should basically remove all the overhead but makes the
> find command uglier and harder to edit
> - we could remove rare and likely irrelevant things from
> completion-ignored-extensions and vc-ignore-dir-regexp (which are used
> to build these lists of ignores)

I like these two approaches.

> - we could use our own recursive directory-tree walking implementation
> (directory-files-recursively), if we found a nice way to pipe its output
> directly to grep etc without going through Lisp.  (This could be nice
> for project-files, at least)

This will probably not work as well. Last I checked, Lisp-native file 
listing was simply slower than 'find'.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 12:44:02 +0000
Resent-Message-ID: <handler.64735.B64735.168985698932761 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168985698932761
          (code B ref 64735); Thu, 20 Jul 2023 12:44:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 12:43:09 +0000
Received: from localhost ([127.0.0.1]:57941 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMSzt-0008WL-Go
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 08:43:09 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:49359)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMSzr-0008W4-0A
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 08:43:07 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id E9E9C5C0089;
 Thu, 20 Jul 2023 08:43:01 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute6.internal (MEProxy); Thu, 20 Jul 2023 08:43:01 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689856981; x=1689943381; bh=7buN++tSPFVB6cJ+5C11cPD2buTnj18XMf+
 6YvmsJHU=; b=Y1CHSHW+cRmEMYugxT1OjEbR9FXLunZN7xPngGXI1IwRL4v/cl/
 ytvU0KrlsmOg7GP7v/H0uQNbRlo43FHcyNkDDEsjFZuv6FmDeQBOijo4JMRd9G59
 4RLQ6Zd/oIaVEKYcmWFYfO7zUDd/oP6qrKz0AwZpf1JfeaVUyuKUDcGM+6DTF+Zk
 2rXG4QbU8TEPe/c8xETSzck/dkogZz9c7+N1K4hJcZqwnbBvp1NqHNVxZSq33LtV
 uZ+KmFuC/NNS3QF8MZKET1gQMUpLQXqgDDOBPCi822Tllyta5GSDI5rJ75nUdZ2/
 g1sSrq+AzxB/jqH3A3yZaPBtZOrtAIyV1gQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689856981; x=1689943381; bh=7buN++tSPFVB6cJ+5C11cPD2buTnj18XMf+
 6YvmsJHU=; b=MGntHloOtIKzkmlzfogInZkqLqeYo5IJLyetuQj+H26XqflBUxh
 5GRzL7thcaMAWWlFxsF4vo2FUIET0hxTtXbMyuzjUm2SsPvLLtzOlDRfi1go+a5U
 tiIJi394QswMJz/kN7psPHXt3eDt3QJJySO5GCEUYdznna7oXj9L0LXF3A73spwN
 bh7TO05SWVuqbPAnQSP1IieVlEfWwjKcAnkGB2RqwAngTRRj2RsAypu1tRJaHqES
 DolTFMrkZSkc6B1IsgWCas/hlOaormJL4C7x1I2b7C9cwzo5gwQu50pBy9cDcwl6
 t9lez3oglwQ0G3An1bELR659aShACKqh3uQ==
X-ME-Sender: <xms:1Su5ZKCLBQ4oD8sunxuRErJdaGoOBIBcqFsMK0xHo3eEvGdytDQQLA>
 <xme:1Su5ZEhPRLMh1UR9m0x47FId56lSCJ9QTVyzD2PYHwP1QWtO8f-HKFszRxQ2eK6to
 2P1CsECVDxgaKHM9GA>
X-ME-Received: <xmr:1Su5ZNnJEDEKUka4JKuFhBnSIqpWiCM_knFYsDjd1h_m6icmEzHScPhJt09L4dY>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgdehhecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:1Su5ZIxTboA1G5kMeLOioJClBjlYOC5blFjm3MKTP7YW-2g9b7JTVA>
 <xmx:1Su5ZPQeXXcXl5vsf70RyElFO8edRKoHEOtctGapF35EQYJLdXTTIw>
 <xmx:1Su5ZDYCTB71e-H5otQbAf8s__qVc7i0ewNFfj8c_tD-rL7TRw_R-g>
 <xmx:1Su5ZKeWSfvXVzZrOM0uZMQlVO6f3eXQSw2ncCBDJWuckGsSR_bSpg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 08:43:00 -0400 (EDT)
Message-ID: <4d4f029f-f32f-13df-ffc3-3952d62d8bb3@HIDDEN>
Date: Thu, 20 Jul 2023 15:42:59 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN> <837cqv41ob.fsf@HIDDEN>
 <87mszqixhh.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87mszqixhh.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 15:22, sbaugh@HIDDEN wrote:
>> I'm not sure we should bother more than these two simple measures.
> Unfortunately those two simple measures help rgrep but they don't help
> project-find-regexp (and others project.el commands using
> project--files-in-directory such as project-find-file), since those
> project commands pull their ignores from the version control system
> through vc (not grep-find-ignored-files), and then pass them to find.

That's only a problem when the default file listing logic is used (and 
we usually delegate to something like 'git ls-files' instead, when the 
vc-aware backend is used).

Anyway, some optimization could be useful there too. The extra 
difficulty, though, is that the entries in IGNORES already can come as 
wildcards. Can we merge several wildcards? Though I suppose if we use a 
regexp, we could construct an alternation anyway.

Another question it would be helpful to check, is whether the different 
versions of 'find' out there work fine with -regex instead of -name, and 
don't get slowed down simply because of that feature. The old built-in 
'find' on macOS, for example.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 13:21:01 +0000
Resent-Message-ID: <handler.64735.B64735.16898592615734 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898592615734
          (code B ref 64735); Thu, 20 Jul 2023 13:21:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 13:21:01 +0000
Received: from localhost ([127.0.0.1]:58014 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMTaW-0001UF-SL
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 09:21:01 -0400
Received: from mout01.posteo.de ([185.67.36.65]:33423)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMTaR-0001Tp-H2
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 09:20:59 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 6C452240029
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 15:20:49 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689859249; bh=g0MjRjEdI72WIvAo0hppkgv+5rtrhexKT12P0v7grYM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=TGWNbr+2TTYBmW18ZUHY5fSKmksDiLo4gcrryCAQYCLRnrhpha5yo4oP5lWgZpFgd
 V7ZLHV0p+A0WHjESuCq9EcsILyOFcG7e8uONYFuR5Ba4LwBACN8NxwRiHtbgIEx1zW
 ND28bk2nxbFXBBBfcwqf0rtiLm9l+d1k/X/1MjHgsx1iGllDQXad2RreIyRvXnJdLZ
 7AlFHGv/JYRDmO+Pl2gEa14XPdnutrprrLQxEtmuhtw0zC+eqB8eJDiF/HkYyozXB6
 nZrCmu8o/XShKZrlvVlU/ihNftYKCQFcmsTpNkUDfET+zuXeoM1nV7DnZSEDyvlNlF
 z27B55ohMNO5A==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6CxD145sz9rxX;
 Thu, 20 Jul 2023 15:20:47 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
Date: Thu, 20 Jul 2023 13:20:59 +0000
Message-ID: <87wmyupvlw.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

> ... Last I checked, Lisp-native file 
> listing was simply slower than 'find'.

Could it be changed?
In my tests, I was able to improve performance of the built-in
`directory-files-recursively' simply by disabling
`file-name-handler-alist' around its call.

See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
(the thread also continues off-list, and it looks like there is a lot of
room for improvement in this area)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 13:45:01 +0000
Resent-Message-ID: <handler.64735.B64735.16898606468065 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898606468065
          (code B ref 64735); Thu, 20 Jul 2023 13:45:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 13:44:06 +0000
Received: from localhost ([127.0.0.1]:58037 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMTws-000261-EF
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 09:44:06 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:34341)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qMTwq-00025Q-Ez
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 09:44:05 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <4d4f029f-f32f-13df-ffc3-3952d62d8bb3@HIDDEN> (Dmitry Gutov's
 message of "Thu, 20 Jul 2023 15:42:59 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <837cqv41ob.fsf@HIDDEN>
 <87mszqixhh.fsf@HIDDEN>
 <4d4f029f-f32f-13df-ffc3-3952d62d8bb3@HIDDEN>
Date: Thu, 20 Jul 2023 09:43:59 -0400
Message-ID: <ierjzuuwvds.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Dmitry Gutov <dmitry@HIDDEN> writes:
> On 20/07/2023 15:22, sbaugh@HIDDEN wrote:
>>> I'm not sure we should bother more than these two simple measures.
>> Unfortunately those two simple measures help rgrep but they don't help
>> project-find-regexp (and others project.el commands using
>> project--files-in-directory such as project-find-file), since those
>> project commands pull their ignores from the version control system
>> through vc (not grep-find-ignored-files), and then pass them to find.
>
> That's only a problem when the default file listing logic is used (and
> we usually delegate to something like 'git ls-files' instead, when the
> vc-aware backend is used).

Hm, yes, but things like C-u project-find-regexp will use the default
find-based file listing logic instead of git ls-files, as do a few other
things.

I wonder, could we just go ahead and make a vc function which is
list-files(GLOBS) and returns a list of files?  Both git and hg support
this.  Then we could have C-u project-find-regexp use that instead of
find, by taking the cross product of dirs-to-search and
file-name-patterns-to-search.  (And this would let me delete a big chunk
of my own project backend, so I'd be happy to implement it.)

Fundamentally it seems a little silly for project-ignores to ever be
used for a vc project; if the vcs gives us ignores, we can probably just
ask the vcs to list the files too, and it will have an efficient
implementation of that.

If we do that uniformly, then this find slowness would only affect
transient projects, and transient projects pull their ignores from
grep-find-ignored-files just like rgrep, so improvements will more
easily be applied to both.  (And maybe we could even get rid of
project-ignores entirely, then?)

> Anyway, some optimization could be useful there too. The extra
> difficulty, though, is that the entries in IGNORES already can come as
> wildcards. Can we merge several wildcards? Though I suppose if we use
> a regexp, we could construct an alternation anyway.
>
> Another question it would be helpful to check, is whether the
> different versions of 'find' out there work fine with -regex instead
> of -name, and don't get slowed down simply because of that
> feature. The old built-in 'find' on macOS, for example.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 15:20:01 +0000
Resent-Message-ID: <handler.64735.B64735.168986639319587 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168986639319587
          (code B ref 64735); Thu, 20 Jul 2023 15:20:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 15:19:53 +0000
Received: from localhost ([127.0.0.1]:59463 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMVRZ-00055q-9s
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:19:53 -0400
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:49533)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMVRW-00055b-Cb
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:19:51 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id C21C55C01A8;
 Thu, 20 Jul 2023 11:19:44 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Thu, 20 Jul 2023 11:19:44 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689866384; x=1689952784; bh=kBQ8rrezP2nvOULyLtlijS6nHHWfTOxYnG5
 cb4dMJb8=; b=MWPD22kYmau0paO2O9wEwuBqZgC06njXTyZFq4GQ52efIwPQ0fY
 cYnpMnpjztmUbpc8NFhWKG0XC8/WXMeE7vRJoCGd0ufY82EgAPoK7dggMU5CSY52
 RXDV4lRaImipI2RVu0a/URwbS3gizVyKQyfoH3kavbYordGOYVm6RSlHzVt2hace
 MGCWezf6mymqV2TTPadHUa6NKSUR9rRxRJ70mRstTOAt/tFWUAf47POyQLCu2vhs
 lZLahPnq7RWim0HG2I8F4Ohp03cxp76CDZRErrnAObQGg29X3QwvGU4V/U23SptK
 TxZdOctFcrfng+kIxvjSXnIawuMTK2JhAXg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689866384; x=1689952784; bh=kBQ8rrezP2nvOULyLtlijS6nHHWfTOxYnG5
 cb4dMJb8=; b=Hxfxu480T/y+kzV5o3kVfWzrkXsUY8u1Smds3M9HGsi82BvWFj9
 gj2v3uLRH7Je64iq7S51xyB/GSHuA5TcfOFCpe9BEgNUzYfTvcL9R14ow55LODvn
 j4GrTopwS1Oj7KoivipbdW/j3JI7X3HWWp+qZCFNMdzybs/a4dUaB7ibff+cu33E
 m+9PA4yKwtw8dJ2K3UOPp/1mJHB43qjSVzH5d016qFkcsy9DhxavE4tce7hP58Q1
 kjwTZL2SxvZ51qg/1Vj/IxFbTPRG5W7R/lJC5o1GYQ/ET3MlolDD2VdT7vGSKs6g
 Nz4433pFlIYZuBshiLrRKaUNQbUGw+a4HJw==
X-ME-Sender: <xms:kFC5ZOOaHf14ylpi52WBtVrGZoIaPrkmEvP2Kn6hsORU0tqzc0FpSA>
 <xme:kFC5ZM8A_Umm1_8ZEOv2jg2vw3MfQJ47s4GyZiE3WTrDmMsjfAXlDzKUGtCN6hI04
 -naFGayenwfTa1sSdk>
X-ME-Received: <xmr:kFC5ZFSqpweNih1powzsUkjHoR5kuggp8PFV7ydrriVHkgWF_ofI9xvk0GnvN8c>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgdekiecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeehleeflefhheeujeduteejveeggeejjefhleeljeettdfhjedthfejgfeivdeu
 leenucffohhmrghinhephihhvghtihhlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd
 enucfrrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:kFC5ZOtpvgdAUMeU1lpjJXkwiqvBdTqUNyH20QDdiW4_8Vu1tT8zBA>
 <xmx:kFC5ZGf_IwrscIXa6sKf-xQ8l64RTwL7HD97Z_eJL3mK1no8mtrgzg>
 <xmx:kFC5ZC3lpO7bNZ0rMs1-9GpYFk2t0l1uklxKrApaK0Pp25PrEMOuRg>
 <xmx:kFC5ZKGnoqL6w9ZjeaAVsZTfGMYaUgfotDwrc0kuguM1N_PNBURURA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 11:19:43 -0400 (EDT)
Message-ID: <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
Date: Thu, 20 Jul 2023 18:19:41 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87wmyupvlw.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 16:20, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@HIDDEN> writes:
> 
>> ... Last I checked, Lisp-native file
>> listing was simply slower than 'find'.
> 
> Could it be changed?
> In my tests, I was able to improve performance of the built-in
> `directory-files-recursively' simply by disabling
> `file-name-handler-alist' around its call.

Then it won't work with Tramp, right? I think it's pretty nifty that 
project-find-regexp and dired-do-find-regexp work over Tramp.

> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
> (the thread also continues off-list, and it looks like there is a lot of
> room for improvement in this area)

Does it get close enough to the performance of 'find' this way?

Also note that processing all matches in Lisp, with many ignores 
entries, will incur the proportional overhead in Lisp. Which might be 
relatively slow as well.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 15:43:01 +0000
Resent-Message-ID: <handler.64735.B64735.168986774421695 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168986774421695
          (code B ref 64735); Thu, 20 Jul 2023 15:43:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 15:42:24 +0000
Received: from localhost ([127.0.0.1]:59487 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMVnL-0005dq-Uh
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:42:24 -0400
Received: from mout02.posteo.de ([185.67.36.66]:51251)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMVnI-0005db-8R
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:42:21 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id D68B3240101
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 17:42:13 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689867733; bh=YZc7W2lILHnFsPNcNeE1M++pUIcYef2prBgPrynZ/3c=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=YfE3crK8tyMvo+wbAFsbjnq9B0Jtmzm+RDsFyfwQQ4XxpWBZBzSU9ApPYaN5X0ph6
 F0dXtEa3zS0cyQv7F2tTI3gRYFKCrhR6hnPhjfK8jAgkLWrmMMVLNsZbEMjIfhE2XI
 gygfT557H3TALI9F5tQDn3xcXP2kjHW80g/+ZnWoL/xjaHTHEEK8JzNWtuLQobKcdG
 8AmFLy2xMS6jmY36Q420L9YnG9NrMlxjKuM+2kXgvNXHZwoRkR39G6sgiKfxkOtZzB
 pKL3xkMqOwYms7K2ZvRxHBC4H/LxNsJJjtEwPulXje2eFb5q5OYbXhq0/lwLBO7Gri
 BbLvUUbfsHtBw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6H4K2HtCz9rxW;
 Thu, 20 Jul 2023 17:42:09 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
Date: Thu, 20 Jul 2023 15:42:17 +0000
Message-ID: <87tttypp2e.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

>>> ... Last I checked, Lisp-native file
>>> listing was simply slower than 'find'.
>> 
>> Could it be changed?
>> In my tests, I was able to improve performance of the built-in
>> `directory-files-recursively' simply by disabling
>> `file-name-handler-alist' around its call.
>
> Then it won't work with Tramp, right? I think it's pretty nifty that 
> project-find-regexp and dired-do-find-regexp work over Tramp.

Sure. It might also be optimized. Without trying to convince find devs
to do something about regexp handling.

And things are not as horrible as 15x slowdown in find.

>> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
>> (the thread also continues off-list, and it looks like there is a lot of
>> room for improvement in this area)
>
> Does it get close enough to the performance of 'find' this way?

Comparable:

(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (directory-files-recursively "/home/yantar92/.data" ""))))
;; Elapsed time: 0.633713s
(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (let ((file-name-handler-alist)) (directory-files-recursively "/home/yantar92/.data" "")))))
;; Elapsed time: 0.324341s
;; time find /home/yantar92/.data >/dev/null
;; real	0m0.129s
;; user	0m0.017s
;; sys	0m0.111s

> Also note that processing all matches in Lisp, with many ignores 
> entries, will incur the proportional overhead in Lisp. Which might be 
> relatively slow as well.

Not significant.
I tried to unwrap recursion in `directory-files-recursively' and tried
to play around with regexp matching of the file list itself - no
significant impact compared to `file-name-handler-alist'.

I am pretty sure that Emacs's native file routines can be optimized to
the level of find.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 15:58:02 +0000
Resent-Message-ID: <handler.64735.B64735.168986863223485 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168986863223485
          (code B ref 64735); Thu, 20 Jul 2023 15:58:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 15:57:12 +0000
Received: from localhost ([127.0.0.1]:59535 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMW1f-00066i-II
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:57:11 -0400
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:43917)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMW1c-00066O-Vu
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 11:57:10 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id 753315C006A;
 Thu, 20 Jul 2023 11:57:03 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Thu, 20 Jul 2023 11:57:03 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689868623; x=1689955023; bh=z1IsZ/XMTPa5iEDsDduIuDCFm7BcSEM4i1N
 RFXergfo=; b=ZNJvCK4k2/XFMedHCEzSJAOeRx2vUWk7HvekHvDxtD40sYafr7E
 7jkr0HOgwxWCNhyWyD50osd+HUufJPyvVUXrK3oA0v2SIZcbYl1fmq/++ZXgnIZZ
 W8Jescl4j20XI0W6gQnS63MlMCc6uEZe1NMNMebkN1WWJn6G7CwA/CB3cREnNji7
 v3dqV6aS8iMaVcNEsx2Q6RWFQ1XGPsCtiQ1SiLnKDmqvmcJjij4HbMvHuRR9LqkP
 MYYv5pHMznERxEmCvLOJw0TILcBY83joI1MfzJOWzm1Lo0k9VxqFXhCGrfZdfIV4
 Sm+i6Bj8yRSXVrPuy0YOYbREZDd1eH/oU+A==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689868623; x=1689955023; bh=z1IsZ/XMTPa5iEDsDduIuDCFm7BcSEM4i1N
 RFXergfo=; b=Ykajmm7JQDfj6pOc5RNaSu6v4Pg9MUdWnIqA4qiZkkNyJDyFo5X
 VWVgRh7DP0mpl9PrXOBCMoYSmkQL5DwRfUZ54YjjsUb+RyAmymhbzGujOdcAIVZi
 hxW9slhl/6i/60Na2akU7u/jpTSuAm+pVBYbjLxmlguMo7HAIrhsf1qmbRjuEFwg
 llgARL2YpaLOs4cCf6Ti7tTSLA542UJlZVk0JlcvVRgVUQsemAievN/gxOcSpN+/
 jLVM0L8Tx/3XTYOQ3pfpn/EZ5TkxCj4Lnigt5FohkH8PEz0V8wLz7+fXrijO55Y/
 0BbBY0qoCV1iccULY+69WraFyMwSambgezw==
X-ME-Sender: <xms:T1m5ZMhS-tBhtfMtPLw2rkCFyyg_16HEPXFII8sPPnHGbmkc-d_qBA>
 <xme:T1m5ZFALS-kse30EhHOtpHyyCWX4nY7Dl9UVxXtZd6IXdNoK5wr4AxVNcQzLprJVb
 IUMF14BJLUB4NOc42A>
X-ME-Received: <xmr:T1m5ZEEJPoZ09XT-BYONdOzDRdaEMCyezHxwfnJReKUO67PXGA5pe2qPY-vFxto>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgdelgecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeffhefgkeevheevvdeutedtkeeijeduueejheethedtgfdutdetveffvdevteef
 ueenucffohhmrghinhephihhvghtihhlrdhorhhgpdihrghnthgrrhelvddruggrthgrne
 cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughmihht
 rhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:T1m5ZNTCiQ8HAWWX4_7zpUFx-P5lS1oqQ1KcJRVrK6QSYOwRn9IChg>
 <xmx:T1m5ZJwwZhLWLyH_SNWjXM_YHXouuw2foW1AFcBRQIYiFnXitNkCcQ>
 <xmx:T1m5ZL525T1YvxwAO7MBHGZ8ohlC9wQzv4Z86_D7p4Pdra1s6cWt_w>
 <xmx:T1m5ZKrUvoDFx0LWrZvZfJCnrBdXf0O-au6G1RpNHvJEYZUt2SK5jQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 11:57:02 -0400 (EDT)
Message-ID: <aba1400a-3d00-44f0-90f1-e5b549de8ae7@HIDDEN>
Date: Thu, 20 Jul 2023 18:57:00 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87tttypp2e.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 18:42, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@HIDDEN> writes:
> 
>>>> ... Last I checked, Lisp-native file
>>>> listing was simply slower than 'find'.
>>>
>>> Could it be changed?
>>> In my tests, I was able to improve performance of the built-in
>>> `directory-files-recursively' simply by disabling
>>> `file-name-handler-alist' around its call.
>>
>> Then it won't work with Tramp, right? I think it's pretty nifty that
>> project-find-regexp and dired-do-find-regexp work over Tramp.
> 
> Sure. It might also be optimized. Without trying to convince find devs
> to do something about regexp handling.
> 
> And things are not as horrible as 15x slowdown in find.

We haven't compared to the "optimized regexps" solution in find, though.

>>> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
>>> (the thread also continues off-list, and it looks like there is a lot of
>>> room for improvement in this area)
>>
>> Does it get close enough to the performance of 'find' this way?
> 
> Comparable:
> 
> (ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (directory-files-recursively "/home/yantar92/.data" ""))))
> ;; Elapsed time: 0.633713s
> (ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (let ((file-name-handler-alist)) (directory-files-recursively "/home/yantar92/.data" "")))))
> ;; Elapsed time: 0.324341s
> ;; time find /home/yantar92/.data >/dev/null
> ;; real	0m0.129s
> ;; user	0m0.017s
> ;; sys	0m0.111s

Still like 2.5x slower, then? That's significant.

>> Also note that processing all matches in Lisp, with many ignores
>> entries, will incur the proportional overhead in Lisp. Which might be
>> relatively slow as well.
> 
> Not significant.
> I tried to unwrap recursion in `directory-files-recursively' and tried
> to play around with regexp matching of the file list itself - no
> significant impact compared to `file-name-handler-alist'.

I suppose that can make sense, if find's slowdown is due to it issuing 
repeated 'stat' calls for every match.

> I am pretty sure that Emacs's native file routines can be optimized to
> the level of find.

I don't know, the GNU tools are often ridiculously optimized. At least 
certain file paths.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 16:05:01 +0000
Resent-Message-ID: <handler.64735.B64735.168986904124440 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168986904124440
          (code B ref 64735); Thu, 20 Jul 2023 16:05:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 16:04:01 +0000
Received: from localhost ([127.0.0.1]:59554 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMW8H-0006Lw-Cz
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:04:01 -0400
Received: from mout01.posteo.de ([185.67.36.65]:55665)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMW8D-0006LS-4O
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:04:00 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id D4494240027
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 18:03:50 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689869030; bh=nRBVd84U/Ajo6awG4TzA4W75Q4Ju8nNFrs/XjjrGsVY=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=iz4LZy3L1GnR72eU3zvT8VUKR6WNviCxANe1kcH48v4KoAz3d+C0MZx7k2gOSxJE6
 3dRa/9r68jMRQcMyoHy+ARkH1AOF8e4bVyaMOvBfUvFNS+D5Ov2lMgF8jQln0192Ah
 yUCQzOoW4BIG1VDTn1kuY98eYlhC+8M4Tl1W62HJp9s01LxZYdikh0WNZask4YMeOU
 pyMFTtKJPsUxoB+NwJFamEvhLg7/xhqgp0npvyfRrIUgI/wwLFaidUZuHTHQxrGmU1
 /s7BDbKwDqTYYe6zQ58oAqQh4F3xpyelXqE1Ul1fIv2NsmrsoUgl50wAYgh/KIHPLw
 /Y0VQ28/FqB+A==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6HYL0lvWz9rxM;
 Thu, 20 Jul 2023 18:03:50 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <aba1400a-3d00-44f0-90f1-e5b549de8ae7@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost>
 <aba1400a-3d00-44f0-90f1-e5b549de8ae7@HIDDEN>
Date: Thu, 20 Jul 2023 16:03:58 +0000
Message-ID: <87r0p2po29.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

>> And things are not as horrible as 15x slowdown in find.
>
> We haven't compared to the "optimized regexps" solution in find, though.

Fair point.

> Still like 2.5x slower, then? That's significant.

It is, but it is workable if we try to optimize Emacs'
`directory-files'/`file-name-all-completions' internals.

>> I am pretty sure that Emacs's native file routines can be optimized to
>> the level of find.
>
> I don't know, the GNU tools are often ridiculously optimized. At least 
> certain file paths.

You are likely right.
Then, what about applying regexps manually, on the full file list
returned by find?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 16:34:02 +0000
Resent-Message-ID: <handler.64735.B64735.168987078627927 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987078627927
          (code B ref 64735); Thu, 20 Jul 2023 16:34:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 16:33:06 +0000
Received: from localhost ([127.0.0.1]:59629 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMWaP-0007GN-Sz
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:33:06 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59722)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMWaO-0007Ft-MB
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:33:05 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMWaI-0006VR-2b; Thu, 20 Jul 2023 12:32:58 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=P7AW6H5hFcHS6VDU4iRlOkQQe7XoZo8sJ8ELI94JyNM=; b=LPAoDG4AZ1sy
 YVdwj6D+797N/ep48CxRP7PwqBm2VfLn9ddwa729JoM+3uXFlONGq0CYhIjRXxHeIn7yaC5uu3lVy
 6+HtZozWLpNKbnCTTXP+vwAtNkWiUzwcHyrN9Iy0rukX91J6ZoxRUtgpLLUfumYukabwFG0IqQz9c
 hy8TT0fcd2DuRRXQOKTDb8T2MorUo7OW3SAIX3HwiPyPzN3lEEjoMhEPewMr2qO/w2fw9QpnbgPu4
 HpSXb6IW9YmeFHBBxstMWZGH1fFtGQ6wnoXWDpti/+6y0DxARskO8TE6HUKmaqH56V9QghKsCOGVM
 mLtEixKSmAurPx524yy4QA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMWaH-0004ia-HE; Thu, 20 Jul 2023 12:32:57 -0400
Date: Thu, 20 Jul 2023 19:33:31 +0300
Message-Id: <837cqu35lw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87tttypp2e.fsf@localhost> (message from Ihor Radchenko on Thu,
 20 Jul 2023 15:42:17 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
> From: Ihor Radchenko <yantar92@HIDDEN>
> Date: Thu, 20 Jul 2023 15:42:17 +0000
> 
> I am pretty sure that Emacs's native file routines can be optimized to
> the level of find.

Maybe it can be improved, but not to the same level as Find, because
consing Lisp strings, something that Find doesn't do, does have its
overhead.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 16:37:02 +0000
Resent-Message-ID: <handler.64735.B64735.168987099628314 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987099628314
          (code B ref 64735); Thu, 20 Jul 2023 16:37:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 16:36:36 +0000
Received: from localhost ([127.0.0.1]:59651 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMWdm-0007Ma-Lv
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:36:36 -0400
Received: from mout02.posteo.de ([185.67.36.66]:46625)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMWdb-0007MC-82
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:36:33 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 28F9F240103
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 18:36:17 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689870977; bh=lZoY/NFHCgKwbBO/IpY2d82D3y7ZM+sPptco6FsjCNg=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=JZ7cqGF0H86x69sakAy3zrW1o47aGJcsUrfMr2ARcpCGboBcVvUkqzf2ByZfFrQU4
 TTSc6FXYvlCDnrpg9JmVD51I7keSTYUelI+rtMn++WG8+FqVebEGaRBq30xYzZF/h0
 awcUcNhUnUg4q02H9QQAP2weXl2G2QOb+6SelUsFvNrIqSlNkNyEQ4TQ2k+kDdtrzR
 Sbvuf3wyy2zO6P4oY7IrLikzYemZsFKR0bjkHDMy79NmsuA92VyFpoKWhGziofQXN3
 a+PkpLKTvpdUO9kuxZSLFISf0lB+alnKZ+YS4odjChP2IDjglUbzm4iedp3/TBCE0z
 fYsWFrsM8ePgQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6JGm0lyMz9rxQ;
 Thu, 20 Jul 2023 18:36:15 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <837cqu35lw.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 16:36:28 +0000
Message-ID: <87o7k6pmk3.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> I am pretty sure that Emacs's native file routines can be optimized to
>> the level of find.
>
> Maybe it can be improved, but not to the same level as Find, because
> consing Lisp strings, something that Find doesn't do, does have its
> overhead.

I am not sure if this specific issue is important.
If we want to use find from Emacs, we would need to create Emacs
string/strings when reading the output of find anyway.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 16:46:02 +0000
Resent-Message-ID: <handler.64735.B64735.168987152629212 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987152629212
          (code B ref 64735); Thu, 20 Jul 2023 16:46:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 16:45:26 +0000
Received: from localhost ([127.0.0.1]:59689 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMWmK-0007b3-8y
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:45:26 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:45838)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMWm9-0007aZ-RC
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 12:45:22 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMWm3-0001k2-UN; Thu, 20 Jul 2023 12:45:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=dro429C2Y35t3PhtT4ea935VNb6hkQBswpgY2OSZIL4=; b=R8RIevVRlVD4
 ortqVTqLw/iZPZ8j0+sDHTh1q3trfZH72bT4i5ZkmAt/6kGk1AjXs1TfJRFW1+mmgRo8ywjiwVwzz
 gTE8TYW1w+CTmUkF77SF40MjONUcCMSX0/1l0BR1WGKbg+wfzSPmmUkDXkNY4xhvkBEdiicI6BU7f
 5NHQkHDpdxYnrz+W3jXkAkURG7MIE+HHvZzMuzEAy2Wq/ummIm4clnlC/hJnyktlvHQASw9RXS4oq
 o/PnPuBGAKjALkOBslwoUdIo2WHeNxnzccDIZp2QtXqTME7461nMNnl6a4zqAQZ6N6oX4NwvKTGtY
 LyZ+PVLYyEa4JUia2dMOvg==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMWm0-000512-It; Thu, 20 Jul 2023 12:45:06 -0400
Date: Thu, 20 Jul 2023 19:45:38 +0300
Message-Id: <834jly351p.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87o7k6pmk3.fsf@localhost> (message from Ihor Radchenko on Thu,
 20 Jul 2023 16:36:28 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: dmitry@HIDDEN, sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Thu, 20 Jul 2023 16:36:28 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > Maybe it can be improved, but not to the same level as Find, because
> > consing Lisp strings, something that Find doesn't do, does have its
> > overhead.
> 
> I am not sure if this specific issue is important.
> If we want to use find from Emacs, we would need to create Emacs
> string/strings when reading the output of find anyway.

So how do you explain that using Find is faster than using
find-lisp.el?

I think the answer is that using Find as a subprocess conses less.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 17:09:01 +0000
Resent-Message-ID: <handler.64735.B64735.168987291531791 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987291531791
          (code B ref 64735); Thu, 20 Jul 2023 17:09:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 17:08:35 +0000
Received: from localhost ([127.0.0.1]:59756 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMX8l-0008Gh-AE
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:08:35 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:32891)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qMX8g-0008GR-RD
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:08:33 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <87tttypp2e.fsf@localhost> (Ihor Radchenko's message of "Thu, 20
 Jul 2023 15:42:17 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost>
Date: Thu, 20 Jul 2023 13:08:24 -0400
Message-ID: <ierfs5iwlx3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:
> Dmitry Gutov <dmitry@HIDDEN> writes:
>
>>>> ... Last I checked, Lisp-native file
>>>> listing was simply slower than 'find'.
>>> 
>>> Could it be changed?
>>> In my tests, I was able to improve performance of the built-in
>>> `directory-files-recursively' simply by disabling
>>> `file-name-handler-alist' around its call.
>>
>> Then it won't work with Tramp, right? I think it's pretty nifty that 
>> project-find-regexp and dired-do-find-regexp work over Tramp.
>
> Sure. It might also be optimized. Without trying to convince find devs
> to do something about regexp handling.

Not to derail too much, but find as a subprocess has one substantial
advantage over find in Lisp: It can run in parallel with Emacs, so that
we actually use multiple CPU cores.

Between that, and the remote support part, I personally much prefer find
to be a subprocess rather than in Lisp.  I don't think optimizing
directory-files-recursively is a great solution.

(Really it's entirely plausible that Emacs could be improved by
*removing* directory-files-recursively, in favor of invoking find as a
subprocess: faster, parallelized execution, and better remote support.)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 17:24:02 +0000
Resent-Message-ID: <handler.64735.B64735.1689873817827 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.1689873817827
          (code B ref 64735); Thu, 20 Jul 2023 17:24:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 17:23:37 +0000
Received: from localhost ([127.0.0.1]:59795 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMXNJ-0000DH-6V
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:23:37 -0400
Received: from mout01.posteo.de ([185.67.36.65]:33023)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMXND-0000D0-LQ
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:23:35 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id C8DBA240028
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 19:23:25 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689873805; bh=4wHMnJ3CJE3RL+TFR2BXOtnSoIN8OM7SMTBeej3VEb4=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=XF8cL+KuwyqHDNkhK32ovQY5Pd5FDsLDKahNh6Natg6Djpi9lY899cCKqyFkPKLsr
 KCAeRIb8gIJNOQYcaF8GwHEjb8Gx7RyA55ztgbGDs0qHptz9cMeUuaC9cfcU7ppjRy
 FQVZJLOJnT8Mxi9bBAiCqcPjU4r+lFchMuTxxX2sozXJBbpzaktQZDclUDLeNi0TlV
 eV2f5fZ6RZxCNLYB+Mgzz/LxbM5kUoKM3RpBqQymmPLogkkPXntQjqbdFcUARfNPrS
 1ElfiljnXHib7zo7JrjiJj5hCcm1kvagge/tWR2rw3+RYNFFcZXMH0rJC3kWLYOCat
 Yoq6ZEzSE7msg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6KJz04Nrz6txN;
 Thu, 20 Jul 2023 19:23:14 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <834jly351p.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 17:23:22 +0000
Message-ID: <87lefapkdx.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> I am not sure if this specific issue is important.
>> If we want to use find from Emacs, we would need to create Emacs
>> string/strings when reading the output of find anyway.
>
> So how do you explain that using Find is faster than using
> find-lisp.el?
>
> I think the answer is that using Find as a subprocess conses less.

No. It uses less excessive regexp matching Emacs is trying to do in
file-name-handler-alist.

(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (find-lisp-find-files "/home/yantar92/.data" ""))))
;; Elapsed time: 2.982393s
(ignore (let ((gc-cons-threshold most-positive-fixnum) file-name-handler-alist) (benchmark-progn (find-lisp-find-files "/home/yantar92/.data" ""))))
;; Elapsed time: 0.784461s


    22.83%  emacs         emacs                            [.] Fnconc
    10.01%  emacs         emacs                            [.] Fexpand_file_name
     9.22%  emacs         emacs                            [.] eval_sub
     3.47%  emacs         emacs                            [.] assq_no_quit
     2.68%  emacs         emacs                            [.] getenv_internal_1
     2.50%  emacs         emacs                            [.] mem_insert.isra.0
     2.24%  emacs         emacs                            [.] Fassq
     2.02%  emacs         emacs                            [.] set_buffer_internal_2


(ignore (let ((gc-cons-threshold most-positive-fixnum) file-name-handler-alist) (benchmark-progn (find-lisp-find-files "/home/yantar92/.data" ""))))
;; Elapsed time: 0.624987s

    12.39%  emacs         emacs                                    [.] eval_sub
    12.07%  emacs         emacs                                    [.] Fexpand_file_name
     4.97%  emacs         emacs                                    [.] assq_no_quit
     4.11%  emacs         emacs                                    [.] getenv_internal_1
     2.77%  emacs         emacs                                    [.] set_buffer_internal_2
     2.61%  emacs         emacs                                    [.] mem_insert.isra.0
     2.47%  emacs         emacs                                    [.] make_clear_multibyte_string.part.0

Non-recursive version of `find-lisp-find-files-internal' is below,
though it provides limited improvement.

(defun find-lisp-find-files-internal (directory file-predicate
						directory-predicate)
  "Find files under DIRECTORY which satisfy FILE-PREDICATE.
FILE-PREDICATE is a function which takes two arguments: the file and its
directory.

DIRECTORY-PREDICATE is used to decide whether to descend into directories.
It is a function which takes two arguments, the directory and its parent."
  (setq directory (file-name-as-directory directory))
  (let (results fullname (dirs (list (expand-file-name directory))))
    (while dirs
      (setq directory (pop dirs))
      (dolist (file (directory-files directory nil nil t))
	(setq fullname (concat directory file))
	(when (file-readable-p fullname)
	  ;; If a directory, check it we should descend into it
	  (and (file-directory-p fullname)
               (setq fullname (concat fullname "/"))
               (funcall directory-predicate file directory)
               (push fullname dirs))
	  ;; For all files and directories, call the file predicate
	  (and (funcall file-predicate file directory)
               (push fullname results)))))
    results))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 17:25:02 +0000
Resent-Message-ID: <handler.64735.B64735.1689873858912 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.1689873858912
          (code B ref 64735); Thu, 20 Jul 2023 17:25:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 17:24:18 +0000
Received: from localhost ([127.0.0.1]:59800 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMXNx-0000Ed-Nb
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:24:18 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:54902)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMXNu-0000EO-KP
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:24:17 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMXNo-0005cY-D5; Thu, 20 Jul 2023 13:24:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=WY48oObOttS85iuWJ3C8cMAvETfvKkT0g1fbrE+ePxM=; b=GvzfFssuXimY
 CsLrPg79A6BXb+q6uBxbs10mptW8j5nsJl89FbpB5obvgNuPqhuv6BzeCZxw99uNko7JjTOQzh5aA
 pypsY/UkuaLPkSB1/2iikOKFihjJN+18JZx3hpwpvuznJAVmjuJ2MouuEA4jk13m7swvXo8xTpeK9
 n9bORHxIQXX9CFm1zR7P1wDBJivBhuq2T5eYUbpfpljJGUr8R/60pFyC6euKOR+UpgtB31KXyPBqN
 GiRybavqSGDNvh/qpE5qhvMOQM1O6ubzhOFvPfg1R6r4tqagIkdu0L/PlyZKgBWAgXEjduisTXSq3
 vtHKII78XMYNnmGOMAHhCA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMXNn-0005hb-Rd; Thu, 20 Jul 2023 13:24:08 -0400
Date: Thu, 20 Jul 2023 20:24:41 +0300
Message-Id: <83351i338m.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <ierfs5iwlx3.fsf@HIDDEN> (message from Spencer Baugh on
 Thu, 20 Jul 2023 13:08:24 -0400)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
> From: Spencer Baugh <sbaugh@HIDDEN>
> Date: Thu, 20 Jul 2023 13:08:24 -0400
> 
> (Really it's entirely plausible that Emacs could be improved by
> *removing* directory-files-recursively, in favor of invoking find as a
> subprocess: faster, parallelized execution, and better remote support.)

No, there's no reason to remove anything that useful from Emacs.  If
this or that API is not the optimal choice for some job, it is easy
enough not to use it.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 17:26:02 +0000
Resent-Message-ID: <handler.64735.B64735.16898739241068 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898739241068
          (code B ref 64735); Thu, 20 Jul 2023 17:26:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 17:25:24 +0000
Received: from localhost ([127.0.0.1]:59809 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMXP0-0000H9-Kl
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:25:24 -0400
Received: from mout01.posteo.de ([185.67.36.65]:54703)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMXOj-0000Fh-Sn
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 13:25:20 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 5AF1E240027
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 19:25:00 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689873900; bh=gRuq5oCQCBhnUAel41GxugDO7R/A9YScDamizVYkx8U=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=OhtPruGy3NSD25fD1PWJzX1ndDTmTEVEgi22E25FwF6i9+wgUAz0Fyrv2Y+lUcwcb
 VoY5xvN5fB25nr7xzBjtBtrc+FR25dJ5j/8GACWbJraX/S1Eq1KecLiZLwC8dCD4UV
 E4VLq1vJ19vgO75ZmuBixeP401IoeqbIHQ4SN8WmxG8a+GCc1pnQHdoo2YiOxDg3IX
 Pi1D2kNAuh/RDUFB1SsbqDYPK/lXsh5qZYr6ijKEjwe91e0XLT5ZhliTo0VjQskCRK
 AwPR4sambX8F6wx7CHa96OlhU1AcozSudG9rerlJyBzDdrjlwWcHk640/gSj5FP/yc
 xqk4qtozzgfgA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6KLz4DQyz9rxR;
 Thu, 20 Jul 2023 19:24:59 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <ierfs5iwlx3.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 17:25:11 +0000
Message-ID: <87ilaepkaw.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Spencer Baugh <sbaugh@HIDDEN> writes:

>> Sure. It might also be optimized. Without trying to convince find devs
>> to do something about regexp handling.
>
> Not to derail too much, but find as a subprocess has one substantial
> advantage over find in Lisp: It can run in parallel with Emacs, so that
> we actually use multiple CPU cores.

Does find use multiple CPU cores?


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:24:02 +0000
Resent-Message-ID: <handler.64735.B64735.16898774407727 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898774407727
          (code B ref 64735); Thu, 20 Jul 2023 18:24:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:24:00 +0000
Received: from localhost ([127.0.0.1]:59848 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYJk-00020Z-4W
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:24:00 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:48944)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMYJg-00020I-OJ
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:23:58 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMYJa-0005os-BD; Thu, 20 Jul 2023 14:23:50 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=GGMHbvHUoxTIXZqRWzh1c6/mWU+i8T1NZX8DHE1s/fU=; b=ic7upQgdezJi
 YVFFf4CihdqzvZqnxZ+I4KKb0zMU/KX9Vbj+O4QxxmofUfU6YtUB/kAUH3kiskrbzZbnAqTISHlk5
 u0FR4I4ncSCfUKLldUpf//IsNK8dmhLLCGRTptFOLdkjNik/6lqSlK0vdbF7uVUMhSDNvAzN+BFhb
 //h/5RQqWdRrMmL793JUcJVTCm4DlhTyC9Um59naiD2cN0uQ64T1RXj8jiEUPfrshvWCfIQMR1FaF
 cYvHFzUV2NOw4QWO1FgyQ8K7/hAMvb91pepzm/wx4zup5cWGjNq/fbQZUP3y6xeD65QT9UzEZZc7O
 iWAjN/zmYHNvVIaVLt+Vgg==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMYJZ-0003H6-Rf; Thu, 20 Jul 2023 14:23:50 -0400
Date: Thu, 20 Jul 2023 21:24:22 +0300
Message-Id: <831qh230h5.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87lefapkdx.fsf@localhost> (message from Ihor Radchenko on Thu,
 20 Jul 2023 17:23:22 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: dmitry@HIDDEN, sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Thu, 20 Jul 2023 17:23:22 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> I am not sure if this specific issue is important.
> >> If we want to use find from Emacs, we would need to create Emacs
> >> string/strings when reading the output of find anyway.
> >
> > So how do you explain that using Find is faster than using
> > find-lisp.el?
> >
> > I think the answer is that using Find as a subprocess conses less.
> 
> No. It uses less excessive regexp matching Emacs is trying to do in
> file-name-handler-alist.

Where do you see regexp matching in the profiles you provided?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:30:02 +0000
Resent-Message-ID: <handler.64735.B64735.16898777868246 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898777868246
          (code B ref 64735); Thu, 20 Jul 2023 18:30:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:29:46 +0000
Received: from localhost ([127.0.0.1]:59853 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYPK-00028w-0q
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:29:46 -0400
Received: from mout01.posteo.de ([185.67.36.65]:52861)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMYPH-00028i-Cu
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:29:44 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 6AB81240028
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 20:29:37 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689877777; bh=mQLb4duNHTj+Rup1cO0lQWX+la/b6pu2r50GOUbAdBM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=Sx7269XC6tXWA+ScQeZbDM+4jzju/qgszq4pw0rLAnCnzukirO8OZxquayNKZxD9p
 Py378yn2SmkziXaFmj3eqVZNZsqS+DonwDdwBQdx/uKLXW3ECU/7DjE8M/EzeJc3MM
 rMSJeH/nPP96lY+z2wEQrXs2xTLqujF+1IiUBt+BfDQ0Xbymyd+8EO3+/H7qdTXNIr
 aUPyBMFF1Ad/sHaTmLIWdeP9aTwFclp4duYztU1iSIIQRxLsL7GWY3M1QvSsS6GLEw
 gWGHSYMZ8mPGaxA2kydg6bH3Sw7tyg65Oy+E6Mv52OplJ1isenPqtX+AZ0d4paxZ6T
 6pCjzrx9Q6log==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6LnX3wTTz9rxg;
 Thu, 20 Jul 2023 20:29:36 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <831qh230h5.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 18:29:43 +0000
Message-ID: <87wmyu8mi0.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> No. It uses less excessive regexp matching Emacs is trying to do in
>> file-name-handler-alist.
>
> Where do you see regexp matching in the profiles you provided?

I did the analysis earlier for `directory-files-recursively'. See
https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/

Just to be sure, here is perf data for
(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (find-lisp-find-files "/home/yantar92/.data" ""))))

    54.89%  emacs    emacs                            [.] re_match_2_internal
    10.19%  emacs    emacs                            [.] re_search_2
     3.35%  emacs    emacs                            [.] unbind_to
     3.02%  emacs    emacs                            [.] compile_pattern
     3.02%  emacs    emacs                            [.] execute_charset
     3.00%  emacs    emacs                            [.] process_mark_stack
     1.59%  emacs    emacs                            [.] plist_get
     1.26%  emacs    emacs                            [.] RE_SETUP_SYNTAX_TABLE_FOR_OBJECT
     1.17%  emacs    emacs                            [.] update_syntax_table
     1.02%  emacs    emacs                            [.] Fexpand_file_name

Disabling `file-name-handler-alist' cuts the time more than 2x.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:43:02 +0000
Resent-Message-ID: <handler.64735.B64735.16898785599822 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16898785599822
          (code B ref 64735); Thu, 20 Jul 2023 18:43:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:42:39 +0000
Received: from localhost ([127.0.0.1]:59878 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYbm-0002YM-QG
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:42:39 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:58592)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMYbk-0002Y6-9K
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:42:37 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMYbe-0001xF-4f; Thu, 20 Jul 2023 14:42:30 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=UAOLTQFKlvjSNejI7YfyNhtOOVwoCwqhzecvWxmLM9E=; b=pEBFyNdjek6+
 BEev7CPSPrwq4Yrw6K51pmI6LHBdHMGh0FfSCDiCGB2Gc1UEgDAILNbxohA6AkCh6ohIzuM4PirJN
 ggBNEuguQ1zMeRWwFyG+kH0Q/sA6yvjbFwtJ2akMUg7DHKKYfuONS89VM0Q8qqQmNVem7YU7ptJzg
 OAyQPrQ1NPAR+EeRensTgMV5EDh837vvzlekn99ZtC56NcQqZ1YaNJCsrAIOQ4WcBwxXMZgtQzMXv
 BOvBBTivQqQwb90c6x043IArI7wNk81Dxg82pnhJXmJe0fCWuhM/3+z6aQeWXv3OSHsTtIjE9PHSY
 DqkGPYeZKBqj3G84jIBZXA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMYbd-0000pN-H6; Thu, 20 Jul 2023 14:42:29 -0400
Date: Thu, 20 Jul 2023 21:43:03 +0300
Message-Id: <83wmyu1l1k.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87wmyu8mi0.fsf@localhost> (message from Ihor Radchenko on Thu,
 20 Jul 2023 18:29:43 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: dmitry@HIDDEN, sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Thu, 20 Jul 2023 18:29:43 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> No. It uses less excessive regexp matching Emacs is trying to do in
> >> file-name-handler-alist.
> >
> > Where do you see regexp matching in the profiles you provided?
> 
> I did the analysis earlier for `directory-files-recursively'. See
> https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
> 
> Just to be sure, here is perf data for
> (ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (find-lisp-find-files "/home/yantar92/.data" ""))))
> 
>     54.89%  emacs    emacs                            [.] re_match_2_internal
>     10.19%  emacs    emacs                            [.] re_search_2
>      3.35%  emacs    emacs                            [.] unbind_to
>      3.02%  emacs    emacs                            [.] compile_pattern
>      3.02%  emacs    emacs                            [.] execute_charset
>      3.00%  emacs    emacs                            [.] process_mark_stack
>      1.59%  emacs    emacs                            [.] plist_get
>      1.26%  emacs    emacs                            [.] RE_SETUP_SYNTAX_TABLE_FOR_OBJECT
>      1.17%  emacs    emacs                            [.] update_syntax_table
>      1.02%  emacs    emacs                            [.] Fexpand_file_name
> 
> Disabling `file-name-handler-alist' cuts the time more than 2x.

Disabling file-handlers is inconceivable in Emacs.  And I suspect that
find-file-name-handler is mostly called not from directory-files, but
from expand-file-name -- another call that cannot possibly be bypassed
in Emacs, since Emacs behaves as if CWD were different for each
buffer.  And expand-file-name also conses file names.  And then we
have encoding and decoding file names, something that with Find we do
much less.  Etc. etc.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:55:02 +0000
Resent-Message-ID: <handler.64735.B64735.168987928721000 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987928721000
          (code B ref 64735); Thu, 20 Jul 2023 18:55:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:54:47 +0000
Received: from localhost ([127.0.0.1]:59891 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYnX-0005Se-7L
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:54:47 -0400
Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:42967)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMYnT-0005SO-CO
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:54:46 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id C871C3202AEB;
 Thu, 20 Jul 2023 14:54:36 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Thu, 20 Jul 2023 14:54:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689879276; x=1689965676; bh=jmu7/QfDNUt5SDBOBPFWjQRSk/qub2wO593
 n2iePy0E=; b=cf6XbX662hXkX442wPgZ1cx4N7CgTaV42f2E0C7uDrEODKm0svp
 CHvlI28ED+wftitd+JFue49DQyIuCwxStee4PFkgU1N9QxP/Hn5iLkSNURelPhzJ
 /U3tvBm+CSf2b908wmdr0J3j9Dacf54OikLjyVX257n+ZeNPBfasIrKVJSMn/ih1
 wsYjd6WEfcM8JYc7wWdGZT5xqnGxKEH3e/51XBUEK5LgLLRBYoAN4igJSmrlvsfP
 320qz368nhNcvJlG7y2EL9G6YjfofcwcLMeuHct4MZs1jRXjrzuW+veXesr5wpyi
 pL6Gj8E1qlK7OSRGIYX9bU+qRUTaBskTbEA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689879276; x=1689965676; bh=jmu7/QfDNUt5SDBOBPFWjQRSk/qub2wO593
 n2iePy0E=; b=NXyEf2CKwbKnGIHmKhlqQxy+v2O4CqWBK1LfjAXNWJW36uMhGHf
 6KTIStziOoBq37qK2wYLVpkj3/IDZIS7SyEIKs0uIo6WWn8I487O3jXqsuq9QVbi
 qY2Zca8kTlwiVa8/s/kHSClP0Ijnxsxj2IBTCf1p34Gk55UAP6NxLKlh6SFrDLoE
 sUNnGKuAwyiTG7ZZR5XSMYiAPVACg7gMAumRqCSXR3j01adUt5wKaNq1WT0Hscuk
 TeDJc0prnb9w1ZM/pSryjYa9c2B/LUba323FEeATvF6MD/h2pUk8scue+UJpmWKf
 1fko45n6+Z7KCeRtmRVPVspZ9gTVmudwqKA==
X-ME-Sender: <xms:64K5ZIMysOxa_qPrZnSMWBL24ZMHDGqwtzykfZC_n0rAJkfYkwlEmg>
 <xme:64K5ZO9PIYcipKAcql7FsBbTYxjAxXDa4cDxTs-xBRy7XpDDfFvm8tChv1_ms7uT0
 5A4-JwUJOHijOXKNac>
X-ME-Received: <xmr:64K5ZPQatGDnkCzM4QOEfrzf2iw_UQYNlMmTN8952FA4d8GI-1UnJMLlJAZx6vw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgddufedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:64K5ZAseurLoQoanAvpcSCmKIhj1aq21qiEVw3G0WdnYOTVrH4yhvw>
 <xmx:64K5ZAdxpMhIZ5ccQpJsGXrJEMl8fSehbkPIqT6wpSpGegqH1EHx_A>
 <xmx:64K5ZE0G6_DxtohTAvwoQO61wJLPyfVmoTv2sPov3fCR2Vdqq8gvXQ>
 <xmx:7IK5ZBr4M2v1PKA9tyX_Oa4oD3AZBjy1krY3h-q4__TOXTBZ_W1u6A>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 14:54:34 -0400 (EDT)
Message-ID: <3a33eb87-4f1c-efee-3d2d-d3ae2c3e7e4b@HIDDEN>
Date: Thu, 20 Jul 2023 21:54:32 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN> <837cqv41ob.fsf@HIDDEN>
 <87mszqixhh.fsf@HIDDEN> <4d4f029f-f32f-13df-ffc3-3952d62d8bb3@HIDDEN>
 <ierjzuuwvds.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <ierjzuuwvds.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 16:43, Spencer Baugh wrote:

>> That's only a problem when the default file listing logic is used (and
>> we usually delegate to something like 'git ls-files' instead, when the
>> vc-aware backend is used).
> 
> Hm, yes, but things like C-u project-find-regexp will use the default
> find-based file listing logic instead of git ls-files, as do a few other
> things.

Right.

> I wonder, could we just go ahead and make a vc function which is
> list-files(GLOBS) and returns a list of files?  Both git and hg support
> this.  Then we could have C-u project-find-regexp use that instead of
> find, by taking the cross product of dirs-to-search and
> file-name-patterns-to-search.  (And this would let me delete a big chunk
> of my own project backend, so I'd be happy to implement it.)

I started out on this inside the branch scratch/project-regen. Didn't 
have time to dedicate to it recently, but the basics are there, take a 
look (the method is called project-files-filtered).

The difficulty with making such changes, is the project protocol grows 
in size, it becomes difficult for a user to understand what is 
mandatory, what's obsolete, and how to use it, especially in the face of 
backward compatibility requirements.

Take a look, feedback is welcome, it should help move this forward. We 
should also transition to returning relative file names when possible, 
for performance (optionally or always).

> Fundamentally it seems a little silly for project-ignores to ever be
> used for a vc project; if the vcs gives us ignores, we can probably just
> ask the vcs to list the files too, and it will have an efficient
> implementation of that.

Possibly, yes. But there will likely remain cases when the project-files 
could stay useful for callers, to construct some bigger command line for 
some new feature. Though perhaps we'll be able to drop that need by 
extracting the theoretically best performance from project-files (using 
a process object or some abstraction), to facilitate low-overhead piping.

> If we do that uniformly, then this find slowness would only affect
> transient projects, and transient projects pull their ignores from
> grep-find-ignored-files just like rgrep, so improvements will more
> easily be applied to both.  (And maybe we could even get rid of
> project-ignores entirely, then?)

Regarding removing it, see above. And it'll take a number of years 
anyway ;-(

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:57:02 +0000
Resent-Message-ID: <handler.64735.B64735.168987938121180 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987938121180
          (code B ref 64735); Thu, 20 Jul 2023 18:57:02 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:56:21 +0000
Received: from localhost ([127.0.0.1]:59899 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYp3-0005VY-0O
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:56:21 -0400
Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:36627)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMYp0-0005V6-82
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:56:18 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.west.internal (Postfix) with ESMTP id 7A8A33200BFF;
 Thu, 20 Jul 2023 14:56:12 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute6.internal (MEProxy); Thu, 20 Jul 2023 14:56:12 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm1; t=
 1689879372; x=1689965772; bh=sUqPPrSfAAds49KCIzaIu4SWGxKzBxO6mq7
 kYqVW7HM=; b=M+NQ7Ot8YD1xDWcoiSZRB/mLdj2xA3fTx9j78kUiAyT+cc/9V7+
 rvmj4dmiQKCFFb0aNI5deUkf0/oCI83YsjGOawUpoElPTECbzPiJx+MEi01O4k5K
 pamOia56mkmt9X2RWe1WUEu4fecXKt24HkywiWAHZFij6YhLicaNICG+JfYSgIhh
 ATVkozciPlbDw+sLL+sM8H56DYIkGU+xB1kCyexGv8zK2C7B7WCKzhTZaDA31rhp
 vtr9FV/aVPGfvQvGTrYDhaH9gM6LM0H3ucMgJ7n8pm1NZ5yPG3tN/a2jmEKBDCUW
 ITPNUCQ+eDuQKVylRCA5ZRqOsB9buL/WPLQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689879372; x=1689965772; bh=sUqPPrSfAAds49KCIzaIu4SWGxKzBxO6mq7
 kYqVW7HM=; b=WJdQfUeVHFLbrqKqos9mIkaP68l1gRsH/9OKWLcGOfzalOhejIP
 Ts9hXwi+6t8phq2Bz3MgTs2W0BtMufBchH/5gf8rFPoFAMFXvUXP34jnp7hfB829
 jU3QH73I9Y8blvdHupCxZFjGPZypK3EbyUyznRvTI3RQq5oKeXhEvgkOle4yKvet
 fVjR3E2hb/a2/ur6ZC3Edcf00c7wMWr9MqwjrrbYeB8v0rknMetl/MJW/SSxB7Ev
 6z37QeqyVYT/2VYPtfyRRX6JmDbEOq3zceSUlARd6QYjuqb+UVicYCTdCNVP4qU2
 As7LrzNYcHJAnFwZthadJuAiOaDSSrzpqDg==
X-ME-Sender: <xms:S4O5ZPNckgOcBmNU8QeKBFuWNKVhO0EviIIfHL2tlde59Uvmamg5EA>
 <xme:S4O5ZJ8BzraAjBNbMBtyJF230BphmZVdT89SfmvAwY9rQUbMlTmGkoQLDyFhR0vsS
 q1afmobwHqXJPzRFYM>
X-ME-Received: <xmr:S4O5ZOTiH0NEtJRaGD6oJWkKYY509SPGFRtsks21UXdBzSOaEKLEPkGiyHRkrzw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedtgddufedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:S4O5ZDvff2GtIwpOMfViplhaEfxyKyowp-TqErsLl0KbdMKvdOPmRQ>
 <xmx:S4O5ZHfSqJZhgzXy4JKPMln8Am9DfuxsnPYB6B3g2EBWxwUu7dxQmQ>
 <xmx:S4O5ZP0ik4tNpwSPOMPl2ad6llIpQGRuiX0DS_PxViWdfz0ye-fMCg>
 <xmx:TIO5ZPF2_wADtSI1wVtR0BnKXAraTzt2_kGaLg5S_E2RNOTDmnBYNw>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 20 Jul 2023 14:56:10 -0400 (EDT)
Message-ID: <615e8997-049b-0685-07db-c87dbbc0ac85@HIDDEN>
Date: Thu, 20 Jul 2023 21:56:09 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <aba1400a-3d00-44f0-90f1-e5b549de8ae7@HIDDEN> <87r0p2po29.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87r0p2po29.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 19:03, Ihor Radchenko wrote:
> Dmitry Gutov<dmitry@HIDDEN>  writes:
> 
>>> And things are not as horrible as 15x slowdown in find.
>> We haven't compared to the "optimized regexps" solution in find, though.
> Fair point.
> 
>> Still like 2.5x slower, then? That's significant.
> It is, but it is workable if we try to optimize Emacs'
> `directory-files'/`file-name-all-completions' internals.
> 
>>> I am pretty sure that Emacs's native file routines can be optimized to
>>> the level of find.
>> I don't know, the GNU tools are often ridiculously optimized. At least
>> certain file paths.

Sorry, I meant "code paths" here.

> You are likely right.
> Then, what about applying regexps manually, on the full file list
> returned by find?

It will almost certainly be slower in cases where several (few) ignore 
entries help drop whole big directories from traversal.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 20 Jul 2023 18:58:01 +0000
Resent-Message-ID: <handler.64735.B64735.168987947121317 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168987947121317
          (code B ref 64735); Thu, 20 Jul 2023 18:58:01 +0000
Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 18:57:51 +0000
Received: from localhost ([127.0.0.1]:59905 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMYqV-0005Xk-Ev
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:57:51 -0400
Received: from mout01.posteo.de ([185.67.36.65]:47081)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMYqR-0005XS-4x
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 14:57:49 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 359F1240028
 for <64735 <at> debbugs.gnu.org>; Thu, 20 Jul 2023 20:57:41 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689879461; bh=lwc38aSJm0wtB/Mnc62i8GtLe7/JMZwJTG3YpzRVRdM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=AlX61wIzULoJDplBECgtlDqpoggQcvaLD7nOihCTX713WjzOSxFB8nbama740p+Lr
 XZ+JzbjeshmZEySiEE9Miudzi8oAXsKVpkxKbBfpCOvMfomKACwTE3ayo5bMEWSUSj
 s4BEit/i2/Tcih4DKkavwws91Sx0kNouwOrIc2WSbVaCf0ruW26v3myn6RfA6gYqDo
 UycodDkfT7KONyxic/1n6J+Yn4HhciWJCgPSE0Ie+IUJahyiRTLhkpnvanH3Wmp1Oz
 04MHsl3O6ewpDjjb4wOb/h/Dzg3sy5FT2ZwSSLQP22anPbamaI+QL0cdzeZVPC6tU/
 dFRNb1EYJNPDA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6MPw2shfz6ty3;
 Thu, 20 Jul 2023 20:57:40 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83wmyu1l1k.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
Date: Thu, 20 Jul 2023 18:57:52 +0000
Message-ID: <87ttty8l73.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Disabling `file-name-handler-alist' cuts the time more than 2x.
>
> Disabling file-handlers is inconceivable in Emacs.

Indeed. But we are talking about Emacs find vs. GNU find here.
In the scenarios where GNU find can be used, it is also safe to disable
file handlers, AFAIU.

> ... And I suspect that
> find-file-name-handler is mostly called not from directory-files, but
> from expand-file-name -- another call that cannot possibly be bypassed
> in Emacs, since Emacs behaves as if CWD were different for each
> buffer.  And expand-file-name also conses file names.  And then we
> have encoding and decoding file names, something that with Find we do
> much less.  Etc. etc.

expand-file-name indeed calls Ffind_file_name_handler multiple times.
And what is worse: (1) `find-lisp-find-files-internal' calls
`expand-file-name' on every file in lisp, even when it is already
expanded (which it is, for every sub-directory); (2) `directory-files'
calls Fexpand_file_name again, on already expanded directory name;
(3) `directory-files' calls Ffind_file_name_handler yet again on top of
what was already done by Fexpand_file_name; (4) `directory-files' calls
`directory_files_internal' that calls `Fdirectory_file_name' that
searches `Ffind_file_name_handler' yet one more time.

There is a huge amount of repetitive calls to Ffind_file_name_handler
going on. They could at least be cached or re-used.

I do not see much of encoding and consing present in perf stats.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Richard Stallman <rms@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 02:43:03 +0000
Resent-Message-ID: <handler.64735.B64735.168990737418992 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: 64735 <at> debbugs.gnu.org
Reply-To: rms@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168990737418992
          (code B ref 64735); Fri, 21 Jul 2023 02:43:03 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 02:42:54 +0000
Received: from localhost ([127.0.0.1]:60280 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMg6X-0004wG-P7
	for submit <at> debbugs.gnu.org; Thu, 20 Jul 2023 22:42:54 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:48852)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rms@HIDDEN>) id 1qMg6V-0004vy-0J
 for 64735 <at> debbugs.gnu.org; Thu, 20 Jul 2023 22:42:52 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <rms@HIDDEN>)
 id 1qMg6P-0000i3-4d; Thu, 20 Jul 2023 22:42:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=Date:References:Subject:In-Reply-To:To:From:
 mime-version; bh=uRns8MqLFonCUBub+9Oj8dRSdji8Hd54ty2rfUcthX4=; b=CWrYhKcLOmHD
 mBdqCBeKUKOMy+CF4SdjFNzAJyAGNZ59jG1npS60kRNMLQVPIfLRhPWRZr+1zmusyjxBPe4cEQxru
 /tb5+7uKNlD1sYaVjhF9htonSMa+SNBuggYuROYo9b3bWbDFHAzDGUnSgCVRoGoQAd6ys/HsL7m++
 IboE8JBIrZqXiLMlKQNoK5T9ZHc/svCk6TdgF+29sBXL7Q9tFZYHKVvkoD6oeqf27lm5/EUMgLs+s
 a5l7jGTA2KZPpB2PnhhnBLSItiXurE2qJN39es74acX/1/V1kMnMbnBC8Q9L4m9UBVU1/Ni8TrpNT
 PjuiCtG8912tPUmzjzB6TA==;
Received: from rms by fencepost.gnu.org with local (Exim 4.90_1)
 (envelope-from <rms@HIDDEN>)
 id 1qMg6O-0003aa-SZ; Thu, 20 Jul 2023 22:42:44 -0400
Content-Type: text/plain; charset=Utf-8
From: Richard Stallman <rms@HIDDEN>
In-Reply-To: <iermszrwqj4.fsf@HIDDEN> (message from Spencer Baugh on
 Wed, 19 Jul 2023 17:16:31 -0400)
References: <iermszrwqj4.fsf@HIDDEN>
Message-Id: <E1qMg6O-0003aa-SZ@HIDDEN>
Date: Thu, 20 Jul 2023 22:42:44 -0400
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I will take a look at this.

In case they are reluctant because of being busy, would anyone like
to help out by writing the code to do the optimization?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 07:46:01 +0000
Resent-Message-ID: <handler.64735.B64735.168992554322121 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168992554322121
          (code B ref 64735); Fri, 21 Jul 2023 07:46:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 07:45:43 +0000
Received: from localhost ([127.0.0.1]:60466 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMkpb-0005kh-9B
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 03:45:43 -0400
Received: from mout.gmx.net ([212.227.17.20]:42393)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMkpY-0005kS-2m
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 03:45:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689925528; x=1690530328; i=michael.albinus@HIDDEN;
 bh=LaAANdTiOwyvYVLhZ/U4cpjZeiP13y74Zb8Gr3kAvQk=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=jjwiVmxbbF+qNxyEoocizZ6zXga58d6u7hr97HrZmWeV/uaMRK9bhC4B2UKwhG8LHW4ajUc
 ftCYlNiOkIR35fl4Cg5EUa1CzM8IZwgGnEbJXj/Yo2QFR24M/BanCBKnV+q+drz3s07cR/0/6
 3wgdVeo1E1k7TmzifJVgQFXrl9p6uYZIbUxMpHTtNZJ26e+UbNNv0QmiXjGDnDrPbbeHcUAYk
 rwJ07niWHY15Dw1HS7EEs7VP5HK2CEq5vgK+bZaTo6Oeu8xCigncmMKi568Iq1CW7L1oco8ZC
 O23I3TqvWlnEJ2FDm8L9DZZy5Gw+BeU4X83Lw3zh86eMK1+B077w==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N5mGB-1ppdop24Wx-017Fm0; Fri, 21
 Jul 2023 09:45:28 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83wmyu1l1k.fsf@HIDDEN> (Eli Zaretskii's message of "Thu, 20 Jul
 2023 21:43:03 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 09:45:26 +0200
Message-ID: <87fs5hemi1.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:b/SmY+B/DfjMtgpc7w5QOzZlx0NVKW5zW8c/dmgIkZBy4x4+fsq
 BdTJHxb7trcu6HjJrQXDtTHkbzetjnKeS+0Ppa5/D76IHljQH09zzeZ90umJe5QXIxC6rZF
 ppkglRAMejmX2XiMtA/nCv9G9ntNgtIlNxKsxjjwPSGKfBSYbtcJwl9Rd1IsRwUj7eLaxk6
 Do7hOnudHPLkhifHOzZzA==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:4306727PI8M=;TirqMR+X4Crd1a37T6jhMaRjden
 4HgVoO3lrTFgVOzGSMVw5SmCc00vwzy1CkXDYrWW7bS4LVmSmRzGHp7SP/+EJFvDHb+6BsxiU
 YsiMDgyt7+4t7mFQucZhlQdNXmXvIXnZIbFdTgnyER9tLSVC4jqglWn+a8ZDjD8Ztq7n+/mBb
 893eS+P8KAgO6fCo9ZvYMaOu9q4/0j/zvvfND/B3Vvb/6MV3smYhO/p059WzSNUS2sf99f0kR
 tm375msFOmFm+ulZfuCSTrraDfT+3QnDSOnMNxWY1zjfEKgSIhE1BffeMfrjKBWsWh+9MS5u+
 jo15TiebbCBqQiegRbtBSO5BjQWT8sL0yq7RKqaA4Ag72FIevG6eHONs68Qqp9ovDCg4oYULV
 J0+tsH/uPJlDFaOleiCup+ZYO/2+cDdKVX+1PRqnhyPdhWeDUTjERTOxMx89cje1VJukeg2/h
 /XY23KoUGG2EcrxpQMs98+hyw5e55d8bhdX8HyVUh3C1QllM2rGCSo2qsyUYPSkpCm0MQwhW6
 hjhpGmUFty50vGcG32KzyelOj/6eVoqc/yR0BAKRVRcO3wMzKoRByQBvQxB5CdppuBAJjecrA
 Tc+gzXEgoUDZZQpMOtV4RZcU2NJIOELbxusAE1jP6+I+qnzPGEeFXnLPKEvmo/L+HQwmLGJq5
 cf0n0hv+vSC1BLBC6Ke0RjNnIF8pMDF6u6sITmeWD0B76QjwSLTqcWdUgbCcSpFLAOnGO57sw
 6RKLdX+0Jx24G80WRltG35DWdpZXqP49x4faxpaM16dI2m/2fMy+ScOZ4Z4QNL2p1FIZQT487
 m0RRDQKfwdJB/WfDNf1IJe7Xi1xsPv6NUFSlzHRxKTtDu5e5PuIh/zM805pJdm560hcZ3AlOw
 glTwHw6oszNNVtO2s82avdCQpcmIYomqKzRfS8gl20J5eB+Wc4PuOV7kn2Tco88yL8ZHWtdh3
 kREWyw==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

Hi Eli,

>> Disabling `file-name-handler-alist' cuts the time more than 2x.
>
> Disabling file-handlers is inconceivable in Emacs.

Agreed.

However, the fattest regexps in file-name-handler-alist are those for
tramp-archive-file-name-handler, tramp-completion-file-name-handler and
tramp-file-name-handler. Somewhere else I have proposed to write a macro
without-remote-files and a command inhibit-remote-files, which disable
Tramp and remove its file name handlers from file-name-handler-alist.
Either temporarily, or permanent.

Users can call the command, if they know for sure they don't use remote
files ever. Authors could use the macro in case they know for sure they
are working over local files only.

WDYT?

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 09:15:01 +0000
Resent-Message-ID: <handler.64735.B64735.168993087931561 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168993087931561
          (code B ref 64735); Fri, 21 Jul 2023 09:15:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 09:14:39 +0000
Received: from localhost ([127.0.0.1]:60554 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMmDf-0008Cz-8o
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 05:14:39 -0400
Received: from mout01.posteo.de ([185.67.36.65]:43201)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMmDc-0008Cl-MN
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 05:14:38 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id B768C240028
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 11:14:30 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689930870; bh=Jv/l9yzMWUP+gQ7eH2Q9XgWsHG+p53GrjjKRe8nGoJM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=BKXVeIQIVULAFObnXo7+g0vFgVMtvVo6E991rs6AVqIh/9GZvmhw6xRBc+FFjMtp8
 1HyHp90CTfyazVM5wdtuWm+pGIzCkgaCXg1VhMX2GuDwqylWXuOfiyBJoLlt//6is5
 WQVTWRpyiYxDZJIlyKT3oTdvDpJMi+9sEfvX0/L31tujIuTH+1TVQwZboivwZANh7n
 nInpYhHTPW8xpy+CypkTjtFTBjspYjuCgIfsWRsYXCuMMsZA8/gsHr/lSzAzV7ssU6
 obOd3zrcS51K5xMULkgrf85Ub1I6BgglHZyzrnc0nDvgbE885WkEYkwg7CzntI/Y+r
 Kjmkd8XadqGHw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6kQY5VV9z9s06;
 Fri, 21 Jul 2023 11:14:29 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <615e8997-049b-0685-07db-c87dbbc0ac85@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost>
 <aba1400a-3d00-44f0-90f1-e5b549de8ae7@HIDDEN>
 <87r0p2po29.fsf@localhost>
 <615e8997-049b-0685-07db-c87dbbc0ac85@HIDDEN>
Date: Fri, 21 Jul 2023 09:14:41 +0000
Message-ID: <87wmyt62ym.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

>> You are likely right.
>> Then, what about applying regexps manually, on the full file list
>> returned by find?
>
> It will almost certainly be slower in cases where several (few) ignore 
> entries help drop whole big directories from traversal.

Right.
Then, what about limiting find to -depth 1, filtering the output, and
re-running find on matching entries?

It gets complicated though, and the extra overheads associated with
invoking a new process may not be worth it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 10:47:01 +0000
Resent-Message-ID: <handler.64735.B64735.16899363989144 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899363989144
          (code B ref 64735); Fri, 21 Jul 2023 10:47:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 10:46:38 +0000
Received: from localhost ([127.0.0.1]:60642 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMnef-0002NQ-N5
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 06:46:38 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:36474)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMneb-0002N8-Dk
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 06:46:37 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMneV-0003zn-O5; Fri, 21 Jul 2023 06:46:27 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=mA/4X9NeX9Pf0zYNgafEeJTngSKAXdHVQV28bgEHkFU=; b=guEr5RE5j6aT
 JG2kuUwi/1TKwMsnCjHxvryAsdY/cd9v1UprMtJ+zr5HawezFipSYvFx6kdE56VlSyQ/Mg0gO6UmD
 wNJmsf/8X97fDsmOKM6C9TdlXaGPP8vFAfqQm0lR8z7PmrdNF+HG8vBgr9Ho38JP4PjAjtKj8ofqC
 O7zJd1szsTIvKt4rTozMf/6/diWkYM7hBsifP+57zgmFOgjG4jgmeR7/8G/400mwXjfyb5oG3au4h
 4ryjg9NqcwtfE3SJWi3iKp7vuvg1RpzzUiF8ZrIOTIBs8RJAdW8F2fKLg2Y+jMjGJHx3Q+j4B/0+1
 Js52vLroPFIGJpuxirLZuA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMneN-00084R-3C; Fri, 21 Jul 2023 06:46:27 -0400
Date: Fri, 21 Jul 2023 13:46:52 +0300
Message-Id: <83edl11qzn.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87fs5hemi1.fsf@HIDDEN> (message from Michael Albinus on Fri, 21
 Jul 2023 09:45:26 +0200)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Cc: Ihor Radchenko <yantar92@HIDDEN>,  dmitry@HIDDEN,
>   64735 <at> debbugs.gnu.org,  sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 09:45:26 +0200
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > Disabling file-handlers is inconceivable in Emacs.
> 
> Agreed.
> 
> However, the fattest regexps in file-name-handler-alist are those for
> tramp-archive-file-name-handler, tramp-completion-file-name-handler and
> tramp-file-name-handler. Somewhere else I have proposed to write a macro
> without-remote-files and a command inhibit-remote-files, which disable
> Tramp and remove its file name handlers from file-name-handler-alist.
> Either temporarily, or permanent.
> 
> Users can call the command, if they know for sure they don't use remote
> files ever. Authors could use the macro in case they know for sure they
> are working over local files only.
> 
> WDYT?

How is this different from binding file-name-handler-alist to nil?
Tramp is nowadays the main consumer of this feature, and AFAIU your
suggestion above boils down to disabling Tramp.  If so, what is left?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 11:34:02 +0000
Resent-Message-ID: <handler.64735.B64735.168993918523929 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168993918523929
          (code B ref 64735); Fri, 21 Jul 2023 11:34:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 11:33:05 +0000
Received: from localhost ([127.0.0.1]:60705 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMoNc-0006Ds-KX
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 07:33:04 -0400
Received: from mout.gmx.net ([212.227.17.20]:47715)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMoNY-0006DM-U1
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 07:33:03 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689939168; x=1690543968; i=michael.albinus@HIDDEN;
 bh=4AYvM/GxjOD43VQAQNyrJyHFc1yV12mOYsZPQDtWwxA=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=ai3V0ZrQIx0/SnVyFvU59KGnk4eUNSvAijrWzSHtxH6CxLkMpB854/rXytVqiVlfuhWXkJy
 kPy3cCC5MwtgS3TlUcnZCjue4JHvNpoOgC4KHbPU//GKEI14+M7p/Ph0P/NhCvdVDB42anu9H
 dnShr08TT7Tnj4YrtBGsXsbI5lIXeQ2Pz6tBxwX3zVXG/MJdRRwNe1sU/BDXLANN++90pSk0i
 zhCT8LUeudahD2vdzbU1dYp4GJdPMQVsGA+o6NTiojnDsKKHgDYa/QLhWeqxEBBhZiUlg+r2N
 AQ5zwEQI0UhPmV2y49srsqKu/mpKmYgZ3G4VtQVCxWpigfaXh3ig==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M5wPb-1qFwtP1GBv-007Rl4; Fri, 21
 Jul 2023 13:32:48 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83edl11qzn.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 13:46:52 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 13:32:46 +0200
Message-ID: <874jlxebz5.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:jujrNfCX3m1C2xpqCl2zIk1mHl8y0EIBWeUWbAbvgQZhtR5RRZh
 P/RDPfFxus6W2HDer1okrZLfHb0JvVJ2IaE9WLbAHb5rbqoUiRVgMQ1G9OeTuyHxeC4k4Ba
 yBiLJRqILSTL0/z0UuxwisdFZQpCT2Bs1GBoiJLxtBxnTijTjLOzl2jtGiorCuRtbIlr+dg
 ljtnuCmOn1AMUYQSQM7AA==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:GdV5zH+8PQo=;8nOyLUXpG/pkB5OvDgIGTFKi7lw
 A/PnKfomnUQoVeZAL8kXwjrgI1jqr0diErC9R3dmshIfuXq4g/StKewKuc0TZz1pvVByZtloD
 qwZ+njcweXF2DWI0AJNagZu26htz44t3/CGZ05meswMHR3K3JyhDEG5jOgF7A+xEIVVzEPAS5
 64828y3hEjROAh8T1gjdOSVdUKdcRERCt/jGTM2GFEOvi3rktRN98Y9cZTd1fC7qA1lbCYYsV
 YUWydDKNwqbK0AGlBd0GSDB8rQoVNkK5pe/Gv/gPjjiKy/vQNdRBAOlt2zjLR+wRmwxNXPbAH
 Vkr27MvPIchhULtooXKeoZS8/CL2+gMxFrYM9REUqyAhb5GPjUtYXm+SQM5/Hn3gQDBGsATW5
 S90ZYG0MZK1yz8pZ8FBbQULie2XDw5lYC06ph44sk0lY4NDwLpyzyOLyxSJdEdwtTK9+pEr8s
 w9Utj3FUNs/Aq64TP/A6xScJuofN6yiSig4mzhbqf+SvwMlHvDfTx2e4xPHbVkbLaVNxjVYCJ
 8CXykBa8bpyygAOzmYa3q9Ayiu5Y7LyjX9jkI0Xt3aTehhBhaZVxL98TRvOHqEszqWQTdc+8h
 E4fCfXX8x1p8nu+mFCD5DAfiAgRBbB+c61izShCYbw9KsMODsfaEts1KKu3wJ2XA78Bc/jjFg
 +rbziAlSf1zAV2g/JI4HOzYTxsfh4qWOnWCxveC407VXX9f1XJgzMTaceC9PdSA37Vxi7A67H
 LM6T9pZU32CgsV+GIIO4AF3WlCCFR6qXgsLkBL+eY320sW+MNVLm8S+nRPSfSZaFV3K/17GyG
 3WlamMPWRXuUyqv8MTFLX+e4p/QAhc5OLfnR6Xz2icPSMw9qeP3xho496FnRsK44ZKmmciByk
 4PkOKRYqgoge5Z+ZtzDF6UlAxfJNCHcm/KHZwhO7puwR1lYIh6A+jmiSIpsMU9/W8BWbJAD1P
 YpiPz3Zha2MeGDFK3zKwlxNnZsg=
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

Hi Eli,

>> However, the fattest regexps in file-name-handler-alist are those for
>> tramp-archive-file-name-handler, tramp-completion-file-name-handler and
>> tramp-file-name-handler. Somewhere else I have proposed to write a macr=
o
>> without-remote-files and a command inhibit-remote-files, which disable
>> Tramp and remove its file name handlers from file-name-handler-alist.
>> Either temporarily, or permanent.
>>
>> Users can call the command, if they know for sure they don't use remote
>> files ever. Authors could use the macro in case they know for sure they
>> are working over local files only.
>>
>> WDYT?
>
> How is this different from binding file-name-handler-alist to nil?
> Tramp is nowadays the main consumer of this feature, and AFAIU your
> suggestion above boils down to disabling Tramp.  If so, what is left?

jka-compr-handler, epa-file-handler and file-name-non-special are
left. All of them have their reason.

And there are packages out in the wild, which add other handlers. Like
jarchive--file-name-handler and sweeprolog-file-name-handler, I've
checked only (Non)GNU ELPA. All of them would suffer from the
bind-file-name-handler-alist-to-nil trick. There's a reason we haven't
documented it in the manuals.

And this is just the case to handle it in Lisp code, with
without-remote-files. According to the last Emacs Survey, more than 50%
of Emacs users don't use Tramp, never ever. But they must live with the
useless checks in file-name-handler-alist for Tramp. All of them would
profit, if they add (inhibit-remote-files) in their .emacs file.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 11:52:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994028825855 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994028825855
          (code B ref 64735); Fri, 21 Jul 2023 11:52:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 11:51:28 +0000
Received: from localhost ([127.0.0.1]:60717 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMofP-0006iv-W0
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 07:51:28 -0400
Received: from mout01.posteo.de ([185.67.36.65]:54241)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMofL-0006if-4Z
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 07:51:26 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 228CF240027
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 13:51:16 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689940277; bh=QC/THAI+BgEmnqDXoXb5znOC5j0bqRw3NkxABy+pxh8=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=EE5HJTVIOknipwh0UI7mtTKZxZkORNVUkS/VftoMudXbXvtAO/QhLjBH16ejh3a5r
 jxgK13sOeJ06fJ3wic+pxUtCYRVQkwOvdz2wHgZTKncn2h2vwRDh0nOp+6KfVP7KRI
 C50anp3HwWo+OnLpxHOgHhaSf0TMBdkJpZ0TSvcUhLCJ7uZ+Z+ifq6AInpCTiNhDQh
 i3BGgbxjZU4V2xjc83wBthhv8CqommcApqz07IArpkz2MP0DL9ariBN0KuYGfPpnh3
 AnJsl2q3TYcEAnj0ncbseLIWdcGULA8+wBCGL44t26Cg4xK+NAAsPvnEvGbCWV9XFu
 bJeqUyWATykqg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6nvR6WYPz6tsg;
 Fri, 21 Jul 2023 13:51:15 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <874jlxebz5.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 11:51:27 +0000
Message-ID: <87lef9mqio.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

> And this is just the case to handle it in Lisp code, with
> without-remote-files. According to the last Emacs Survey, more than 50%
> of Emacs users don't use Tramp, never ever. But they must live with the
> useless checks in file-name-handler-alist for Tramp. All of them would
> profit, if they add (inhibit-remote-files) in their .emacs file.

May tramp only set file-name-handler-alist when a tramp command is
actually invoked?

Or, alternatively, may we fence the regexp matches in
`file-name-handler-alist' behind boolean switches?
I examined what the actual handlers do, and I can see
`jka-compr-inhibit', `epa-inhibit', `tramp-archive-enabled',
and `tramp-mode' are used to force-execute the original handler. If we
could make Emacs perform these checks earlier, the whole expensive
regexp matching phase could be bypassed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:02:02 +0000
Resent-Message-ID: <handler.64735.B64735.168994090426909 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994090426909
          (code B ref 64735); Fri, 21 Jul 2023 12:02:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:01:44 +0000
Received: from localhost ([127.0.0.1]:60721 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMopM-0006zx-7M
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:01:44 -0400
Received: from mout.gmx.net ([212.227.17.22]:42975)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMopH-0006z1-38
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:01:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689940884; x=1690545684; i=michael.albinus@HIDDEN;
 bh=QcLiGo/W3UV7LcMJBeDurtVoqkz/s2rAoMIjLHKgiA4=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=q07kKQXwbTdynr2O+e45L3D/JdBh3VFHx51UYtkudsGqntl9TIPSv2ZLw63/rf0h4d4ldn6
 +AY6ms8f5S5f4zd0QAhZ7R4g4QxRJxXuXZS3/06KTdrU5l/sJCKc7gKTjlOSFKdVAwXUQbY3U
 6iLchaIHBUO/LstYtm7ow/iLOCmNbKIV8FyVtS9Qq85F2c2Xvsa7NCAucd0MeThQeeZSHpJrl
 Oe32Qr+wCsRC2c4dQ0k8tNpMawXBB11SKuO+nDqCYoE9MWvWCp46kUXIZggrr3BA7mjf4hhue
 Oa0rfNJmAkJHoNBkHXuV/oDgtp6HgAbg5yB7JPvQBAMcSqDl/PoA==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MfYLQ-1ph9LQ0VCq-00fxI9; Fri, 21
 Jul 2023 14:01:24 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87lef9mqio.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 11:51:27 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
Date: Fri, 21 Jul 2023 14:01:23 +0200
Message-ID: <87edl1scbw.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:f1YXj636mHpw+6Y3TVPGruGdz/kNXGxBZDGfKcTcGg7LHenqwbr
 uTHXPkMoPumPqg/pPe8NbZE1a/R4ucxYMpYdgOq36vLn+WEJED3LiYE46DOHtBfFKIbwFl+
 AdRij82V7EMAb3PtykF6xKElGP3Asqhkde4iNgkufrT5c2Q3aRKqsjxT6CSHSXhtIAXlAsL
 nrCdVP4hDNo4MwezePtiA==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:zFm1Vi+vbhc=;V1XnBEtLh6oAsdcRPABYMcfUwTQ
 PONtNRjxhtTYLecq2k+Z/wmzkD3Slr9YM1ZtK6UESYP/rZEMlc/I+/9R6/AS+N1msOD/YWojk
 0qEJmTa/usUDsEIbmml3toEijdELvZJZuXm9d59FS7M7K0x08cZ66sQTZUKPqNCQ5jDV9jrUd
 33PFmoc9rIeFiEV8G96CQOBBpJHAb5ell3RJN6XchOqG4LYw3tOj1y6zMpDjOSUCoJLvX2UHs
 kY95m64Zlq10/J80m+gkRH37p+Tl3piMd6HI3xQiG3FHja1YRyIbXbzauXx4o9ovYuoGQCy3Z
 apie3ZIDYe9qBF9NBK/U4es8dgoE3MGfdaKPF4ff/CK2mgAFj6W9N4WRUIK+82DGFcmji7d1h
 MO/6RNJS4tRl+/Ufm3dvl3UMl/2nUZPWRbBL5sFodTQUwtO54kO1rGjkCPAx6m3esarKTkj3h
 +0GtCPOnRhRin9YoYB5MDw0ZtGy04o8sTX3SNhkNZ4RB+mx/D1+Ia07KPP6g9vAOUAch5gsuM
 s+ol0tiXReR+34ud2CYoKgn2p/ffeucP7VEKJ18ZElLvBV9y29rQntoi6wIk3vDo4ZFXRlok2
 Ewpm3LTZjxYjEDpEltYA+7OOJtAUv5vRKNeAaTi7uCPLP0jagCyjxBe5/eiAG9oBaUXuguc9N
 0Qi2iWhDprx7gUqtXVQ14Rgn01k8X+De6wkHJo6+LQmFQElP2w/jnBT4aFvrOQm/fOUM6Yg/N
 frTtLTs2Tj4R+zcKmQ+afimyj/tYJYqvZHEDLnfvXQywaz6P3Ay5lgIS/YkBi0S+3vLY8gXia
 RqRPhNQnNEhm3YmnE0dawzRNJYohp6OsOCA1O85DYrSHDKbuVmkviY+f2FakcjzKvZ4gMNkte
 PPoAr28oqoKC0YG9ji3FZkl0iHNqwr9+RA+2VKwneLB5AeY6En7IdOmobnRnvOIYQVlHw/0KN
 rYa/gg==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

> May tramp only set file-name-handler-alist when a tramp command is
> actually invoked?

It's the other direction: Tramp is only invoked after a check in
file-name-handler-alist.

> Or, alternatively, may we fence the regexp matches in
> `file-name-handler-alist' behind boolean switches?
> I examined what the actual handlers do, and I can see
> `jka-compr-inhibit', `epa-inhibit', `tramp-archive-enabled',
> and `tramp-mode' are used to force-execute the original handler. If we
> could make Emacs perform these checks earlier, the whole expensive
> regexp matching phase could be bypassed.

Hmm, this would mean to extend the file-name-handler-alist spec. Instead
of a regexp to check, we would need to allow a function call or
alike. Don't know whether this pays for optimization.

And there is also the case, that due to inhibit-file-name-handlers and
inhibit-file-name-operation we can allow a remote file name operation
for a given function, and disable it for another function. Tramp uses
this mechanism. The general flag tramp-mode is not sufficient for this
scenario.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:21:02 +0000
Resent-Message-ID: <handler.64735.B64735.168994201829136 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994201829136
          (code B ref 64735); Fri, 21 Jul 2023 12:21:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:20:18 +0000
Received: from localhost ([127.0.0.1]:60779 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMp7K-0007Zs-0l
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:20:18 -0400
Received: from mout02.posteo.de ([185.67.36.66]:46591)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMp7I-0007ZZ-5b
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:20:17 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 6FB43240104
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 14:20:10 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689942010; bh=bj6Pbz+tDJVbigiJXKSn27FKQVj2T2ypIk4qjwHBemg=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=Tl118zM9dAHlbHJ0BbDzoHuNpHaAkkr0QNcRrGsOt7cnArfror8mydG6wGgmK1801
 QHJLOSE76EnHiTjjN1uqEolGAzpH4gbYkTR3lme1tRSLIj3YiHXfvfHImsPEdN5UU5
 YFpVmq97Ul+nkydjL4J13ePpasH8ao9GL/RJuE2m1qChkRPQ8dMIC7R+kZfJr8Omjz
 b9ArkNcuKjijNAuasT6tmTqy4SqfmBHJ3pU+9a8AQQqrfhz7vDW6jCYq9FYBcmuEzu
 p5IlFuhyQkUnsgPThP9hTdCxqcDVY6gfLX3/u0FozTCPGtq+k6UxOYQI51h1bZN4+A
 fGsaAWJskFlwQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6pXn32Qlz9rxM;
 Fri, 21 Jul 2023 14:20:09 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87edl1scbw.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 12:20:21 +0000
Message-ID: <87fs5hmp6i.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> Or, alternatively, may we fence the regexp matches in
>> `file-name-handler-alist' behind boolean switches?
>> I examined what the actual handlers do, and I can see
>> `jka-compr-inhibit', `epa-inhibit', `tramp-archive-enabled',
>> and `tramp-mode' are used to force-execute the original handler. If we
>> could make Emacs perform these checks earlier, the whole expensive
>> regexp matching phase could be bypassed.
>
> Hmm, this would mean to extend the file-name-handler-alist spec. Instead
> of a regexp to check, we would need to allow a function call or
> alike. Don't know whether this pays for optimization.

The question is: what is more costly
(a) matching complex regexp && call function or
(b) call function (lambda (fn) (when (and foo (match-string- ... fn)) ...))

> And there is also the case, that due to inhibit-file-name-handlers and
> inhibit-file-name-operation we can allow a remote file name operation
> for a given function, and disable it for another function. Tramp uses
> this mechanism. The general flag tramp-mode is not sufficient for this
> scenario.

I am not sure if I understand completely, but it does not appear that
this is used often during ordinary file operations that do not involve
tramp.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:26:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994232629681 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994232629681
          (code B ref 64735); Fri, 21 Jul 2023 12:26:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:25:26 +0000
Received: from localhost ([127.0.0.1]:60791 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpCI-0007if-3n
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:25:26 -0400
Received: from mout01.posteo.de ([185.67.36.65]:42283)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMpCF-0007iK-Ue
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:25:24 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 23A81240027
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 14:25:17 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689942318; bh=trV0crLwf1UqcRt+UAMceMZcJX4t5oopw9xIAI/9cww=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=fQbbywY9hNHWg4mLFUwEfFPdTtUIgYcv7faneB9q0b8HKVoZYiiA1VyKaIwrZX95u
 UHWpC/Hw8LbLFDdL9WhB+zfW1VjRDI5Nw6zmuh/WswSH5viG0S/9FuVIPKtT9VpL27
 UFdrxelBVRBYvHekACdanV1QrHgTr0SLz/jWxbh3L+lzBAfy1JS7wkLuBwUiDpq/v2
 x83WmLLEReXMbVgG7KPT4vATJYrSjiC0HQHwHsdU4Y5oS7izLmx9EVYwT9Q5etNnsv
 mKZRHLtHPhjspjumi9+F4nqXjpMSR5nVInTpOYl4tnct7BvEFfdUVvG8+cDbBjuIxS
 w5X7rDR3/m7sQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6pfj1hN8z9rxB;
 Fri, 21 Jul 2023 14:25:16 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87fs5hmp6i.fsf@localhost>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost>
Date: Fri, 21 Jul 2023 12:25:29 +0000
Message-ID: <87cz0lmoxy.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Ihor Radchenko <yantar92@HIDDEN> writes:

>> Hmm, this would mean to extend the file-name-handler-alist spec. Instead
>> of a regexp to check, we would need to allow a function call or
>> alike. Don't know whether this pays for optimization.
>
> The question is: what is more costly
> (a) matching complex regexp && call function or
> (b) call function (lambda (fn) (when (and foo (match-string- ... fn)) ...))

(benchmark-run-compiled 10000000 (string-match-p (caar file-name-handler-alist) "/path/to/very/deep/file"))
;; => (1.495432981 0 0.0)
(benchmark-run-compiled 10000000 (funcall (lambda (fn) (and nil (string-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
;; => (0.42053276500000003 0 0.0)

Looks like even funcall overheads are not as bad as invoking regexp search.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:28:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994245229898 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994245229898
          (code B ref 64735); Fri, 21 Jul 2023 12:28:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:27:32 +0000
Received: from localhost ([127.0.0.1]:60800 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpEJ-0007mA-UI
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:27:32 -0400
Received: from mout.gmx.net ([212.227.17.21]:41949)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMpEH-0007lv-F1
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:27:30 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689942437; x=1690547237; i=michael.albinus@HIDDEN;
 bh=PdWE4sqdgCU6TXGxrtApxXqAb+IvIMteqiGJLLSrV5U=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=bh7srkFGGXpPFLR0dnlpbW8zFjzSxTffEeSNZVsmRcqf/bll1NsPOKHAbuWEYc3P3LGZyq0
 QH/ZXWYrtj8Mjfe9vQj/DBgPh9GWV+uFGYoA2L12IlvWRVSms1X/jlVO9Ab5jyChh3DlMx2VP
 x/QhQoSaTG7hcNDBIhJ8X9ib0Nc7E6mgG+j4gev3dcZxzoolV/O/6kAkmbR/QhdMC7k8/PD39
 RxoTuKRasB+MLody17K3SL6rBKenNZpM+5H2D/yzQopuLV8++mboy5YbOYUcLMX+jw7GN9pOp
 w1IvVRnw74lLx/aeVLTY57dAGFmtKMan3xCwRuF0JJpuHlTgTc0g==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MlNtF-1pe6H21y3B-00lkG7; Fri, 21
 Jul 2023 14:27:17 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87fs5hmp6i.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 12:20:21 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
Date: Fri, 21 Jul 2023 14:27:16 +0200
Message-ID: <87zg3pcuvv.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:MbCHBgWcexySIXXbWFTXnfLWfJ/TBafSqIfayEd4z2Glr4vihPt
 eGo8f7MOKhl0Kqc2ppJ5wuGsKM7sKWXeoZXsfhBVrZ3M1DNH//F2hMjbujaFOwSkRumrVh2
 TTlz8fKGBeVJNsk9ab5l01ZW3cbk3BzTy00r8RVblwnNyXp2aYoWR7cjP3r82fI8lyh3rDJ
 7MfOLlHL0VqQuQXqcB4Fg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:0wKlWK00M0k=;y6VHWtQ5zMY1BVZe2ieGbXnbkq+
 ytaptkaSvkt5SKZrQ57OCTfTsX8AL7DyEHE4/IKk4KPoCidsSQTnBFSJWjuF3DnoZZUM7JbSt
 njCOps7B8frYV9hrfUgmH+UIWOAYqm1M4NVwtdpFyXGs58RrlI4TtVLzfsMzzwxwVFN920Suc
 dTj/FpUhX0CY+LX1MN459hFK2jdZSB3giKzUdZKo04ktlntK4gMxABObSb88+iCEJZKM72iO0
 2p1Ki9IXOJsaixXUFng0ec0rhN15v6mNobJo6QR0+PRYioJZkSUsbzy16fF36fyyXcuiVTohu
 9aSaCWeh2k50NDrCu9wPOzSFdKz6MXuuz5H95VsYLtUovKQOh334V1EX5Ef4/f+tFbyafIlgs
 KDkRACG+/Ed6aOgqe+kegj8UCQajoe60TKrsPrmqxartg4KE/eZQnhoIErFUixbPPqNVT5ptv
 gMwQT+0E9wV/OMDGh2nN6iP/7VJIA2V03Pz/xQdoEMdts3Ebls2pvpKBjOHsf5m2VF6CiTmor
 mZNt1tgg/Do3Ls3hItRwORVJj3Aqs0B+iUcwdeRJwUP1tP6rLPLmLoAZ7T3vzNAyub0AEbc/e
 tE+gvOw7t5+XrO3/f2qPCEdHo52cSlmQU+b+Eb1y7WSRooHq5xqRCw/Ei6yRvpsvxtoinLwPZ
 hTxbIlvlQxEI0+5RVwBVkbJEV3NVLxlue78ket2i2yMRjnkzTpJro1g3zuoooejfSPSe7cDnB
 0XBctmSnPVoLTAYEqs4jHxk5n3kGR3Kv9ZDxOpWU+qjBLCqf9l9rRIc1mOkzdmU8V6Vu46gva
 3n7PnlqB7zTwoZcGP4xJH0vWy9+1rzK0WdIVe+fQm4Zu6jXKSlJ81kQ4iUFByrqNSb7z+iI6y
 GBQnY3s5WyCeMiOSWK4vkkIL2uE4PZmtQZ+rDSpI70yFR2s4LXeEjw1qZHMnG4Sz4749twFY7
 ithQ1aEGmbTKGbh5f/x+/nKpEeI=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

>> And there is also the case, that due to inhibit-file-name-handlers and
>> inhibit-file-name-operation we can allow a remote file name operation
>> for a given function, and disable it for another function. Tramp uses
>> this mechanism. The general flag tramp-mode is not sufficient for this
>> scenario.
>
> I am not sure if I understand completely, but it does not appear that
> this is used often during ordinary file operations that do not involve
> tramp.

Don't know, but it is a documented feature for many decades. We
shouldn't destroy it intentionally. If we have an alternative, as I have
proposed with without-remote-file-names. What's wrong with this? You
could use it everywhere, where you let-bind file-name-handler-alist to
nil these days.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:31:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994263930337 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994263930337
          (code B ref 64735); Fri, 21 Jul 2023 12:31:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:30:39 +0000
Received: from localhost ([127.0.0.1]:60817 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpHL-0007tF-30
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:30:39 -0400
Received: from mout01.posteo.de ([185.67.36.65]:59723)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMpHI-0007t1-DU
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:30:37 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 814BC240029
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 14:30:30 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689942630; bh=3REpDgzWakRA6nIlYtKgvy2DHPEtlvUW/pYE1dukDIQ=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=RiEyIuy0/kDr4SDvJISs3xurw1bdyspFuq9lto7FTVU/H8pcio208p+4FkaOth+D6
 k7s+XRutRy32BvuxizmO/66KmsCUnKnPAUYW9Sc7eQ4tbN/ljf79wa3ykl7e0lvru8
 AdaT/pZAd/aL7gUnwJ+JyaF4rHGaTDcYm076JuQdWnCymXPxsQfmXoPg3ZNjjTpElA
 Dk0G/y0qkG0FczrEcMFwUQjoBlEnAuLqr9BMqUfJIwqZmsKYzsQGqMFOxD/3UwAaR+
 T+RIgiOAkafX8pd2wT/0EJ0HKCF2fIJuq4l0vQ9e5KGA0ZDBrgKZGsnHu0WqZqhzVa
 lww5g3hJMuHYg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6pmj5gjQz9rxW;
 Fri, 21 Jul 2023 14:30:29 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87zg3pcuvv.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87zg3pcuvv.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 12:30:41 +0000
Message-ID: <87a5vpmopa.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> I am not sure if I understand completely, but it does not appear that
>> this is used often during ordinary file operations that do not involve
>> tramp.
>
> Don't know, but it is a documented feature for many decades. We
> shouldn't destroy it intentionally. If we have an alternative, as I have
> proposed with without-remote-file-names. What's wrong with this? You
> could use it everywhere, where you let-bind file-name-handler-alist to
> nil these days.

The idea was to make things work faster without modifying third-party
code.

And what do you mean by destroy?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:39:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994309731003 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994309731003
          (code B ref 64735); Fri, 21 Jul 2023 12:39:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:38:17 +0000
Received: from localhost ([127.0.0.1]:60822 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpOg-00083v-2L
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:17 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:58505)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMpOa-00083c-AR
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:12 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id 8019D32001BB;
 Fri, 21 Jul 2023 08:38:02 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Fri, 21 Jul 2023 08:38:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689943082; x=1690029482; bh=RsEw8kz80yGAbUVUnq8QI+goFsNzdHjT5B8
 YJkWevZE=; b=oTQR3kiI2Vbd6ncYOpu3qjR/3+jiUDGpY3mQFEN41AEuxCZmKQo
 6coAjkxkc+J5peKuWlIaxBj0WEn9vPlZcnFxl81CQZ4Ik+l5V2i7MiC0yA7FWFbJ
 wxZOa0Xiig/ydmO+uX1ais0YJWMuzFzNLhf2L8hNvx5F5RjZcU/0W9fjGIqkdp1v
 FizpKqIFZaLviaq6S2S0ebRMowabGC/OeuRLc+JlaISnlVF10hL69lTLNB7hlmRF
 xuQaCVdh2Ht/fAUDzUIZ0AhfpR+bwqUBgZuU6ACZn5xZlVcSEzMl4oG1jEv/3Z9F
 36GZnonWogiM/Cxkit5ahEosrAk1mA6Nn+g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689943082; x=1690029482; bh=RsEw8kz80yGAbUVUnq8QI+goFsNzdHjT5B8
 YJkWevZE=; b=Xs8eaVGFCgU5yocLqYGSxKm9La5FFnpZgj2fJRiXFaXnjkGKbu6
 fpJu8tRfaBQ8WhOtSlUgsJjEZKbj6Rp9fxAj4taO/lrWEwGLn5ktSeyCSBI/YHPy
 uCMoI2UPByWiIHjeWAdBbS9H6Gay4gEL0odHzWSK5ZyBvBc5G6U1fSDko/TzZQf0
 seuGMgeNgE2HnZfCytisBikCYf8RXhBcMgx3oEx9UBKGSCZGFs8wFwx1nWP7EV2f
 29qGwVGC0EYcZ28PnBC5UXxCuAND2huL2gGQzqYTC+oY2pgV0Slz5/OygoHwKfWk
 Jp6stgxTLQlTbwTTntGsYqmnbkhZbQzR9ow==
X-ME-Sender: <xms:KXy6ZNl67cGjVYnKSfk-tWa80-u0Wgi9H6ut5GesMi234dSAnqMprQ>
 <xme:KXy6ZI1gfqQ8m473Gmlr_qL8k6x02XI07vKbetTn1RW9GxQXFbW-7XjwZFaNGuSvN
 x9WR9VSej1z3TplNko>
X-ME-Received: <xmr:KXy6ZDoi6tOfT6UfaUKsUxJa7HCNfS-1ns5-f17CX2Hzjkzbj4ZvfslMcWTgGuM>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgdehudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:KXy6ZNkazLM-cpyDV_737Bi7DxG64uJKcOrqbXJfgrxadnyxtO8_5w>
 <xmx:KXy6ZL0oVhgm8l5kbrV1fK9LCJ2G3XelWx2gG4RFQEOR4DuKVchEbA>
 <xmx:KXy6ZMvCa7pv2yZHN9vNJUcMXtmmMyxVaqIo1Ool9bjXnKjP25xCLQ>
 <xmx:Kny6ZJCwhKlIuwpg3fk7OKRGYkQt7SvEFJeb7HGx5dPe6_l4i5R24g>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 08:37:59 -0400 (EDT)
Message-ID: <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
Date: Fri, 21 Jul 2023 15:37:56 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87ttty8l73.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 20/07/2023 21:57, Ihor Radchenko wrote:
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
>>> Disabling `file-name-handler-alist' cuts the time more than 2x.
>>
>> Disabling file-handlers is inconceivable in Emacs.
> 
> Indeed. But we are talking about Emacs find vs. GNU find here.
> In the scenarios where GNU find can be used, it is also safe to disable
> file handlers, AFAIU.

GNU find can be used on a remote machine. In all the same cases as when 
it can be used on the local one.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:39:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994312031042 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>, Michael Albinus <michael.albinus@HIDDEN>
Cc: sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994312031042
          (code B ref 64735); Fri, 21 Jul 2023 12:39:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:38:40 +0000
Received: from localhost ([127.0.0.1]:60826 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpP5-00084c-MM
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:39 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:44387)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMpP4-00084M-4j
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:38 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.west.internal (Postfix) with ESMTP id 3C2A23200942;
 Fri, 21 Jul 2023 08:38:32 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Fri, 21 Jul 2023 08:38:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689943111; x=1690029511; bh=413mKa+MZEDBWIGIfkiXPnltNP9SNs7dKSL
 kOe/d5GU=; b=IUQR+J9qQ6YZyh9uARi/dIRO6ECVrVL7aw+0nXlOo7r3uYJ294n
 EsShLuntm3t+vU0S9Lv6CiJLyqidJbKPNlGXRYbbnyIusuoXG483zz1DZ4SV2vJI
 a/glqphBvg+8df0BXCAmGc/gMQxIalg3n4p0ItcuCUPhjN3sYZxVIKfTdWlG7Gv8
 DqChX8wAiSzKeT7irQFwoO7+fy22GasrPRDzB9d8gnSn3tmfsRD4UAYyZIkRfaus
 aveQglvbD11nCQ2dHfcHQIRvmtEBn4vTzsRxUC8DLPbTjpm7dNAOXW+rKOpCno2P
 fKy8WIR7oiCeujSg0X0gEMd/uAPVliYYeQg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689943111; x=1690029511; bh=413mKa+MZEDBWIGIfkiXPnltNP9SNs7dKSL
 kOe/d5GU=; b=Vdsw+Pcl4kc9TgFdC8gaxFEUS/a/QBum3XtL2/XhlqxHLtmtG5I
 7ilL4tmNnvIRMYjO8DK2vG+HWw6hrFtuCEGeq1toyEGYBergkP3kne8whATF/yDm
 5+uoWmGEfKCuJS5BXKwNk/mc2NI/h6sDgKRCyKkdIBhmj5HiEhyaNbW3YaO98G/a
 aMHZynrV+pDKp4mh6XcnErrZBpsoqHYkvEXlJb5c4C0mb3JKh/5Z1E23dGz0GHJl
 vuBac+nheGE7ZSkapyodLtF7UU5A+jwjBvTBa2v8nxJ0RENkoIz82H+olVDAficw
 6sTpzLObMEEczt21ySQFiW1ZhA0fyDS814Q==
X-ME-Sender: <xms:R3y6ZNFlSvLnoRYHTbTZ0VR7pGDlEazMtJz9Hd4x1CoIW-C9qmzmlQ>
 <xme:R3y6ZCU8yWN_E95BB6UZ5AeSiwGwucLEMBjdGauuog75mThEBggFqZv6DSEy0hT3U
 ZRCzFtNGS8r2vxn-FI>
X-ME-Received: <xmr:R3y6ZPIxNMjDiuziaQbv3NSLTBNG1fZL4-H-fSzv19Px0jm4xA2qV_qSj_80ROw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgdehudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:R3y6ZDEcFE8DK0vRGB07Km0ORhdxc34DNOHilEsO4wTSnXB-5rtLfw>
 <xmx:R3y6ZDXLU3QOJqfzmuG80eb4rlHuJEvaMz6pCcXNvkG5QXcJn0Aesw>
 <xmx:R3y6ZOO_AZwMvMprkKDGWfk2Eis1Y3YzevuEa3StKWwWR0yEfCTZ6g>
 <xmx:R3y6ZOQe39DpOIA70qvuj1WgKFwr7X87IqepwCUxXI1ZA32297hH-Q>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 08:38:30 -0400 (EDT)
Message-ID: <a4e7c1b6-483f-7cde-96df-03012c241fb2@HIDDEN>
Date: Fri, 21 Jul 2023 15:38:29 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83edl11qzn.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 13:46, Eli Zaretskii wrote:
> How is this different from binding file-name-handler-alist to nil?
> Tramp is nowadays the main consumer of this feature, and AFAIU your
> suggestion above boils down to disabling Tramp.  If so, what is left?

I don't understand this either.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:39:02 +0000
Resent-Message-ID: <handler.64735.B64735.168994312931072 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994312931072
          (code B ref 64735); Fri, 21 Jul 2023 12:39:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:38:49 +0000
Received: from localhost ([127.0.0.1]:60830 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpPE-000855-Uf
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:49 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47620)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMpPA-00084b-IB
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:38:46 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpP3-0003He-7M; Fri, 21 Jul 2023 08:38:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=hyEisAq6Nd4xnGbML/VwTNxtJuSZgWcVONyZWancEI0=; b=SwtOBz6qqWfE
 jG6koNJ6kI+gASSUBQnFyIsow9Ij5+872PstKmeYt/wi9ACp3/NHtG+RDH3aQeOfRCtjSdcXbF8d8
 WtFZ8AkGUajuAGrqK8xdfI0jmGMyJFgd6qvPvMeGRUTn+xT3NX8b21U4up8Y6Yf3qPxGSMnkhA5i0
 yGyRL4LYGH9dx4/dn5JraIwpH9CjI4T+sScGOjzM81rCu9ZRwt3CQSr5i06220MFm+eYcQC8qmLdB
 lauv3L9/IL6ZiX57jjSVjx7iyK3LXr0HxlLXbG+xEX4/S9kgZMOd3dVXJzGCE8EUAr55Ao0Y9rJtP
 Xta30zchkiDx1XDChAclVg==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpP2-0005Hq-NH; Fri, 21 Jul 2023 08:38:37 -0400
Date: Fri, 21 Jul 2023 15:39:11 +0300
Message-Id: <83y1j9zbf4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <874jlxebz5.fsf@HIDDEN> (message from Michael Albinus on Fri, 21
 Jul 2023 13:32:46 +0200)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Cc: yantar92@HIDDEN,  dmitry@HIDDEN,  64735 <at> debbugs.gnu.org,
>   sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 13:32:46 +0200
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> Users can call the command, if they know for sure they don't use remote
> >> files ever. Authors could use the macro in case they know for sure they
> >> are working over local files only.
> >>
> >> WDYT?
> >
> > How is this different from binding file-name-handler-alist to nil?
> > Tramp is nowadays the main consumer of this feature, and AFAIU your
> > suggestion above boils down to disabling Tramp.  If so, what is left?
> 
> jka-compr-handler, epa-file-handler and file-name-non-special are
> left. All of them have their reason.

I know, but when I wrote that disabling file-handlers is
inconceivable, I meant remote files, not those other users of this
facility.

Let me rephrase: running Emacs commands with disabled support for
remote files is inconceivable.

IMO, if tests against file-name-handler-alist are a significant
performance problem, we should look for ways of solving it without
disabling remote files.

In general, disabling general-purpose Emacs features because they
cause slow-down should be the last resort, after we tried and failed
to use smarter solutions.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:47:01 +0000
Resent-Message-ID: <handler.64735.B64735.168994357731932 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168994357731932
          (code B ref 64735); Fri, 21 Jul 2023 12:47:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:46:17 +0000
Received: from localhost ([127.0.0.1]:60851 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpWT-0008Iy-Bd
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:46:17 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:35306)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMpWR-0008Io-OP
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:46:16 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpWI-0005pr-KM; Fri, 21 Jul 2023 08:46:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=pA2D+kQCW2pjhu8iqOuIoNirmB1G0z71J5unhP3zcBw=; b=rUVHr4cIgeSQ
 VDx1gBiDvX5dR8hhZQwW8gVdrtAhDpNzIcHpMM1u9gpLUHPFo+tagGM0/8A/SicaO5AS7hvgVmEuM
 45jfL6u7ezqo68KPFZS7qK3yW81DspJvXE/XDVCe9YYUaCEqd+WFWQZmKdq3pjns4qgAWsz4LMmhT
 Qet7uLX+ZE0Fjm69+7mffRAO8cKNx0JbuV5bV0k10vyc19Bl3hIENiQaVhFSqr13wNLDxom8mGGO2
 QtZpvzfddUUF/qmHYuuGp4ywR9dRuywmdMZDHyjl3xxyEKVBOHQfrLCbNUM1ew17y2UYaYtaSPH6K
 vfMHiJObUosgRxSLXxBjJw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpW1-0006Ea-R3; Fri, 21 Jul 2023 08:46:05 -0400
Date: Fri, 21 Jul 2023 15:46:26 +0300
Message-Id: <83v8edzb31.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87cz0lmoxy.fsf@localhost> (message from Ihor Radchenko on Fri,
 21 Jul 2023 12:25:29 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org,
>  sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 12:25:29 +0000
> 
> Ihor Radchenko <yantar92@HIDDEN> writes:
> 
> > The question is: what is more costly
> > (a) matching complex regexp && call function or
> > (b) call function (lambda (fn) (when (and foo (match-string- ... fn)) ...))
> 
> (benchmark-run-compiled 10000000 (string-match-p (caar file-name-handler-alist) "/path/to/very/deep/file"))
> ;; => (1.495432981 0 0.0)
> (benchmark-run-compiled 10000000 (funcall (lambda (fn) (and nil (string-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
> ;; => (0.42053276500000003 0 0.0)
> 
> Looks like even funcall overheads are not as bad as invoking regexp search.

But "nil" is not a faithful emulation of the real test which will have
to be put there, is it?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:57:01 +0000
Resent-Message-ID: <handler.64735.B64735.1689944191930 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.1689944191930
          (code B ref 64735); Fri, 21 Jul 2023 12:57:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:56:31 +0000
Received: from localhost ([127.0.0.1]:60870 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpgG-0000Ee-Vh
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:56:31 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47372)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMpgC-0000E4-VB
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:56:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpWI-0005pr-KM; Fri, 21 Jul 2023 08:46:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=pA2D+kQCW2pjhu8iqOuIoNirmB1G0z71J5unhP3zcBw=; b=rUVHr4cIgeSQ
 VDx1gBiDvX5dR8hhZQwW8gVdrtAhDpNzIcHpMM1u9gpLUHPFo+tagGM0/8A/SicaO5AS7hvgVmEuM
 45jfL6u7ezqo68KPFZS7qK3yW81DspJvXE/XDVCe9YYUaCEqd+WFWQZmKdq3pjns4qgAWsz4LMmhT
 Qet7uLX+ZE0Fjm69+7mffRAO8cKNx0JbuV5bV0k10vyc19Bl3hIENiQaVhFSqr13wNLDxom8mGGO2
 QtZpvzfddUUF/qmHYuuGp4ywR9dRuywmdMZDHyjl3xxyEKVBOHQfrLCbNUM1ew17y2UYaYtaSPH6K
 vfMHiJObUosgRxSLXxBjJw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMpW1-0006Ea-R3; Fri, 21 Jul 2023 08:46:05 -0400
Date: Fri, 21 Jul 2023 15:46:26 +0300
Message-Id: <83v8edzb31.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87cz0lmoxy.fsf@localhost> (message from Ihor Radchenko on Fri,
 21 Jul 2023 12:25:29 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org,
>  sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 12:25:29 +0000
> 
> Ihor Radchenko <yantar92@HIDDEN> writes:
> 
> > The question is: what is more costly
> > (a) matching complex regexp && call function or
> > (b) call function (lambda (fn) (when (and foo (match-string- ... fn)) ...))
> 
> (benchmark-run-compiled 10000000 (string-match-p (caar file-name-handler-alist) "/path/to/very/deep/file"))
> ;; => (1.495432981 0 0.0)
> (benchmark-run-compiled 10000000 (funcall (lambda (fn) (and nil (string-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
> ;; => (0.42053276500000003 0 0.0)
> 
> Looks like even funcall overheads are not as bad as invoking regexp search.

But "nil" is not a faithful emulation of the real test which will have
to be put there, is it?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 12:59:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899443221282 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899443221282
          (code B ref 64735); Fri, 21 Jul 2023 12:59:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 12:58:42 +0000
Received: from localhost ([127.0.0.1]:60876 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpiT-0000Kc-OX
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:58:42 -0400
Received: from mout01.posteo.de ([185.67.36.65]:44675)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMpiR-0000KF-Dz
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 08:58:40 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id D7F6824002A
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 14:58:33 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689944313; bh=EKe2XiBJlS9VdwpiD7r5P2c95uMxpRUKrmKcF23M0KM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=CeZPDWXFBBwJdA9/YvL2lLP7g3MP6xl6unkTDdPZf5RlyTq0RPNl/TwbQ1Za48RAM
 KydziKyIrDxCBwSJoZIGwqQW2D/NSssWZnG2EHZzwmSVvwZNm+TOxSoZITkoCht6QQ
 AMjuc4h6dTMo/JHJEEPyuPKqqKlljoe19VIvDpPHttoVeoN7OzZzcQg1RpH6iG95IQ
 P7DKKacS0pisS0KGXJgALzHM2g1p7fpvJcagmVF4SbzF941zuESCTXCACbuAmmg7W7
 0RlkGP8HjkiyhdnOZtJ7aSDX1Q6dsENunAQZlbuc0TtbEZd3mzP9NH5hbwL6kLh+5K
 cIEpHzczJo27Q==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6qP51dtHz6tw3;
 Fri, 21 Jul 2023 14:58:32 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
Date: Fri, 21 Jul 2023 12:58:45 +0000
Message-ID: <87y1j9l8u2.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

>>> Disabling file-handlers is inconceivable in Emacs.
>> 
>> Indeed. But we are talking about Emacs find vs. GNU find here.
>> In the scenarios where GNU find can be used, it is also safe to disable
>> file handlers, AFAIU.
>
> GNU find can be used on a remote machine. In all the same cases as when 
> it can be used on the local one.

But GNU find does not take into account Emacs' file-handlers for each
directory when traversing directories. 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:01:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899444601743 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899444601743
          (code B ref 64735); Fri, 21 Jul 2023 13:01:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:01:00 +0000
Received: from localhost ([127.0.0.1]:60881 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpki-0000S3-Cl
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:01:00 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:52801)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMpkf-0000RZ-CB
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:00:58 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.west.internal (Postfix) with ESMTP id 85C5F3200925;
 Fri, 21 Jul 2023 09:00:50 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Fri, 21 Jul 2023 09:00:50 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689944450; x=1690030850; bh=sY0D68lTqr/ynw2XUE4suwbwfPW+oxwP+QB
 VJArAoV4=; b=eh8tNW4KF08VnaC78zNhHFQwXlscbHE+E8FfufG3crEk97q1n0I
 8BqVvYVhE2O22U8oo4vw0KZTKdzjC4EdjdI8k44VQbuK2EGUEc+Y5d7YsGtpUokn
 lSgtdMyxVrbCSI/yEi0sTV0f8ke6C5ePdPRaF9EL7RyFEb4EgL2R5MKUUFaBjbQP
 mNGu6lr3cQyPN9HB2f9kjZVgew8rx5mIsRFZhYLVWSVHdYAVlATRUhKll+HKKd+A
 3VixBFKr/uYRkWHLbvJwwbgRgwYlhCOjKK3LttQEizDICvOZK1IJoi3ls1eyUbm4
 zxt0zFnTUXBpxnS80ksV0iyR/9XSYxA/O8w==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689944450; x=1690030850; bh=sY0D68lTqr/ynw2XUE4suwbwfPW+oxwP+QB
 VJArAoV4=; b=gHuwnthOfVBS3u9JoCmwQXrZDnhKoUaG/ZMCTxKUyQL/9HHKXk4
 m9rypFbeyjY9EuG1MYhy62673pW0EBZtM6l046fnRedDBfAL2yNSkkl9e66y7/rI
 y/upjk9VYJ7nUVrjvePdd7ofniuByTgOOrFHlrrBFi1VNhU8DTwL+KG172/b5W0L
 3RM1IuOXzrcLj6D27f6pCye56QHE1VlK+OOr2d9VRVptXPNbQ8S4B3LMc4aNnDi4
 tpkDzFwsZaz6fEcDE+QPsFvHCWN5oZFTLiMZhAuF4Q/9Cjw6QnTikhWqvjXq8dtP
 3LXta0XDebQPDSqPVbV3mtNs0sk91ZfVMoA==
X-ME-Sender: <xms:gYG6ZM1ufL-31Vd4TLcKGRv8PaJ6JkZMu-DzfLs8h21yxYl5ciQoDw>
 <xme:gYG6ZHF3eYZAexW-zpAZpkMg0pUdSh5jMO6CTZhd_39e2NGcVnhmgAcwCxS-3erVb
 tqGzL-PAhqOyuXnx9g>
X-ME-Received: <xmr:gYG6ZE65k3RvhT2DQ9sGufRllCVju45yI2B-MwQQTUHfkb2HeT5glWpt2KRVvXQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgdehiecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:gYG6ZF1mLRhvGvJrco8ic9_JrcHD7_3pA57w9hIUmw2FA5orokvITA>
 <xmx:gYG6ZPHnS1WJ_XNocUmLmt-asbzoQV-JCb91kvDtXB0xkNz9BbVmcQ>
 <xmx:gYG6ZO_jxtTmxnMGT_na8NS_8dWlfKn5yNIf57NtEK6CSfHCE2hAXA>
 <xmx:goG6ZESidxXWGlm-6SvWm6b7UA4DUSfMah9DD5PTIuB6EuiEWp0qRA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 09:00:47 -0400 (EDT)
Message-ID: <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
Date: Fri, 21 Jul 2023 16:00:46 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost> <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87y1j9l8u2.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 15:58, Ihor Radchenko wrote:
> Dmitry Gutov<dmitry@HIDDEN>  writes:
> 
>>>> Disabling file-handlers is inconceivable in Emacs.
>>> Indeed. But we are talking about Emacs find vs. GNU find here.
>>> In the scenarios where GNU find can be used, it is also safe to disable
>>> file handlers, AFAIU.
>> GNU find can be used on a remote machine. In all the same cases as when
>> it can be used on the local one.
> But GNU find does not take into account Emacs' file-handlers for each
> directory when traversing directories.

Indeed. Such usage always assumes the initial invocation and each 
visited directory belong to the same remote host. Which is usually a 
correct assumption.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:03:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899445412104 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899445412104
          (code B ref 64735); Fri, 21 Jul 2023 13:03:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:02:21 +0000
Received: from localhost ([127.0.0.1]:60885 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpm0-0000Xr-QO
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:02:21 -0400
Received: from mout.gmx.net ([212.227.15.18]:47201)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMplr-0000XA-QZ
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:02:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689944518; x=1690549318; i=michael.albinus@HIDDEN;
 bh=CrA1BJulU/5ibdeCNbGQ+hDEjrXAKL3OmN0+uEZwIe0=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=miYFMKG5gh3Grf4LpJtNHCO3jBfhkcvxsWWzOx6a0XLJD1aUUeByXifYyfJuZoVKfTSPh2g
 8jCFIibS1F4oLRelJbHaosuBME7RwB5IbibFyJB5FUiWlXyfG6pbJvG37VVDFiYQxDbglsVBw
 j207VPOyycm3Eyng0ZG/obwS8ahFt0ed6I4Wibgozxg5g/1TMA2cEXN+qmfu8Uj9tws/gNWKN
 9CwJ1kuPZx86Ns6W3mX9ABsFo32Kd39q6k8RVPXvgtzfD+svSlg6bv3jmPV7v56xKkEUHJN0J
 R3yZm3406LWSx7Q0nyYukdifWS7cSQpr1n7LJGjn7xmbMKDFd0sQ==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx004
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MoO2E-1pY9xG0gXI-00ojoE; Fri, 21
 Jul 2023 15:01:58 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83v8edzb31.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 15:46:26 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87cz0lmoxy.fsf@localhost> <83v8edzb31.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 15:01:56 +0200
Message-ID: <87r0p1cta3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:rrfDIcNewcxn2AyGKRlVyVU0RKUnxdVBaiInX+Nual2vbMMsMy9
 KwyoEZ5P0+Rc+fKFwiSKGw9MuSQMvTnga79OEVqCqo9JsG8NRRDFnynpl7j1RpRb2UN2Pfe
 s3pzhPf2HHCYbH7P0sLNpCS9e96U1PrQzS4DSK5cHsO1bNca6BRt3Gx7LiSJ8TdpSRV8X2e
 r+ktqDmzhBpYFHJsCmGBQ==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:kO388ASvuTQ=;3l80Xhr7SG1qDsAAiN8b0MJNhV8
 ghOcG6USDkQCO9AJo1aQ+i50QDZt1ngb06t2mpPldWFhio8bsGzCfV2GTa/05v3oHGwjXrQFd
 IRh5IKjga8xTS5CDwkOQ+xks3cYCOrBrnvpwHotqB2Qy1Y0J9U+Fd35jheOwC1Ny1At9cw7HI
 YLNKezDEb7JTkVPm6XlEsfKxT7cT2tcnxasiA9KhMD8KL4ZcIg5KVweK/iNZKBVSmAGOVytH8
 ucmb1tOoZVQGQIr5TdNaa2cERPfeZLjyTJyikXD2RJOWiBhoaaJfvbYpPVgiLgT8hMYqTTllA
 1s7xb/8fN1Tk5kc4jOSiVFWoFdSqc/85LKQvH9/nOaXGMpwGc60a3Tt5XSkcIg627sgyFmRSl
 PJt4d7/FaiXqouhakZMW8RTX9dgyNgnKZ3gVZfm8XseH+IowgwvQYBVCFjT7uiDrswv54QoTZ
 KCC2gsejjHAdFCS9TjNkU2Yi/K/nMfrwbbhI8whzUnQKV+gatMnBrmP02gHz1TJ51V2dEqAUn
 ZcuoWC78r0ZAZcm3HlB71CD3x9814SQN6AvdRzndObJtjnB35wqtRwfcUNpglSnZRK3cnC8nq
 IOV6M45IOEwkzU8vnWTJRwNEv2JuG2FP+Q4x7PF/AcmBRM4TEXuQwzUSPjFXdQfG+bhQJ1H1A
 dlWau/ClJU1MvY2dAeeDQHmjw1niQIAcdTnBJG098eCEHexv8TvASKvsCAYI1eRI+FJrwwqc4
 8EMkbELVhTj8tvt2X6epOQeJ74TtSbQVg0yGvum10djAt2qdfGnP3mm8d8spkVOtpOlCEyazl
 qTXBau2/YSA/Mkcf1n4nZzXoxKPJT8CtVHv+jUiiHnvfXr6P6LJ2KdWlzGF9ACx/gNFpjXpIO
 U8i+KEjbBdMl4/bZdZYIpRJX1b6fQLISARPml7lrPwpwCSlRuonCMBtvDy1UgWihiGN3rydJT
 /2Yvp5N/BMuYuMbePZ+MfSEQZs8=
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Ihor Radchenko <yantar92@HIDDEN>
>> Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735@HIDDEN=
rg,
>>  sbaugh@HIDDEN
>> Date: Fri, 21 Jul 2023 12:25:29 +0000
>>
>> Ihor Radchenko <yantar92@HIDDEN> writes:
>>
>> > The question is: what is more costly
>> > (a) matching complex regexp && call function or
>> > (b) call function (lambda (fn) (when (and foo (match-string- ... fn))=
 ...))
>>
>> (benchmark-run-compiled 10000000 (string-match-p (caar file-name-handle=
r-alist) "/path/to/very/deep/file"))
>> ;; =3D> (1.495432981 0 0.0)
>> (benchmark-run-compiled 10000000 (funcall (lambda (fn) (and nil (string=
-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
>> ;; =3D> (0.42053276500000003 0 0.0)
>>
>> Looks like even funcall overheads are not as bad as invoking regexp sea=
rch.
>
> But "nil" is not a faithful emulation of the real test which will have
> to be put there, is it?

Here are some other numbers. The definition of inhibit-remote-files and
without-remote-files is below.

=2D-8<---------------cut here---------------start------------->8---
(length (directory-files-recursively "~/src" ""))
146121
=2D-8<---------------cut here---------------end--------------->8---

A sufficient large directory.

=2D-8<---------------cut here---------------start------------->8---
(benchmark-run-compiled 1 (directory-files-recursively "~/src" ""))
(38.133906724000006 13 0.5019186470000001)
(benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-=
recursively "~/src" "")))
(32.944982886 13 0.5274874450000002)
=2D-8<---------------cut here---------------end--------------->8---

There are indeed 5 sec overhead just for file name handler regexp checks.

=2D-8<---------------cut here---------------start------------->8---
(benchmark-run-compiled 1 (without-remote-files (directory-files-recursive=
ly "~/src" "")))
(33.261659676 13 0.5338916200000003)
=2D-8<---------------cut here---------------end--------------->8---

Removing just the Tramp file name handlers comes near to let-binding
file-name-handler-alist.

=2D-8<---------------cut here---------------start------------->8---
(inhibit-remote-files)
nil
(benchmark-run-compiled 1 (directory-files-recursively "~/src" ""))
(34.344226758000005 13 0.5421030509999998)
=2D-8<---------------cut here---------------end--------------->8---

And that's for the innocents, which aren't aware of Tramp overhead, and
which don't need it. As said, ~50% of Emacs users. Just adding
(inhibit-remote-files) to .emacs gives them a performance boost. W/o
touching any other code.

=2D-8<---------------cut here---------------start------------->8---
;;;###autoload
(progn (defun inhibit-remote-files ()
  "Deactivate remote file names."
  (interactive)
  (when (fboundp 'tramp-cleanup-all-connections)
    (funcall 'tramp-cleanup-all-connections))
  (tramp-unload-file-name-handlers)
  (setq tramp-mode nil)))

;;;###autoload
(progn (defmacro without-remote-files (&rest body)
  "Deactivate remote file names temporarily.
Run BODY."
  (declare (indent 0) (debug ((form body) body)))
  `(let ((file-name-handler-alist (copy-tree file-name-handler-alist))
         tramp-mode)
     (tramp-unload-file-name-handlers)
     ,@body)))
=2D-8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:05:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899446922524 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899446922524
          (code B ref 64735); Fri, 21 Jul 2023 13:05:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:04:52 +0000
Received: from localhost ([127.0.0.1]:60896 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMpoR-0000ed-IJ
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:04:51 -0400
Received: from mout.gmx.net ([212.227.17.22]:38723)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMpoP-0000eE-MH
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:04:50 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689944678; x=1690549478; i=michael.albinus@HIDDEN;
 bh=mEojxwmWoCYnd2KjUMuFxY1QKg/SmUVZDnPH7nwHO0Y=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=l2AztjWJ+psNo7jPo/NSd3YCStiN8j9nAZeCUaqn0nACaK4ab2UMu0833vCtkb9DMmKs0Ua
 z9e5lEUXzbMMWqOPOTY+dnHMmRmgI9j7ejatIO6f64sEIn6N5zuQXcXdcxgl0gLNcI0Czm/98
 bPYp8NwGwJ8cahplao1OyA9dnMU7F8EoMOPm+GklxOTE15X65MmYqM4avfstaehsfVAVhX1XC
 Om15luApAo/556EvesrTAqRTvxhdrFj8/Koi6j+gnRZ8IIU/CIGli92WhFTZhINEMCW219Vou
 4wnsMKYDy9D8Ofj6OJSsC0cN1pj/DmrPXOqikRQ0iW7oLeIfDWVg==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MQ5vc-1qa7TJ3eBU-00M7Xm; Fri, 21
 Jul 2023 15:04:37 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87a5vpmopa.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 12:30:41 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87zg3pcuvv.fsf@HIDDEN> <87a5vpmopa.fsf@localhost>
Date: Fri, 21 Jul 2023 15:04:36 +0200
Message-ID: <87mszpct5n.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:Hx1FSL6gvCifCKe6+4TZ+XsBqTdBx8tfAb7gpqbv/KPrffphCbz
 E8q4O7ymKkbLDJCQD1i6lViof1TLrNbMjs14PqYqhZCGW4U6MhGrCL6lvHmrz6vanpiUZV5
 /MG20FnpwJorwhXWk8/EpoW6b1KpWaNe8s29/C/sKnR5Vc29YrdnP+eko6zg0N1skWibDLG
 llozUYPPOkOpQZfg04Vhg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:wwPfCQh3k1o=;WT7CsVauPABU9ltsvaM0cZhzGm+
 uJDmH/zOJ7mhxCeCXAVu+GmZz+L4sk6NnI8miUOk4Xa7ji2WQDYRi41RMPd/mzKAWUFrdXKtb
 ZE8sAYrOgDPzu474eJgwiD49izz6PFY3FWAD3jtHBJkBtPXaQhxKGtZ3/GHk80/6QtC4fk1+G
 o2O6bsVwOXX1uSKQnszUurGXroDJIKsaHV86fbNbyRa1t7+eQqx2VTDltiIYY3IeSsXw5Tt+3
 MzknyzMeQzBGvv/jDco2+R70TNopbT6PuiSubTrTs+NsDaN2DaKbs/ztElK1l2WmFP5nRJjCG
 +3oH4ZtIck14UaM37kYqadVn45ZH2W0k/ZKhapHLSVmHBSTquZJkn1GWapSz3GSMECBeLn2SO
 san55fCptVWZlxBgawjZRUPLsEaWPZ1KXbCmyo4cNZOTDbOfFtCrU5whNRrn1AaeEF8ytzC0J
 CY3ubH4ajYLT8HdG9gbv2dGRqr2fkUR+2DegSrLNn5hdKO4FmPFU4s3KsxAqEe+sy6NawStdT
 SuC3qt8z795lK7V/82ldGYjGTtjE+2FGY/ufwjiBtnV9SEwQOrmTVpxx4SPbsS9D80XB8prM0
 1WklpXJFI67w3krnFY46SOcm55M97uzcOJn6WQZw/IdJUpiXS4Lra7vnI9jsxB9OxMkZu63rb
 L5lATEdST9wjgRqToi+JdZCc8DOYwn0v4Dc02hsg0fuc4Lz/zGpm3GcNpuwAKG2mfOZgN7vlU
 WmFWP8/tCDJ93qsinkz065NJ+WlES5la2PzhuAHGNT052TdkIsaOYmXupdSdOXyLADms0yx78
 Qg1jgvoaZ7CFwVom7G6J8rGbT8F9tuxDcLEoQZA3wAcvAm90seKlU7KAshAYwkTZl3WKYKk0J
 lWU14LOLb8yaOwKGj0rmZxhudILuq7/qprxTICGClkBBNPbdJr4ACq7se7sm5+r0fZTZMW4uz
 rdrQBeSN27YcKKaTK2LwoSKgnIU=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

>>> I am not sure if I understand completely, but it does not appear that
>>> this is used often during ordinary file operations that do not involve
>>> tramp.
>>
>> Don't know, but it is a documented feature for many decades. We
>> shouldn't destroy it intentionally. If we have an alternative, as I have
>> proposed with without-remote-file-names. What's wrong with this? You
>> could use it everywhere, where you let-bind file-name-handler-alist to
>> nil these days.
>
> The idea was to make things work faster without modifying third-party
> code.

People use already (let (file-name-handler-alist) ...). As I have said,
this could have unexpected side effects. I propose to replace this by
(without-remote-files ...)

> And what do you mean by destroy?

"Destroy the feature".

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:11:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899450243437 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899450243437
          (code B ref 64735); Fri, 21 Jul 2023 13:11:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:10:24 +0000
Received: from localhost ([127.0.0.1]:60907 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMptn-0000tL-Ng
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:10:24 -0400
Received: from mout.gmx.net ([212.227.17.22]:58745)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMpta-0000sI-8R
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:10:21 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689944999; x=1690549799; i=michael.albinus@HIDDEN;
 bh=V2AEHKSmR2LvJ4OjauRfXgJu7JYYdg1Z55WMKPZm7B4=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=kh4yP/pMcDQpTeQ5N/o5jGEqfVE6dN6/HSE0nYkhW01+Xqv7EwbFISMzvsC4lu4QXejW7kK
 vGNPk4KE4ZkF4kC4QM5dx4J8PaZy6/Lvzn2ApE8b0b+cGtjeJ97dLD0g4LQSF8VsD+f8xU9H9
 s75E/SNIrd/x4UqmRiV3usQq/z6DXdIHBoZcvkykyU/I54Y8BHO9EThooHBimLXo9CM1j4TcI
 xY0ZBnA86bp0whtnUmC5Q9Yd2sLuB97XZ3hM4OjLyHf0Xh/RsgCQrqhKcHp/z8t+j+AaMhP0o
 PNbmoSgnjTsI29q1pc8svA9VKSmlAOWIto4kJM7FoBF9wJlQf/9A==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MXGr8-1qPiWH2zyI-00Ylno; Fri, 21
 Jul 2023 15:09:59 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83y1j9zbf4.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 15:39:11 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <83y1j9zbf4.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 15:09:58 +0200
Message-ID: <87iladcswp.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:dgC8RwxbuQW+qX1PQ4QrhYm3xutSC21WOgpGHUm4L+50Zmzp/Of
 O7ksIGV2Ky4gLNNc22PwrQ3RlRN5QpfLfxP+ageJ0EfWrS4lxCkHo7/KhwUJuFhkJR7Amf2
 KRmQNeD7JkjHg3BCPMKQP/CNtPObGqXq7M1/dagX7kTcLT4yOCXy9IyB7rM98RXKoY3PEyo
 k77s65khMJY4QC4KJFmLQ==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:3PHJccNSvLQ=;/oEG/t3jAUQzah16TjPUmyxAGt5
 MPH0u3miisRs1WDXT3W2djnjyL/jKgTads7I2Vv5DdaDHiQl0r2xb3SAbRuQaJkewauTvQO73
 LNzEe4tnhBhhaPhnFvMBGB76u7lPBQuTJIlkQYEk7L7nKAJCTD4Fx6nn/wardWOUQ4J+DN6VS
 a36+p5wVSZHyogMErbpXW5Aq0Bt/71bTz+jrSqh4FDWzeU5Y1d1VM+EsLBPZAifm9Ia7X9WCf
 3/zJ9YAJe2V7ujo1a+uL4tyR80SwAfu/KO7aFCkaSyxqUVaqqhvov8iJ94fwNUjBhn4e4KYZX
 nakCaqZ8yB7fsqu/J/aEiurZSf2uO3s6icY23rDGopHJv0+VC79L60FSBigLAT1NBheQhBddZ
 V545Xd323JT3Ca0KCoUEIn3Qk9LzAcHs+Pnn+Q1FfM8KPYe84fbI5zMD8y4wGqljKqfZIgs75
 WnzmrhA7Jbes88KwiKgil9o670pUhCcfFCCQee9zSAyB2CuDSdfZ8w0kuXsezmZ5ygM+zA7OT
 TmKfZyguI7IDezNZzOCy135mup6lIFkLdMiQqWFH6z45+aGGHoZ1bBXdZOiOnpVeShMiLGzRC
 lRri/DYHjbw+4seb6u1qxxDB9wI1VSaHKKsblXld4fNKYqrZ3wUlw2rRutaRQu5NGUPP/cx+t
 PAsPQwHRJhdIw0DJARW49g6Iv+QZh/ZiAiAlItgxfSIiC0oN0vqHETBYR1k5zEobA/Hgp+ZT3
 MS2vMJiiWL2kOHEiUAcFtwTWzVzIngcX75LTawf7aef7V92bkRU8EPbSQVqLVxWoaoqL0uROi
 tCTgIzS0g2mv1M8NU/wTq/ELBvGVTnMEhsyCBe3GPGFbdK8JCsI1dLICypIm9Fgb/tMQiM7ZH
 20fpkcFMt7mcS0OCqLGSMswtCmt7UrW5gBmiH4J7RcMrD08ZN4W8EnKKAlDI+7dGdiXwbpD+O
 xvxZTQ==
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > How is this different from binding file-name-handler-alist to nil?
>> > Tramp is nowadays the main consumer of this feature, and AFAIU your
>> > suggestion above boils down to disabling Tramp.  If so, what is left?
>>
>> jka-compr-handler, epa-file-handler and file-name-non-special are
>> left. All of them have their reason.
>
> I know, but when I wrote that disabling file-handlers is
> inconceivable, I meant remote files, not those other users of this
> facility.
>
> Let me rephrase: running Emacs commands with disabled support for
> remote files is inconceivable.

Agreed. My proposal was to provide a convenience macro to disable Tramp
when it is appropriate. Like

(unless (file-remote-p file) (without-remote-files ...))

Instead of binding file-name-handler-alist to nil, as it is the current
practice.

And the command inhibit-remote-files shall be applied only by users who
aren't interested in remote files at all. Again and again: these are
~50% of our users.

> IMO, if tests against file-name-handler-alist are a significant
> performance problem, we should look for ways of solving it without
> disabling remote files.

Sure. If there are proposals ...

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:18:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899454434451 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899454434451
          (code B ref 64735); Fri, 21 Jul 2023 13:18:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:17:23 +0000
Received: from localhost ([127.0.0.1]:60911 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMq0Y-00019h-0L
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:17:22 -0400
Received: from mout02.posteo.de ([185.67.36.66]:56085)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMq0O-00019M-3T
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:17:21 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 8E5DE240105
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 15:17:06 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689945426; bh=YVjgHITeyH1oF6ZfygheH9eD2bdEs4ri7WCKlYXrJ1Y=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=lIBF0dr9Y7Ag25IOlo1QBA3kDUkL8pMR7krtr2lPleYJ/tiy6Rpbj0Tqa+eB1ILmG
 gSb8OhOmMgMHdmTDYn39+8Lbwlsbv/G/Zgj4yw/U6XpNGTwopqb1ZBns7z7saOsBGn
 Bj/LBYBRn8KNP7q4+lDr13dp+apHcw+z5BH19kULb/rHlPKJL7nVDfvkA21bZlTq/J
 dzRamagX4YbkrCs4PVSTr2UUXdS3w0r4VK8BN+2ZkcYrMxFYMDX0qB+3RxuKT3uEEj
 1B0GiY1WoCJs8u/XqEcJ+Id6e1HtlJqsOZPneFj7K/Ct5xZuGEijYIm0cP0sniYJpo
 WiyEtJwBa/aoA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6qpT5rtQz6txC;
 Fri, 21 Jul 2023 15:17:05 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83v8edzb31.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 13:17:17 +0000
Message-ID: <87v8edl7z6.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > The question is: what is more costly
>> > (a) matching complex regexp && call function or
>> > (b) call function (lambda (fn) (when (and foo (match-string- ... fn)) ...))
>> 
>> (benchmark-run-compiled 10000000 (string-match-p (caar file-name-handler-alist) "/path/to/very/deep/file"))
>> ;; => (1.495432981 0 0.0)
>> (benchmark-run-compiled 10000000 (funcall (lambda (fn) (and nil (string-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
>> ;; => (0.42053276500000003 0 0.0)
>> 
>> Looks like even funcall overheads are not as bad as invoking regexp search.
>
> But "nil" is not a faithful emulation of the real test which will have
> to be put there, is it?

It is, at least in some cases. In other cases, it is list lookup, which
is also faster:

(benchmark-run-compiled 10000000 (funcall (lambda (fn) (and (get 'foo 'jka-compr) (string-match-p (caar file-name-handler-alist) fn))) "/path/to/very/deep/file"))
;; => (0.5831819149999999 0 0.0)

Let me go through default handlers one by one:

file-name-handler-alist is a variable defined in fileio.c.

Value
(("..." . jka-compr-handler)
 (".." . epa-file-handler)
 ("..." . tramp-archive-file-name-handler)
 ("..." . tramp-completion-file-name-handler)
 ("..." . tramp-file-name-handler)
 ("\\`/:" . file-name-non-special))

---- 1 -----

(defun jka-compr-handler (operation &rest args)
  (save-match-data
    (let ((jka-op (get operation 'jka-compr)))
      (if (and jka-op (not jka-compr-inhibit))
	  (apply jka-op args)
	(jka-compr-run-real-handler operation args)))))

skips when `get' fails, and also puts unnecessary `save-match-data'
call, which would better be inside if.

---- 2 -----

(defun epa-file-handler (operation &rest args)
  (save-match-data
    (let ((op (get operation 'epa-file)))
      (if (and op (not epa-inhibit))
          (apply op args)
  	(epa-file-run-real-handler operation args)))))

again checks `get' and also epa-inhitbit. (and again,
`save-match-data' only needed for (apply op args)).

Side note: These handlers essentially force double handler lookup
without skipping already processed handlers when they decide that they
need to delegate to defaults.

---- 3 -----

    (if (not tramp-archive-enabled)
        ;; Unregister `tramp-archive-file-name-handler'.
        (progn
          (tramp-register-file-name-handlers)
          (tramp-archive-run-real-handler operation args))
          <...>

Note how this tries to remove itself from handler list, by testing a
boolean variable (nil by default!). However, this "self-removal" will
never happen unless we happen to query a file with matching regexp. If
no archive file is accessed during Emacs session (as it is the case for
me), this branch of code will never be executed and I am doomed to have
Emacs checking for regexp in front of this handler forever.

------ 4 ------

(defun tramp-completion-file-name-handler (operation &rest args)
  "Invoke Tramp file name completion handler for OPERATION and ARGS.
Falls back to normal file name handler if no Tramp file name handler exists."
  (if-let
      ((fn (and tramp-mode minibuffer-completing-file-name
		(assoc operation tramp-completion-file-name-handler-alist))))
      (save-match-data (apply (cdr fn) args))
    (tramp-run-real-handler operation args)))

is checking for tramp-mode (t by default) and
minibuffer-completing-file-name (often nil).

-------- 5 --------

(defun tramp-file-name-handler (operation &rest args)
  "Invoke Tramp file name handler for OPERATION and ARGS.
Fall back to normal file name handler if no Tramp file name handler exists."
  (let ((filename (apply #'tramp-file-name-for-operation operation args))
     <...>
    (if (tramp-tramp-file-p filename) ;; <<--- always nil when tramp-mode is nil
    <do staff>
    ;; When `tramp-mode' is not enabled, or the file name is quoted,
      ;; we don't do anything.
      (tramp-run-real-handler operation args))

this one is more complex, but does nothing when tramp-mode is nil.

--------- 6 -------

file-name-non-special is complex.
The only thing I noticed is that it binds tramp-mode as

(let ((tramp-mode (and tramp-mode (eq method 'local-copy))))

So, other handlers checking for tramp-mode variable early would benefit
if they were able to do so.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:24:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899458055057 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899458055057
          (code B ref 64735); Fri, 21 Jul 2023 13:24:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:23:25 +0000
Received: from localhost ([127.0.0.1]:60920 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMq6O-0001JU-Ss
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:23:25 -0400
Received: from mout02.posteo.de ([185.67.36.66]:35693)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMq6L-0001JB-9U
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:23:23 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 82ED3240101
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 15:23:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689945795; bh=bZ5Hxl/95taiQ8rPNMgMc7+PpRit3CgWxJKAfN07dOU=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=bavoBXHQfIQBsyMj3WbokD8UxAPLwbu5HXyRXKPn/Wnl/60GMCWZNsc/F1qlLxQED
 sBs01iwxdUc3YnyBU+1YLlV0HYEtxSAjGKoSr4VBVoll6Se9Fpl01WBYOPS94fP5EX
 ylwuRqULOoDXkbPnza1/PrBmK2U0uq8AuNJBYq5P3PxQ0486afdBJLArEs44HA8MO2
 mKb8q6skzXlCJ8irPDSIvybA4i8/F97rY498nwr8EBPLAz444OGD+gpWBK7cMKcmh2
 RPZYQgS6Y7x857gkIPRk1RbDVHl96drG+vEJ83y5SfffteNGI58TyFZ26Ow5Ot+Uoa
 Ii5Gplrm8kYkQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6qxZ58KQz9rxL;
 Fri, 21 Jul 2023 15:23:14 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87r0p1cta3.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 13:23:26 +0000
Message-ID: <87pm4ll7ox.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

> --8<---------------cut here---------------start------------->8---
> (benchmark-run-compiled 1 (directory-files-recursively "~/src" ""))
> (38.133906724000006 13 0.5019186470000001)
> (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/src" "")))
> (32.944982886 13 0.5274874450000002)
> --8<---------------cut here---------------end--------------->8---

Interesting. Apparently my SSD is skewing the benchmark data on IO:

(length (directory-files-recursively "~/Git" ""))
;; => 113628

(benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
;; => (1.756453226 2 0.7181273930000032)
(benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/Git" "")))
;; => (1.202790778 2 0.7401775709999896)

Would be interesting to see profiler and perf data for more detailed
breakdown where those 30+ seconds where spent in.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:25:01 +0000
Resent-Message-ID: <handler.64735.B64735.16899458645183 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899458645183
          (code B ref 64735); Fri, 21 Jul 2023 13:25:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:24:24 +0000
Received: from localhost ([127.0.0.1]:60929 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMq7L-0001LW-MN
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:24:24 -0400
Received: from mout02.posteo.de ([185.67.36.66]:46703)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMq7J-0001LJ-LL
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:24:22 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 21165240104
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 15:24:16 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689945856; bh=D7yayK0H7HX5SPCQ63yYWvruDCkDamJOU+YQ9EgS1vw=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=kejeiALzT8/+omWQtIE/WJWEKDgo32jdgPR3mzM+SPTc9qT4Oz4Yoq0hHm53Q83q5
 7nII7AQv+hwMV/EoS6E73T+8EXXp0gc7LUEyjsoBwRbjVSQmYl6t2idk5TfSiuYwg5
 FTfwWiLc4ivq31Uzna56IqJYA2g/grWGyy/UW8N1fB8BYJhja41XgDg7GOyEyaNa4D
 dtEOlDyFMFIe4G49l7vtwM5Z1hTZ6TPb2kx572eyPrNLqqpsDWd04CbrJmHvAqIvXJ
 sBWj9aXjVWdj4wu6DmPjqw11YRHvPIPZCC3nsG12P1+HB2gRzo8pqY2o0J8nOhoRNT
 T5B1fT57rEwTQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6qyl2qpFz6tm4;
 Fri, 21 Jul 2023 15:24:15 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87mszpct5n.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87zg3pcuvv.fsf@HIDDEN>
 <87a5vpmopa.fsf@localhost> <87mszpct5n.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 13:24:27 +0000
Message-ID: <87mszpl7n8.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> And what do you mean by destroy?
>
> "Destroy the feature".

I am sorry, but I still do not understand how what I proposed can lead
to any feature regression. May you please elaborate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:35:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899464756035 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899464756035
          (code B ref 64735); Fri, 21 Jul 2023 13:35:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:34:35 +0000
Received: from localhost ([127.0.0.1]:60937 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMqHC-0001ZG-4o
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:34:34 -0400
Received: from mout01.posteo.de ([185.67.36.65]:34515)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMqH6-0001Yy-1D
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:34:32 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 08C92240028
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 15:34:21 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689946462; bh=X97yuJL+rnGQxdHuTpyMndNoPTCbWbn9FJCbKlIiL7g=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=hP5UnPkC9mycx1V+uig4ePalGMumbKCFpv0xLPwzU1E07D8p1DK5WlyFXABbaOmfj
 uVVxma3+LG3Q/+RCJofd0PPDh7cIH9Rd+ByvsuQuxZ3DVqRAhu+IGIVLCDNqShsYjf
 USW7Ex28HNEzQhE1h5hUIXSZYC4NzsIL45UD5wEG65Hey+7jN+DI1a0PfsaY0Gt5/5
 c+/J1XIg1BNkzxZ/gxpJypkWQ50xEAzcpTjXFIJvdNIVNhDxYAdrChUZ6585dVMqID
 JexMDrhGZrJNrQUuiCie2wH6dGb4sPzkcex6bbdvJNOqTvSwwmOinrdjkj7vrEEhp8
 HX/SEYceRt8jw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6rBP0JRJz6tyD;
 Fri, 21 Jul 2023 15:34:20 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
Date: Fri, 21 Jul 2023 13:34:33 +0000
Message-ID: <87jzutl76e.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

> On 21/07/2023 15:58, Ihor Radchenko wrote:
>> Dmitry Gutov<dmitry@HIDDEN>  writes:
>> 
>>>>> Disabling file-handlers is inconceivable in Emacs.
>>>> Indeed. But we are talking about Emacs find vs. GNU find here.
>>>> In the scenarios where GNU find can be used, it is also safe to disable
>>>> file handlers, AFAIU.

So, we agree here? (I've read your reply as counter-argument to mine.)

>>> GNU find can be used on a remote machine. In all the same cases as when
>>> it can be used on the local one.
>> But GNU find does not take into account Emacs' file-handlers for each
>> directory when traversing directories.
>
> Indeed. Such usage always assumes the initial invocation and each 
> visited directory belong to the same remote host. Which is usually a 
> correct assumption.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:38:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899466316307 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899466316307
          (code B ref 64735); Fri, 21 Jul 2023 13:38:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:37:11 +0000
Received: from localhost ([127.0.0.1]:60945 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMqJi-0001de-WA
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:37:11 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:39547)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMqJg-0001dM-6c
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:37:08 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.west.internal (Postfix) with ESMTP id B0BB8320095C;
 Fri, 21 Jul 2023 09:37:01 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Fri, 21 Jul 2023 09:37:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689946621; x=1690033021; bh=c3L5A39AlpayP5p7EpvNHs9rY60AJ9U5PcQ
 FhoWhpk0=; b=OwcwNql1STncKOtYCb0UavEu3LiKeW82Gvr+VhIIhBIeHrQ/Eam
 aGYj2GUFK/ozLm/OxggFngcCsWIT7pqOwNBvGaJyy5xmUQkDiG7TfF84+J+gtPv8
 bmpR+guSj7oBSbgfJFu10t3KnItujO345eRX5diAFUzwUMIWyUOIKqHdgUpt5L1T
 6qKm+ketAY/kye7NdDnFx5FZEkg2EHWMf8WVMCWjn6QLA+pI2d2NUjGga4EDWywW
 shld4eNUidPbJ3EFU9iK7qcx8YKPFVW61n937zhuou8dkmC/0jmKJYWTcPibWysz
 oRh8RV5VfgNf9XHZ4MAm3RHyBDAntiouT9w==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689946621; x=1690033021; bh=c3L5A39AlpayP5p7EpvNHs9rY60AJ9U5PcQ
 FhoWhpk0=; b=jgk6N9YqJI7CcZ28CKF7HWDuCcVWGzuBRjFw1jMRtzEYbnqkZAE
 A0tPt53EvO6SIXdhiou4v8Y3sPLXJdFLrdZegzks3SnpeY94wxPqNr1xVHAihzqM
 McHE9KQN0kCI80S408mvNLTG4DKmrjTT0nLnWrtpLQjJb4gwV3jqC4eDbZmHCm5Y
 WbXVJXlBLHPt1buzVrKw8wJUZWHas3PuuwRyN5YEsJjeNKbqizj6lO0q8OKeLV56
 h5vPf2VlRcM4tZiJ4aInxyLQxO6J1kUl1jzAkiWe8WBoY60MbucVtiE0W41KKOCJ
 dR9eppNsk4iWn68wCFWvWGBAlR3bnYTcAbg==
X-ME-Sender: <xms:_Im6ZHijxcJWP_goHK_jAq2GZBFVf-xrVagMzhGtV_EJURHyHhDTww>
 <xme:_Im6ZECGmonvr2NxhZVQwPNuVPBFZFvqvdDv3Abjl0u2_KYUvWk6GcV8tjqm4cfv9
 H_IJeTBELH6eLptU_o>
X-ME-Received: <xmr:_Im6ZHEByaV4FQgU_0_gOndQYvGWY7w6NK1gqOYpQRpvhBzOCH0h1FOn4zRQz_8>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgdeifecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:_Im6ZESP8MS2f9ZBPAXr7A8EY6sxJ_Nxx6iJb_t17Ic4oKBPfML3SQ>
 <xmx:_Im6ZEzcF0Ik1fzlljSUVcFOcN0M2poIthBPwQUWe3EdWRRUtta8vA>
 <xmx:_Im6ZK6wCxNRxm6t8FobkPdZLrHG-dahgXWKx_D-N8-gVulpS4CHtQ>
 <xmx:_Ym6ZP-mrR7p_-YACgFZEA4ExaRC7LuANjJ5Jo3N4Q6Gcf3K6oKErw>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 09:36:59 -0400 (EDT)
Message-ID: <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
Date: Fri, 21 Jul 2023 16:36:57 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost> <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost> <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87jzutl76e.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 16:34, Ihor Radchenko wrote:
> Dmitry Gutov<dmitry@HIDDEN>  writes:
> 
>> On 21/07/2023 15:58, Ihor Radchenko wrote:
>>> Dmitry Gutov<dmitry@HIDDEN>   writes:
>>>
>>>>>> Disabling file-handlers is inconceivable in Emacs.
>>>>> Indeed. But we are talking about Emacs find vs. GNU find here.
>>>>> In the scenarios where GNU find can be used, it is also safe to disable
>>>>> file handlers, AFAIU.
> So, we agree here? (I've read your reply as counter-argument to mine.)
> 

We don't, IIUC.

To use GNU find on a remote host, you need to have the file handlers 
enabled.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 13:47:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899472027587 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899472027587
          (code B ref 64735); Fri, 21 Jul 2023 13:47:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 13:46:42 +0000
Received: from localhost ([127.0.0.1]:60960 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMqSv-0001yJ-PB
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:46:42 -0400
Received: from mout02.posteo.de ([185.67.36.66]:55647)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMqSs-0001we-DI
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 09:46:40 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 644D4240104
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 15:46:32 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689947192; bh=kreoqqOdrt4S+qGU5+IDB8131y5kmZ1PQiW6teC/qHs=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=WgT5TMlk4y43lU414cTDOCKcOoyxwO+bLJGvRPjTvZ4JVCQyDDSvrJljYpVk3Qw2Y
 H1j6Q53b4QLOsFCwBxebJr9XxRTXRVuz7guSczAN240GvXxOw3f0sNw9DWAK6ZSGwZ
 hsoJG5PfmHoDtwXnAdTmn/6UNER6DSQpt/Q+2NBzULrsk6BdjuNbLNXqX5PhBpl5iT
 ydXXxriBL/8NvdhtIBeCnkZPGHLaIsXxU5PjxEvtWHnqalxASYLz8XoexWY1kKCdt7
 PAyGC3+t5Luz8HnbRgEtQ7/LLpY3uLhjLFdVMPXlrunUi8gWgCC5JXTXaMLBLsgSz4
 h+oArHvUw0hgA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6rSR5ZkYz6tvt;
 Fri, 21 Jul 2023 15:46:31 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
Date: Fri, 21 Jul 2023 13:46:43 +0000
Message-ID: <87h6pxl6m4.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

> On 21/07/2023 16:34, Ihor Radchenko wrote:
>> Dmitry Gutov<dmitry@HIDDEN>  writes:
>> 
>>> On 21/07/2023 15:58, Ihor Radchenko wrote:
>>>> Dmitry Gutov<dmitry@HIDDEN>   writes:
>>>>
>>>>>>> Disabling file-handlers is inconceivable in Emacs.
>>>>>> Indeed. But we are talking about Emacs find vs. GNU find here.
>>>>>> In the scenarios where GNU find can be used, it is also safe to disable
>>>>>> file handlers, AFAIU.
>> So, we agree here? (I've read your reply as counter-argument to mine.)
>> 
>
> We don't, IIUC.
>
> To use GNU find on a remote host, you need to have the file handlers 
> enabled.

Let me clarify then.
I was exploring the possibility to replace GNU find with
`find-lisp-find-files'.

Locally, AFAIU, running `find-lisp-find-files' without
`file-name-handler-alist' is equivalent to running GNU find.
(That was a reply to Eli's message that we cannot disable
`file-name-handler-alist')

On remote host, I can see that `find-lisp-find-files' must use
tramp entries in `file-name-handler-alist'. Although, it will likely not
be usable then - running GNU find on remote host is going to be
unbeatable compared to repetitive TRAMP queries for file listing.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:33:01 +0000
Resent-Message-ID: <handler.64735.B64735.168995353220110 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995353220110
          (code B ref 64735); Fri, 21 Jul 2023 15:33:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:32:12 +0000
Received: from localhost ([127.0.0.1]:34517 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMs71-0005EI-Io
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:32:11 -0400
Received: from mout.gmx.net ([212.227.15.19]:57139)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMs6w-0005Di-83
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:32:10 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689953514; x=1690558314; i=michael.albinus@HIDDEN;
 bh=BJL3GZoLlk7Mf/rlVLiS7h4DTwp2PrNHLXaMF7ez0yA=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=RCpO5W4kSjGtARaHskkK33RBsQIwbyiEwUoDxLPXfopVdbsJRxuBOCDeNGGXwB8GZdyjZBi
 1PS/DEhSBT3GNfXUI9G5SUMiyRG7SbTccyGw2Mj6xLnDpX599k4mEULoWPU78g3urM4NoOged
 kgnmal9YTNTMR9eGpjQ7YugrsYBPMRAxB4wJgkLaHiu4jIEzzHCJIDF47mFkz3ognPsgrNLsx
 oOiUlJg79xso6DNLYcwAMyWlWSXehjaTgsoHbHY4+taqjCsWBtp6JFYau9Yx0s1mAoGVpy6PV
 NFWY1WpkhWO/GabQWoMixuutlijl6rKlohoKdxC1o8bHv0bv4ZsQ==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MKbkM-1qdRYM08Tc-00L13q; Fri, 21
 Jul 2023 17:31:54 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87pm4ll7ox.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 13:23:26 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87cz0lmoxy.fsf@localhost> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost>
Date: Fri, 21 Jul 2023 17:31:52 +0200
Message-ID: <87a5vpcmc7.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:d5FVuHYZ2I1Gyj98I0uYKnMThYGOfbBDhhDm4aXBx2e9w+FDIge
 BeoiAuW8U5KEeAmW2wRVPZpOHopDBPSFT8tIv/KYSwmlS6NI2BZlauvn6ehIdEDmOp330TY
 g2SrZ3vfpLbeg5m254pVZ+tbUhUFG/uLSrnzWAFkALNQyH2bSwTqybGLZx5c+UzZr5ZwGeZ
 pe6kNpaAId5j3NJIJ36EQ==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:xtZmWgotrKw=;7sfwBaWP6qn+2b+R7a2QRpxbp8k
 FxUakwmyVTV3D/cM4CiB7nTMz6EkOzQ9w8Zrr5NmNBa7JbSMZ9OF+jzutf74PyhBX+CMLMxRE
 LbBrHoA5hom7wUeanSmp7XQ3SUT7T/WZ2ccAhI9kAnF/hcGRU/W/pNw/CORLxzbC03aQ5mUtw
 HrgQ32Lb5Bf/LTJT9xfqNgNGLqfGvwSodv8jz4EY4sbL7AcowOlblyu62tLHwfBx7tt6wIT72
 s1gkhPKkEoGZgCI0JgIOy4mshTa7mHsIZoeXDEQhGdjJaAy1HKl730yTNl4srMG5PlEsDGpMR
 u03eb2rLCCRruQuJjRmX0ibfAc+YPwrLGCUJqxaX04OWCuem0OKdtYusGsTDcWNXG9Be1mrq9
 DkWv7WI1xTjd7COgZaU2Kc+zZZJwue+Xm7xnDvlWryCtz8tZMfE90PcIWFTbQtauuHy4kjOiA
 PYZuOGAE2JKyXO5R09uIGNLr6/B1b5bhL9/QyRn+lAWO27udArNfXlQb7sxjfRRpFXmi0+Xhr
 vfyrpUj2WXX1fzgmmIOuSVWG76wtaSQ6DVpYOf+bdKQZMGZfTFOBMXVcrXrG1wachTZJ+48VQ
 qR71cvOJjjk647FZy5y8nBPKmyX7ML9xhS0EIJsl/tVdxz49+DHKQaDzkn2L1JGPKx6rmFoJs
 jNVFC0FUdYtYjkIgghEKPlP/e78V0KrAQZwPSdGdMD3twp83i9WSuoYIrqtocXlXEklqE9+ix
 pp7bA31L786BLfiUkAMWcl5D48Ru7ysgoGscy/Y6lW21FbHGpOeu/GlwOXGyLQ94juDouGcE1
 ZZxzgQJecftlnutF8FdpjZLu9hW3icUL6Qqs1vasa/nnm6NMRtlCj8N9PyAkiboc9L6KG53CD
 C27XcEpQLNAgGbKAtt1ZOTdvZ+kXLb66ebMuXZdofL2KIumPBdwbR5T0FJgw9KeDlTNVjT0+n
 wJjbcp5/MrGmYpOVe9D6MusA9fM=
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

> Interesting. Apparently my SSD is skewing the benchmark data on IO:
>
> (length (directory-files-recursively "~/Git" ""))
> ;; =3D> 113628
>
> (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
> ;; =3D> (1.756453226 2 0.7181273930000032)
> (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-file=
s-recursively "~/Git" "")))
> ;; =3D> (1.202790778 2 0.7401775709999896)
>
> Would be interesting to see profiler and perf data for more detailed
> breakdown where those 30+ seconds where spent in.

I have no SSD. And maybe some of the files are NFS-mounted.

=2D-8<---------------cut here---------------start------------->8---
[albinus@gandalf emacs]$ sudo lshw -class disk
  *-disk
       description: ATA Disk
       product: SK hynix SC311 S
       size: 476GiB (512GB)
=2D-8<---------------cut here---------------end--------------->8---

My point was to show the differences in the approaches. Do you have also
numbers using without-remote-files and inhibit-remote-files?

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:37:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995379320612 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995379320612
          (code B ref 64735); Fri, 21 Jul 2023 15:37:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:36:33 +0000
Received: from localhost ([127.0.0.1]:34530 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsBF-0005MO-AY
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:36:33 -0400
Received: from mout.gmx.net ([212.227.15.19]:51077)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMsBE-0005M8-1A
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:36:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689953781; x=1690558581; i=michael.albinus@HIDDEN;
 bh=o9fGz98QEezYBH21+4TMGkfUCAG0F70cZDiUPEa+Ba0=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=PSd1TIvHCAvG4JIIcM7saVDMNad9bvGtQcsICdunzGYH7Z4wOmEPpO4nA/g+1iplh6NEDEc
 edT4N5zWrOJDgQSFVeo6i+jCI+7yZr6QHY4SRlnuSpfywCDknO09zp84kzDxJ3xQ3PcSHt0FB
 d7BCfgNbjsowhiX7TfPtkj098GiVXHFJ8KyqH2T2JbF8dZTO2Q6g11bYWTY3robeQgBwCghwd
 KVkNLKcRC982OY9jjRCf+5HJXqbddnQnrPSKRJ8RoSzIToAcp27LVjRR5uKritveJ6L0sao5V
 yzfSOs7/AZkmvIrWauCfgAdD5BYUm2LJuvRrscSFHmi89kde4Xqg==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MhU5R-1pisek1uRn-00ecBB; Fri, 21
 Jul 2023 17:36:21 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87mszpl7n8.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 13:24:27 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87zg3pcuvv.fsf@HIDDEN> <87a5vpmopa.fsf@localhost>
 <87mszpct5n.fsf@HIDDEN> <87mszpl7n8.fsf@localhost>
Date: Fri, 21 Jul 2023 17:36:20 +0200
Message-ID: <875y6dcm4r.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:k8yMWVKDr9rWQCVe1syGQeKE3vR4NKii5hpFKwa3568XolouOwB
 16yC7KYw+BxXAVnPt0AT6ZnM0PhH0kDFo8w9WaXj2f8qwuUt2lZ6+3L48gmLGy+0lR33RHo
 C1RgdAnRr8SGMDWUA/v2F5/0ItbwG5UlWai582mhuS9gTqBXOq0D+YbT2lKlDoEcW8PqITj
 C+Drmo0hr9Kr40i77sCOg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:gAn+pdGIjnM=;5ZQCkpyKGhnuQmzhetdjlv6MPbq
 ERC10c4TWrF+Vey870310W/ViayUFOLPL3Xmzn5CAJyy8bqoE44voqPimQHcQ4B1cNdicGyah
 nZHpBzWozi4BBfeTVghP8qE2cXELPPweSRIRhg5i9Ca8Iv3DvvSDjWnsq8/W88nhdpWVdy30h
 yq6QWjhQHrI/ngD8KS5p12LgOIL0I2y9MKbUdliWx5Rc32hPnaWzxPmFCA2cKiTO9azJftWoR
 fqZbFb9wSXYbSjSVVHaanHZNMgFCecJ6n61J1kd5mLWQX1gQymYtOe/Coa2n6kTad9MAwyfv7
 gZDgiS34wf2tnOMKDfo/88lHPuE4tBpM3rd1ika8QX4+1Dh3HzQqmUWv8+V+b8SvAJxZzdtWX
 YZ5/oPX3Q64oGPrZCA0/J2sX35qA2BxliqiF5K+sWp0v1cjX9XOWLriH9kM9Ns7KUBAEJhoPM
 epXGMck217i42t17H+Yb4VMa9UxjhSwe2Kkhn2srsNI1Z56N8eYBCllXRIdnHqIf0bOYEj64a
 WiT+rzz2jBw4Pqzs2uMCrPus2SwlpdiV4094+I/c0+D1njdsI2rpPR1N8xXeQfe1bc/ZrCUR5
 IBGWYHVMw9fcmCZfzHavzK8ZqTlDr6PuB6GhxcGZhZOCObLIeQgn+oYG76g+1kW1Z20FhDC4O
 cpjZp1vwHLm6JS6vtPZIb7DI3XIlnTmw9GC7aRa747jVs9ceCSJDj1Kx36Ue6/UmRJWpp6l3r
 MrDz3YPoxDhXDRCdsUa+8LJSrXv2O8aIO4wBxZ6hPPVnOmyzNxQs3hr8XfxbmFJY8OgZuQe4r
 gJbxJwjo5gMgYRln5oW16CA2EYhiDPR17uZdnWSkkErNa6s6MHXwaaJf2FbQEnxDUsdqUXvVw
 h3RhA1t9xM4Uib6fumEpjNfwHIYSDxTwKYn1bxTjBnEM6bQd8oQs75xC1PekD6MfAYzv6gyAV
 IBO4ig==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

>>> And what do you mean by destroy?
>>
>> "Destroy the feature".
>
> I am sorry, but I still do not understand how what I proposed can lead
> to any feature regression. May you please elaborate?

When you invoke a file name handler based on the value of a variable
like tramp-mode, either all file operations are enabled, or all are
disabled.

The mechanism with inhibit-file-name-{handlers,operation} allows you to
determine more fine-grained, which operation is allowed, and which is
suppressed.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:39:01 +0000
Resent-Message-ID: <handler.64735.B64735.168995393420871 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995393420871
          (code B ref 64735); Fri, 21 Jul 2023 15:39:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:38:54 +0000
Received: from localhost ([127.0.0.1]:34537 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsDW-0005QZ-C4
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:38:54 -0400
Received: from mout01.posteo.de ([185.67.36.65]:51809)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMsDT-0005QG-Ot
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:38:53 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id B31D7240029
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 17:38:45 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689953925; bh=lg6z0Re0P0faPg21zfW+vKSpJmokcY7HxP4cXbaEXD4=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=IXORiBCpjHzEsqvQJ+QUKdGtMFvE1avn6WOb+iRAzeXrZ2oGWt6eIuWCaR5bmhKxV
 z5Q8JVFkRwV4C889/Xm3zuIOwDO18q/F0PHogcYpAmeYRJn4H83jlEVqsGm2tYcjG7
 WsMqoyu2rSXBGEhLOcyil7tjS+KnKQLe5nCC2v9ePYVbiK78Joci9xZPHhMjtlqX82
 ax2/ypuQmbgvU81wrZx1BkTRurpkflPsjdpZdgCqe9zmhdBM5oS9j6E+pv0LpQ0FBV
 6Se/IrzitJyTy1fzPxw2+LCWMGlV6Gfq6waBUhPUbmLdjtUq3JlJ7t56JwlhJLzU4v
 J1/T2tvMiQnHA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6txw2W9Lz9rxB;
 Fri, 21 Jul 2023 17:38:44 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87a5vpcmc7.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 15:38:54 +0000
Message-ID: <878rb9l1f5.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> Would be interesting to see profiler and perf data for more detailed
>> breakdown where those 30+ seconds where spent in.
>
> I have no SSD. And maybe some of the files are NFS-mounted.

That's why I asked about profile data (on emacs).

> My point was to show the differences in the approaches. Do you have also
> numbers using without-remote-files and inhibit-remote-files?

(length (directory-files-recursively "~/Git" ""))
;; => 113628
(benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
;; => (1.597328425 1 0.47237324699997885)
(benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/Git" "")))
;; => (1.0012111910000001 1 0.4860752540000135)
(benchmark-run-compiled 1 (without-remote-files (directory-files-recursively "~/Git" "")))
;; => (1.147276594 1 0.48820330999998873)
(inhibit-remote-files)
(benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
;; => (1.054041615 1 0.4141427399999884)


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:42:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995410221236 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995410221236
          (code B ref 64735); Fri, 21 Jul 2023 15:42:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:41:42 +0000
Received: from localhost ([127.0.0.1]:34556 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsGE-0005WS-FI
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:41:42 -0400
Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:33213)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMsGC-0005W8-C8
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:41:41 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id C67523200907;
 Fri, 21 Jul 2023 11:41:33 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Fri, 21 Jul 2023 11:41:34 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689954093; x=1690040493; bh=jFONDpMu3wNqG5wY5OAv/HyJvLf/5+pq9GU
 fg5Ufzmc=; b=Xdkl0HO15qiSXZb5smXY9MoFhAx49VszbA9RkxTOGOtPrZ1/NF2
 eWHYS34uv32DjslXlEiph1ojITbh9tvja8GCoppTlFAuWrFdtdGa71HAP+xcB6Zz
 6zwWBQdQLFAmRzSyf09CYyO9dNdEQs8AYfYnvwVnqfP4PfPlOEulggZcjAt5Nc+a
 6Tulte7+gEVBHqBbBFpQxBmfpvxEeuNZR1Ti6xlXUly9HOs6t+a4h+g1Cy/UWgLd
 AaZyqR8bdgaXNhNhvq7cPvkIIiMQXDZfNrc/CFJu3va8+tGU4v8PBxBndAm+lE8y
 u2AUJgSvSK69QX9q5nI2ZzwIHXRxvAHTtRw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689954093; x=1690040493; bh=jFONDpMu3wNqG5wY5OAv/HyJvLf/5+pq9GU
 fg5Ufzmc=; b=wAxetCVuQqqFPJfKlWV8TwXsWaGthLUtgGZtSVuQ44RIUJyyX3t
 fwFEYsTXsDelmMN2EyrQnHmE3j3U8/ZpZ5cORv/1uxnb/8bpVxqoYBC5UjWPQ9RF
 ru5+1CcxuY4C38ukUetxeBSH6+ph0fyfyhu/tsdC7aUmeS1Vz41KHzB3vQO8qn1S
 BABhcdTM4Gp4CMZJ1YnAIpe1Kr4lKE1SSNUT/VkSdqzPqrbxtkcNloDWdUwKId2N
 WjMj2IpcW56QZQJKllf/vX04TjGqJwhlJ3bkjgPGVvyBrC6/pjsCrm4J1nIsVY3S
 v3ytE8Y+Y5LUyAo8+gF1XEUrXUsS0GRmJ4Q==
X-ME-Sender: <xms:LKe6ZFqt0gNDUxOXHRwK-E22bxdpcwj-eAc2kc02QMnGMg-nrLpuyQ>
 <xme:LKe6ZHrHYAvgLR5r6MEXOlp6FYSBeom-dkmRNcTPbhq2NnH2z6HtfeuzXZdD5XwAC
 qH2xWZaTJyIAeHqQok>
X-ME-Received: <xmr:LKe6ZCPjRHM3tJqyrlQZ6w7DYeyqAqwU88NJibEKMpcNbw6Bqc8R38tzvU8CeJk>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgdekiecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:Lae6ZA6UjfugklP0kBPqQ_i0Bg-KaPzdN7rEw1QKHP7c6E3NL_WHCg>
 <xmx:Lae6ZE5ieCsAJXfj2tcw9i9HlWM1JHglNAHXS35fq3462u-SOmNA2g>
 <xmx:Lae6ZIiLYm85z3TqklWmL9ciQiWDLvOlmYGJdla_sSNA7wgZV13Pjw>
 <xmx:Lae6ZEErpMMIcAbxsMpoMRfFZhAZkk8qPCMHIm5oOAfceSGj28N8AQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 11:41:31 -0400 (EDT)
Message-ID: <ea2332cd-ec57-794d-d662-ff522a4dd404@HIDDEN>
Date: Fri, 21 Jul 2023 18:41:29 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost> <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost> <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost> <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87h6pxl6m4.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 16:46, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@HIDDEN> writes:
> 
>> On 21/07/2023 16:34, Ihor Radchenko wrote:
>>> Dmitry Gutov<dmitry@HIDDEN>  writes:
>>>
>>>> On 21/07/2023 15:58, Ihor Radchenko wrote:
>>>>> Dmitry Gutov<dmitry@HIDDEN>   writes:
>>>>>
>>>>>>>> Disabling file-handlers is inconceivable in Emacs.
>>>>>>> Indeed. But we are talking about Emacs find vs. GNU find here.
>>>>>>> In the scenarios where GNU find can be used, it is also safe to disable
>>>>>>> file handlers, AFAIU.
>>> So, we agree here? (I've read your reply as counter-argument to mine.)
>>>
>>
>> We don't, IIUC.
>>
>> To use GNU find on a remote host, you need to have the file handlers
>> enabled.
> 
> Let me clarify then.
> I was exploring the possibility to replace GNU find with
> `find-lisp-find-files'.
> 
> Locally, AFAIU, running `find-lisp-find-files' without
> `file-name-handler-alist' is equivalent to running GNU find.
> (That was a reply to Eli's message that we cannot disable
> `file-name-handler-alist')

But it's slower! At least 2x, even with file handlers disabled. 
According to your own measurements, with a modern SSD (not to mention 
all of our users with spinning media).

> Although, it will likely not
> be usable then - running GNU find on remote host is going to be
> unbeatable compared to repetitive TRAMP queries for file listing.

That's right.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:45:01 +0000
Resent-Message-ID: <handler.64735.B64735.168995426821564 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995426821564
          (code B ref 64735); Fri, 21 Jul 2023 15:45:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:44:28 +0000
Received: from localhost ([127.0.0.1]:34572 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsIu-0005bk-9p
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:44:28 -0400
Received: from mout01.posteo.de ([185.67.36.65]:56355)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMsIn-0005bK-1r
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:44:26 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 235C3240027
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 17:44:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689954255; bh=hnsj4PIJpZgWLlL5oi0st/WosZwvA/G5bZBEWGeEk5E=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=Rt8Vd1xAQR3KvG2efiJDseOumolWV65Gz4KjCBfZOqwcVXyfj+wQ6HflnioSawUDu
 KXS7LMjpZ8gE4dx+31PNfKRNzLBmQjucP+d2eeYKezUwhFwKLKadsFu0cUefeyZP8o
 wLYhyg0k/Ye11vgTX7QKVkuO4K7LMpdZQ1GyQn8sCAU02WBljOArpe5wPni/wHlGZG
 RWg1e6ybVWjJeSc+y04kgEeZyKPu0eN8cysCOBvXHuQ+sNsnIK6Dix2C/kWlwjPAyx
 8hpM6USuNTQRs88mv8gNQY2+IJ7d7e3arCJoIaKZ9vXXTWzL2VHQ5S+qj6VHIbkUvK
 6GWSI5owA2WUA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6v4G2wYhz6tvm;
 Fri, 21 Jul 2023 17:44:14 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <875y6dcm4r.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87zg3pcuvv.fsf@HIDDEN>
 <87a5vpmopa.fsf@localhost> <87mszpct5n.fsf@HIDDEN>
 <87mszpl7n8.fsf@localhost> <875y6dcm4r.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 15:44:27 +0000
Message-ID: <875y6dl15w.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> I am sorry, but I still do not understand how what I proposed can lead
>> to any feature regression. May you please elaborate?
>
> When you invoke a file name handler based on the value of a variable
> like tramp-mode, either all file operations are enabled, or all are
> disabled.
>
> The mechanism with inhibit-file-name-{handlers,operation} allows you to
> determine more fine-grained, which operation is allowed, and which is
> suppressed.

I did not mean to remove the existing mechanisms.
Just wanted to allow additional check _before_ matching filename with a
regexp. (And I demonstrated that such a check is generally faster
compared to invoking regexp search)

Also, note that `inhibit-file-name-handlers' could then be implemented
without a need to match every single handler against
`inhibit-file-name-handlers' list. Emacs could instead have
handler-enabled-p flag that can be trivially let-bound. Checking a flag
is much faster compared to (memq handler inhibit-file-name-handlers).

Of course, the existing mechanism should be left for backward
compatibility.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:49:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995452822333 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995452822333
          (code B ref 64735); Fri, 21 Jul 2023 15:49:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:48:48 +0000
Received: from localhost ([127.0.0.1]:34576 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsN6-0005o6-71
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:48:48 -0400
Received: from mout01.posteo.de ([185.67.36.65]:51345)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMsN3-0005ns-Iu
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:48:46 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 09815240029
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 17:48:40 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689954520; bh=14Yz82ehhieUwDUN96d87bZ2yhNl3xXvPVhzN9KPkOw=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=Fy8eqpy2kZCq6EA5XtxRnfC59zYNa9aSENoLSHR6j1Hp8DWP4k0wWXgo3shRSRZ58
 bA02MgpbREzzmk+wTm0RgeKcz+hkEManwOzrbFbPHwFCZ+Px5X1dU8fy/Ht2xaJkFK
 yVka5gAnIGttH4uiXYDJWizX+uFIk6ATUErrYVimN0EFD+THeOPmNzbwTnOpp/TMUZ
 sAYavke7G5Nc4uJlR+ImHsAEyT++SI8ZiPvU3X0u4FYuLIVBKvLkQ2b1C/BY+SGt1U
 ZgPyhGxut7Je2Oncoti06zeAwmeOHmvkgkC6Inwn68EkYnsvlWZRgdx8xH2eMODRrX
 +2q7eDx3iLkXQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6v9M3t1pz9rxV;
 Fri, 21 Jul 2023 17:48:39 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <ea2332cd-ec57-794d-d662-ff522a4dd404@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost>
 <ea2332cd-ec57-794d-d662-ff522a4dd404@HIDDEN>
Date: Fri, 21 Jul 2023 15:48:52 +0000
Message-ID: <87351hl0yj.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

>> Locally, AFAIU, running `find-lisp-find-files' without
>> `file-name-handler-alist' is equivalent to running GNU find.
>> (That was a reply to Eli's message that we cannot disable
>> `file-name-handler-alist')
>
> But it's slower! At least 2x, even with file handlers disabled. 
> According to your own measurements, with a modern SSD (not to mention 
> all of our users with spinning media).

Yes, but (1) there is room for optimization; (2) I have a hope that we
can implement better "ignores" when using `find-lisp-find-files', thus
eventually outperforming GNU find (when used with large number of
ignores).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:50:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995457122422 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995457122422
          (code B ref 64735); Fri, 21 Jul 2023 15:50:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:49:31 +0000
Received: from localhost ([127.0.0.1]:34580 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsNm-0005pa-LX
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:49:31 -0400
Received: from mout.gmx.net ([212.227.15.18]:35533)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMsNj-0005pJ-6K
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:49:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689954556; x=1690559356; i=michael.albinus@HIDDEN;
 bh=3AIB1AI5v0K18NoKohqFeeXgcYtFMKryUIGZNx4Yhb8=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=opLyPz0B+cddl+P74OhcmKqN7Pm34hutYbGSbnEiPDPGYHdHzlqIN33ggI0b0ep0NRkBHuG
 37xaf13cqTlBZU+cRRiG0vOWtTiZAUw4yvI+7lLHWJsZOGSE1bI4VMhCRAHHseFmkOYJTv2va
 zODjkYzD3EVSnltfyr1nRQbxmKwlgSne11ZnTnp7x4cQy5SVtLtPCsQHw6mQr+E1GvPB4jtFO
 XLRj5w6DupQe69AH5XrKs+ElsLJjKb8x4N8MT9NbWJnKzTVCDGnSGJkKG7XPmrOA0mVLV8Pzy
 cOpWbfiWD+wPmfSRMODXCMEaN6Yp60rVcKIwBe/OC8EWFNS9q3oQ==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N3siA-1pwNER42b2-00zoGZ; Fri, 21
 Jul 2023 17:49:16 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <878rb9l1f5.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 15:38:54 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost>
Date: Fri, 21 Jul 2023 17:49:14 +0200
Message-ID: <87zg3pb6yt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:wJXoA08r1VTJ7kNDyJfoKYsmjrzJZOuFNzkJ5fQs9BGi1Z+Bo7n
 PIVd30+sLN5XYn0J5stFgKbHWI3o81xqSktWZ6Z4K34MJ5C/JNSi7lnvRbcocclyJw1fXqo
 xIlzkO1T/0hb5ynvyO8bmU4UzwfqusDjG1glLJ6TeuqP4oI5Na0/EnkrSRHcZU4myHJmXI5
 V81c4N5d9p5V4vHB/SEHA==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:2DZ0apATdA8=;X4qjKf6dko7WBESta2Pn5vYMfKG
 bXRmByMLciOAKm3zU52qyg1ov2wNGhSgQDuX5nz6eOuHE2mbBjjwJ1CPqBLB+lRRoiseDUHXg
 REdH9s2dnxUN5fBDsPOa5xHGbGQ4YDQ0Iu02h8shwQ0O4+ndS5ZAwfRAeOpz/BUS2xtkPGDHA
 OnPfveiFbKcsWHA8zI8tH9znfD0U2xU6z1QyAf6OC9MbVzKy+ZxbCm5Dmv+FZPebl8TpLa6mq
 CKgrJWFCwMKwxMJXq7n+naUupWWTVMa2QpYuz5WkF+rlvW6KBhCGxm29Rl3Il5/xFuWUbTg1d
 DUuGO0pxt/YK4bURKXb6z3xLPqiyIB9R+3DL9fPJ3+yCkwx7MWF4MvrRkbwDO31MtlBykAOyV
 vvDw9qiOcSPLyQlfjZX2nUpa4lIt/1OCaKylbIlpcFXkf58p/yygc6eBTVyY7I8w1UlSk4nF4
 MGHSBUkI7mfMRF1Ktf8LxP+CWZo3iabl+XkvsYsQEGQeXsdgs/QSZCRO1SfXBkc5XIBpDjGBW
 9kKh2COFWjNdIwSVbIkljM8TQZOdrYT/fFe+YC4xjm8WkXwSfsSDY6G/89vefNedn66I/H6vz
 oYb4NWj+xih8rS1ti6m0OieK9r/4a7BUlGLNotU6dSqgGRFXe8TG4d5kt047MtZJQ3N0xYLzN
 ymhCBSzostt3567qwKTcG/XLGDK/8Q7CBXotM4q1s7oXG2HvhPQstzBFQnZ8Ia0koFW85CRpv
 AgTQi9wc3a9InZ/XT5QtXgjElcTnLTecCGmOUmt1d65W9GApRX79PTWyZGRS7jJvzZhF4xsy+
 2XoJpum9FnZdO23Z+hZ4MpG5i74qww+vNexxOJRC7iwa6P6SP4Dg+XG1TfC3Hv9nxmOjlM4Wy
 L5KJZUpIt5nAE4c8p3yCH63JrhuzXKEIzq1yVel6uAJyixQybc+cfpHc9xm41pwpJH9N167/C
 R4ENrg==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

>> My point was to show the differences in the approaches. Do you have also
>> numbers using without-remote-files and inhibit-remote-files?
>
> (length (directory-files-recursively "~/Git" ""))
> ;; => 113628
> (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
> ;; => (1.597328425 1 0.47237324699997885)
> (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/Git" "")))
> ;; => (1.0012111910000001 1 0.4860752540000135)
> (benchmark-run-compiled 1 (without-remote-files (directory-files-recursively "~/Git" "")))
> ;; => (1.147276594 1 0.48820330999998873)
> (inhibit-remote-files)
> (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
> ;; => (1.054041615 1 0.4141427399999884)

Thanks a lot! These figures show, that both without-remote-files and
inhibit-remote-files are useful. Of course this shouldn't stop us to
find further approaches for performance optimizations.

I'll wait for some days whether there's opposition, before installing
them in master.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 15:56:01 +0000
Resent-Message-ID: <handler.64735.B64735.168995493923018 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995493923018
          (code B ref 64735); Fri, 21 Jul 2023 15:56:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 15:55:39 +0000
Received: from localhost ([127.0.0.1]:34597 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsTi-0005zC-MM
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:55:39 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:40948)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMsTg-0005yy-3g
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 11:55:37 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMsTQ-0003RM-3z; Fri, 21 Jul 2023 11:55:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=Ttl1E6ptYzqju6Jetua3CqswqmmOTyheJ1HRao74IT8=; b=F1jOrrqkw4i7
 dZr2tHQ6SBjl+hewiPuasfLSoAFOs0zzybPRLDkaIFkSbP07ps+iRV8wlow5Stk4IRcsDSX3hkvuM
 dbYsSoaE4Irul5skTwBxspZlvt3P+HakrhKYSSv/yXSHpdqAuT66d1S/iwtdg7R/k/DxGph38s6gv
 dVVQpuJ++bEvw/Pk4PcWGwc1eBLxMUrA7YfT4fb5HeTzh4fwU0aDpgPko0JN4860TXaDqSN5LnSyb
 Msf9sePSN2qjQHGAiNOgn1Gnmzx8iffqd4UbeXxoyV0vkakiknwi4V7IBJrW7z4AoyRZiijZo8AHC
 vz4CDwoG0/xCyvDbaNsf1Q==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMsTO-0000WQ-1H; Fri, 21 Jul 2023 11:55:18 -0400
Date: Fri, 21 Jul 2023 18:55:54 +0300
Message-Id: <83zg3p9s39.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87zg3pb6yt.fsf@HIDDEN> (message from Michael Albinus on Fri, 21
 Jul 2023 17:49:14 +0200)
References: <iermszrwqj4.fsf@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Cc: Eli Zaretskii <eliz@HIDDEN>,  dmitry@HIDDEN,  64735 <at> debbugs.gnu.org,
>   sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 17:49:14 +0200
> 
> Ihor Radchenko <yantar92@HIDDEN> writes:
> 
> Hi Ihor,
> 
> >> My point was to show the differences in the approaches. Do you have also
> >> numbers using without-remote-files and inhibit-remote-files?
> >
> > (length (directory-files-recursively "~/Git" ""))
> > ;; => 113628
> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
> > ;; => (1.597328425 1 0.47237324699997885)
> > (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/Git" "")))
> > ;; => (1.0012111910000001 1 0.4860752540000135)
> > (benchmark-run-compiled 1 (without-remote-files (directory-files-recursively "~/Git" "")))
> > ;; => (1.147276594 1 0.48820330999998873)
> > (inhibit-remote-files)
> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
> > ;; => (1.054041615 1 0.4141427399999884)
> 
> Thanks a lot! These figures show, that both without-remote-files and
> inhibit-remote-files are useful. Of course this shouldn't stop us to
> find further approaches for performance optimizations.
> 
> I'll wait for some days whether there's opposition, before installing
> them in master.

Can you spell out what you intend to install?

The figures provided in this thread indicate speedups that are modest
at best, so I'm not sure they justify measures which could cause
problems (if that indeed could happen).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 16:09:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995571024322 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995571024322
          (code B ref 64735); Fri, 21 Jul 2023 16:09:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 16:08:30 +0000
Received: from localhost ([127.0.0.1]:34644 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsg9-0006KE-IX
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:08:29 -0400
Received: from mout.gmx.net ([212.227.15.19]:51917)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMsg7-0006Jx-EZ
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:08:28 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689955695; x=1690560495; i=michael.albinus@HIDDEN;
 bh=i56p7OaV5d+8wLROiMiof/k1K+E6k0Y+/BJk4dUv2SM=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=GFDqnHCAI2xAYKCSUaakDkwdKX6sp9nevWyu8q21uc2wRFSYKn9vm9dzmKg8INvYww3OAM4
 l2d69alue1SsZ71Ob2H9OT4u9gEv5s3vz7agmBGDItBYNxo1IxKsc9iAYaU5q2iQo7kQ3mF1p
 MpeivcLXUL0NCb4X4bZi0zv2HOHxnl+VxKPsX5OxmwUbhFgaztZdwdwPHODE7dRYc5FhKRwdI
 EEpfAYsU/ym6a8Lq4TP1IRp3fI76muw1/5Y+lIzSChwUq5DfMwBXo/exVP3ZHrvsEFWAtLQ2U
 Yy7tEqXO/sz/29VPTilFPIcIGpiU9VdNGpswSMk7LfewbojExK3w==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M26rD-1qKwjO2mM5-002WQl; Fri, 21
 Jul 2023 18:08:15 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83zg3p9s39.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 18:55:54 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 18:08:14 +0200
Message-ID: <87v8edb635.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:S5cxs2b/fHV5MePDrlvge3joUI0ZN4SaKncPOSaWddIY3MtwAsd
 9z66YVfkYExDiuLkcA/snL+Pc3BA0Tjg6/M8zAhY1gVa1Frd55KX5EK2dmDG6rkVI0qiW5K
 M5RK5JPHT9u5Yd013azVcAsxAhSW++6hOKifDeZD7oRGUTmaYAxXqr5kD9dCoaPuYvdUnLA
 FQCoKzPd/IUPiZKedutrg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:s9Y0A2U2xvI=;R4MPshp++0+Ho3ys5ffaLRiRRU1
 PY17MSZ50DOmogQ2b6/D2m7lZqScJLoMaWnNfyggSYh8RaL9wtv4YY85I0S0ZNkRFVRj+sRfJ
 /CDvV9Krwp2toxGATEbytsFsX6eXq7eERrK2VL1D5eu7hZ/bwDUhIk2wLUnxXWnoaiJaa3Ucs
 TfzoWkJSrtrHepN2dVat0UGyTcHj1QZiljvuEOo0/Bbf/yes7VFdF+1HWHAmOQIHSz28suYoO
 6GdmS71gjc3cq7kdFDOMTg4+zRTd5jePbYuKd8GDLwkrHO8IX3aF0i/AirlKw2qnpURhCa7A+
 ye3AaYxmN3et+fIXvAhZqATfH2+58qPu27o9Gp+7OWkwWVv3LtxL/JwZJQTJkEIDglh0nGOR6
 LuTty/jp3ZUZjb6my0U+NSPU8tmRjgin98kKU106tK/rRDCycj38W3xB/fvSU8bB8/UK+btdY
 lrj3kwxbd3Pucc6mGQ6CJ6jgpheSvFBJUcJ04lUYBzPTOO9OdponBUQBrRVtIQxf/BbE0f/Ks
 mj52w0lie2Ekoc1MMgi5zzzo3UIMkZ4O9a7LcXv61tSMicUQzczysR/K5ke9qbGMD4R7Qle96
 u/gQ7TvEzhGeUZa3T8T0fSdbrONff4ebLAAXnM/0HzrfSKicvUwhWJoczzYiZ5F9o48SATO67
 XUyOq/akgB5hfz2srFjQuFwZEwBt8LHoTXPg+Aml4aK2u732blQE1K3rgN/PfzsFImOHqGffM
 IW2iOq7UzM+sEtVIiWhYoH9sQu4WxpGVN//CMPlmdfK50zPrGqYTqXTnWl8wNr3IUpTVIOk+6
 LMi4pKAFXeXru+lz4UtZeEoW6quzneewHYYF5sKVRJOQwuhnGex6kSP0ztKJv9hGYgIu4madF
 en2cBbcYwKfUxZD9sp5LgHms5WUm8A1XOPm8rglykRB+u4PVYAi/gwHJ7J9LTeegj3/asyTRr
 /T15dw==
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

Hi Eli,

>> > (length (directory-files-recursively "~/Git" ""))
>> > ;; =3D> 113628
>> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
>> > ;; =3D> (1.597328425 1 0.47237324699997885)
>> > (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-f=
iles-recursively "~/Git" "")))
>> > ;; =3D> (1.0012111910000001 1 0.4860752540000135)
>> > (benchmark-run-compiled 1 (without-remote-files (directory-files-recu=
rsively "~/Git" "")))
>> > ;; =3D> (1.147276594 1 0.48820330999998873)
>> > (inhibit-remote-files)
>> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
>> > ;; =3D> (1.054041615 1 0.4141427399999884)
>>
>> Thanks a lot! These figures show, that both without-remote-files and
>> inhibit-remote-files are useful. Of course this shouldn't stop us to
>> find further approaches for performance optimizations.
>>
>> I'll wait for some days whether there's opposition, before installing
>> them in master.
>
> Can you spell out what you intend to install?

I intend to install without-remote-files and inhibit-remote-files, which
I have shown upthread. Plus documentation.

> The figures provided in this thread indicate speedups that are modest
> at best, so I'm not sure they justify measures which could cause
> problems (if that indeed could happen).

>> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
>> > ;; =3D> (1.597328425 1 0.47237324699997885)

1.59 seconds.

>> > (benchmark-run-compiled 1 (without-remote-files (directory-files-recu=
rsively "~/Git" "")))
>> > ;; =3D> (1.147276594 1 0.48820330999998873)

28% performance boost.

>> > (inhibit-remote-files)
>> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
>> > ;; =3D> (1.054041615 1 0.4141427399999884)

34% performance boost.

I believe it is more than a modest speedup. And without-remote-files
mitigates problems which could happen due to let-binding file-name-handler=
-alist.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 16:16:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995614225082 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, Michael Albinus <michael.albinus@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995614225082
          (code B ref 64735); Fri, 21 Jul 2023 16:16:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 16:15:42 +0000
Received: from localhost ([127.0.0.1]:34682 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMsn7-0006WU-Vj
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:15:42 -0400
Received: from mout02.posteo.de ([185.67.36.66]:43329)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMsn5-0006WG-4K
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:15:41 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 634B5240101
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 18:15:33 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689956133; bh=RN4+cEWqYGuroyNt2YKaOielEQOFdXNB3WmdrAO3nrU=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=qvrCFkCp1z8/wldLRLrTjfiZQzjay/H1jGL1fuJ3tmWqpkM4vxUS9iYjakTnnikL1
 na++iKE6r6AePtT5NolYs3UDN6QD+If1xojw+HHx3Bsz0XZNzjgC8vLPv7sxmN2JCO
 bX/P90XBCJ1xzfO+YGBcw0qzGBO5aIJBh03M4VfElTKfUmPFLLmeJqp8k25WT+IPP+
 tmY6YuY0CNsPRk9R/rovasmeBGDU5ID7A9OFz9E+gzO3b9BApTq401/4DI59gjE/9X
 i5siw5gx+oZfmdELA8wdQ7MoRGIy7VMUZezWnXTD2RhgGp3CSjeHz3S1dbZRtFMpFo
 GjLE4A9MdFEAA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6vmN1mdCz9rxY;
 Fri, 21 Jul 2023 18:15:32 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83zg3p9s39.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 16:15:41 +0000
Message-ID: <878rb944wi.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > (length (directory-files-recursively "~/Git" ""))
>> > ;; => 113628
>> > (benchmark-run-compiled 1 (directory-files-recursively "~/Git" ""))
>> > ;; => (1.597328425 1 0.47237324699997885)
>> > (benchmark-run-compiled 1 (let (file-name-handler-alist) (directory-files-recursively "~/Git" "")))
>> > ;; => (1.0012111910000001 1 0.4860752540000135)
> ...
> The figures provided in this thread indicate speedups that are modest
> at best, so I'm not sure they justify measures which could cause
> problems (if that indeed could happen).

Not that modest. Basically, it all depends on how frequently Emacs file API is
being used. If we take `find-lisp-find-files', which triggers more file
handler lookup, the difference becomes more significant:

(benchmark-run-compiled 1 (find-lisp-find-files "/home/yantar92/.data" ""))
;; (3.853305824 4 0.9142656910000007)
(let (file-name-handler-alist) (benchmark-run-compiled 1 (find-lisp-find-files "/home/yantar92/.data" "")))
;; (1.545292093 4 0.9098995830000014)

In particular, `expand-file-name' is commonly used in the wild to ensure
that a given path is full. For a single file, it may not add much
overheads, but it is so common that I believe that it would be worth it
to make even relatively small improvements in performance.

I am pretty sure that file name handlers are checked behind the scenes
by many other common operations.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 16:38:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995746627265 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995746627265
          (code B ref 64735); Fri, 21 Jul 2023 16:38:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 16:37:46 +0000
Received: from localhost ([127.0.0.1]:34733 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMt8U-00075g-9w
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:37:46 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:33036)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMt8O-00075P-Sd
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:37:44 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMt8H-0005CX-T7; Fri, 21 Jul 2023 12:37:33 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=j63R0ElzseoRPxTXKoYo4/bMg1JDMiOJ44bgcz5ec6M=; b=iw4lcbWZBR8e
 /Me8xe4O+DxrigjpPDMvL33PIRqGY0KGIBHp60XbUT86kdLp0kBWv/GWSifUA+8dsGvQehJWHZi1q
 DryC8/jhshXeG6hjj8Z+z3BGjYmEcbqUh0nDREyM7IXiHhWwGnc6EQiQWwQo3UCSi7WX6KU6ZDgqW
 IPToDJOhNfI8y4fYBe8CR8ZRDplw+GSI+V7CSaSh+sNzpj1vWDjb5pqBrUWKGW1W1YVQZCHpM2nJm
 IC5zInziSA5k39Viz5ltIOyssbSVP2K1aiUQssdli1zLJLBbw2FEv/9oU5iOVV9Em9/NNUO9p0KJz
 1auBYv346ntoV2z5JnmHnw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMt8H-00066l-6s; Fri, 21 Jul 2023 12:37:33 -0400
Date: Fri, 21 Jul 2023 19:38:08 +0300
Message-Id: <83tttx9q4v.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <878rb944wi.fsf@localhost> (message from Ihor Radchenko on Fri,
 21 Jul 2023 16:15:41 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: Michael Albinus <michael.albinus@HIDDEN>, dmitry@HIDDEN,
>  64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 16:15:41 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > The figures provided in this thread indicate speedups that are modest
> > at best, so I'm not sure they justify measures which could cause
> > problems (if that indeed could happen).
> 
> Not that modest. Basically, it all depends on how frequently Emacs file API is
> being used. If we take `find-lisp-find-files', which triggers more file
> handler lookup, the difference becomes more significant:
> 
> (benchmark-run-compiled 1 (find-lisp-find-files "/home/yantar92/.data" ""))
> ;; (3.853305824 4 0.9142656910000007)
> (let (file-name-handler-alist) (benchmark-run-compiled 1 (find-lisp-find-files "/home/yantar92/.data" "")))
> ;; (1.545292093 4 0.9098995830000014)

The above just means that find-lisp is not a good way of emulating
Find in Emacs.  It is no accident that it is not used too much.

> In particular, `expand-file-name' is commonly used in the wild to ensure
> that a given path is full. For a single file, it may not add much
> overheads, but it is so common that I believe that it would be worth it
> to make even relatively small improvements in performance.

The Right Way of avoiding unnecessary calls to expand-file-name is to
program dedicated primitives that perform more specialized jobs,
instead of calling existing primitives in some higher-level code.
Then you can avoid these calls altogether once you know that the input
file names are already in absolute form.

IOW, if a specific job, when implemented in Lisp, is not performant
enough, it means implementing it that way is not a good idea.

Disabling file-name-handlers is the wrong way to solve these
performance problems.

> I am pretty sure that file name handlers are checked behind the scenes
> by many other common operations.

I'm pretty sure they aren't.  But every file-related primitive calls
expand-file-name (it must, by virtue of the Emacs paradigm whereby
each buffer "lives" in a different directory), and that's what you
see, by and large.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 16:44:02 +0000
Resent-Message-ID: <handler.64735.B64735.168995780927860 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995780927860
          (code B ref 64735); Fri, 21 Jul 2023 16:44:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 16:43:29 +0000
Received: from localhost ([127.0.0.1]:34756 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMtE0-0007FI-Nj
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:43:29 -0400
Received: from mout02.posteo.de ([185.67.36.66]:58499)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMtDw-0007Ew-TC
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:43:27 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id F4062240103
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 18:43:18 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689957799; bh=cdXSD+R+tj+Uy4EfRAZSd/RBzmT4HE4W8XTDPRozTEM=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=YxRWId09knQZngm75QHnD89Ya2o4PrOXIBIH97Gx2+ZXDeZeWgL7kks6lT/TbaqlA
 HAFXOM0ZNdg613jF9imoAC8KjgfoS8NWUAr2nHH19A5iD68vHUiozIcdIFZXGuJ6Dp
 jOQrgYV25r3ZL+5+aBkzMvxc63ixoM3xuAPL9ohJHR6WGX5Iy7mDeW9b8Rn0k+WtOO
 QzVDkLM22yjdg2VWfScIXdsgq8Cq0hqOKZ5l8vGLMf1R9nOPdI+j361u1qkH4fQG0y
 HAqht8iU0IVf/+becNU1O6Exw7mTxyTxaz41qHx9wZR2cZ/EfPJy28L/U35IQd8g7w
 kh9x+B+agfeQQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R6wNQ3MCNz9rxV;
 Fri, 21 Jul 2023 18:43:18 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83tttx9q4v.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 16:43:31 +0000
Message-ID: <87fs5hp64s.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> Disabling file-name-handlers is the wrong way to solve these
> performance problems.

We are in agreement here.
Note that I am talking about optimization.
And Michael proposed to provide a way for disabling only the
tramp-related handlers, when it is appropriate.

>> I am pretty sure that file name handlers are checked behind the scenes
>> by many other common operations.
>
> I'm pretty sure they aren't.  But every file-related primitive calls
> expand-file-name (it must, by virtue of the Emacs paradigm whereby
> each buffer "lives" in a different directory), and that's what you
> see, by and large.

The end result is the same - file handlers are searched very frequently
any time Emacs file API is used.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 16:45:03 +0000
Resent-Message-ID: <handler.64735.B64735.168995785127953 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168995785127953
          (code B ref 64735); Fri, 21 Jul 2023 16:45:03 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 16:44:11 +0000
Received: from localhost ([127.0.0.1]:34760 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMtEh-0007Gn-3k
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:44:11 -0400
Received: from mout.gmx.net ([212.227.17.20]:46681)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMtEe-0007GJ-3O
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 12:44:09 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689957834; x=1690562634; i=michael.albinus@HIDDEN;
 bh=WNc5Z1pDrLwFNrhF2Jo6gqIxSJpwW9GkdAW+fFYuvP8=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=USTL/jD625nXl7UoDiPC1UWGbMmkWdOFuFrCoempWD11nVb0SaHLDRvZYkiC4BSw9eoq5ZV
 uzznbgtc96AjLneA85VIMy5828Y32zXL5aOHoHSxEIuu8J/97LQyai6GJiAO7SNt3BYmsNvLN
 be0mL4K3tpSPw87UWkwszRucK6pZZ8aElXTQ3OFedhugvr0ybrTXelMuubx9Md4LamsHk90X/
 ww0lzYn6iIjGK6v+0hae5CoYuc33Y7ERYszFWC9/TY9MQErOxpfTNpEWUhjHjbOmX6gimhPow
 XrfQGB6Fvc11x3r42fuUgK0i4Fbl+d05Hoj4uVZ2lgTaIPlqBbMA==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N3KPq-1pvq8W0xzm-010OqC; Fri, 21
 Jul 2023 18:43:54 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83tttx9q4v.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 19:38:08 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 18:43:52 +0200
Message-ID: <87pm4lb4fr.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:OGPJuh3/48G58nYpZP0GZZFck2t/YaUVeSvJYgik1C47Jnwk5ZQ
 3zyTfgZccPoov8SicmZhVMuY3VPqfabhE6lu9iWcKbr8lqDsZYroBda1xEawDjtJurh9aN0
 ZwU4zBO4nJhxpoCfNJlspmOVgj5nBldXlTezABKy70w7lUaOXTthaxqA/+K77hpFU5p+3/9
 NjgEqzapiXghTFaqBT6+g==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:g+Hdt2PwdYg=;4zwIPQxifJi/Db8qv1byuvZTD6o
 vPp41yCPyMFqUMz+z82wq/2si6HlXQSPVI6mG+kqaO28fzJj08PGk1YHDj3OCNqR8sIAwynIo
 tWApg4xHokYE01YMttcZp5GGZLrkx733rYZ0kbGhDgJSoxtfxqlPPPMhQHfcTSVG6hZGJxb9k
 +8zwUG8mY/GCCNz8yPVHD0e1NE9c+B2/RaTH+bK5oNaEwck7j7a6caI0l0ZUBl+pjBufRxKKq
 5MX9MQoxfVu+B0zm+qc0QnfzPP1OSUhGqeJZP5ZT7QaL8SHNOjqCycsq/JiEDxZY2y0BINazL
 kFHPBSGA/Mb9nhSKEpR4qvLwBOdp7NwOYFGSRykHRz35L28ETt0mBfyIOZgnVb+yoH5g5cmej
 J+U6/aiER4QNFJCpq5ZTyjcAr+kc4u7mNUusz35PM6REW9tdRmL0qdai+imy1HUtjzolHv/a1
 keYnxlJxIXXMyaGIqmWGSd9kp0akJZD72LyoBRiCo2zPormf/3UyKKHoihVdgFJxXAaHl1U4w
 //KVbiTXUwpgIT7YHDSA99xtd/ssSxM2NH/xijPbObRO/977BjuxExOwtNeseGDlG6NPJWpCW
 Ft4nWAhvBEEqsSr6LHWj+K3YQEaYpt5RzVnhCwd+5oCMY5Zh3V/SGHVNdKApuwSDiCViF8nMG
 6IsW+xiiGwnUqwqu2VHVikMfOVKHPqmb/euIim9P0v58pR820epByUjicWnz4/m27cJqUDlea
 e6mOWMbxpnxAF8ab56whbTepfIMPFsAe5CD+ZqXpyHtxiTcrMfJWhk/dztQhkYsDFoL6a7+XB
 QxpiEQ3MK1M9IAemMp4wxytxm2pCiOzw96KFVcVwKke8NKgWVHNTd7TzHjg8QfU5KLvjY4E3x
 1jbaZSK3doa7BVlwdNeHtdEradC3W6D4YPp2b2iKVcY2FMTEKpB3R2UP8DJ/WC/TYiSHhu6Wu
 ScLU1fglD83ROgbp2MgPs9qCaxk=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

Hi Eli,

> Disabling file-name-handlers is the wrong way to solve these
> performance problems.

Does this mean you disagree to install the two forms I have proposed?
Although not perfect, they are better than the current status-quo.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 17:46:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899615032228 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899615032228
          (code B ref 64735); Fri, 21 Jul 2023 17:46:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 17:45:03 +0000
Received: from localhost ([127.0.0.1]:34819 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMuBb-0000Zr-7J
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 13:45:03 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:38064)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMuBX-0000Z3-GP
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 13:45:01 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMuBR-0001hd-C7; Fri, 21 Jul 2023 13:44:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=5sHJSDe6ZjHMkbtphaUN/q7HV1JVVPWj2jzdv5dCO48=; b=NzzL0FnIsEAZ
 w/H0rBFOKd/+SxJxrb3/IcDDwUVZVG/vxRv1sfJwkjbSpuq6ZOw+luDXe1bf7+BwUvGsxsll+7ZrB
 3Fo/ATs+WybgurppskzA3V2MyUGy2AHaMEVqj7M03hHc5i9AV8s//pJ3ypkD1/0H/iP6dugUWZ5f1
 r77XPVbeTJzjbgEQQlSGl6RCng33us0JTSnYW2/o5YQwD/2BkWhdjnvVf0vplrXKQKDxs98pLcQ07
 IiPywb5iu6FVYhoNGqH/4EViy6gXFY3ToVkZR5v5D2HKR24/oEob1Nlm7Ez1VU9uYr88FJ8ERlSgz
 sTkiiu7T/YCbbChiqWvnrg==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMuBQ-0003qY-S0; Fri, 21 Jul 2023 13:44:53 -0400
Date: Fri, 21 Jul 2023 20:45:27 +0300
Message-Id: <83pm4l9n0o.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87pm4lb4fr.fsf@HIDDEN> (message from Michael Albinus on Fri, 21
 Jul 2023 18:43:52 +0200)
References: <iermszrwqj4.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Cc: Ihor Radchenko <yantar92@HIDDEN>,  dmitry@HIDDEN,
>   64735 <at> debbugs.gnu.org,  sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 18:43:52 +0200
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > Disabling file-name-handlers is the wrong way to solve these
> > performance problems.
> 
> Does this mean you disagree to install the two forms I have proposed?
> Although not perfect, they are better than the current status-quo.

No, I just disagree that those measures should be seen as solutions of
the performance problems mentioned here.  I don't object to installing
the changes, I only hope that work on resolving the performance issues
will not stop because they are installed.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 17:56:01 +0000
Resent-Message-ID: <handler.64735.B64735.16899621373526 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899621373526
          (code B ref 64735); Fri, 21 Jul 2023 17:56:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 17:55:37 +0000
Received: from localhost ([127.0.0.1]:34838 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMuLp-0000uo-FF
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 13:55:37 -0400
Received: from mout.gmx.net ([212.227.15.18]:50549)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qMuLm-0000uX-KN
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 13:55:36 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1689962124; x=1690566924; i=michael.albinus@HIDDEN;
 bh=/L+LiFIUDjAWfV3Q0X0jByP4OL6MKkRgeMa0hyzz/t8=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=ESg6Z2vwiX60T95vvtiB1oH1t21Tlx5ZyJq9oNRi3gPdZPlg7DHgGKVtLoJgCoIEbHaIykI
 lexp0IA9ttD5xNFQ6uPRnJsUvNtcRAzlKdgCdG6TSNSviHjt5tyL7AxW5ZEiVjv2KpaW5a/Pn
 1VqrzY3wZzqG/x/UJySS11YGw+uqorMmuo0apU2piSN5ET4HJPQqCuQxOfA/FfuU8Tw/dXauR
 w+OUK+9R6HmVWTAm6A254V5DxVpExBFu5FDvhHK20EvnQQLTQ4oZwJUPpifWgq8ZhzJ7boPZe
 5AtRFNGae45typ7ZD59KaDnG3kWkBSclOGmUZ+jC/vGBLFn7H2fg==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx004
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MFsZ3-1q8OA83Uel-00HKPF; Fri, 21
 Jul 2023 19:55:23 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <83pm4l9n0o.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 20:45:27 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 19:55:22 +0200
Message-ID: <87jzutb14l.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:fb479R4z+XEMOEARPv/3zBDiJ9aYxBCHsZTlb3gGS9TApjEbiNx
 zxGXP84difRSTKd7wPqYVdK96YapA/Z2ULRmcupm7SU3SpM8uhUrQKwQVPumWLNdYbnLLnv
 8+pbo8M8lbAuDWCkaH6MwfA2MdG4jp2g6pqm+CqVZZO3rY1EJ145QJUEUcE2rHNawEOL+jB
 IT92Mi9yfTI/edBnvk2Mg==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:6Ydkd2DPM/g=;mvO9zL+qMDJXL5xU7IhS1qpzFY4
 sFVqVk0mzloFarcuvnzjWOFqKr6zPPMP+FPeeKGnhPHRLKgcOPHtSOFnvzfA3FmVUdchzUcMp
 l2jFHC5niXMT8IZVAwRVvUVS5weoxdq/mAWBSTMvKFyiK3pwADMtN4OhO22ZbZ64j0tV29E9w
 4vQz48Ft/Gf0p4Sl9O3VaGNyIuBypDYx7lQAXCwy3Hj0/NcUv5eQQJNtKYyg8v5+QjERqBNRG
 Us4/CVsli9HGCHkW4GaTGdkr7hesXJYx/EntShwB5pL9JbG8ax9BKBp8tesKMAHInRh8ZKLMP
 0W7SsgVr92J9aFpnp9B6PG0lu3OHTcTy5bZ9clcHIwYKmYJ0LesQ8nMn7ULh4zOvSLeUsSRj5
 bHxI5lCVkx/BvkDTmqudFpc28fDO9s1zLZUMR5G5rTNMBlOhnT4qyHxfhlW76Cz8sPyu2ZFyV
 EIJbw0X7Pyyn5uL+psVHSjl7JSeZE64H3Fzv73uJIXFKK7nbKj40hVYCksyldNkRkvSLuxhYS
 6XfkvXxLbeUrjr+smxLUCexWY1/19jTev0Soc9ANyH4u6qz38rVFQkQJZJIqKUVGkAbTbioLe
 Cy2bnyeKjleQwLQx4EnPBaez569zEAb7km7xeF9ypXVR9yKzReGZ6LfToAmMmwO9VWrnOdhBN
 +R5MVdc3f58zQt5W5N4cGNtgX61sk48tNqp/Bzbby4hO33PFKrqvfWxlrbomh3z4lnAUgPuud
 uafdaIUwZT92IiBKoEGSFoEMxWw5/K+DGrZAO3Nx83H+NUZXQxkBGQBPyM1PNYJt8WBYU6JM4
 A2heYrG8+kmB3MRcgugG8R7tAT5zS4gFCNIWLdNAj19esW/ux6IDeF+MAYVyEaJDjtL6C46jT
 RoCEW+fS2NhJlLKg3X/z6qUTne87NVl4tGCAwHsJkxoUkKyyaTfpqmjf+tY2dQ4r27GF8L4Nq
 MsbJSo2kAAyV36INOkt9Y7vyWwU=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

Hi Eli,

>> Does this mean you disagree to install the two forms I have proposed?
>> Although not perfect, they are better than the current status-quo.
>
> No, I just disagree that those measures should be seen as solutions of
> the performance problems mentioned here.  I don't object to installing
> the changes, I only hope that work on resolving the performance issues
> will not stop because they are installed.

Thanks. I'll install tomorrow.

I'm open for any proposal in solving the performance problems. But since
I'm living in the file name handler world for many years, I might not be
the best source for new ideas.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 18:38:02 +0000
Resent-Message-ID: <handler.64735.B64735.16899646547350 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16899646547350
          (code B ref 64735); Fri, 21 Jul 2023 18:38:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 18:37:34 +0000
Received: from localhost ([127.0.0.1]:34864 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMv0P-0001uS-RG
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 14:37:34 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34024)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qMv0N-0001uE-5V
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 14:37:32 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMv0H-0003Lu-Hm; Fri, 21 Jul 2023 14:37:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=j46TbXnEu0HRty5w3f4yCJVVKLZpOgHI4UAKN4Cc1iM=; b=LTUOIXyBwiTU
 7mfsrFQOpet03PfP28l5+EOi+/DitX6FfgKvzj9dKtjUWcbc7kTvu9F2ruEVzcArBzS4NDPKizpj8
 QZxufeyyHNi3fgfmmtDiJKyZmkBpeJRpxaTtO1IUxysdk8KR86rLXc1+gp6WJNSYOnehsCOhzbMiU
 i1vbzxqCOIA6vZLd/vMwT5IYRNg1hbj43HhrxbbS4vV4RTk/xDfg2SDD/9w/OCOW+DXMAAcel/0W8
 +H6d0A/CQanWwfgYsnsCGsPRmJdDEvPjTm+OM94t2L4MXs7AEtYKlZn1nAcj5pU/xWfBnsS36aDLp
 yzmq7oc1b7u5giKkhiuBuw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qMv0H-0000Hj-1r; Fri, 21 Jul 2023 14:37:25 -0400
Date: Fri, 21 Jul 2023 21:38:01 +0300
Message-Id: <83mszp9kl2.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87jzutb14l.fsf@HIDDEN> (message from Michael Albinus on Fri, 21
 Jul 2023 19:55:22 +0200)
References: <iermszrwqj4.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Cc: yantar92@HIDDEN,  dmitry@HIDDEN,  64735 <at> debbugs.gnu.org,
>   sbaugh@HIDDEN
> Date: Fri, 21 Jul 2023 19:55:22 +0200
> 
> I'm open for any proposal in solving the performance problems. But since
> I'm living in the file name handler world for many years, I might not be
> the best source for new ideas.

The first idea that comes to mind is to reimplement
directory-files-recursively in C, modeled on how Find does that.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 19:32:02 +0000
Resent-Message-ID: <handler.64735.B64735.168996790612691 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168996790612691
          (code B ref 64735); Fri, 21 Jul 2023 19:32:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 19:31:46 +0000
Received: from localhost ([127.0.0.1]:34924 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMvqs-0003Id-A5
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:31:46 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:37927)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qMvqp-0003IK-Bc
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:31:44 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <87ilaepkaw.fsf@localhost> (Ihor Radchenko's message of "Thu, 20
 Jul 2023 17:25:11 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87ilaepkaw.fsf@localhost>
Date: Fri, 21 Jul 2023 15:31:37 -0400
Message-ID: <ierbkg5ukme.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:
> Spencer Baugh <sbaugh@HIDDEN> writes:
>
>>> Sure. It might also be optimized. Without trying to convince find devs
>>> to do something about regexp handling.
>>
>> Not to derail too much, but find as a subprocess has one substantial
>> advantage over find in Lisp: It can run in parallel with Emacs, so that
>> we actually use multiple CPU cores.
>
> Does find use multiple CPU cores?

Not on its own, but when it's running as a separate subprocess of Emacs,
that subprocess can (and will, on modern core-rich hardware) run on a
different CPU core from Emacs itself.  That's a form of parallelism
which is very achievable for Emacs, and provides a big performance win.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 19:34:02 +0000
Resent-Message-ID: <handler.64735.B64735.168996800212846 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, Michael Albinus <michael.albinus@HIDDEN>, Richard Stallman <rms@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168996800212846
          (code B ref 64735); Fri, 21 Jul 2023 19:34:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 19:33:22 +0000
Received: from localhost ([127.0.0.1]:34928 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMvsP-0003L8-OY
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:33:22 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:34453)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qMvsM-0003Ku-KZ
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:33:19 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <83mszp9kl2.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 21 Jul
 2023 21:38:01 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 15:33:13 -0400
Message-ID: <ier8rb9ukjq.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:
>> From: Michael Albinus <michael.albinus@HIDDEN>
>> Cc: yantar92@HIDDEN,  dmitry@HIDDEN,  64735 <at> debbugs.gnu.org,
>>   sbaugh@HIDDEN
>> Date: Fri, 21 Jul 2023 19:55:22 +0200
>> 
>> I'm open for any proposal in solving the performance problems. But since
>> I'm living in the file name handler world for many years, I might not be
>> the best source for new ideas.
>
> The first idea that comes to mind is to reimplement
> directory-files-recursively in C, modeled on how Find does that.

If someone was thinking of doing that, they would be better off
responding to RMS's earlier request for C programmers to optimize this
behavior in find.

Since, after all, if we do it that way it will benefit remote files as
well.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 19:38:01 +0000
Resent-Message-ID: <handler.64735.B64735.168996826513250 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168996826513250
          (code B ref 64735); Fri, 21 Jul 2023 19:38:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 19:37:45 +0000
Received: from localhost ([127.0.0.1]:34939 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMvwf-0003RY-9K
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:37:45 -0400
Received: from mout02.posteo.de ([185.67.36.66]:54303)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qMvwb-0003Qz-GK
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:37:42 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id A911D240106
 for <64735 <at> debbugs.gnu.org>; Fri, 21 Jul 2023 21:37:35 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1689968255; bh=PG6jDwRXJXDtlB0rwfiITmTxfc//YDZ9XPhZ+wjGZ6E=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=MThnlsn4Jua+nJNT0byBa6dLtGEhNur/heQQyg/M1n99LYTCzINgay5vBOImtLf2M
 s1jgEsEQPDo9Sus28zkkEi6iIaMXFEhBUru3w7dEhC3rhQPX7Qg/PM8ki6I5TK/f/c
 B3sv4Wa1GoUFy/6e1rCiB6ohTeVkOT9yZT/n1YZ4uao/HpFqX2zPqKD7TyWyy6zEib
 3AfE7Xhqz4cl6gSyRQIv6P5SHvNJFiTNsIb2mtv7vK5+w2nx51klXUPMJWy94ti2i+
 4+HBHsfXAELuKIkj6s+uBhkdYgvfVLyDAye09R8Wh881+QGlO/AFHKL4Irx889TTea
 TY0yOd/BUJ2dA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R70FV5zm0z9rxN;
 Fri, 21 Jul 2023 21:37:34 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <ierbkg5ukme.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87ilaepkaw.fsf@localhost> <ierbkg5ukme.fsf@HIDDEN>
Date: Fri, 21 Jul 2023 19:37:48 +0000
Message-ID: <871qh1oy2b.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Spencer Baugh <sbaugh@HIDDEN> writes:

>> Does find use multiple CPU cores?
>
> Not on its own, but when it's running as a separate subprocess of Emacs,
> that subprocess can (and will, on modern core-rich hardware) run on a
> different CPU core from Emacs itself.  That's a form of parallelism
> which is very achievable for Emacs, and provides a big performance win.

AFAIU, the way find is called by project.el is synchronous: (1) call
find; (2) wait until it produces all the results; (3) process the
results. In such scenario, there is no gain from subprocess.

Is any part of Emacs is even using sentinels together with find?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 19:55:01 +0000
Resent-Message-ID: <handler.64735.B64735.168996925315012 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168996925315012
          (code B ref 64735); Fri, 21 Jul 2023 19:55:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 19:54:13 +0000
Received: from localhost ([127.0.0.1]:34952 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMwCa-0003u2-CD
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:54:13 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:49719)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMwCV-0003tn-8T
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:54:11 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 1A8ED5C0090;
 Fri, 21 Jul 2023 15:54:02 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute6.internal (MEProxy); Fri, 21 Jul 2023 15:54:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689969242; x=1690055642; bh=8mRF5AHmBaLcWh8sMzoKBdXdlWWQdYJHgWG
 pCGKqgTo=; b=eEswCrQ4nYKMug5OWTkKNlgZNcLyLsi5E/8Q5f1pOWBeXh1CtEM
 g6TBT2CekwD8LtU8Q5tYcJ8HhzovCIKkagUShdUyIaX5Rjxz0VNrIlJupDqcvvV8
 s53Q2ZToRJa4BVVae6CHQUbX17BKjy7OU3kuqBLzS8tHR7rjloTW2eCk7+O7sW2M
 +P1IU1cPgretTRCo9WtEL43QRqfCTYkbhx2bTK9qzSuPwhpUmno3gDB+90xWT4PU
 aKSf+Ceduvyn5zCREx71YDrfp8jkMwUrkB004KjdTMufgaxdjXpVbeh9+NPX7mSM
 g7VZs4grEerjiXA8guTOMH5XPG5TSB/0U3A==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689969242; x=1690055642; bh=8mRF5AHmBaLcWh8sMzoKBdXdlWWQdYJHgWG
 pCGKqgTo=; b=UMAypXmwtfRwPlDeOHuYtgF3oO5foss62miweR6xPMGS8Vz/EX1
 7SV7BPHS2zwGOBhnHyf08uOHjgwrKxBPnplIW0jEX1OuVk1HbipIAq8iPmV/8xE/
 2Aw9oggQ0hjXoISQWHa6K2vr4yeIogs0OHLL2Ut8cp/ZHRCfD2CT0LmSUR24TOXs
 Vn9+Od7DGv0jnde16yx81Z23keKnjSXbvF+VCOsfJRQyKbmHQRVRMrHtnYlgnmWE
 l4JhI4t6hvwJCnoRH1ZQ5UKME96A/JPRLvdxFDo5ZGDDzhbPAAh/W5tCjVGWrmoc
 KJPvN3Ky/kE3H7qEpPXUXoZuFII3u0EFP/A==
X-ME-Sender: <xms:WeK6ZBKTyiJAAu7kvjhispVDEzyAE7HZk7R2nY9gsvfgwcvLzkOi0w>
 <xme:WeK6ZNLm8EjhrJ_sUS_1bKE8DWArPVxCaxLPepXhvhmUiTukc2Im-ZyDzv7PhNOcT
 Fx9magTnIS5TCQhyII>
X-ME-Received: <xmr:WeK6ZJtL61pyIwJUA_uCdnogSrzYiQcFxt-eQM1C8hTsh9iOJRrd5G4qEmWVoho>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgddufeejucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:WeK6ZCbf433taEQW5fmCHE5CGJu54UHWjHcemUgIiTarrjTnT6m5Ag>
 <xmx:WeK6ZIbrzTsWCFtoj8HRbsE2atExIUX_5sd4G8OvKwt_S6ZtT6BbyQ>
 <xmx:WeK6ZGBItkxivXztBw10ZI9kH6h1e76D1AKc_2Fn-R2n41bTCcgeLQ>
 <xmx:WuK6ZHmj_JvDok-HXAtaYH0pw-3hGJSYBE69yY8fNdDsDpaJlSQGxg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 15:54:00 -0400 (EDT)
Message-ID: <ce17fd10-f119-62b7-e69e-dcc76885e5ea@HIDDEN>
Date: Fri, 21 Jul 2023 22:53:58 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost> <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost> <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost> <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <ea2332cd-ec57-794d-d662-ff522a4dd404@HIDDEN>
 <87351hl0yj.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87351hl0yj.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 18:48, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@HIDDEN> writes:
> 
>>> Locally, AFAIU, running `find-lisp-find-files' without
>>> `file-name-handler-alist' is equivalent to running GNU find.
>>> (That was a reply to Eli's message that we cannot disable
>>> `file-name-handler-alist')
>>
>> But it's slower! At least 2x, even with file handlers disabled.
>> According to your own measurements, with a modern SSD (not to mention
>> all of our users with spinning media).
> 
> Yes, but (1) there is room for optimization; (2) I have a hope that we
> can implement better "ignores" when using `find-lisp-find-files', thus
> eventually outperforming GNU find (when used with large number of
> ignores).

There are natural limits to that optimization, if the approach is to 
generate the full list of files in Lisp, and then filter it out 
programmatically: every file name will need to be allocated. That's a 
lot of unnecessary consing.

But you're welcome to try it and report back with results. Tramp is easy 
to disable, so you should be fine in terms of infrastructure.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 19:57:02 +0000
Resent-Message-ID: <handler.64735.B64735.168996940515229 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>, Spencer Baugh <sbaugh@HIDDEN>
Cc: 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168996940515229
          (code B ref 64735); Fri, 21 Jul 2023 19:57:02 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 19:56:45 +0000
Received: from localhost ([127.0.0.1]:34957 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMwF3-0003xZ-Ho
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:56:45 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:40677)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qMwF1-0003xM-RZ
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 15:56:44 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id C03DF5C00FC;
 Fri, 21 Jul 2023 15:56:38 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Fri, 21 Jul 2023 15:56:38 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1689969398; x=1690055798; bh=hyK0HyT/9j757p2K1pbxym7ydTJH63OM4OQ
 +m6/jmzg=; b=EPmGIGzAsvYqXQhvtLLkq4eIg4dXeGuupxpivWd4+0PLdYRWB2r
 N8uNt5zFPYitnfObKm4oT3nyCPxDZZ0Y6WGT5/8YhMtMhzUSkaqUta0w8avLfRc3
 FVb5h/B9+4I+Itz9hcnWG2+XBkJpNT9fBipa/7gaAwTyaMvc19clZ8zFVNsNV2Np
 HoNxO169YAYU7yWbnwpfBFTcy7ouY5nbvt4Mgkb0BeJilgrNlT1uGLt4H+aGK4T7
 jgLPFI2nLkGwI30L7xI8qgarvuPZTVwxI9puGV0I6ta7f6WAhN/zZLftjMfxeALH
 3iMSFKTIf40sA+PeGhRn3bnB91gdnDbbmtQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1689969398; x=1690055798; bh=hyK0HyT/9j757p2K1pbxym7ydTJH63OM4OQ
 +m6/jmzg=; b=BY83L8kRyp5kg6oI/ZRDpgIr6cNQKLGyWrxr7cdzLVzk945uQwS
 /WhQFiEn1aW7nL7WbIiBPnyyPc8C9EBqgR7HX7Vtt7UrNCg349nOEp41vJxT4gxx
 jJpfR7vJO2K19KDrmDFqPkBiYpWoLpkXu5VEmhBIJMhpkNzBDzbGG/qARKe1Q+Xg
 m2+IxBREYpq91GqouqI2lKZraCjQwZLWrGj6UZcWzbI7R3SbBmlU1va3MVMJRQS9
 hUJ4LREdQK3cmiW/5+yyZcpWB4eAvxeag0opqoQfnzu+wzqg1MUzOlOqyY2W/7ju
 T7K/H5ZRQ03lR02v711rvxTNpSgJREg30Zw==
X-ME-Sender: <xms:9uK6ZGQEgWO0bJ4wr64FtHSGHy79kQ31dx8birY6Q_t3H87PNZqvlw>
 <xme:9uK6ZLzSBlSoBqCAC37UIk_33bCmSejaM-qm7w9KkfzjXAkAL4KTJHwIbWNSw5wwz
 ou0akCEMaxTgIhEpOI>
X-ME-Received: <xmr:9uK6ZD367CxdGc7zPgxcf9za0ykRR0pgHgp6OT4IumHijyEo2kCTo2Q1q6fgA9U>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrhedvgddufeekucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:9uK6ZCDG2OSOBTzg917-HHr7dkZupZYvTVxDyavTFEtUdJUCHQ5j6A>
 <xmx:9uK6ZPjfENR86-XGrqJ6NbiMYy_RCzXhgo6xCY-Rv0WtPROP5sv-4Q>
 <xmx:9uK6ZOrI7n0Nn7dOMDHIrDLWnjHaf4NEUy_-9taYmHKr1XzuX1JVVA>
 <xmx:9uK6ZHZ19sC8IIh525ul_k8c0nQ9MNqPQUglBmm8URK1c8Go2PIs1Q>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 21 Jul 2023 15:56:37 -0400 (EDT)
Message-ID: <debbee40-8f60-999e-98a2-7f99f810500a@HIDDEN>
Date: Fri, 21 Jul 2023 22:56:35 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87ilaepkaw.fsf@localhost>
 <ierbkg5ukme.fsf@HIDDEN> <871qh1oy2b.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <871qh1oy2b.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 21/07/2023 22:37, Ihor Radchenko wrote:
> AFAIU, the way find is called by project.el is synchronous

For now.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Jul 2023 20:12:01 +0000
Resent-Message-ID: <handler.64735.B64735.168997030016701 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168997030016701
          (code B ref 64735); Fri, 21 Jul 2023 20:12:01 +0000
Received: (at 64735) by debbugs.gnu.org; 21 Jul 2023 20:11:40 +0000
Received: from localhost ([127.0.0.1]:34970 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qMwTT-0004LI-OW
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 16:11:40 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:51263)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qMwTS-0004L2-0M
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 16:11:39 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <871qh1oy2b.fsf@localhost> (Ihor Radchenko's message of "Fri, 21
 Jul 2023 19:37:48 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87ilaepkaw.fsf@localhost> <ierbkg5ukme.fsf@HIDDEN>
 <871qh1oy2b.fsf@localhost>
Date: Fri, 21 Jul 2023 16:11:31 -0400
Message-ID: <ier5y6duirw.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:
> Spencer Baugh <sbaugh@HIDDEN> writes:
>
>>> Does find use multiple CPU cores?
>>
>> Not on its own, but when it's running as a separate subprocess of Emacs,
>> that subprocess can (and will, on modern core-rich hardware) run on a
>> different CPU core from Emacs itself.  That's a form of parallelism
>> which is very achievable for Emacs, and provides a big performance win.
>
> AFAIU, the way find is called by project.el is synchronous: (1) call
> find; (2) wait until it produces all the results; (3) process the
> results. In such scenario, there is no gain from subprocess.
>
> Is any part of Emacs is even using sentinels together with find?

rgrep.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Richard Stallman <rms@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 02:40:01 +0000
Resent-Message-ID: <handler.64735.B64735.168999359223603 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
Reply-To: rms@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.168999359223603
          (code B ref 64735); Sat, 22 Jul 2023 02:40:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 02:39:52 +0000
Received: from localhost ([127.0.0.1]:35128 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN2XA-00068d-1e
	for submit <at> debbugs.gnu.org; Fri, 21 Jul 2023 22:39:52 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47780)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rms@HIDDEN>) id 1qN2X5-00068H-DG
 for 64735 <at> debbugs.gnu.org; Fri, 21 Jul 2023 22:39:51 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <rms@HIDDEN>)
 id 1qN2Wz-0001tQ-Sr; Fri, 21 Jul 2023 22:39:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=Date:References:Subject:In-Reply-To:to:From:
 mime-version; bh=h/FbF2fHgC77hFUoYIIUucSYk9Axy+/OwzXcMW0NHyY=; b=GxNvALdmCe9u
 B7T4Ov562pFVEOq6/jcrabfFK45k5qQSjsL87+EjX8i8Z+539SSSlxIVnThKJzcjYLyiqXm+4hSSF
 ZKt+L+GDnWjx0dOlPJbkqqDGVz5P1lCtvTI/4jhXo4p97x0D0i68kOn06oeixPGz/tlkXPfn/LewO
 kHornn7otcpJErDudfR+oIUjPL168d/5mkq3WU5ZG0Xxkv3TOj7/U6xKjeRW/lnyKyOI3gG+jXy0W
 9BaPDFRapp2ESR6FoEOFZSCGRI5anyEnbI1XpmZLEOcIwTOhx1mjntmrpkJbBeFrRRVMy14FZ40/U
 oiSD1IgfX8DNTlck4T8Ngg==;
Received: from rms by fencepost.gnu.org with local (Exim 4.90_1)
 (envelope-from <rms@HIDDEN>)
 id 1qN2Wz-0008Jo-L8; Fri, 21 Jul 2023 22:39:41 -0400
Content-Type: text/plain; charset=Utf-8
From: Richard Stallman <rms@HIDDEN>
In-Reply-To: <E1qMg6O-0003aa-SZ@HIDDEN> (message from Richard
 Stallman on Thu, 20 Jul 2023 22:42:44 -0400)
References: <iermszrwqj4.fsf@HIDDEN>
 <E1qMg6O-0003aa-SZ@HIDDEN>
Message-Id: <E1qN2Wz-0008Jo-L8@HIDDEN>
Date: Fri, 21 Jul 2023 22:39:41 -0400
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Since people are making a lot of headway on optimizing this
in Emacs, I won't trouble the Find maintainers for now.

I wonder if it is possible to detect many cases in which
the file-name handlers won't actually do anything, and bind
file-name-hander-list to nil for those.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 05:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169000364321157 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169000364321157
          (code B ref 64735); Sat, 22 Jul 2023 05:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 05:27:23 +0000
Received: from localhost ([127.0.0.1]:35275 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN59G-0005VA-M7
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 01:27:23 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:38316)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qN59B-0005Uq-U6
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 01:27:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qN595-0004yf-S5; Sat, 22 Jul 2023 01:27:11 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=XX7Ug0FB9iYtRvRRVmZkR7izyZKOi6hYkCTIpmCHzwQ=; b=MPTodv4gRJaS
 3EFB/wTDxCkNv5vmjJwTvk5vvNQkGuG19yHhXwCnHFGxXhTjIuHOAnDV+4Mbni6V9QFRTA+dMM58P
 BOO36aOkYiIE89vSaGvkRhkS6i8iLxBVjuyYTjV+AheNZESVqne4F9/HWp5d/Wh9nuC3bZyYfe3JY
 l3BR+q9bLst5nOEhBGHqnDaW6nBapKTrGUO1XNK9JA2HeLcWlCW5mQCqfKgPw1hNiTQYp0qgQK4kO
 jOZrktgnJMeX7hydNVwOPeyJgYYTyuRwXNFHlIRkGYRJ6pMLKb8fQDgzjBToxbGkq64NMXoKj3XUy
 1wOzyznetbyKXUuVVf/DmQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qN58r-0007YI-Cl; Sat, 22 Jul 2023 01:26:58 -0400
Date: Sat, 22 Jul 2023 08:27:32 +0300
Message-Id: <83h6pwa52z.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <ier8rb9ukjq.fsf@HIDDEN> (message from Spencer Baugh on
 Fri, 21 Jul 2023 15:33:13 -0400)
References: <iermszrwqj4.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Spencer Baugh <sbaugh@HIDDEN>
> Cc: Michael Albinus <michael.albinus@HIDDEN>,  dmitry@HIDDEN,
>    yantar92@HIDDEN,  64735 <at> debbugs.gnu.org, Richard Stallman
>   <rms@HIDDEN>
> Date: Fri, 21 Jul 2023 15:33:13 -0400
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> > The first idea that comes to mind is to reimplement
> > directory-files-recursively in C, modeled on how Find does that.
> 
> If someone was thinking of doing that, they would be better off
> responding to RMS's earlier request for C programmers to optimize this
> behavior in find.

No, the first step is to use in Emacs what Find does today, because it
will already be a significant speedup.  Optimizing the case of a long
list of omissions should come later, as it is a minor optimization.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 05:50:01 +0000
Resent-Message-ID: <handler.64735.B64735.169000497124118 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: rms@HIDDEN, Michael Albinus <michael.albinus@HIDDEN>
Cc: sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169000497124118
          (code B ref 64735); Sat, 22 Jul 2023 05:50:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 05:49:31 +0000
Received: from localhost ([127.0.0.1]:35293 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN5Ug-0006Gw-Mr
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 01:49:30 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:54898)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qN5Uf-0006Gi-Dy
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 01:49:29 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qN5UZ-00051X-PF; Sat, 22 Jul 2023 01:49:23 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=zRn9FZgcZFeJbByOx9xZRrElfIKZubDLYkRSZpIV5HA=; b=DDTQQhwH9FTR
 piZ9BBrS9QWLusQc27BP7g1/66caNOUhyn5HYNOA2EC9PzJNvgPNtwthxqB9uCisNW1edf7N60PB0
 B+p6/o3Z0bBIAQ/Vvcoz3zPXw3YlFoYHnNx5hiZKWH7fGmsbaiK870+Ybg/1rrto1eoeBvm0aPWp6
 IQsOywvoHXJtLEuXT54xmfJh6YiSZMkDU2UvhfksH5JgwNt2Ma3hDOnOdbbNRSz4qDJmCpcZn6kPi
 be+f1eQ72pp5u9UEzssKFGe9tNaHrDf4XU1wek4AD0v48Ouzi4XZwqPHgcaY9sVljgSlju0ofNKUk
 ee15sNFo9RB2SfIdd5rngQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qN5UP-0005Cu-Ih; Sat, 22 Jul 2023 01:49:13 -0400
Date: Sat, 22 Jul 2023 08:49:51 +0300
Message-Id: <83bkg4a41s.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <E1qN2Wz-0008Jo-L8@HIDDEN> (message from Richard
 Stallman on Fri, 21 Jul 2023 22:39:41 -0400)
References: <iermszrwqj4.fsf@HIDDEN>
 <E1qMg6O-0003aa-SZ@HIDDEN> <E1qN2Wz-0008Jo-L8@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Richard Stallman <rms@HIDDEN>
> Date: Fri, 21 Jul 2023 22:39:41 -0400
> 
> I wonder if it is possible to detect many cases in which
> the file-name handlers won't actually do anything, and bind
> file-name-hander-list to nil for those.

I think we already do, but perhaps we could try harder.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Po Lu <luangruo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 06:36:02 +0000
Resent-Message-ID: <handler.64735.B64735.169000773328112 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, dmitry@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169000773328112
          (code B ref 64735); Sat, 22 Jul 2023 06:36:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 06:35:33 +0000
Received: from localhost ([127.0.0.1]:35310 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN6DF-0007JM-Fn
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 02:35:33 -0400
Received: from sonic317-34.consmr.mail.ne1.yahoo.com ([66.163.184.45]:37505)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <luangruo@HIDDEN>) id 1qN6DB-0007J8-W4
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 02:35:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1690007724; bh=tSj4J9q3molIIO2jERfQxF75sUyV0FmJXnnWpIBn37U=;
 h=From:To:Cc:Subject:In-Reply-To:References:Date:From:Subject:Reply-To;
 b=VbA7BpgP2J9O5uXNECboAvU0S9wfYzR4pLC/RTgvjUWrX7+NB+uFczMFgLaKugWK9o6CNX2IgqPkQvwzmY+0k8Gq+/TBoutlggvMKXXSZnakxObYSMeaLZ122PvkIK56Y8MH2xwgpxtO73cOtZFGi2QbVlo3VFJLUzHGcmcujWvcNnGVrL21H5G+IFh2oinV1smJIw8gQXLVs55fxPkdFJFLRT2HC9ogPjm9bJwFctEOk9DT60V9nlBGthFsYmim6bH549OdWdk4kHhuVP94wJoy3KU7z3i6aGYjE7e+vCDh5+BHcLSp8dOPOnDsVq4JLgVGm/RNHEZ2/eY2rghpWQ==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1690007724; bh=nnXrmyC4hduJU4nBtBkKJ0mqEcFxp/9cnD38p9Y81q4=;
 h=X-Sonic-MF:From:To:Subject:Date:From:Subject;
 b=YFqaY86v4XtjGNJv/k4/qc4aPvDZW1OBAm/llSO80+HQs6kpIhfVo9ruUcKg4fW3lcJvSrvzOZUg/HEXRv6NSiltEZh7kCrpmupSEsqoAhWnh422STM/0norp5KIUJmpGPsQewmrclW/kIMg7uEoUYxwePISvlYp9jxW2CVPMFNsUc0ipO7QWV47CwEutIKtLcQ8bNEkNkIOKIj1PSTmhGmOBFl1S/r1XMb7enmLCyhd2HsvPR/bk2eKZqxAULMrY9pbav85wIuT3nLw6GrbKw2JH5tmmT81xtwD6goE3yFSMz401DaaL0xteBMzSu0eBz9c1JUEGvFxluVt0ZLi8Q==
X-YMail-OSG: ZCo3DDcVM1mTFKiG2pOjVUNLtm3fQpvxXwEhHtjRBkevNnLIx8Skl0XnF2zaifQ
 _F6dBDexC2Os4SaFsBtxbgv00bw9ks9TIE4i3E7yrKbsdTRCqWu0uHhzhwkH45Vjm8aeuB6fB0Al
 AiHt.niXBVFcd92s4g4m8gQkQTHTrKqfCg_j1LGkHlPJKneB2p9.Fw3XgcvLmKRPWKwBRJGh1Za7
 qENkupx0.yYnLU01zdEfwGWKL8AMHJOasbJPHdPXtbuzIITHKViovo.7TjQS2ZQtCP2weeEAQl_Y
 mHxjWs4XnyKxx3FvvMBGHaxT4wDvlAoWbcCrm31yyGF.2SWCri3IXyV3fTVqsA8KsX86GeTRc.Sj
 P5K6er9p2xZKvflfbJXUqpRapcEidzxiv5ypoHZvk.RBaIjd4_MXAq_Hsq2MzPCCB6v9tCd2qx5a
 E0Kpvzue499e1JCVvfkOSOeVfc9TwTeSwq4rE20x3Wo2CKutJrEygQAjbGvs4zfCG7rzCVsEVpQK
 7n03jfhXl3P2RUpmsCSNvb310vV2TvXArqf_0LM_5aJ7zCLdM8I9F6.lSWuxBTQAJp7.tFMitZsp
 444.s4tR94F6ztlJNssl6zN0uCU5Mls5JkzFsGj4Ad3VuroSGKd9adsk4sKraV7re9FYi_SrZx2y
 mJCuGGCLJHkmYrvLy23dZGHc0EsS6Pi3qreJPHGzxHrfRwVZa9DpJOx_YDN0GsMJSNnUtpPxIxSu
 uoLLra8stf6VY1ewGvjpV37.Xf2Ln_6MsuQR0FWbaMZHX4m1uVhLb2AgA_XI_bahLd4SiMVopyQm
 VA2dbPlJyDvKxqFEX.kb7kfZPFs1hRUBY8Q1DNdgUEVLnyDeeBiD07GqJlju7ElFE_4RM0vrH.Um
 X4tzgW1qm2k.uPd2a5KGMwJ2dcWDHJEvq_9DWlIsQ6I.YY4m8fktUPmLDKMxtGU_yEfTOu0Vun.J
 kA4NkkmYAl2XSb7v4ercqukLsn0CSNc36d4KnkAlg4jKOcEJdIaG8O6rBZuQu5UqAGxjeeH51TSS
 rDr9YdXoeeLbdx4NSy85OFEipsBNTG2DRxpluxxrTnDiSUbjzFytCvG9y.LnKZsMatB.LzyfNxf7
 l2vzgA20qYb0L7OFA0mRXwhm8raBEg28iamY2dPZu_fwzKXN.L9Br8g53plGcEN67qEgrcnvy2kG
 N34RObJvu8TJ8YvQeLT.wQ4ZlW7iWZDWyJnzTyB5TzArNh0L0HTW3epQ8Pyn.F5jWF0OS9T6DQNk
 sd.oJHywDPFZ_4.AxSJz6UkFDvT4qRwmT2DLrmUt_kPgRW__IUNuMSH7ZqKcGHYXSXCUxsHFXqKc
 ekzrvt_sMTT1N9nmapXKDFtbismCd.XRbX0STSyigu5YzgyNKWqcGJMTIbpdTtRBao7FQSJhOmFR
 82n23Z1zE9KC8eD0xUWIFCuGSpkKTTXLduP.jle6C161opz4qAA9sjL01F642QLbTbr2BLhDe82e
 7zV4hlAgF9ZsA_5epHK6857FviVf4Ryf9Um15GCyz6zZV8N56Knc5DuVTj3zmjyqhOthDMvUbRt5
 PsBjKnxnwAsngrcgwxf98lXynRTPHuVD6rSgGmJyTNQeroLEDDGSfsXJr5BAKqIXtBNZFWKeAGzm
 iaBoY6TlraY.cNWO.fuwcjbqQL5C8w_cV7GBbzM.WDJ9IASonDWVReJfX1E8ts99s.GcWU909SML
 Zfm.FEdZnfO34uY6IKmD0DDh8Dqb7D1Gf5gJJuF8IKGPBA971kcXZ7jyORIn3eVBJP2y_q6AOId1
 Q7lCjdwaYWAOyEjdKnJ.ORP1XzIZEGxtU8Yah60evu2VFMKJrSGSJvRJREQpT7Dlm_ufhnFvr9y_
 .rro10TUQ653WhewlzvMMaPyS5E_yEMO512doG0AhiEQcsXzrAR9BZ8IyR1q1CVrmMt1Apcff15I
 HYq8W5B2EzjKF8X_tzL9ypBdmtp5C.8seWLRkTWT3SEOLo5K5PRIQccHUdf26.KHcAdGSL_zhonX
 dqZW_cPFE8gU00NYzO33KFOwrjt64ysPns4u.HQWLCZTT9Vh0tr.dqjU5_LFGNLQncogn9J8pfAz
 2NydmzEo5sE7vPyRWdklUtDV4oUe1CJ9ahIaYmImcXJ_GYLn.CflnSdR7X9IUdFDWiJ1P_p41q_h
 74K8qhUQGpLbrlzLLjIyGbgEQiXJj9vS7nZ3KEIl2vSnqHsdn0crNDUF_GxBIovclKkVcgFqFZsW
 wYYLfVGhOUKtK65d44vaYvvoMd.iNZVHs4TvOtDjhgO7pANgD_PXrexinPoXmTtBkyg--
X-Sonic-MF: <luangruo@HIDDEN>
X-Sonic-ID: a74230b5-1c24-4fe9-8632-014ce33b17ac
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic317.consmr.mail.ne1.yahoo.com with HTTP; Sat, 22 Jul 2023 06:35:24 +0000
Received: by hermes--production-sg3-9dc5f54fc-zbsxq (Yahoo Inc. Hermes SMTP
 Server) with ESMTPA ID 75a8bfcd853f39f57b16aa0b3aa78f6c; 
 Sat, 22 Jul 2023 06:35:20 +0000 (UTC)
From: Po Lu <luangruo@HIDDEN>
In-Reply-To: <83351i338m.fsf@HIDDEN> (Eli Zaretskii's message of "Thu, 20 Jul
 2023 20:24:41 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <83351i338m.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 14:35:13 +0800
Message-ID: <87v8ec313y.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Mailer: WebService/1.1.21647
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo
Content-Length: 824
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Cc: Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org
>> From: Spencer Baugh <sbaugh@HIDDEN>
>> Date: Thu, 20 Jul 2023 13:08:24 -0400
>> 
>> (Really it's entirely plausible that Emacs could be improved by
>> *removing* directory-files-recursively, in favor of invoking find as a
>> subprocess: faster, parallelized execution, and better remote support.)
>
> No, there's no reason to remove anything that useful from Emacs.  If
> this or that API is not the optimal choice for some job, it is easy
> enough not to use it.

Indeed.

I would like to add that subprocesses remain unimplemented on MS-DOS,
and the way find is currently invoked from project.el and rgrep makes
both packages lose on Unix, indicating that correct portable use of find
is decidedly non-trivial.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Po Lu <luangruo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 06:41:02 +0000
Resent-Message-ID: <handler.64735.B64735.169000801728528 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169000801728528
          (code B ref 64735); Sat, 22 Jul 2023 06:41:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 06:40:17 +0000
Received: from localhost ([127.0.0.1]:35320 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN6Hp-0007Q4-Gh
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 02:40:17 -0400
Received: from sonic307-56.consmr.mail.ne1.yahoo.com ([66.163.190.31]:34785)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <luangruo@HIDDEN>) id 1qN6Hn-0007Pp-6X
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 02:40:15 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1690008009; bh=rC5s1XRkh2ESHgHxu6vgvj12iN9HsKCICKMUPJvqZZY=;
 h=From:To:Cc:Subject:In-Reply-To:References:Date:From:Subject:Reply-To;
 b=A8wT1PVb7mWN/4MypeZhW/Qd6OAZWGfInzX5IYfMwfowRXr0/2Kc18R/Uq1me7wTpinK0+OK6zICuIP7hp1QMMVcn1yAEHf6FuanBU4InhdOJme8jdBad5b5YL83uviMjpUBVW8oz9ysqJL6KQ52eXTEW4rjvFFneoRI/GTwurCZEljTOX5qyuEPQJTnGTUiKQzyKuV9D/IKngF/wcHjLQzJVO6x5Hg6LNh8XMWml6r64vVQPmOZdDosZbnPd5m+cbjLAH2+0YDLkJX+vV7G/8NWWS+hK0h2eCXIpbDL/O30pWeXhCwFyf2Jo6sZZ0D4TDgMS3zx+NwyB944ujtNFg==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1690008009; bh=SHOSC88P2nuZNVRnQSwAc4OzBKZLdsG68g7hGxrujdY=;
 h=X-Sonic-MF:From:To:Subject:Date:From:Subject;
 b=sNTJhI/ll7234FHei3fqovSEAy9DebNzl+cbXKs64f1bnnxGmm22Of1xUJ8gr/3031tc195vhPGrhIC5ncEPkvpB+diOr3rnQWXI5w6d5ZQnnYkALvSWx+FeXr6EhKQFkrqyVoNVwvnqfvlHbKkPTQmI4I+SwRmtmBH30x5iwIlDIoMgrxh0UlsT03t9paygxaVZNQz6u7ZbmrDZ7TCng2u7/IBdywEGIQdY9deNYFDbmsWKotz7ICOKcI2mpEJ7I3AUeSkcAj2fRLrG3PSTSIFU58pVw/miWKtelR/t5yCiY5KDw9a8BpDsTqbKT+QzUnKA8R4MWIKoQO3GDOGyWQ==
X-YMail-OSG: Boms5wwVM1mMsj07SqS9mCaeU7oGCpRa893uSoqSPnOGzkd7BZNzXV8zmzejSmZ
 OjzIm.P_otasz8Qb.yT1GUEwg91_uK9CgJht4xKdT2Ji2azdgQZHksoi6B3RIKOmMtWdlC4k.Gxv
 Q46yICVA8pHjT0ETDlGuNQPTceV0K8nF61VUIrQA5UOk99Kh1o3fczHMUpkiq4rwz93fndoHwbti
 VSvqq1TbExiuLdcxDFO4YyGd4e4Py7ofrKkQOUUyjfSk2Y2uftvCfGHxeTP1QjccYsSUqiaIj2sJ
 ASaj7O_GyQGzL3cxQ2Hp7vbBvHxF2fe91ekWkRjIQh3JHLsSyIBkOmu.0ba.P9IWvHygKs7Mnupy
 gakTJwzJxbHWhIyzluRhGyXPKqWcWixKindbzaQe0Pssrbj9joEZyNtbOg._yc2aI.L1gAa7X3Mz
 hCZefm7IGgAmvzjBfW65.0nDiktlQxjD_K19A.3dYy6X2nbp_t5OTiC5NT5qyKLut6up6oUK.6oN
 fRiJt9tBWhfjQrKdUbLkAsn8lzoHMjb_RxY8GHt.0mOiuZQXrtz1QTjycUBZL2gGcuBarASR1Eja
 M0WD1AIfEJ1dSp9Of9iUAi9IpKMCJI2FKtvsLJhiULLw4CZD5rgt0lQuYxWZ4qfhN17e_VD6yL3n
 18Tu8XI5TJRgFoRRGsyHro5Ybb5gPz7BU8pZDy2p2dwnKm._T..9igSrO7x5N9D6h3sMfgd.OrUR
 9grAPTTaH1g0fBX9y2HKgarg63Orx5Pr56s0FJNymzicrRJD_mCfTFXSvfNYF.4WwOiakfGkX2XC
 5MED4u5wXS8ANyycrg_cbM7hr1DAYirB0w1MK0.2SgKOVPScwGdmRhLeDvXg49RdTG_TBdSm9QOe
 Ri9LBaueDgGMzgGr9lyz.k4CT.I5wFYUOFbuSvsuI9zN9Y7AgS_wzsuJwkirVYfAMaMqmovhT3LQ
 EWY9jWgbjtb8BnJ3dWLUyj.KGInCKn2oCcRBr2E5DRzzcag1xnvNLbTgYtYhuBmT0dTEZ4Bj.uCg
 IZ3WcNHvGXInXYUf4TwFX9g3zqYpHFUPpFIGbV3ao4nmkFl33dLz_RwapXHpJHeSg4P5e9Togfqq
 VC1aGlN46DVFQqHVyQqT.415LxjRIKkiLkjuS.5TiHsFrsc_.IWs1BVbPgsIfCkV11mL.nnH.4Zk
 8tIJde1O.aT6fZQBo3iW95jQdFhHAuKwedHfoLcQNo0NZ8tsXlBwDSxCArzj52iQsn9FdJNYd51L
 EyUAnXf6MLcJ45Hu_j.nbCHdBPT9DLw8EUgQATVBQ9ykL7nQdOqFM5Zui.0DenW0YDG8BoUzQfMJ
 km_RCOV.G_gRTgvok4bqaGLLkv0Tg449wZLpHwynru7LZmlkgywj21Rgzrhl6hI1J0qtUUf8j370
 eFwRihmQqLxfFMqivGcAklalqZWRnUk9wGheXKiNWc9ctg0R2HSsQzsBM_u7wQfML5Yfxa8cfklr
 yV81F6M_cBgcM8FVqLE5.nYpTiH0D5k65NeHglpZGi0T8BeufIzC6iiZNTJ2J4ktmn_YyYmmmGxS
 _aN0xVqrMJ49LDvxWY_hh87dpAKAoPtZifcCICFv4MOuAEL2IqPE03QJq_wL9G1V39IXRb9.rXw_
 7eHfg.XhKkj4bhp4mc2KuZCn8dAE4Tl9CHT3CA2gFMBKd8oWEW4jGdk4rWAV0XjAAXB9qrYQ4whr
 4fKwfib9Vm8HmP6UMdSudgJ6_tAST.tCMWHgwoCmPQZhq78jRevvV.82pkcwWA5D9PedOaVylSu4
 z6mpzLDCzDDz4M9mbgWF2JiHdKOWpiix7RPInKKZs6JHgVwigXiYbbeaYjgXG71axTjmLACiWGMw
 Pyff9qibf6QWwzZDdit4g0tw5Om0BJPA0kNHI2Ecytb4n1fk6ibMk8aCCyp7I5QaT41GQtGDlQDV
 X1tVDCDqEdfUZVtyJKt9AQbxWYT9blWPn4LAmx8QcihRnb.3jfigmFmhxjJ.Z92KpSF7Pc14_t_x
 xOs5I6nakLJYEwxSZIK5jWHNTI7lqETOYrw384XHBvBCCQ.3tMv1rhupH7HMcSF8cV0LH37cy4jc
 MH5dE4qHNdMqUEYWy3lg4Yla6IYxZFAE0YhXB4AamkMlHXLwUXihTUuD70jIQmIu2xGdqCdaPKKg
 .7Jnm0kBL0cKw5i_UGsAIzXA8Figu0._i_SYqIEQcezcZWYAMT1z.XAkYs1uf28XJZtayRzBc6Xn
 iVyolh9hYBTlQzxd5L.Z9IN8DGP96tw7qbLqt0x34DS0sWDTuAKjRZUUahiggWyt.rPI-
X-Sonic-MF: <luangruo@HIDDEN>
X-Sonic-ID: 5b9575fe-9afd-4efb-8bb4-8f9ce28d83d2
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic307.consmr.mail.ne1.yahoo.com with HTTP; Sat, 22 Jul 2023 06:40:09 +0000
Received: by hermes--production-sg3-9dc5f54fc-9mrgp (Yahoo Inc. Hermes SMTP
 Server) with ESMTPA ID a55ae7f91a86cab175db34b82a92cc2e; 
 Sat, 22 Jul 2023 06:40:04 +0000 (UTC)
From: Po Lu <luangruo@HIDDEN>
In-Reply-To: <ierfs5iwlx3.fsf@HIDDEN> (Spencer Baugh's message of
 "Thu, 20 Jul 2023 13:08:24 -0400")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 14:39:59 +0800
Message-ID: <87r0p030w0.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Mailer: WebService/1.1.21647
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo
Content-Length: 987
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Spencer Baugh <sbaugh@HIDDEN> writes:

> Not to derail too much, but find as a subprocess has one substantial
> advantage over find in Lisp: It can run in parallel with Emacs, so that
> we actually use multiple CPU cores.
>
> Between that, and the remote support part, I personally much prefer find
> to be a subprocess rather than in Lisp.  I don't think optimizing
> directory-files-recursively is a great solution.
>
> (Really it's entirely plausible that Emacs could be improved by
> *removing* directory-files-recursively, in favor of invoking find as a
> subprocess: faster, parallelized execution, and better remote support.)

find is only present in the default installations of Unix-like systems,
so it doesn't work without additional configuration on MS-Windows or
MS-DOS.  project.el and rgrep fail to work on USG Unix because they both
use `-path'.

Programs that use find should fall back to directory-file-recursively
when any of the situations above are detected.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 08:18:02 +0000
Resent-Message-ID: <handler.64735.B64735.169001388115877 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: dmitry@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169001388115877
          (code B ref 64735); Sat, 22 Jul 2023 08:18:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 08:18:01 +0000
Received: from localhost ([127.0.0.1]:35418 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN7oP-000481-A0
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 04:18:01 -0400
Received: from mout.gmx.net ([212.227.17.20]:50245)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qN7oL-00047e-2b
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 04:18:00 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1690013854; x=1690618654; i=michael.albinus@HIDDEN;
 bh=TUiBtIN0DG5tYdHIHc8e4CtiWpBMZNEXEOBucQ0tvV0=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=WLkIrG6MbS8OrZaCKR+yqp53PpGLUiGvHjsLYnp33j3BscTsjFEZBgrdvw6JwRBLx/zMmtr
 zRIw82h53HQb84O3sYYXAZ3OFPG1J+wJhr/UKWl/WykAvupA2uxRyBVO5cwfew/1kXsiKcvLt
 dK+7/IglGo75vDQbPOSSazUedPuN0uVoS6u/ezQhGPNexjL+wQm9xk1+9V928sX2uIU/hBhhn
 9tF5YDv72HwKC4Ha1il81kk06tJW2wEZoDsYV0XW6KOTBq3Ctg3xNT5t70+cpslHVFpFd1pst
 vbdSrkE4WXfXBi5WOa4hvajhaGbd++MHHEK/ZUvIR35qh2fRUkrQ==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M2O6Y-1qLURM04re-003rpl; Sat, 22
 Jul 2023 10:17:34 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87jzutb14l.fsf@HIDDEN> (Michael Albinus's message of "Fri, 21
 Jul 2023 19:55:22 +0200")
References: <iermszrwqj4.fsf@HIDDEN> <83wmyu1l1k.fsf@HIDDEN>
 <87fs5hemi1.fsf@HIDDEN> <83edl11qzn.fsf@HIDDEN>
 <874jlxebz5.fsf@HIDDEN> <87lef9mqio.fsf@localhost>
 <87edl1scbw.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87cz0lmoxy.fsf@localhost> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost>
 <87a5vpcmc7.fsf@HIDDEN> <878rb9l1f5.fsf@localhost>
 <87zg3pb6yt.fsf@HIDDEN> <83zg3p9s39.fsf@HIDDEN>
 <878rb944wi.fsf@localhost> <83tttx9q4v.fsf@HIDDEN>
 <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 10:17:33 +0200
Message-ID: <87cz0kbbs2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:i/W+U1HfSO1RN++UM91BCPNEGx4BoW9e9O3ydLv2wEtyKzgy4G0
 F+Et6dcVoDv8OOUCqeqRQUXpQ072aHV6bcFGjT0puc2VhdtJQw8EmbMVHhsOtplMd5ofaK7
 th7JJfbrL36XG2kiMNaUIIlSzSB96cBZqjwRZLYItNVfvTsUt66GOen8la1X0DtMpA7N79U
 q0DgADs4Ed+nv9q5PQMtQ==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:4qie1BiGz6g=;+XAKsG6WLTK0nf3AovYNEvQI3T4
 KK58tkmj+c7E1M7HCH/nynKv5zpfM0de+oDVzxis2hOFt6rim3EdTYh2saGF2x+49jRrK/oc4
 NwyvCKelidzZiJo0OBqtzSqqZ09/+vgLp0S7cKGwuus/HLD9wW85Fu1ev0AqsK0RiwTBY7mR6
 vQYtmMxi7yxDZP+QsT5bqwWDU6ia1rP7Oc0jinpeqsUVnOz89NCu/UByCmMIeZaBuBXJ+LJPc
 0NAZBBxQKX2PhAa2AzaDSwi1G9TQUKRh250V3Lte7/Sfz+MhTUfbhKfskpz7iGbo7CX1cgbgN
 TFvQy/u/qzf5vs5iMs43YkVpXjXOp/go6Om1BPhLu8vSyCbMBCZHf1vu8/UEMTSlVuZUrGger
 kOyJ+rt+ICSw9EAbOffH4MrD87GE8nI0mn1HKM7EpBFgFwOlmFUkaVRoL9f7VSk0H2qaT5XvH
 c17DBIUhuzl+qhZdjNzrzwpgoeEsIBFGEOo2FeJjcWnZOob3sUVCjnk/xUntzuI3pHkPuuPEf
 Y7aIrRvyfuKO0mDNrXbci5RZvmkqV5lyhf65+ca1Fhs+5OYRu07o/pHuIFUuMcrUkFoSNSvb9
 or2m0TEp0h6lb0p3AXqz9GSV1IaxQ+gKA+IMaQEhZfPcG9uGEJnVDyKQabiFzurirXaedHvQQ
 X86ayBN6Kwi8Wy6ECL9nfYjKGzVkd/jjsX1GUGbBS/XQ2X+SElztBrwAOIPrXYx7vI8By0d4W
 MCdUdphKYJyimcNKhGMPbtWEEPdggTddIfSqRs315zdCgSIKBa8B+MGb86czpCkX2Po8KO5Wc
 xX1vfANuzad4gguNI0UykYJHbEvd1FHsGKpuy3/RDagfqrJv/E+9YYtn6XVYD7jTO0oKs7vX4
 ngxS/anyBbewBofgLbwEIMBoVstG3Vdq6RPfINbjx5Z6nCZzMTCGu+ehwhWiov92IhQ0ylhpy
 OzmQlQ==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> No, I just disagree that those measures should be seen as solutions of
>> the performance problems mentioned here.  I don't object to installing
>> the changes, I only hope that work on resolving the performance issues
>> will not stop because they are installed.
>
> Thanks. I'll install tomorrow.

Pushed to master.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 10:19:02 +0000
Resent-Message-ID: <handler.64735.B64735.169002110028100 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169002110028100
          (code B ref 64735); Sat, 22 Jul 2023 10:19:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 10:18:20 +0000
Received: from localhost ([127.0.0.1]:35498 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qN9gq-0007JA-83
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:18:20 -0400
Received: from mout01.posteo.de ([185.67.36.65]:34431)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qN9gn-0007Iv-6n
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:18:19 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 18260240029
 for <64735 <at> debbugs.gnu.org>; Sat, 22 Jul 2023 12:18:10 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690021091; bh=eYLrO6lc80gVNDdorkRuqWnTN7qgE7++42Xj7c5twtE=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=Vea9mm9cL5+cbLIyEIEI+DLccbLPwF+S8LY2TfYMYM3IoJMoBLMZVUc4smNn8iam3
 av9TrUtpgJnAXAW5q/JmK8CWu4++v+ED/58aD2TDDEz7xIFRVyjx7f2f+V4og0zn+z
 jV3fpe7R6c+IvZiZt5FVc6nUIVVp1KkdfO9Laz41pJhEZ5kNzY/xCGUYCSee33UMpL
 z3i3CYuZaBOc7W7ylS9AUA4SxkM1DmB+Ea9Y1Nx+q86sHm6iMC6PF9Jkc/uHuz/dUS
 JYGSqfKCBGxNaQDsd7pQu0CwYEind4bKdGq7BYqSpZ8LIMk0EgeCLZVHeuPuTxFBMy
 /iHO80x96Q98A==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7MnZ2j77z6tw5;
 Sat, 22 Jul 2023 12:18:10 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <iermszrwqj4.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 10:18:22 +0000
Message-ID: <87edl08d1t.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Spencer Baugh <sbaugh@HIDDEN> writes:

> - we could use our own recursive directory-tree walking implementation
> (directory-files-recursively), if we found a nice way to pipe its output
> directly to grep etc without going through Lisp.  (This could be nice
> for project-files, at least)

May you elaborate this idea?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 10:39:01 +0000
Resent-Message-ID: <handler.64735.B64735.169002232630626 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169002232630626
          (code B ref 64735); Sat, 22 Jul 2023 10:39:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 10:38:46 +0000
Received: from localhost ([127.0.0.1]:35526 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNA0c-0007xt-5S
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:38:46 -0400
Received: from s.wrqvtbkv.outbound-mail.sendgrid.net ([149.72.123.24]:2592)
 by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from
 <bounces+21787432-b76c-64735=debbugs.gnu.org@HIDDEN>)
 id 1qNA0Z-0007xd-1a
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:38:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=catern.com;
 h=from:subject:in-reply-to:references:mime-version:to:cc:content-type:
 content-transfer-encoding:cc:content-type:from:subject:to;
 s=s1; bh=JmkGW7uOEyQXvznsKLr+w9z4YgZvxGiiW74fdpKaJ4k=;
 b=ckq71Yddpzk8yJDHWs+5/9quS+fMHiMfRpazBtAW8aW1GWpjmz00eYr3ZvbwZc/SALmZ
 qBchfmJVYQG/auYq6R070X2mTczO9ok/39tF2seU6jhFQdrCqtcOF5KJ8fI9QIRElq3jF/
 dPU0tnz2oOE8qVJWjBdRB9zs76a+gB9+lguESmztetryf66YSESSsCnyJ4DMEzRFidMVAV
 3k8o3aSLl6bvqLGuCpuB5bFeYGpP1N0ApP04WVtEQDONzCowYYJlY7in6kBNf4HEc546H7
 Ng7/kEmfL1qPP/If0FdeDySmpLeGFEmmPiZlgqRR7zAh18nJVN37N+WyYVZIHWqw==
Received: by filterdrecv-d7bbbc8bf-hf4lw with SMTP id
 filterdrecv-d7bbbc8bf-hf4lw-1-64BBB1AD-F
 2023-07-22 10:38:37.481541034 +0000 UTC m=+6260331.354104113
Received: from earth.catern.com (unknown) by geopod-ismtpd-19 (SG) with ESMTP
 id M4_2X6gUQg2Lbq6FjHaRfA Sat, 22 Jul 2023 10:38:37.286 +0000 (UTC)
X-Comment: SPF check N/A for local connections - client-ip=::1; helo=localhost;
 envelope-from=sbaugh@HIDDEN; receiver=gnu.org 
Received: from localhost (localhost [IPv6:::1])
 by earth.catern.com (Postfix) with ESMTPSA id B145B60166;
 Sat, 22 Jul 2023 06:38:36 -0400 (EDT)
From: sbaugh@HIDDEN
In-Reply-To: <83h6pwa52z.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 22 Jul
 2023 08:27:32 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 10:38:37 +0000 (UTC)
Message-ID: <87ilaci637.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
X-SG-EID: ZgbRq7gjGrt0q/Pjvxk7wM0yQFRdOkTJAtEbkjCkHbIEq3B8VB07aQyhxWj0nfMRL7FquXRq3RnCJN0YKQQwVDjUc530FyHIavQw4ZLWEgDhKAvYRiAi9KjAQ3MYqbVeqAIYZTeC7gUkD1LrFWCCpgAD5zYmeo/3VdlOJP1UGGrM2TjQTXalnoB+Ey4svepamu3kVHUTOY9bTTjtj/xq2w==
X-Entity-ID: d/0VcHixlS0t7iB1YKCv4Q==
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:
>> From: Spencer Baugh <sbaugh@HIDDEN>
>> Cc: Michael Albinus <michael.albinus@HIDDEN>,  dmitry@HIDDEN,
>>    yantar92@HIDDEN,  64735 <at> debbugs.gnu.org, Richard Stallman
>>   <rms@HIDDEN>
>> Date: Fri, 21 Jul 2023 15:33:13 -0400
>> 
>> Eli Zaretskii <eliz@HIDDEN> writes:
>> > The first idea that comes to mind is to reimplement
>> > directory-files-recursively in C, modeled on how Find does that.
>> 
>> If someone was thinking of doing that, they would be better off
>> responding to RMS's earlier request for C programmers to optimize this
>> behavior in find.
>
> No, the first step is to use in Emacs what Find does today, because it
> will already be a significant speedup.

Why bother?  directory-files-recursively is a rarely used API, as you
have mentioned before in this thread.

And there is a way to speed it up which will have a performance boost
which is unbeatable any other way: Use find instead of
directory-files-recursively, and operate on files as they find prints
them.  Since this runs the directory traversal in parallel with Emacs,
it has a speed advantage that is impossible to match in
directory-files-recursively.

We can fall back to directory-files-recursively when find is not
available.

> Optimizing the case of a long
> list of omissions should come later, as it is a minor optimization.

This seems wrong.  directory-files-recursively is rarely used, and rgrep
is a very popular command, and this problem with find makes rgrep around
~10x slower by default.  How in any world is that a minor optimization?
Most Emacs users will never realize that they can speed up rgrep
massively by setting grep-find-ignored-files to nil.  Indeed, no-one
realized that until I just pointed it out.  In my experience, they just
stop using rgrep in favor of other third-party packages like ripgrep,
because "grep is slow".

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 10:43:01 +0000
Resent-Message-ID: <handler.64735.B64735.169002254930953 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169002254930953
          (code B ref 64735); Sat, 22 Jul 2023 10:43:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 10:42:29 +0000
Received: from localhost ([127.0.0.1]:35531 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNA49-000838-VS
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:42:29 -0400
Received: from s.wrqvwxzv.outbound-mail.sendgrid.net ([149.72.154.232]:27498)
 by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from
 <bounces+21787432-b76c-64735=debbugs.gnu.org@HIDDEN>)
 id 1qNA3v-00082i-BF
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 06:42:24 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=catern.com;
 h=from:subject:in-reply-to:references:mime-version:to:cc:content-type:
 content-transfer-encoding:cc:content-type:from:subject:to;
 s=s1; bh=RdHRp03vjHa+1ZfbHGooFdIIB5sGQ+tAY9sAqKbuNss=;
 b=teAeOAl+Bw5jkoZDMlz0ASTHWOLaY1bSkOJ36erP8BzuWP/jjauhssnapqTMo/9L2kPS
 9fti2VD4exrIqLIZVoS2/9axFwCIfJ+pItr/obBuczU/UHlzHL6DTQL07QsWUvzrWdIrLE
 Z2wZha2O8vKGEi+LOOlKvIhfAwQshQSbfjy2Mdh81qcwJtLYB1iHcPN87iMMXTmt6kkl4X
 rHIZ09eAqW+5fdsBe8277uamtolaM+PxmrYVCt0hyHS5tGHWOTy38s5zFNfOYTVtTNM9Mr
 7vjUBJ5EKjF0qI1r9OyIPr5UpaIe+aQtuIHE5p5HkjISkcbdu71doOzwV0YwRufw==
Received: by filterdrecv-66949dbc98-9d8bm with SMTP id
 filterdrecv-66949dbc98-9d8bm-1-64BBB27E-1
 2023-07-22 10:42:06.033193656 +0000 UTC m=+6260549.422128204
Received: from earth.catern.com (unknown) by geopod-ismtpd-3 (SG) with ESMTP
 id xquYvXL7Q9OuYzrhCwRkmA Sat, 22 Jul 2023 10:42:05.909 +0000 (UTC)
X-Comment: SPF check N/A for local connections - client-ip=::1; helo=localhost;
 envelope-from=sbaugh@HIDDEN; receiver=posteo.net 
Received: from localhost (localhost [IPv6:::1])
 by earth.catern.com (Postfix) with ESMTPSA id 6A48F60166;
 Sat, 22 Jul 2023 06:42:05 -0400 (EDT)
From: sbaugh@HIDDEN
In-Reply-To: <87edl08d1t.fsf@localhost> (Ihor Radchenko's message of "Sat, 22
 Jul 2023 10:18:22 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87edl08d1t.fsf@localhost>
Date: Sat, 22 Jul 2023 10:42:06 +0000 (UTC)
Message-ID: <87fs5gi5xe.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
X-SG-EID: ZgbRq7gjGrt0q/Pjvxk7wM0yQFRdOkTJAtEbkjCkHbJSRTbRpqF3T0QMtzR7x2X/ZGgD15eka72bRqGmmDcanJaKvC7AHYUCAp5gnzOkw2hqgQFhz3Q5rpmKy3fhBbuFZHXG+B8S3yuJkdllctoDdgIHhC8O59lrCy4i+l715yexau9eTYj2AIlp1P2lZDrXXZfKUS0XTrBDa2mZo2aYfw==
X-Entity-ID: d/0VcHixlS0t7iB1YKCv4Q==
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

Ihor Radchenko <yantar92@HIDDEN> writes:
> Spencer Baugh <sbaugh@HIDDEN> writes:
>
>> - we could use our own recursive directory-tree walking implementation
>> (directory-files-recursively), if we found a nice way to pipe its output
>> directly to grep etc without going through Lisp.  (This could be nice
>> for project-files, at least)
>
> May you elaborate this idea?

One of the reasons directory-files-recursively is slow is because it
allocates memory inside Emacs.  If we piped its output directly to grep,
that overhead would be removed.

On reflection, though, as I've posted elsewhere in this thread: This is
a bad idea and is inherently slower than find, because
directory-files-recursively does not run in parallel with Emacs (and
never will).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 11:59:02 +0000
Resent-Message-ID: <handler.64735.B64735.169002710416141 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN
Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169002710416141
          (code B ref 64735); Sat, 22 Jul 2023 11:59:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 11:58:24 +0000
Received: from localhost ([127.0.0.1]:35588 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNBFg-0004CH-3t
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 07:58:24 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:50762)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNBFd-0004C1-E1
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 07:58:22 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNBFX-0007bw-8W; Sat, 22 Jul 2023 07:58:15 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=84MqFbzxw4wvGtn4ZxLkS/dj2BZ3tK/v8PlCYFhzn5A=; b=lLUWEPZ9KhHh
 tlja7emkSn/e2lniSK3zAJXA5Zh+7w5vzqRmxLi6rMtIHx++HzNN7nYU74cnuYTwJDzZpLKFhZvGB
 rZ3k5ap1cYCYc6REjKrn8zh1TGTA6Pcy+fhyvAxF4KLPA81ZhmcgEPbypdyvf9ww1HCI0Y76/x2tl
 0UwMhERUgiA2n92gfrpRbJff0RcPD9XJeitlRnO0K8eSlA09KNfTEvgDR36q7AnCKNohqOIlwHeKH
 S8pLEk08lLIOGoi0f1Eeel8K8hDykjvGPjMV2XA24l/Bnq+HkxG5sDM9u5I6lVOZs3QfsjOoPdxZU
 j1o0I/3aIfCgwaQYy025jQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNBFR-0006sJ-5p; Sat, 22 Jul 2023 07:58:09 -0400
Date: Sat, 22 Jul 2023 14:58:46 +0300
Message-Id: <83sf9g88eh.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87ilaci637.fsf@HIDDEN> (sbaugh@HIDDEN)
References: <iermszrwqj4.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: sbaugh@HIDDEN
> Date: Sat, 22 Jul 2023 10:38:37 +0000 (UTC)
> Cc: Spencer Baugh <sbaugh@HIDDEN>, dmitry@HIDDEN,
> 	yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN,
> 	64735 <at> debbugs.gnu.org
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> > No, the first step is to use in Emacs what Find does today, because it
> > will already be a significant speedup.
> 
> Why bother?  directory-files-recursively is a rarely used API, as you
> have mentioned before in this thread.

Because we could then use it much more (assuming the result will be
performant enough -- this remains to be seen).

> And there is a way to speed it up which will have a performance boost
> which is unbeatable any other way: Use find instead of
> directory-files-recursively, and operate on files as they find prints
> them.

Not every command can operate on the output sequentially: some need to
see all of the output, others will need to be redesigned and
reimplemented to support such sequential mode.

Moreover, piping from Find incurs overhead: data is broken into blocks
by the pipe or PTY, reading the data can be slowed down if Emacs is
busy processing something, etc.

So I think a primitive that traverses the tree and produces file names
with or without attributes, and can call some callback if needed,
still has its place.

> Since this runs the directory traversal in parallel with Emacs, it
> has a speed advantage that is impossible to match in
> directory-files-recursively.

See above: you have an optimistic view of what actually happens in the
relevant use cases.

> We can fall back to directory-files-recursively when find is not
> available.

Find is already available today on many platforms, and we are
evidently not happy enough with the results.  That is the trigger for
this discussion, isn't it?  We are talking about ways to improve the
performance, and I think having our own primitive that can do it is
one such way, or at least it is not clear that it cannot be such a
way.

> > Optimizing the case of a long
> > list of omissions should come later, as it is a minor optimization.
> 
> This seems wrong.  directory-files-recursively is rarely used, and rgrep
> is a very popular command, and this problem with find makes rgrep around
> ~10x slower by default.  How in any world is that a minor optimization?
> Most Emacs users will never realize that they can speed up rgrep
> massively by setting grep-find-ignored-files to nil.  Indeed, no-one
> realized that until I just pointed it out.  In my experience, they just
> stop using rgrep in favor of other third-party packages like ripgrep,
> because "grep is slow".

Making grep-find-ignored-files smaller is independent of this
particular issue.  If we can make it shorter, we should.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 12:00:02 +0000
Resent-Message-ID: <handler.64735.B64735.169002717316260 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN
Cc: sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169002717316260
          (code B ref 64735); Sat, 22 Jul 2023 12:00:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 11:59:33 +0000
Received: from localhost ([127.0.0.1]:35593 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNBGm-0004EC-NR
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 07:59:33 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47920)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNBGk-0004Dy-Uv
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 07:59:31 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNBGd-0007ka-Nu; Sat, 22 Jul 2023 07:59:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=Zpat+D4go7rtCXaY2yPAbzi5PQ4ZzVDUz3OKW2hExaM=; b=XN3Pvf9rKELv
 B4MizkQzWxRttWvfcOrhO4Nwva1mrJCFbZbpiKps0NbpXdN2YxudB4C6G56pE2oCoRAGTpdAKwaeh
 dNb9jdn0xOUv10Vlm8z7369Ydwe6w3J1rDBcCDZEjl/6WO75wNZS0aXAzPoF43Ydws7rGK0oAAx52
 sqA3JjHx1SAomHDHMq6OI6GcrJs8K+qfB0RgkUq79hWzT+fovdT+Xqw9Dhj1ornIJVPS0rqhvIUbV
 Co83/y/RuMJ5wRiTJjqUzt9nU/SETmoYclWc8D3bH1jsX4pFrLX+ASz991Z6kra8DThKOB+hUhAEJ
 f2dO27EVKRBLgMuxwfg8Iw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNBGb-0008RB-SW; Sat, 22 Jul 2023 07:59:22 -0400
Date: Sat, 22 Jul 2023 15:00:00 +0300
Message-Id: <83r0p088cf.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87fs5gi5xe.fsf@HIDDEN> (sbaugh@HIDDEN)
References: <iermszrwqj4.fsf@HIDDEN> <87edl08d1t.fsf@localhost>
 <87fs5gi5xe.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Spencer Baugh <sbaugh@HIDDEN>, 64735 <at> debbugs.gnu.org
> From: sbaugh@HIDDEN
> Date: Sat, 22 Jul 2023 10:42:06 +0000 (UTC)
> 
> Ihor Radchenko <yantar92@HIDDEN> writes:
> > Spencer Baugh <sbaugh@HIDDEN> writes:
> >
> >> - we could use our own recursive directory-tree walking implementation
> >> (directory-files-recursively), if we found a nice way to pipe its output
> >> directly to grep etc without going through Lisp.  (This could be nice
> >> for project-files, at least)
> >
> > May you elaborate this idea?
> 
> One of the reasons directory-files-recursively is slow is because it
> allocates memory inside Emacs.  If we piped its output directly to grep,
> that overhead would be removed.

How can you do anything in Emacs without allocating memory?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 14:15:01 +0000
Resent-Message-ID: <handler.64735.B64735.169003526331847 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169003526331847
          (code B ref 64735); Sat, 22 Jul 2023 14:15:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 14:14:23 +0000
Received: from localhost ([127.0.0.1]:37395 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNDNH-0008Ha-Ex
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 10:14:23 -0400
Received: from mout01.posteo.de ([185.67.36.65]:47837)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNDNF-0008HK-Fn
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 10:14:22 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id E5E81240029
 for <64735 <at> debbugs.gnu.org>; Sat, 22 Jul 2023 16:14:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690035255; bh=WIZexkpDDEDHYR9XRHB19c3FQey8YwWQK60VhlimEdI=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=RIcoeJ8KVwsrIteNQqyHJVBsKhn1JmH4oAoDHVFmW0k3UWeiIjZYEhptpdtRLTHZX
 bt6+cO6KrNAzusGrNAXt0HRfOcgtjJW/HYcAhof0E2DDdACXv7zzhRDY+z0dH/NSCI
 YuxpzfYaSnchxd3GAc5MhB9Iy9WUS8S+ZsUfdUORnZIt3u80qejMtcXwvixwMd/tJ0
 mdNFOPkyuE9koj++38xX00MsXMncDVHPwlLDjZ78t1nwrvv/MzTASLUovXIWgkAFrg
 OD/xFFxdHF24xBfK69O998xHTZqDGbhzbce2AX6+8mnnLVU1PF78fa2CxVKzUir5Q0
 l2uZeOdd1wvlQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7T1x3V86z9s3H;
 Sat, 22 Jul 2023 16:14:13 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83sf9g88eh.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 14:14:25 +0000
Message-ID: <87pm4krq2m.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> So I think a primitive that traverses the tree and produces file names
> with or without attributes, and can call some callback if needed,
> still has its place.

Do you mean asynchronous primitive?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 14:32:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900363151205 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900363151205
          (code B ref 64735); Sat, 22 Jul 2023 14:32:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 14:31:55 +0000
Received: from localhost ([127.0.0.1]:37419 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNDeF-0000JN-BJ
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 10:31:55 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42254)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNDeC-0000J5-TB
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 10:31:54 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNDe7-0004XA-AJ; Sat, 22 Jul 2023 10:31:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=/yg88qKF3g4MXeeh40OVFNqkGdDc926dGxx6141b9qI=; b=D2rVKk8FwHX5
 LkHZGwPJKI+LHeAaxxSjJ9tDdJSZfyjhQp64JbDcbrcd1Y1/b3u1RvVtpINGGBQxKz1p25c4fZVWZ
 lv2a1LUebIJUMSdANOqLTMJgiXkp39CHxDzxob92y8TziRjAQwVuWh1Q5+U30exX4wbrL8p+J3jrz
 w/MOkVFRKfWWe2XORz96m+Cjs03Wayji3Q5Gyz9bdmXJLV349/nRx3UbH6uM/a3esQNWwxdZvhGFe
 D3i4cS0SoOd5SzQ1NJtWeWovaEURdUKjdQrX2hBSzN4MMgYkICljsyZ2Qe9tf1F4r5gR7YoTJSu9l
 gfg9qJCox3jOnHmUXUATAA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNDe1-00016D-4T; Sat, 22 Jul 2023 10:31:41 -0400
Date: Sat, 22 Jul 2023 17:32:19 +0300
Message-Id: <838rb881ak.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87pm4krq2m.fsf@localhost> (message from Ihor Radchenko on Sat,
 22 Jul 2023 14:14:25 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sat, 22 Jul 2023 14:14:25 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > So I think a primitive that traverses the tree and produces file names
> > with or without attributes, and can call some callback if needed,
> > still has its place.
> 
> Do you mean asynchronous primitive?

No, a synchronous one.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 15:08:01 +0000
Resent-Message-ID: <handler.64735.B64735.16900384675003 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900384675003
          (code B ref 64735); Sat, 22 Jul 2023 15:08:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 15:07:47 +0000
Received: from localhost ([127.0.0.1]:37466 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNECw-0001Id-PZ
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 11:07:47 -0400
Received: from mout01.posteo.de ([185.67.36.65]:37631)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNECu-0001IQ-G4
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 11:07:45 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id A759D240028
 for <64735 <at> debbugs.gnu.org>; Sat, 22 Jul 2023 17:07:38 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690038458; bh=PGIo2UYi1ciachkniOJ0fjVbYtnYL7qfxQpyPwTp0Qs=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=K4UcUHb6Uh1c7uw4hbhwa8UP0TXjVXyUh5nryEjjj5JCWjU24lTBLYZnwN0DXZpwZ
 Gto2YrrTq/sxTAs5IlMT6sNhYL+lPQKpA5q+hPvnHNxOeJu0IdnuNSDZw6akGkgZBt
 Y1MEulduz/BHOGKR8iu5OIGdke6bPuCwyHYdKRiJYyElyVSZbqfhbVhjKt0+BLWINo
 pqgoYrMSBYHq7ppVjpi/Culjs9/X48Ee9eZN73Jmbig2BsnQ7AlyyCXq0R/Hu2VOms
 yle3SnT/FnKD2gsSAItAdGiPJ84sRitve96dya5wK+HtHxqemnjyf1uh8msYxayao8
 CTJRSAWNv2opw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7VCY4Qknz6tsg;
 Sat, 22 Jul 2023 17:07:37 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <838rb881ak.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 15:07:45 +0000
Message-ID: <87mszornlq.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Ihor Radchenko <yantar92@HIDDEN>
>> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
>> Date: Sat, 22 Jul 2023 14:14:25 +0000
>> 
>> Eli Zaretskii <eliz@HIDDEN> writes:
>> 
>> > So I think a primitive that traverses the tree and produces file names
>> > with or without attributes, and can call some callback if needed,
>> > still has its place.
>> 
>> Do you mean asynchronous primitive?
>
> No, a synchronous one.

Then how will the callback be different from
(mapc #'my-function (directory-files-recursively ...))
?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 15:30:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900397527047 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900397527047
          (code B ref 64735); Sat, 22 Jul 2023 15:30:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 15:29:12 +0000
Received: from localhost ([127.0.0.1]:37480 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNEXf-0001pb-LU
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 11:29:12 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:45576)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNEXd-0001pD-CO
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 11:29:10 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNEXW-000332-O7; Sat, 22 Jul 2023 11:29:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=Rggkwh1/pJ8WUMA6W5kQZr/BtER1xtpzAdD4yE/9Ne8=; b=o6JdU9Cd906v
 X10wRem8hgXnQWEPK2uT89F7kLlH7ts+M11+Yc9Sb9fkOnGEWz74ZLW7nFIRAkxHtBzk6J+mmZPUa
 Ry8+/bG7mw9r7jRKb+lQl3oPx3W57Cgt0pWafaMf84sL9TcdYUf5LLcwXCWI/WKaO6M0SejRYvbdI
 5Fd7ElUp8U6uxpv0ZEyytJHWKmIhq/dXd1ciOiMNjNCWoCtKqg8pW7w3vwclCT2HHGlf0aETP4hNF
 L/8LefC0kSCS5rnjdSq/JVfhxOsdAjKSdpfWhdWlZR4agy1ra67wFX/zWiLTLmkL8orWUJtgIYhu1
 h84xJRNxIJ15sldsfQiRng==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNEXQ-0002FP-OC; Sat, 22 Jul 2023 11:28:57 -0400
Date: Sat, 22 Jul 2023 18:29:34 +0300
Message-Id: <83351g7yn5.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87mszornlq.fsf@localhost> (message from Ihor Radchenko on Sat,
 22 Jul 2023 15:07:45 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN> <87mszornlq.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sat, 22 Jul 2023 15:07:45 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> > So I think a primitive that traverses the tree and produces file names
> >> > with or without attributes, and can call some callback if needed,
> >> > still has its place.
> >> 
> >> Do you mean asynchronous primitive?
> >
> > No, a synchronous one.
> 
> Then how will the callback be different from
> (mapc #'my-function (directory-files-recursively ...))
> ?

It depends on the application.  Applications that want to get all the
data and only after that process it will not use the callback.  But I
can certainly imagine an application that inserts the file names, or
some of their transforms, into a buffer, and from time to time
triggers redisplay to show the partial results.  Or an application
could write the file names to some disk file or external consumer, or
send them to a network process.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 17:19:02 +0000
Resent-Message-ID: <handler.64735.B64735.169004630917619 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169004630917619
          (code B ref 64735); Sat, 22 Jul 2023 17:19:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 17:18:29 +0000
Received: from localhost ([127.0.0.1]:37526 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNGFQ-0004a6-FT
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:18:29 -0400
Received: from s.wrqvtzvf.outbound-mail.sendgrid.net ([149.72.126.143]:13978)
 by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from
 <bounces+21787432-b76c-64735=debbugs.gnu.org@HIDDEN>)
 id 1qNGFO-0004Zt-1I
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:18:27 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=catern.com;
 h=from:subject:in-reply-to:references:mime-version:to:cc:content-type:
 content-transfer-encoding:cc:content-type:from:subject:to;
 s=s1; bh=eD958oiQjQIrk0Ld6XP5QZWP3il36+j1PcE1wIQrrRM=;
 b=mD0zv01PsNCOs3TTg4SQLP9DQBEgEkYhptmAkpZBN/wSEP8nYe0pY2ZKa2KtNxyfoHFZ
 fn7j3RtrVpNfsNytcb890LE6O+Y7ycfFxURednm90k3u4YvqX5jLM3Vy2e4TqGC+j/6rm5
 2T0tHbV8PwRnEInateAHQn/Qnxu6/vJ57EL0RicHLxrp9MwDleVlKDhtvCxg1w6Jyu/I3U
 QzGOjs92Eu7BkyalVfpY8dgtMlO+yJNUsdIwJGal0CWawXr1qiH3PMt87aZd/wSsAC3s4X
 DIkCXSvFyd2MPmxhk1Y3YE7zt4rumiGq6w4JR+YvqXkjSBwP5T8DaIHhN/kxkkeg==
Received: by filterdrecv-84b96456cb-b5mzh with SMTP id
 filterdrecv-84b96456cb-b5mzh-1-64BC0F5B-A
 2023-07-22 17:18:19.641567595 +0000 UTC m=+6284401.883235998
Received: from earth.catern.com (unknown) by geopod-ismtpd-8 (SG) with ESMTP
 id b_UL3K0eRH2wUziXbaGQpw Sat, 22 Jul 2023 17:18:19.449 +0000 (UTC)
X-Comment: SPF check N/A for local connections - client-ip=::1; helo=localhost;
 envelope-from=sbaugh@HIDDEN; receiver=gnu.org 
Received: from localhost (localhost [IPv6:::1])
 by earth.catern.com (Postfix) with ESMTPSA id 1AA0A60077;
 Sat, 22 Jul 2023 13:18:19 -0400 (EDT)
From: sbaugh@HIDDEN
In-Reply-To: <83sf9g88eh.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 22 Jul
 2023 14:58:46 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 17:18:19 +0000 (UTC)
Message-ID: <87cz0jj25g.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
X-SG-EID: ZgbRq7gjGrt0q/Pjvxk7wM0yQFRdOkTJAtEbkjCkHbJf+I76yiUOSLF77crDivv8+uvwSZTFgqtahh2tqM+F7M2mOi6ydlNgAw8AFwCkWrnWqEAmfE88qKoTvOwipr7IJTIoYAGJEQeq2qPZMhJhYkAh84G8xLIJ4nPFW7SRJ+7EHZ9DEBNKsEgNyyJpTWsQpm2Kbqh1cmrcAk3QRKS6TQ==
X-Entity-ID: d/0VcHixlS0t7iB1YKCv4Q==
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: sbaugh@HIDDEN
>> Date: Sat, 22 Jul 2023 10:38:37 +0000 (UTC)
>> Cc: Spencer Baugh <sbaugh@HIDDEN>, dmitry@HIDDEN,
>> 	yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN,
>> 	64735 <at> debbugs.gnu.org
>> 
>> Eli Zaretskii <eliz@HIDDEN> writes:
>> > No, the first step is to use in Emacs what Find does today, because it
>> > will already be a significant speedup.
>> 
>> Why bother?  directory-files-recursively is a rarely used API, as you
>> have mentioned before in this thread.
>
> Because we could then use it much more (assuming the result will be
> performant enough -- this remains to be seen).
>
>> And there is a way to speed it up which will have a performance boost
>> which is unbeatable any other way: Use find instead of
>> directory-files-recursively, and operate on files as they find prints
>> them.
>
> Not every command can operate on the output sequentially: some need to
> see all of the output, others will need to be redesigned and
> reimplemented to support such sequential mode.
>
> Moreover, piping from Find incurs overhead: data is broken into blocks
> by the pipe or PTY, reading the data can be slowed down if Emacs is
> busy processing something, etc.

I went ahead and implemented it, and I get a 2x speedup even *without*
running find in parallel with Emacs.

First my results:

(my-bench 100 "~/public_html" "")
(("built-in" . "Elapsed time: 1.140173s (0.389344s in 5 GCs)")
 ("with-find" . "Elapsed time: 0.643306s (0.305130s in 4 GCs)"))

(my-bench 10 "~/.local/src/linux" "")
(("built-in" . "Elapsed time: 2.402341s (0.937857s in 11 GCs)")
 ("with-find" . "Elapsed time: 1.544024s (0.827364s in 10 GCs)"))

(my-bench 100 "/ssh:catern.com:~/public_html" "")
(("built-in" . "Elapsed time: 36.494233s (6.450840s in 79 GCs)")
 ("with-find" . "Elapsed time: 4.619035s (1.133656s in 14 GCs)"))

2x speedup on local files, and almost a 10x speedup for remote files.

And my implementation *isn't even using the fact that find can run in
parallel with Emacs*.  If I did start using that, I expect even more
speed gains from parallelism, which aren't achievable in Emacs itself.

So can we add something like this (with the appropriate fallbacks to
directory-files-recursively), since it has such a big speedup even
without parallelism?

My implementation and benchmarking:

(defun find-directory-files-recursively (dir regexp &optional include-directories _predicate follow-symlinks)
  (cl-assert (null _predicate) t "find-directory-files-recursively can't accept arbitrary predicates")
  (with-temp-buffer
    (setq case-fold-search nil)
    (cd dir)
    (let* ((command
	    (append
	     (list "find" (file-local-name dir))
	     (if follow-symlinks
		 '("-L")
	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
	     (unless (string-empty-p regexp)
	       "-regex" (concat ".*" regexp ".*"))
	     (unless include-directories
	       '("!" "-type" "d"))
	     '("-print0")
	     ))
	   (remote (file-remote-p dir))
	   (proc
	    (if remote
		(let ((proc (apply #'start-file-process
				   "find" (current-buffer) command)))
		  (set-process-sentinel proc (lambda (_proc _state)))
		  (set-process-query-on-exit-flag proc nil)
		  proc)
	      (make-process :name "find" :buffer (current-buffer)
			    :connection-type 'pipe
			    :noquery t
			    :sentinel (lambda (_proc _state))
			    :command command))))
      (while (accept-process-output proc))
      (let ((start (goto-char (point-min))) ret)
	(while (search-forward "\0" nil t)
	  (push (concat remote (buffer-substring-no-properties start (1- (point)))) ret)
	  (setq start (point)))
	ret))))

(defun my-bench (count path regexp)
  (setq path (expand-file-name path))
  (let ((old (directory-files-recursively path regexp))
	(new (find-directory-files-recursively path regexp)))
    (dolist (path old)
      (should (member path new)))
    (dolist (path new)
      (should (member path old))))
  (list
   (cons "built-in" (benchmark count (list 'directory-files-recursively path regexp)))
   (cons "with-find" (benchmark count (list 'find-directory-files-recursively path regexp)))))

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 17:27:02 +0000
Resent-Message-ID: <handler.64735.B64735.169004681818382 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN
Cc: sbaugh@HIDDEN, rms@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169004681818382
          (code B ref 64735); Sat, 22 Jul 2023 17:27:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 17:26:58 +0000
Received: from localhost ([127.0.0.1]:37530 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNGNd-0004mQ-PR
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:26:58 -0400
Received: from mout02.posteo.de ([185.67.36.66]:49073)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNGNb-0004mC-CF
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:26:55 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 6846E240104
 for <64735 <at> debbugs.gnu.org>; Sat, 22 Jul 2023 19:26:48 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690046808; bh=/3KqGjvuZ/WucD2J9KgOIvjIMwrXz8FbfcoMS0yENDI=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=SU7ZoSmTUGk76ZjwuVmtBNEf5ptBHv1R1hPfmClfH+aenjDRt+37YeY/aKlDruKMb
 51IClcgJXPSITmibMX6qXs2vcihezawvl5Agqqp7dVQK24IiHQROtOBjM/kO+pHtLt
 JrVOwvIVE5RY5I7Tlu25aUFXQC68/9sqlNdhr0R2hXSxrkAACccGsTYw46HA+p5/Gi
 vZTpcZXLhg1sZN9bqCaLmaCSTBMZ1cso0FEMIcSFM9OL/c66LhTkAYIBotErDEJygJ
 YDniLbgAvUqY5dyGgedXPsAbtWT00toyaAaX7LZxWS4VwTUqPwoGgXe/hhTC4YEBzq
 H9cNO7fkQ26ow==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7YJ71gjnz6txs;
 Sat, 22 Jul 2023 19:26:46 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87cz0jj25g.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87fs5hmp6i.fsf@localhost>
 <87cz0lmoxy.fsf@localhost> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 17:26:54 +0000
Message-ID: <87tttvsvq9.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

sbaugh@HIDDEN writes:

> I went ahead and implemented it, and I get a 2x speedup even *without*
> running find in parallel with Emacs.
>
> First my results:
>
> (my-bench 100 "~/public_html" "")
> (("built-in" . "Elapsed time: 1.140173s (0.389344s in 5 GCs)")
>  ("with-find" . "Elapsed time: 0.643306s (0.305130s in 4 GCs)"))
>
> (my-bench 10 "~/.local/src/linux" "")
> (("built-in" . "Elapsed time: 2.402341s (0.937857s in 11 GCs)")
>  ("with-find" . "Elapsed time: 1.544024s (0.827364s in 10 GCs)"))

What about without `file-name-handler-alist'?

> (my-bench 100 "/ssh:catern.com:~/public_html" "")
> (("built-in" . "Elapsed time: 36.494233s (6.450840s in 79 GCs)")
>  ("with-find" . "Elapsed time: 4.619035s (1.133656s in 14 GCs)"))

This is indeed expected.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 17:46:02 +0000
Resent-Message-ID: <handler.64735.B64735.169004794120128 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN
Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169004794120128
          (code B ref 64735); Sat, 22 Jul 2023 17:46:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 17:45:41 +0000
Received: from localhost ([127.0.0.1]:37544 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNGfk-0005EZ-VU
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:45:41 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:40836)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNGfi-0005EM-4B
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 13:45:39 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNGfb-0004iT-Pt; Sat, 22 Jul 2023 13:45:31 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=bKTVOF9y9vUbCMp9Lv9+yw8MxYmMaeqfltn0EszC6lQ=; b=EtqU/6GBD/w1
 PYE0VK+VuJg+i+qBVz6675PKmkmh9CxZIAVr2uf0YspSnJM3u8F3ul2+1b4KpDF5onx0av+E7L1ja
 Kd+SaVp16qA50D+W1EN4hu0GcFrzhnW3ydplpxQw23cikhUFVlCDyMj5zb1kgAfDf8tX37wCzFIAJ
 5VEmeXAOEN/I+65A/73NTKQ0gTqmGwqJ2XQTgQ0qdJSbkdPjsQtVhSNIPI+jwyXq92yDC7RLHyJnZ
 fZkPVx20E17vEUbkO/0YnHFAJVuZKdZaSDLxgvdUSpFZAs7wbGlvA1puAU/2kWsyN4AafaJawkVyk
 ySNc1qdcIRX8rEy8wBo2xw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNGfU-0005FA-KZ; Sat, 22 Jul 2023 13:45:24 -0400
Date: Sat, 22 Jul 2023 20:46:01 +0300
Message-Id: <83wmyr7sbq.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87cz0jj25g.fsf@HIDDEN> (sbaugh@HIDDEN)
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: sbaugh@HIDDEN
> Date: Sat, 22 Jul 2023 17:18:19 +0000 (UTC)
> Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN,
> 	michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
> 
> First my results:
> 
> (my-bench 100 "~/public_html" "")
> (("built-in" . "Elapsed time: 1.140173s (0.389344s in 5 GCs)")
>  ("with-find" . "Elapsed time: 0.643306s (0.305130s in 4 GCs)"))
> 
> (my-bench 10 "~/.local/src/linux" "")
> (("built-in" . "Elapsed time: 2.402341s (0.937857s in 11 GCs)")
>  ("with-find" . "Elapsed time: 1.544024s (0.827364s in 10 GCs)"))
> 
> (my-bench 100 "/ssh:catern.com:~/public_html" "")
> (("built-in" . "Elapsed time: 36.494233s (6.450840s in 79 GCs)")
>  ("with-find" . "Elapsed time: 4.619035s (1.133656s in 14 GCs)"))
> 
> 2x speedup on local files, and almost a 10x speedup for remote files.

Thanks, that's impressive.  But you omitted some of the features of
directory-files-recursively, see below.

> And my implementation *isn't even using the fact that find can run in
> parallel with Emacs*.  If I did start using that, I expect even more
> speed gains from parallelism, which aren't achievable in Emacs itself.

I'm not sure I understand what you mean by "in parallel" and why it
would be faster.

> So can we add something like this (with the appropriate fallbacks to
> directory-files-recursively), since it has such a big speedup even
> without parallelism?

We can have an alternative implementation, yes.  But it should support
predicate, and it should sort the files in each directory like
directory-files-recursively does, so that it's a drop-in replacement.
Also, I believe that Find does return "." in each directory, and your
implementation doesn't filter them, whereas
directory-files-recursively does AFAIR.

And I see no need for any fallback: that's for the application to do
if it wants.

>   (cl-assert (null _predicate) t "find-directory-files-recursively can't accept arbitrary predicates")

It should.

> 	     (if follow-symlinks
> 		 '("-L")
> 	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
> 	     (unless (string-empty-p regexp)
> 	       "-regex" (concat ".*" regexp ".*"))
> 	     (unless include-directories
> 	       '("!" "-type" "d"))
> 	     '("-print0")

Some of these switches are specific to GNU Find.  Are we going to
support only GNU Find?

> 	     ))
> 	   (remote (file-remote-p dir))
> 	   (proc
> 	    (if remote
> 		(let ((proc (apply #'start-file-process
> 				   "find" (current-buffer) command)))
> 		  (set-process-sentinel proc (lambda (_proc _state)))
> 		  (set-process-query-on-exit-flag proc nil)
> 		  proc)
> 	      (make-process :name "find" :buffer (current-buffer)
> 			    :connection-type 'pipe
> 			    :noquery t
> 			    :sentinel (lambda (_proc _state))
> 			    :command command))))
>       (while (accept-process-output proc))

Why do you call accept-process-output here? it could interfere with
reading output from async subprocesses running at the same time.  To
come think of this, why use async subprocesses here and not
call-process?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 18:31:01 +0000
Resent-Message-ID: <handler.64735.B64735.169005065424636 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN, sbaugh@HIDDEN
Cc: dmitry@HIDDEN, yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169005065424636
          (code B ref 64735); Sat, 22 Jul 2023 18:31:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 18:30:54 +0000
Received: from localhost ([127.0.0.1]:37571 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNHNV-0006PI-T3
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 14:30:54 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59894)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNHNT-0006Os-3R
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 14:30:52 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNHNN-00056Z-4q; Sat, 22 Jul 2023 14:30:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=wwIJ3hXVgxXXSiWcZNJmiLbgxASrQvLWP9j/bmN2Oy8=; b=f3oSE/PxgaxX
 Ssk+ayhioFbald7Zxify3E4izcP52CIcMQKHo56GcV+i051Z1YIzZY2OrKo8MhMVf2l6G3w79U5CD
 Ao4vPWM1qaPAIcQ8BE5i9rZklORdSf+r2hc9avyskwX2KrzQyY+h5wEMbl8ul/7j4iiVCdKKY4rFO
 xX3JZOq0DRaZALHeWqaKZL1SeO/RKIg1BnEg8K4CbsRoDW2d9kVzE+yPSAdW6tNtMw3NriMyXYKPX
 sGLo9LkJqQdkfbtxgP8Tnuzm/9tsYyvmlhf735f6tGAZdY0utmNYSuYtHYBKGsvKrRZMHXuqlxCza
 t7CkTsdcr2TJJ6bm6xEHjw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNHNE-0008Lv-On; Sat, 22 Jul 2023 14:30:37 -0400
Date: Sat, 22 Jul 2023 21:31:14 +0300
Message-Id: <83v8eb7q8d.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <83wmyr7sbq.fsf@HIDDEN> (message from Eli Zaretskii on Sat, 22
 Jul 2023 20:46:01 +0300)
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sat, 22 Jul 2023 20:46:01 +0300
> From: Eli Zaretskii <eliz@HIDDEN>
> 
> > First my results:
> > 
> > (my-bench 100 "~/public_html" "")
> > (("built-in" . "Elapsed time: 1.140173s (0.389344s in 5 GCs)")
> >  ("with-find" . "Elapsed time: 0.643306s (0.305130s in 4 GCs)"))
> > 
> > (my-bench 10 "~/.local/src/linux" "")
> > (("built-in" . "Elapsed time: 2.402341s (0.937857s in 11 GCs)")
> >  ("with-find" . "Elapsed time: 1.544024s (0.827364s in 10 GCs)"))
> > 
> > (my-bench 100 "/ssh:catern.com:~/public_html" "")
> > (("built-in" . "Elapsed time: 36.494233s (6.450840s in 79 GCs)")
> >  ("with-find" . "Elapsed time: 4.619035s (1.133656s in 14 GCs)"))
> > 
> > 2x speedup on local files, and almost a 10x speedup for remote files.
> 
> Thanks, that's impressive.  But you omitted some of the features of
> directory-files-recursively, see below.

My results on MS-Windows are less encouraging:

  (my-bench 2 "d:/usr/archive" "")
  (("built-in" . "Elapsed time: 1.250000s (0.093750s in 5 GCs)")
   ("with-find" . "Elapsed time: 8.578125s (0.109375s in 7 GCs)"))

D:/usr/archive is a directory with 372 subdirectories and more than
12000 files in all of them.  The disk is SSD, in case it matters, and
I measured this with a warm disk cache.

So I guess whether or not to use this depends on the underlying
system.

Btw, you should not assume that "-type l" will universally work: at
least on MS-Windows some ports of GNU Find will barf when they see it.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 19:07:01 +0000
Resent-Message-ID: <handler.64735.B64735.169005279628625 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN, sbaugh@HIDDEN
Cc: dmitry@HIDDEN, yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169005279628625
          (code B ref 64735); Sat, 22 Jul 2023 19:07:01 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 19:06:36 +0000
Received: from localhost ([127.0.0.1]:37583 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNHw3-0007Rb-V4
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 15:06:36 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:41700)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNHw1-0007RP-De
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 15:06:34 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNHvu-0008CH-6Q; Sat, 22 Jul 2023 15:06:26 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=NNm6DMgiXIKdfH4b8Iqyn4idxdxjF3oxB9t+OnorII0=; b=dqNOz+Gp8JBa
 wWF7J7egrnSYdJxeqtT/0viwYdOzC/Y5CxuBm3aIOdZwtdCGIhp0TGzfOJk/fBnhzwu4+hVQ/TK3k
 ZgOfYJX1Z4e5Waz0Rcc5/ysRoCckNONmlU97y8fiG+hPN86KUutpYROWqt2TdWrO/3HhhnszDc8rx
 be7ROoz0PiuE+5MIZliftV9rfm6jyDZeNTlCipNpczSI5QxWc/XhiYP5z03q+I7zOt1wznlZGfCLq
 bUqJwivYM8Aogz36pAEX7v6jm1ucgfRDh9LwuJwib7xO4AS0vR4jDQ4ImmortMoSbAV02u5v51H6y
 p4Wnscu9rT8Zs6sIYJNA7A==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNHvm-0008KD-Ci; Sat, 22 Jul 2023 15:06:18 -0400
Date: Sat, 22 Jul 2023 22:06:55 +0300
Message-Id: <83tttv7okw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <83v8eb7q8d.fsf@HIDDEN> (message from Eli Zaretskii on Sat, 22
 Jul 2023 21:31:14 +0300)
References: <iermszrwqj4.fsf@HIDDEN> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <83v8eb7q8d.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: dmitry@HIDDEN, yantar92@HIDDEN, michael.albinus@HIDDEN, rms@HIDDEN,
>  64735 <at> debbugs.gnu.org
> Date: Sat, 22 Jul 2023 21:31:14 +0300
> From: Eli Zaretskii <eliz@HIDDEN>
> 
> My results on MS-Windows are less encouraging:
> 
>   (my-bench 2 "d:/usr/archive" "")
>   (("built-in" . "Elapsed time: 1.250000s (0.093750s in 5 GCs)")
>    ("with-find" . "Elapsed time: 8.578125s (0.109375s in 7 GCs)"))

And here's from a GNU/Linux machine, which is probably not very fast:

  (my-bench 10 "/usr/lib" "")
  (("built-in" . "Elapsed time: 4.410613s (2.077311s in 56 GCs)")
   ("with-find" . "Elapsed time: 3.326954s (1.997251s in 54 GCs)"))

Faster, but not by a lot.

On this system /usr/lib has 18000 files in 1860 subdirectories.

Btw, the Find command with pipe to some other program, like wc,
finishes much faster, like 2 to 4 times faster than when it is run
from find-directory-files-recursively.  That's probably the slowdown
due to communications with async subprocesses in action.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Spencer Baugh <sbaugh@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 20:54:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900591947518 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: yantar92@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900591947518
          (code B ref 64735); Sat, 22 Jul 2023 20:54:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 20:53:14 +0000
Received: from localhost ([127.0.0.1]:37623 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNJbF-0001xB-BH
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 16:53:13 -0400
Received: from mxout5.mail.janestreet.com ([64.215.233.18]:51239)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <sbaugh@HIDDEN>) id 1qNJbD-0001wy-27
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 16:53:12 -0400
From: Spencer Baugh <sbaugh@HIDDEN>
In-Reply-To: <83wmyr7sbq.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 22 Jul
 2023 20:46:01 +0300")
References: <iermszrwqj4.fsf@HIDDEN> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN>
 <83wmyr7sbq.fsf@HIDDEN>
Date: Sat, 22 Jul 2023 16:53:05 -0400
Message-ID: <ierwmyru0r2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: sbaugh@HIDDEN
>> Date: Sat, 22 Jul 2023 17:18:19 +0000 (UTC)
>> Cc: sbaugh@HIDDEN, yantar92@HIDDEN, rms@HIDDEN, dmitry@HIDDEN,
>> 	michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
>> 
>> First my results:
>> 
>> (my-bench 100 "~/public_html" "")
>> (("built-in" . "Elapsed time: 1.140173s (0.389344s in 5 GCs)")
>>  ("with-find" . "Elapsed time: 0.643306s (0.305130s in 4 GCs)"))
>> 
>> (my-bench 10 "~/.local/src/linux" "")
>> (("built-in" . "Elapsed time: 2.402341s (0.937857s in 11 GCs)")
>>  ("with-find" . "Elapsed time: 1.544024s (0.827364s in 10 GCs)"))
>> 
>> (my-bench 100 "/ssh:catern.com:~/public_html" "")
>> (("built-in" . "Elapsed time: 36.494233s (6.450840s in 79 GCs)")
>>  ("with-find" . "Elapsed time: 4.619035s (1.133656s in 14 GCs)"))
>> 
>> 2x speedup on local files, and almost a 10x speedup for remote files.
>
> Thanks, that's impressive.  But you omitted some of the features of
> directory-files-recursively, see below.
>
>> And my implementation *isn't even using the fact that find can run in
>> parallel with Emacs*.  If I did start using that, I expect even more
>> speed gains from parallelism, which aren't achievable in Emacs itself.
>
> I'm not sure I understand what you mean by "in parallel" and why it
> would be faster.

I mean having Emacs read output from the process and turn them into
strings while find is still running and walking the directory tree.  So
the two parts are running in parallel.  This, specifically:

(defun find-directory-files-recursively (dir regexp &optional include-directories _predicate follow-symlinks)
  (cl-assert (null _predicate) t "find-directory-files-recursively can't accept arbitrary predicates")
  (cl-assert (not (file-remote-p dir)))
  (let* (buffered
         result
         (proc
	  (make-process
           :name "find" :buffer nil
	   :connection-type 'pipe
	   :noquery t
	   :sentinel (lambda (_proc _state))
           :filter (lambda (proc data)
                     (let ((start 0))
                       (when-let (end (string-search "\0" data start))
                         (push (concat buffered (substring data start end)) result)
                         (setq buffered "")
                         (setq start (1+ end))
                         (while-let ((end (string-search "\0" data start)))
                           (push (substring data start end) result)
                           (setq start (1+ end))))
                       (setq buffered (concat buffered (substring data start)))))
	   :command (append
	             (list "find" (file-local-name dir))
	             (if follow-symlinks
		         '("-L")
	               '("!" "(" "-type" "l" "-xtype" "d" ")"))
	             (unless (string-empty-p regexp)
	               "-regex" (concat ".*" regexp ".*"))
	             (unless include-directories
	               '("!" "-type" "d"))
	             '("-print0")
	             ))))
    (while (accept-process-output proc))
    result))

Can you try this further change on your Windows (and GNU/Linux) box?  I
just tested on a different box and my original change gets:

(("built-in" . "Elapsed time: 4.506643s (2.276269s in 21 GCs)")
 ("with-find" . "Elapsed time: 4.114531s (2.848497s in 27 GCs)"))

while this parallel implementation gets

(("built-in" . "Elapsed time: 4.479185s (2.236561s in 21 GCs)")
 ("with-find" . "Elapsed time: 2.858452s (1.934647s in 19 GCs)"))

so it might have a favorable impact on Windows and your other GNU/Linux
box.

>> So can we add something like this (with the appropriate fallbacks to
>> directory-files-recursively), since it has such a big speedup even
>> without parallelism?
>
> We can have an alternative implementation, yes.  But it should support
> predicate, and it should sort the files in each directory like
> directory-files-recursively does, so that it's a drop-in replacement.
> Also, I believe that Find does return "." in each directory, and your
> implementation doesn't filter them, whereas
> directory-files-recursively does AFAIR.
>
> And I see no need for any fallback: that's for the application to do
> if it wants.
>
>>   (cl-assert (null _predicate) t "find-directory-files-recursively can't accept arbitrary predicates")
>
> It should.

This is where I think a fallback would be useful - it's basically
impossible to support arbitrary predicates efficiently here, since it
requires us to put Lisp in control of whether find descends into a
directory.  So I'm thinking I would just fall back to running the old
directory-files-recursively whenever there's a predicate.  Or just not
supporting this at all...

>> 	     (if follow-symlinks
>> 		 '("-L")
>> 	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
>> 	     (unless (string-empty-p regexp)
>> 	       "-regex" (concat ".*" regexp ".*"))
>> 	     (unless include-directories
>> 	       '("!" "-type" "d"))
>> 	     '("-print0")
>
> Some of these switches are specific to GNU Find.  Are we going to
> support only GNU Find?

POSIX find doesn't support -regex, so I think we have to.  We could
stick to just POSIX find if we only allowed globs in
find-directory-files-recursively, instead of full regexes.

>> 	     ))
>> 	   (remote (file-remote-p dir))
>> 	   (proc
>> 	    (if remote
>> 		(let ((proc (apply #'start-file-process
>> 				   "find" (current-buffer) command)))
>> 		  (set-process-sentinel proc (lambda (_proc _state)))
>> 		  (set-process-query-on-exit-flag proc nil)
>> 		  proc)
>> 	      (make-process :name "find" :buffer (current-buffer)
>> 			    :connection-type 'pipe
>> 			    :noquery t
>> 			    :sentinel (lambda (_proc _state))
>> 			    :command command))))
>>       (while (accept-process-output proc))
>
> Why do you call accept-process-output here? it could interfere with
> reading output from async subprocesses running at the same time.  To
> come think of this, why use async subprocesses here and not
> call-process?

See my new iteration which does use the async-ness.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 22 Jul 2023 21:02:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900597018381 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Po Lu <luangruo@HIDDEN>, Spencer Baugh <sbaugh@HIDDEN>
Cc: Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900597018381
          (code B ref 64735); Sat, 22 Jul 2023 21:02:02 +0000
Received: (at 64735) by debbugs.gnu.org; 22 Jul 2023 21:01:41 +0000
Received: from localhost ([127.0.0.1]:37628 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNJjQ-0002B7-Mf
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 17:01:40 -0400
Received: from out2-smtp.messagingengine.com ([66.111.4.26]:48027)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNJjO-0002Ao-P7
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 17:01:39 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 5E84A5C008C;
 Sat, 22 Jul 2023 17:01:32 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute6.internal (MEProxy); Sat, 22 Jul 2023 17:01:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690059692; x=1690146092; bh=3+viGp4phQ1xFQgQJEcz49ii6AEfod4tzmj
 vuJyEo0I=; b=oDCM9iwQW6iyHKoHjVffh7S+dzIV4f9C5lEx+rp9+IAe9kj6eAY
 p5CC2KIKQ5LYy5j1R7CDjkOyMcI9gNN6HwrBEmRyxkzV1TNYI7vzJ1skdrzjBUyH
 dGxUnX80au/f3CMmWiS7C2lHtNpmF4ISzF8WDf5O4GowKd0NhILv/iYESKivYbUL
 zOJCIyKagv93AlGT0aVXzx+isoHt/0GtvICntFbXc0vDifQa5cTcNSuxqRE/4XC3
 Rh2c1vxXduxzTiRDQbGB7b0eiwQTUWn/0Y7kxfgYsjplX5OuTO0iLlbTgWu3hxeh
 0Tv82/chjst3SIl7WZWZ+dksyeEXLTcJpUA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690059692; x=1690146092; bh=3+viGp4phQ1xFQgQJEcz49ii6AEfod4tzmj
 vuJyEo0I=; b=gsi3S9Q5wzyqNqjogSnZaehus2X4xkxgAI2Ik8inn17+76veW1K
 jnBG/PYJWvWNkcJIYfkqXeyvq0j8fzYg6199nAcmm7Onlyh6z4b4589cbdz7ejSk
 18tGW8OfzVY3bF6CxGIKMn5Crpn3DnKWNrtJm5BazqKCh5FD2AN13DYQ+tx9cJkK
 EoTrDHqQTMqOHRGpDPBiJn4UT6cCnnGfaRAlS7R7Cm2Yc3daHOJQZMEzP4MVEKHs
 8ZBvq/i/syR8bA4Pw2Nnjo8RtisYB3mJT9pTEQzJxwQOO3ZulyPRMemSSksNDZ+X
 nCONtUW7v1D/X7CMtrO53ULv2SrNyZhnr7A==
X-ME-Sender: <xms:rEO8ZHN2JGI0xLvzpcx0j1LZNOHLmRjwbpAhV63MSZ2z4d1UTtWRiA>
 <xme:rEO8ZB_d67-7L33z_lDBT_HJNa1V2Gm8lkPG3dTW59eVTwfzL5PYSKZ189nSD4gMb
 rNcAOyyqW6NWmy3JwI>
X-ME-Received: <xmr:rEO8ZGRYC1L7dxI0of4T8DDnljBkuHSuPK4xunv6bKKgB-IsDMByDqrFSehmX_c>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeggdduheegucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:rEO8ZLvmUZnuMmA_mpq40ZGcx1ikRErY4S35kmebn2HkT6PxOCtAjQ>
 <xmx:rEO8ZPcD0_c-ooKr98x8qm-xwiq7icMZUs4PokK6F4fdVRFRIeprGQ>
 <xmx:rEO8ZH02A3WrgqJGsSMgrbhhBXgvEfL4HIRQSP8aYuTNcOcvHHV-VA>
 <xmx:rEO8ZAr5J7reLZ5Ir9hmM72B1CDSuRU_ciyIJqyRl6FgWapmx03AJg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat,
 22 Jul 2023 17:01:30 -0400 (EDT)
Message-ID: <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
Date: Sun, 23 Jul 2023 00:01:28 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87r0p030w0.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 22/07/2023 09:39, Po Lu wrote:
> Spencer Baugh <sbaugh@HIDDEN> writes:
> 
>> Not to derail too much, but find as a subprocess has one substantial
>> advantage over find in Lisp: It can run in parallel with Emacs, so that
>> we actually use multiple CPU cores.
>>
>> Between that, and the remote support part, I personally much prefer find
>> to be a subprocess rather than in Lisp.  I don't think optimizing
>> directory-files-recursively is a great solution.
>>
>> (Really it's entirely plausible that Emacs could be improved by
>> *removing* directory-files-recursively, in favor of invoking find as a
>> subprocess: faster, parallelized execution, and better remote support.)
> 
> find is only present in the default installations of Unix-like systems,
> so it doesn't work without additional configuration on MS-Windows or
> MS-DOS.  project.el and rgrep fail to work on USG Unix because they both
> use `-path'.
> 
> Programs that use find should fall back to directory-file-recursively
> when any of the situations above are detected.

Perhaps if someone implements support for IGNORE entries (wildcards) in 
that function, it would be easy enough to do that fallback.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Richard Stallman <rms@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 03:00:02 +0000
Resent-Message-ID: <handler.64735.B64735.169008115211357 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: 64735 <at> debbugs.gnu.org
Reply-To: rms@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169008115211357
          (code B ref 64735); Sun, 23 Jul 2023 03:00:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 02:59:12 +0000
Received: from localhost ([127.0.0.1]:37723 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNPJP-0002x6-TY
	for submit <at> debbugs.gnu.org; Sat, 22 Jul 2023 22:59:12 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:57856)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rms@HIDDEN>) id 1qNPJM-0002wq-V3
 for 64735 <at> debbugs.gnu.org; Sat, 22 Jul 2023 22:59:10 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <rms@HIDDEN>)
 id 1qNPJH-0004Az-17; Sat, 22 Jul 2023 22:59:03 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=Date:References:Subject:In-Reply-To:To:From:
 mime-version; bh=IXrGDaWxAQrwyJxip0erJsgL0sUlcn8DTEE7zFxxrCw=; b=i+hl7POsP/jB
 GblENesjP/9HudIrQ7edKxGDGoX2n5QIEh/5NmSt7r5T373uVtiCEimycJBldEe3Y2moj3iuZ9ESD
 0jswU0urWj7gG5BxIHtsuhTcZnGCqrYR5rX0CtB+hONEPbvcF/FajwettvB050Qy2JLjoUPBLm9kv
 X42gIb9Mj1jzVtrToRDl4qkH+kHkYa4C1CzNC6HYragjZVIFyYeVcrsc2F1gjB5OaMANobgVeAtEh
 9AOaYjFhBgWGpC5iLK5TjqoUm5z0ui6LByKO3V1Y3Svb1WkTDvDPN+hUOAeZzR9/XkI+kHiiwDEmg
 obaP53xDArtOKnao36WHrw==;
Received: from rms by fencepost.gnu.org with local (Exim 4.90_1)
 (envelope-from <rms@HIDDEN>)
 id 1qNPJG-0007gO-Km; Sat, 22 Jul 2023 22:59:02 -0400
Content-Type: text/plain; charset=Utf-8
From: Richard Stallman <rms@HIDDEN>
In-Reply-To: <ier8rb9ukjq.fsf@HIDDEN> (message from Spencer Baugh on
 Fri, 21 Jul 2023 15:33:13 -0400)
References: <iermszrwqj4.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
Message-Id: <E1qNPJG-0007gO-Km@HIDDEN>
Date: Sat, 22 Jul 2023 22:59:02 -0400
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If someone was thinking of doing that, they would be better off
  > responding to RMS's earlier request for C programmers to optimize this
  > behavior in find.

  > Since, after all, if we do it that way it will benefit remote files as
  > well.

I wonder if some different way of specifying what to ignore might make
a faster implementation possible.  Regexps are general but matching
them tends to be slow.  Maybe some less general pattern matching could
be sufficient for these features while faster.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 05:11:02 +0000
Resent-Message-ID: <handler.64735.B64735.169008905324600 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169008905324600
          (code B ref 64735); Sun, 23 Jul 2023 05:11:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 05:10:53 +0000
Received: from localhost ([127.0.0.1]:37784 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNRMq-0006Oh-MC
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:10:53 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:54298)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNRMg-0006OM-II
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:10:51 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNRMX-0007yn-2n; Sun, 23 Jul 2023 01:10:34 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=PP/nBa58ywXZy/Nzu9OSSCrUpR7PO6qUPorW/0DafgM=; b=P4Iu5RaRlxTY
 Pyoy7mAe5G1fKg10VGUPNYTlEQNLa/JZi+FnCC+pZAcLdxBMfAy7Nxj+GajuomWpCR0gUYpNSmCqE
 9VV9xvP2WP1K6G+JWb6qWaMqmE5n5K769tXHRK8HIT1aYfo5bTGVI1+XK58xg78QThdN75FHJV4fp
 oMA7qI9DvtVE59usYID7NGHfNveVTtiKjqJS9K4CtYiIJcdoJbIzMyb+DUz5htpGeAH10YCs6QExv
 2taISrNjLjmPua4BI+fSb4Rw0gQMY/q4O4ibuisyyJRVSdDUoZRN1Gopzyn0UQegV8EB/7iK3WhdP
 BFFEKofrB/TL20VtLtSkJA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNRMP-00006F-At; Sun, 23 Jul 2023 01:10:32 -0400
Date: Sun, 23 Jul 2023 08:11:03 +0300
Message-Id: <83sf9f6wm0.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 00:01:28 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: Ihor Radchenko <yantar92@HIDDEN>, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 00:01:28 +0300
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 22/07/2023 09:39, Po Lu wrote:
> > 
> > Programs that use find should fall back to directory-file-recursively
> > when any of the situations above are detected.
> 
> Perhaps if someone implements support for IGNORE entries (wildcards) in 
> that function, it would be easy enough to do that fallback.

Shouldn't be hard, since it already filters some of them:

    (dolist (file (sort (file-name-all-completions "" dir)
                        'string<))
      (unless (member file '("./" "../"))  <<<<<<<<<<<<<<<<<<<

Even better: compute completion-regexp-list so that IGNOREs are
filtered by file-name-all-completions in the first place.

Patches welcome.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 05:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169009006426349 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: rms@HIDDEN
Cc: sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169009006426349
          (code B ref 64735); Sun, 23 Jul 2023 05:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 05:27:44 +0000
Received: from localhost ([127.0.0.1]:37815 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNRdA-0006qu-Dx
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:27:44 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34136)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNRd8-0006qi-Tt
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:27:43 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNRd3-0003ey-D4; Sun, 23 Jul 2023 01:27:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=ozftes0KIR0dDxaZkvM2gnV4RFqZC4+i5S/16eEB4bA=; b=Ts1y1erjTOLv
 edKkrpMOkQTCB9pnHozMMtn3W/V/4vvCK1407flW0XazyPqJi6TpS3DfKB7zpgFrJKw+12Su4mv27
 SoIcIejVwTujGKIcwIpqzju1u9qpHz1j1YD9lDSMGHhKvCY6lZQjHd4ijbQswqtJ4xaLuJ+25KhL5
 bFKJL4cHZIHVTycEqy5Qkoe1iQ6dZMmzrtAf4j7gx5EVsOGdRof6kW7jebNuXH5khz7GoPSwzgrJb
 JPwFEYBVarG/54UlxbsPETkXZaAya9HZY97zWWKyFLgu5X4hUXZgoZEUc0SURQL1tYhUdw/CvlYbQ
 DZx+9XPLUsmk4pl40GrUfA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNRcv-0001jI-DD; Sun, 23 Jul 2023 01:27:29 -0400
Date: Sun, 23 Jul 2023 08:28:10 +0300
Message-Id: <83o7k36vth.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <E1qNPJG-0007gO-Km@HIDDEN> (message from Richard
 Stallman on Sat, 22 Jul 2023 22:59:02 -0400)
References: <iermszrwqj4.fsf@HIDDEN> <87fs5hemi1.fsf@HIDDEN>
 <83edl11qzn.fsf@HIDDEN> <874jlxebz5.fsf@HIDDEN>
 <87lef9mqio.fsf@localhost> <87edl1scbw.fsf@HIDDEN>
 <87fs5hmp6i.fsf@localhost> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <E1qNPJG-0007gO-Km@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 64735 <at> debbugs.gnu.org
> From: Richard Stallman <rms@HIDDEN>
> Date: Sat, 22 Jul 2023 22:59:02 -0400
> 
>   > If someone was thinking of doing that, they would be better off
>   > responding to RMS's earlier request for C programmers to optimize this
>   > behavior in find.
> 
>   > Since, after all, if we do it that way it will benefit remote files as
>   > well.
> 
> I wonder if some different way of specifying what to ignore might make
> a faster implementation possible.  Regexps are general but matching
> them tends to be slow.  Maybe some less general pattern matching could
> be sufficient for these features while faster.

You are thinking about matching in Find, or about matching in Emacs?

If the former, they can probably use 'fnmatch' or somesuch, to match
against shell; wildcards.

If the latter, we don't have any pattern matching capabilities in
Emacs except fixed strings and regexps.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 05:41:01 +0000
Resent-Message-ID: <handler.64735.B64735.169009084427861 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>, Michael Albinus <michael.albinus@HIDDEN>
Cc: sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169009084427861
          (code B ref 64735); Sun, 23 Jul 2023 05:41:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 05:40:44 +0000
Received: from localhost ([127.0.0.1]:37825 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNRpk-0007FJ-3S
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:40:44 -0400
Received: from mout02.posteo.de ([185.67.36.66]:49817)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNRpi-0007F3-AG
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 01:40:42 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 62CC9240103
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 07:40:36 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690090836; bh=/O98D43j93kzOB5g4OjE58C3gA5aOBcP1LlVc5HALB4=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=hSoTdkQf5heem9Jgn/awG/POjasM5+9yikb0ETiTuI38/fsWPNrzdY6KgbriddSf1
 HUus9PTc0Z+V4yvh9X/KrYCLvPscI86AsoqwsJeo5+ZK4kkCwRkDxU8BKSPHKLipa3
 Vs1ZZu4v6XygRm21oRPK/8WXIP9AAg7HILYWkr9e4egxL/4Jdfw3vxLkuOPFvMQoWs
 EIyzB0k6Yglib9LRA+/SJ/K2qDqhBPUpIMXmmxY4xfeGCVdsgd6PNIXvp3bSkcaL+r
 5EtZOnTs9AFwVYpSsQkxrbPXpHl76Ye65Cmi7CGcIdXieS34BbcKpEuYKX6TEW0euF
 si+9Xm4Tl9JPQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7sZq2BDhz6v3P;
 Sun, 23 Jul 2023 07:40:35 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <87h6pxl6m4.fsf@localhost>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost>
Date: Sun, 23 Jul 2023 05:40:41 +0000
Message-ID: <87pm4jrxra.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Ihor Radchenko <yantar92@HIDDEN> writes:

> On remote host, I can see that `find-lisp-find-files' must use
> tramp entries in `file-name-handler-alist'. Although, it will likely not
> be usable then - running GNU find on remote host is going to be
> unbeatable compared to repetitive TRAMP queries for file listing.

That said, Michael, may you please provide some insight about TRAMP
directory listing queries. May they be more optimized when we need to
query recursively rather than per directory?

GNU find is faster simply because it is running on remote machine itself.
But AFAIU, if TRAMP could convert repetitive network request for each
directory into a single request, it would speed things up significantly.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 06:15:02 +0000
Resent-Message-ID: <handler.64735.B64735.169009290032220 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: yantar92@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169009290032220
          (code B ref 64735); Sun, 23 Jul 2023 06:15:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 06:15:00 +0000
Received: from localhost ([127.0.0.1]:37840 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNSMu-0008Nc-1V
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 02:15:00 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:39096)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNSMr-0008NN-KW
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 02:14:59 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNSMj-0001Be-VO; Sun, 23 Jul 2023 02:14:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=877rEBDiCdCpXvd5zPvzgYUFox7/ScyEiBA2QRapeh8=; b=J7lysJzc3f1g
 EkDuqXW1rQmPdGgtY2GOmgGvcCcVc+Fnyv6GtU1PeM7Uk7CtZeUXgSzD5ak0jl0G975sSrxocgswE
 JZKR8P5wkXH9Fm7xsTnCGg5UGN+NNSZMRoqaPULBgH0KAmI+hiXMbItkKXFrs5ESIxtNjv0dduVU2
 /jAiw9ejWe+tjes7L8n3Yn+LWXuLlSXSC1qHjhfgHRqYM7wKeTPpeKxkJ1KFwLTqTLv3CmRd0O8jB
 cwWJC0idjoP7y9E2nGr6Os6tYznvTPLnP5HP5VxFgxooOyxO6IH7DwbgYEPDWi4VRzAXNzM5hLiMK
 LlpU6mSx4vMrVHRj9RoqsA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNSMc-0005zf-QX; Sun, 23 Jul 2023 02:14:43 -0400
Date: Sun, 23 Jul 2023 09:15:22 +0300
Message-Id: <83lef76tmt.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <ierwmyru0r2.fsf@HIDDEN> (message from Spencer Baugh on
 Sat, 22 Jul 2023 16:53:05 -0400)
References: <iermszrwqj4.fsf@HIDDEN> <87cz0lmoxy.fsf@localhost>
 <83v8edzb31.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN>
 <83wmyr7sbq.fsf@HIDDEN> <ierwmyru0r2.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Spencer Baugh <sbaugh@HIDDEN>
> Cc: sbaugh@HIDDEN,  yantar92@HIDDEN,  rms@HIDDEN,
>    dmitry@HIDDEN,  michael.albinus@HIDDEN,  64735 <at> debbugs.gnu.org
> Date: Sat, 22 Jul 2023 16:53:05 -0400
> 
> Can you try this further change on your Windows (and GNU/Linux) box?  I
> just tested on a different box and my original change gets:
> 
> (("built-in" . "Elapsed time: 4.506643s (2.276269s in 21 GCs)")
>  ("with-find" . "Elapsed time: 4.114531s (2.848497s in 27 GCs)"))
> 
> while this parallel implementation gets
> 
> (("built-in" . "Elapsed time: 4.479185s (2.236561s in 21 GCs)")
>  ("with-find" . "Elapsed time: 2.858452s (1.934647s in 19 GCs)"))
> 
> so it might have a favorable impact on Windows and your other GNU/Linux
> box.

Almost no effect here on MS-Windows:

  (("built-in" . "Elapsed time: 0.859375s (0.093750s in 4 GCs)")
   ("with-find" . "Elapsed time: 8.437500s (0.078125s in 4 GCs)"))

It was 8.578 sec with the previous version.

(The Lisp version is somewhat faster in this test because I
native-compiled the code for this test.)

On GNU/Linux:

  (("built-in" . "Elapsed time: 4.244898s (1.934182s in 56 GCs)")
   ("with-find" . "Elapsed time: 3.011574s (1.190498s in 35 GCs)"))

Faster by 10% (previous version yielded 3.327 sec).

Btw, I needed to fix the code: when-let needs 2 open parens after it,
not one.  The original code signals an error from the filter function
in Emacs 29.

> >>   (cl-assert (null _predicate) t "find-directory-files-recursively can't accept arbitrary predicates")
> >
> > It should.
> 
> This is where I think a fallback would be useful - it's basically
> impossible to support arbitrary predicates efficiently here, since it
> requires us to put Lisp in control of whether find descends into a
> directory.

There's nothing wrong with supporting this less efficiently.

And there's no need to control where Find descends: you could just
filter out the files from those directories that need to be ignored.

> So I'm thinking I would just fall back to running the old
> directory-files-recursively whenever there's a predicate.  Or just not
> supporting this at all...

We cannot not support it at all, because then it will not be a
replacement.  Fallback is okay, though I'd prefer a self-contained
function.

> >> 	     (if follow-symlinks
> >> 		 '("-L")
> >> 	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
> >> 	     (unless (string-empty-p regexp)
> >> 	       "-regex" (concat ".*" regexp ".*"))
> >> 	     (unless include-directories
> >> 	       '("!" "-type" "d"))
> >> 	     '("-print0")
> >
> > Some of these switches are specific to GNU Find.  Are we going to
> > support only GNU Find?
> 
> POSIX find doesn't support -regex, so I think we have to.  We could
> stick to just POSIX find if we only allowed globs in
> find-directory-files-recursively, instead of full regexes.

The latter would again be incompatible with
directory-files-recursively, so it isn't TRT, IMO.

One other subtlety is non-ASCII file names: you use -print0 switch to
Find, which produces null bytes, and those could inhibit decoding of
non-ASCII characters. So you may need to bind
inhibit-null-byte-detection to a non-nil value to get correctly
decoded file names you get from Find.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 07:49:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900985228577 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900985228577
          (code B ref 64735); Sun, 23 Jul 2023 07:49:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 07:48:42 +0000
Received: from localhost ([127.0.0.1]:37887 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNTpa-0002EH-6M
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 03:48:42 -0400
Received: from mout01.posteo.de ([185.67.36.65]:35445)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNTpX-0002E2-7D
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 03:48:40 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 664BC240029
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 09:48:33 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690098513; bh=kwC49HbA+CFEF5i/2//Ly2M2KXKP7RE5AvzY+NmpUgE=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=VkuHlz/RW/EkqQCWeTtkP0EomoQBCItB1xEoSL/eb/vR3iQBkd+EXop5igkMJuta+
 H5EREpkNAL474oBWlBALzx9YAAHBdOREf7R6vZMbrabkqJ5iP2sJnchWafaPd/YJFj
 msEtL4L11pT5bV5g756wnfBF24T0A8hJx2IGakqBFqqkIYxmY2ZRQV1YnwIx2trR6+
 p8I3B0QwqYrxBqHMG4LhhTL/1jmLKGou1fnZbZPBGnJn2YuyY8IEkbONfapi8RoBk5
 DYxZiVG0hXaYRUaYZ23eQsgVZGU7b38tU2HnpskkkJYdsqsUAonwrjMVU9M++Cn0V7
 nTPd03EApYDuQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7wQS0lLCz6twG;
 Sun, 23 Jul 2023 09:48:31 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <ierwmyru0r2.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <ierwmyru0r2.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 07:48:45 +0000
Message-ID: <878rb7rrtu.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Spencer Baugh <sbaugh@HIDDEN> writes:

> Can you try this further change on your Windows (and GNU/Linux) box?  I
> just tested on a different box and my original change gets:

On GNU/Linux, with slight modifications 

(defun my-bench (count path regexp)
  (setq path (expand-file-name path))
  ;; (let ((old (directory-files-recursively path regexp))
  ;; 	(new (find-directory-files-recursively path regexp)))
  ;;   (dolist (path old)
  ;;     (should (member path new)))
  ;;   (dolist (path new)
  ;;     (should (member path old))))
  (list
   (cons "built-in" (benchmark count (list 'directory-files-recursively
					   path regexp)))
   (cons "built-in no handlers"
	 (let (file-name-handler-alist)
	   (benchmark count
		      (list 'directory-files-recursively path
			    regexp))))
   (cons "with-find" (benchmark count (list
				       'find-directory-files-recursively path regexp)))))


(my-bench 10 "/usr/src/linux/" "")

(("built-in" . "Elapsed time: 7.134589s (3.609741s in 10 GCs)")
 ("built-in no handlers" . "Elapsed time: 6.041666s (3.856730s in 11 GCs)")
 ("with-find" . "Elapsed time: 6.300330s (4.248508s in 12 GCs)"))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 07:53:01 +0000
Resent-Message-ID: <handler.64735.B64735.16900987468909 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900987468909
          (code B ref 64735); Sun, 23 Jul 2023 07:53:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 07:52:26 +0000
Received: from localhost ([127.0.0.1]:37892 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNTtB-0002Jd-V1
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 03:52:26 -0400
Received: from mout02.posteo.de ([185.67.36.66]:57049)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNTtA-0002JL-CI
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 03:52:24 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id CB29A240101
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 09:52:18 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690098738; bh=z4aKAFp8VnAm2PusYShpVkSIeEv0GiQgMWrtBdHXfAA=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=PDaBsbQtt0DjNttxIXAfhfljM0cRUMVDKUjMUOyyAx/d5qrulcIiQXFhV8vsEE7KJ
 L5HAwkIiRZ2eRC/9t7RYlzWtVIs3Ss6MW+GS0eOTRYJw3h5gKiO5Wt2KBlkkq1TN8Q
 raTmoPGNrnqVm9lK5PmUsRG8G/zq9tLfUHImynVodzR/Qt4NNfNVQR8fO83NYNXs5O
 QmfsEUWYCoZmFPw6f2uV60vgT5MyZrw20KQCJkwLeQezNglhzZVW4R6sqxXAtPWoer
 gKGrNiAHTZJYQFnt9EP3zoF0beMk4L8IOz5KqWDHPDYu6l3KFqfA4TZZJVPMzT1MCE
 SkLjMTFSsF5JQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7wVp1Hzyz6txs;
 Sun, 23 Jul 2023 09:52:18 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83351g7yn5.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 07:52:31 +0000
Message-ID: <875y6brrnk.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Then how will the callback be different from
>> (mapc #'my-function (directory-files-recursively ...))
>> ?
>
> It depends on the application.  Applications that want to get all the
> data and only after that process it will not use the callback.  But I
> can certainly imagine an application that inserts the file names, or
> some of their transforms, into a buffer, and from time to time
> triggers redisplay to show the partial results.  Or an application
> could write the file names to some disk file or external consumer, or
> send them to a network process.

But won't the Elisp callback always result in a queue that will
effectively be synchronous?

Also, another idea could be using iterators - the applications can just
request "next" file as needed, without waiting for the full file list.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 08:02:02 +0000
Resent-Message-ID: <handler.64735.B64735.16900992859913 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16900992859913
          (code B ref 64735); Sun, 23 Jul 2023 08:02:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 08:01:25 +0000
Received: from localhost ([127.0.0.1]:37915 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNU1t-0002Zp-3i
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:01:25 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42688)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNU1q-0002Zb-Ut
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:01:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNU1k-0000aY-3K; Sun, 23 Jul 2023 04:01:16 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=IO3kwVpRdxe3KTO8kIoGaSYgBDBajNlIK1L1p+sNBZQ=; b=hln9IJciBXgu
 K39hjY+Pb5jn1HXneMu/xde6ZI0LwzrpDhIsN4ENvfcdeX7ch8eqPweMkcRYX31gjixF7Kklv2VwO
 oiy8XAhjV5fLVA/B65SAiOBDqW21Dn+54EJU5nLDpq27lYZZBO7po+UdQS7mV7lIN3foxPgyNDo/b
 HpNdF53U6fd5LBjmnmNtGU7AS5JNQGQ+DYPmH5WO/L1naBl4jTV9dUbkkTnraUCIcRh9XOM/7Lwmt
 NOjzkSTUc3QWa2ZJYCMg0lhi3plAQ6YpnzIIGS19fJ37xnEU7eXGwZi8aw94fTC6V3pRlWlGcabjL
 0mQZ6v3cuOVqeyEL4beCAw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNU1c-0005jx-Ce; Sun, 23 Jul 2023 04:01:09 -0400
Date: Sun, 23 Jul 2023 11:01:47 +0300
Message-Id: <83fs5f6opg.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <875y6brrnk.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 07:52:31 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87r0p1cta3.fsf@HIDDEN>
 <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN> <875y6brrnk.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 07:52:31 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> Then how will the callback be different from
> >> (mapc #'my-function (directory-files-recursively ...))
> >> ?
> >
> > It depends on the application.  Applications that want to get all the
> > data and only after that process it will not use the callback.  But I
> > can certainly imagine an application that inserts the file names, or
> > some of their transforms, into a buffer, and from time to time
> > triggers redisplay to show the partial results.  Or an application
> > could write the file names to some disk file or external consumer, or
> > send them to a network process.
> 
> But won't the Elisp callback always result in a queue that will
> effectively be synchronous?

I don't understand the question (what queue?), and understand even
less what you are trying to say here.  Please elaborate.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 08:06:01 +0000
Resent-Message-ID: <handler.64735.B64735.169009955910320 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169009955910320
          (code B ref 64735); Sun, 23 Jul 2023 08:06:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 08:05:59 +0000
Received: from localhost ([127.0.0.1]:37920 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNU6I-0002gN-TE
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:05:59 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:38038)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNU6H-0002gB-LV
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:05:58 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNU6B-0001Gi-Ig; Sun, 23 Jul 2023 04:05:51 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=h3I7YMWCjCO4nGQwk5X66WSyEAEiffK2jrYwghV/2dE=; b=kN9SQYVOqmt3
 gzK/+NQwZV6O1t/eURbf5fXnDgoESwemNGJzY9q5EtfW5HCCHAbQjaDtQoovoVpIRlp8GXq9PVURj
 BXa7s0DPF64lJPQRHdNF5Az6FKGJYHcOG7Ofb5YhLqYbCwjJXLudEFebZXNqbNXOAqDg5fmucsZ/G
 qsDNzfrbZjXz1unDtNs4JcswFONSi44JtHux9XAWSUARq6GAlqXGwsGhQpC0MBWOIbA+fbWhAFGX1
 j5WaHCaqVNcn8ksy6og2jOfRTNy9UMhWvpeO/ym2TlN7QOH8kGoT4O7iptR0w+axXx2e4MFs6IU4D
 25wJpJdkhzfHclv3wI1BeA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNU65-0007yu-Gz; Sun, 23 Jul 2023 04:05:45 -0400
Date: Sun, 23 Jul 2023 11:06:25 +0300
Message-Id: <83edkz6ohq.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <878rb7rrtu.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 07:48:45 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <ierwmyru0r2.fsf@HIDDEN> <878rb7rrtu.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: Eli Zaretskii <eliz@HIDDEN>, sbaugh@HIDDEN, rms@HIDDEN,
>  dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 07:48:45 +0000
> 
> (my-bench 10 "/usr/src/linux/" "")
> 
> (("built-in" . "Elapsed time: 7.134589s (3.609741s in 10 GCs)")
>  ("built-in no handlers" . "Elapsed time: 6.041666s (3.856730s in 11 GCs)")
>  ("with-find" . "Elapsed time: 6.300330s (4.248508s in 12 GCs)"))

Is this in "emacs -Q"?  Why so much time taken by GC?  It indicates
that temporarily raising the GC thresholds could speed up things by a
factor of 2 or 3.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 08:12:02 +0000
Resent-Message-ID: <handler.64735.B64735.169009991310960 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169009991310960
          (code B ref 64735); Sun, 23 Jul 2023 08:12:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 08:11:53 +0000
Received: from localhost ([127.0.0.1]:37930 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNUC0-0002qg-Sg
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:11:53 -0400
Received: from mout02.posteo.de ([185.67.36.66]:36859)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNUBy-0002qR-4W
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:11:51 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 2B705240104
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 10:11:44 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690099904; bh=OuBq0iEkxwNimnYc05k4ItVLvYsucmOrhiyXYRC3F9w=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=bALxblN2eu1XB878ibZ90/t+56aSUKZckksgjmYTI+8STC5eDtAP/GwXpb8ckCsIq
 qhyK9bZm0sfSXhIEI6/Oe/Xp/5pIRcmXNAaw7aa63ah9f51dMXsU9nRn+pbOCN3EzP
 RMxS+p56P8XOHhRqrGZ7yi93J88CHhPYwhxuQauCvi67ca9dA7BNPt0arceZ1ouCR/
 wf5BB/qngoGPe4uBPY4eO1flVIKSjYx6i1mKiTUYF3949y30W7pTySg71yJX9pzdpY
 G8KppGzdd+4NEf723I6kNz05ZofyCWJ/ol2O24BT3DzKCt2hsvmkH5Tk9UtCVb+G9F
 Y9GqDqr5/thvQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7wxB60mfz9ryh;
 Sun, 23 Jul 2023 10:11:42 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83fs5f6opg.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 08:11:56 +0000
Message-ID: <87351frqr7.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> >> Then how will the callback be different from
>> >> (mapc #'my-function (directory-files-recursively ...))
>> >> ?
>> >
>> > It depends on the application.  Applications that want to get all the
>> > data and only after that process it will not use the callback.  But I
>> > can certainly imagine an application that inserts the file names, or
>> > some of their transforms, into a buffer, and from time to time
>> > triggers redisplay to show the partial results.  Or an application
>> > could write the file names to some disk file or external consumer, or
>> > send them to a network process.
>> 
>> But won't the Elisp callback always result in a queue that will
>> effectively be synchronous?
>
> I don't understand the question (what queue?), and understand even
> less what you are trying to say here.  Please elaborate.

Consider (async-directory-files-recursively dir regexp callback) with
callback being (lambda (file) (start-process "Copy" nil "cp" file "/tmp/")).

`async-directory-files-recursively' may fire CALLBACK very frequently.
According to the other benchmarks in this thread, a file from directory
may be retrieved within 10E-6s or even less. Elisp will have to arrange
the callbacks to run immediately one after other (in a queue).
Which will not be very different compared to just running callbacks in a
synchronous loop.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 08:16:01 +0000
Resent-Message-ID: <handler.64735.B64735.169010016021322 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010016021322
          (code B ref 64735); Sun, 23 Jul 2023 08:16:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 08:16:00 +0000
Received: from localhost ([127.0.0.1]:37975 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNUG0-0005Xq-Hg
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:16:00 -0400
Received: from mout01.posteo.de ([185.67.36.65]:58511)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNUFy-0005Pt-9X
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 04:15:59 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id B6C6E24002A
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 10:15:52 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690100152; bh=OEgX9EtsdGK9/dIQ1lkCaREWAVADYBy4nszJGFUkR2o=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=ptMWYC7oFQLEHRQ7O9FbIi0EV2mKDjNxuDuN8r7Z4TxC+KhO5LLJDEp+fdeLC/kal
 BsEvKyInofungXFzs2eY56lCLLFQUC0ksnsxldF2h6PrzQ5v6N/Sg6qBZkOylURHTb
 KN7VRdWHVs/elrC0ItOjG1C1Xn7coL6onRsr5wj/B3EROa5gLHhwBySmjS9eLznFmh
 2afSiCewdXf1yM6Nn7N+h5Su0zt1XkWOa64Exey8gtG8g/9V/+T06ewV3GRAN3Bxww
 OUMg1J0TPnNSi2ABjV0fJ/b6a2sIGpksXmz6xq69d1tcvobhkXPvpxT9NgZH680AcP
 MUsM/jvJzPnNg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7x1z6vk2z6tvy;
 Sun, 23 Jul 2023 10:15:51 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83edkz6ohq.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost>
 <87a5vpcmc7.fsf@HIDDEN> <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <ierwmyru0r2.fsf@HIDDEN> <878rb7rrtu.fsf@localhost>
 <83edkz6ohq.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 08:16:05 +0000
Message-ID: <87zg3nqbzu.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> (("built-in" . "Elapsed time: 7.134589s (3.609741s in 10 GCs)")
>>  ("built-in no handlers" . "Elapsed time: 6.041666s (3.856730s in 11 GCs)")
>>  ("with-find" . "Elapsed time: 6.300330s (4.248508s in 12 GCs)"))
>
> Is this in "emacs -Q"?  Why so much time taken by GC?  It indicates
> that temporarily raising the GC thresholds could speed up things by a
> factor of 2 or 3.

With emacs -Q, the results are similar in terms of absolute time spent
doing GC:

(("built-in" . "Elapsed time: 5.706795s (3.332933s in 304 GCs)")
 ("built-in no handlers" . "Elapsed time: 4.535871s (3.161111s in 301 GCs)")
 ("with-find" . "Elapsed time: 4.829426s (3.333890s in 274 GCs)"))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:11:02 +0000
Resent-Message-ID: <handler.64735.B64735.169010344727409 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010344727409
          (code B ref 64735); Sun, 23 Jul 2023 09:11:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:10:47 +0000
Received: from localhost ([127.0.0.1]:38188 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNV71-000780-6b
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:10:47 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:46646)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNV6z-00077o-46
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:10:46 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNV6s-0005X3-98; Sun, 23 Jul 2023 05:10:38 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=XyFizPQv1TXWRTtIdqBl0CT17U4awXxv1d0aZh8do2g=; b=b46MJ6HVUqPB
 K26MRhoJLG8ruNDAu6ViMU9UJe48UxvsI9arFi9DsDOsnL/jV7hrxBjiJlqMH8tGQRfhPFeLUWK1Z
 iSbDN5zXxYCCazmkPCHzAIYXfiMSFC+HFTfvRA/zXUiKlBCpMtY2uJX1JTr+5Yb6he3Ew7HZB+LKK
 nVZe5M8SpHPMZ5rFZwwBbV7alaV2tbG1oAus0hxDN9tf+I31tNX2oeMVl0du57CXN8B9MVut36Vc9
 Axmht3NoXylQETApefn0gCHTMXM69Ok2MC+NTepFvMtOZ40W3rLSkOij91gKxXFcNF5aK4pyYzya8
 vzMQTQVWtR3C4kkJOVI3Eg==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNV6e-0005cX-Ch; Sun, 23 Jul 2023 05:10:32 -0400
Date: Sun, 23 Jul 2023 12:11:00 +0300
Message-Id: <831qgzuh5n.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87351frqr7.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 08:11:56 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87a5vpcmc7.fsf@HIDDEN>
 <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN> <87351frqr7.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 08:11:56 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> But won't the Elisp callback always result in a queue that will
> >> effectively be synchronous?
> >
> > I don't understand the question (what queue?), and understand even
> > less what you are trying to say here.  Please elaborate.
> 
> Consider (async-directory-files-recursively dir regexp callback) with
> callback being (lambda (file) (start-process "Copy" nil "cp" file "/tmp/")).

What is async-directory-files-recursively, and why are we talking
about it?  I was talking about an implementation of
directory-files-recursively as a primitive in C.  That's not async
code.  So I don't understand why we are talking about some
hypothetical async implementation.

> `async-directory-files-recursively' may fire CALLBACK very frequently.
> According to the other benchmarks in this thread, a file from directory
> may be retrieved within 10E-6s or even less. Elisp will have to arrange
> the callbacks to run immediately one after other (in a queue).
> Which will not be very different compared to just running callbacks in a
> synchronous loop.

Regardless of my confusion above, no one said the callback must
necessarily operate on each file as soon as its name was retrieved,
nor even that the callback must be called for each file.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:13:02 +0000
Resent-Message-ID: <handler.64735.B64735.169010358027779 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010358027779
          (code B ref 64735); Sun, 23 Jul 2023 09:13:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:13:00 +0000
Received: from localhost ([127.0.0.1]:38228 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNV9A-0007Dy-4M
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:13:00 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42972)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNV98-0007DU-4Q
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:12:58 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNV91-0005tP-Vq; Sun, 23 Jul 2023 05:12:51 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=gu6Y93dE8rkqypWonem802WzrkkpVaRtr3Tke3xuog8=; b=NQE38lWn2tpi
 0U4m33pRt79nuiaQi1YAIxZuiT9Z+6KqduM9ExpIxXIdh3EfyYFlBKMszyRuoUVOz/n4rzv77oslY
 C3IQpxof7q/A5Sj3vLTqUoIB0GNsNJvyfdJ2BD6kfbzmLx23IveOMS4wkZ9LasEiOJOWiW+AFlPCg
 GmGdqxha0dccVfirx7/bjt6MTwHrzZquwxt/aLUZRuNTC93mz2ve0g+0hJ0KMf/7DQXKV1M4s5ok9
 YTZMlTrl0BXnmWJHSE+YjupT57PuXk2dNilqU9zgM0llvh+el7q6eVP3J3TBL6pA7UF0AWWgjBDwA
 ZBIXT1yR/8qsP+dRdidt4g==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNV8w-0005rG-0F; Sun, 23 Jul 2023 05:12:46 -0400
Date: Sun, 23 Jul 2023 12:13:26 +0300
Message-Id: <83zg3nt2h5.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87zg3nqbzu.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 08:16:05 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost>
 <87a5vpcmc7.fsf@HIDDEN> <878rb9l1f5.fsf@localhost> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <ierwmyru0r2.fsf@HIDDEN> <878rb7rrtu.fsf@localhost>
 <83edkz6ohq.fsf@HIDDEN> <87zg3nqbzu.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, rms@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 08:16:05 +0000
> 
> With emacs -Q, the results are similar in terms of absolute time spent
> doing GC:
> 
> (("built-in" . "Elapsed time: 5.706795s (3.332933s in 304 GCs)")
>  ("built-in no handlers" . "Elapsed time: 4.535871s (3.161111s in 301 GCs)")
>  ("with-find" . "Elapsed time: 4.829426s (3.333890s in 274 GCs)"))

Strange.  On my system, GC takes about 8% of the run time.  Maybe it's
a function of how many files are retrieved?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:17:02 +0000
Resent-Message-ID: <handler.64735.B64735.169010379828230 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010379828230
          (code B ref 64735); Sun, 23 Jul 2023 09:17:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:16:38 +0000
Received: from localhost ([127.0.0.1]:38240 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNVCf-0007LG-Pi
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:16:38 -0400
Received: from mout02.posteo.de ([185.67.36.66]:59877)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNVCd-0007Ky-BZ
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:16:36 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 4BB4C240105
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 11:16:28 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690103788; bh=rF7PD5bqsHqSjJtB3XArwjW93kixk3qz+l8Ud0BWxV8=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=hxItngyDe2KJxT/fqxYwzsmPy/ggMCpwCbH97Nd19epWIY14u9oaZ6kg7A7ChG6jO
 fKiLuYgH9RwHlTu58NIFGHIqGENq4QMp/+mhlhabqdOtbbqoH9dKXDSXOqCpfFG8CX
 yjU7W19dlHJFIVUjw8cUv02tm+AzTfbkiDO0nOoo2Xoxua4KBdMN51CC1alPAvKROa
 aMXXX9ZGie9kzvi9w0wh+DO76jaITQn9O465bwHvQrMRpBVl313jthW+76aMhdsO1G
 grTSkZ8W4/3HnNL30Yv7TYwci/iTqt09jZZpd/12sGNMpD5BB60ZRaQUAcxiPmWGSv
 O13vcA8wRzqqQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7yMt5X8Cz6v34;
 Sun, 23 Jul 2023 11:16:26 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83zg3nt2h5.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <878rb9l1f5.fsf@localhost>
 <87zg3pb6yt.fsf@HIDDEN> <83zg3p9s39.fsf@HIDDEN>
 <878rb944wi.fsf@localhost> <83tttx9q4v.fsf@HIDDEN>
 <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87cz0jj25g.fsf@HIDDEN>
 <83wmyr7sbq.fsf@HIDDEN> <ierwmyru0r2.fsf@HIDDEN>
 <878rb7rrtu.fsf@localhost> <83edkz6ohq.fsf@HIDDEN>
 <87zg3nqbzu.fsf@localhost> <83zg3nt2h5.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 09:16:40 +0000
Message-ID: <87wmyrq96v.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> With emacs -Q, the results are similar in terms of absolute time spent
>> doing GC:
>> 
>> (("built-in" . "Elapsed time: 5.706795s (3.332933s in 304 GCs)")
>>  ("built-in no handlers" . "Elapsed time: 4.535871s (3.161111s in 301 GCs)")
>>  ("with-find" . "Elapsed time: 4.829426s (3.333890s in 274 GCs)"))
>
> Strange.  On my system, GC takes about 8% of the run time.  Maybe it's
> a function of how many files are retrieved?

Most likely.
(length (directory-files-recursively "/usr/src/linux/" "")) ; => 145489

My test is producing a very long list of files. 10 times for each test
and for each function variant.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:35:01 +0000
Resent-Message-ID: <handler.64735.B64735.169010485630289 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010485630289
          (code B ref 64735); Sun, 23 Jul 2023 09:35:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:34:16 +0000
Received: from localhost ([127.0.0.1]:38290 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNVTk-0007sS-7e
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:34:16 -0400
Received: from mout02.posteo.de ([185.67.36.66]:53409)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNVTi-0007sA-1z
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:34:14 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 1C195240104
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 11:34:08 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690104848; bh=QBw7MsDGIGZlRQu65Gv4tqoLRdlZs/8VxZr+L286Lq4=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=QOpb67BzjPtKuQAzMczKdy1c/1JZWvrtxqfav+Go6uSJTTtZM1b3V1p2dqx2bwUuN
 GquEgBwR6APHIYsqfxZrJ78Cf9uxCVSNzOH73TNSRKNIOP2C/CTjxrENQtLR/2N0rF
 tyZlwLcNEZsqPR6pwlqFs6wbmphGNxezpqTdh6/BytDQMucpfUZL6HR+5fkhggKh9A
 znLcQpAMUWLwEy90oJa2nzV7R60elJ07zjitLRKimHHr67Sl1t1akG442ikHat0uDK
 IPMNbn3TF2JzhCToPvxzFz6Qh/a+oNMN2nB68j3DTkiWC3s+B2iI87FKACvU4YLq0F
 D6JERKUTSZP3w==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7ymH0R8rz9rxg;
 Sun, 23 Jul 2023 11:34:06 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <831qgzuh5n.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 09:34:20 +0000
Message-ID: <87tttvq8df.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Consider (async-directory-files-recursively dir regexp callback) with
>> callback being (lambda (file) (start-process "Copy" nil "cp" file "/tmp/")).
>
> What is async-directory-files-recursively, and why are we talking
> about it?  I was talking about an implementation of
> directory-files-recursively as a primitive in C.  That's not async
> code.  So I don't understand why we are talking about some
> hypothetical async implementation.

Then, may you elaborate about how you imagine the proposed callback
interface?
I clearly did not understand what you had in mind.

>> `async-directory-files-recursively' may fire CALLBACK very frequently.
>> According to the other benchmarks in this thread, a file from directory
>> may be retrieved within 10E-6s or even less. Elisp will have to arrange
>> the callbacks to run immediately one after other (in a queue).
>> Which will not be very different compared to just running callbacks in a
>> synchronous loop.
>
> Regardless of my confusion above, no one said the callback must
> necessarily operate on each file as soon as its name was retrieved,
> nor even that the callback must be called for each file.

The only callback paradigm I know of in Emacs is something like process
sentinels. Do you have something else in mind?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:40:02 +0000
Resent-Message-ID: <handler.64735.B64735.169010515830980 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010515830980
          (code B ref 64735); Sun, 23 Jul 2023 09:40:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:39:18 +0000
Received: from localhost ([127.0.0.1]:38337 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNVYc-00083c-6M
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:39:18 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:56810)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNVYZ-00083M-5n
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:39:16 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNVYT-0004KL-15; Sun, 23 Jul 2023 05:39:09 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=IGX+/bxKZxxUf1iz/6sv4rK/YsQAy3SCtB+aTgAZthw=; b=Artk5X431Kux
 3nYRi7WLbjwzOy2kX3svtuj45NfRBbeXZQPc4Va6rG36eNREGb/eSCGAcy8oZ7pluWFLQOO+Lgz3O
 Cx3qh0gIlT11rL5FGdWw3eyMdWU5+LUPZjxuJcttn2EL7dLdoZ/B+AAEw22vijixWlElYVB0dlbYW
 yZ+gAQojhMM+uH7dowfhZAFZefr8dSXHfjl/Tttm5LIUALBGm/P5iJ3n3DPJC1x+qUtDOqFkRHlpE
 OOroCJYITj+sseutaEd8Ym8usdVUg5y4gv6Fv46uWWrip3Tu6XihHVGksByAM1dJraeRK59dsGKXM
 1NliMtkNxPLSI71BR+fbmQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNVYM-0004Ft-Jl; Sun, 23 Jul 2023 05:39:02 -0400
Date: Sun, 23 Jul 2023 12:39:42 +0300
Message-Id: <83y1j7t19d.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87tttvq8df.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 09:34:20 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87zg3pb6yt.fsf@HIDDEN>
 <83zg3p9s39.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN> <87tttvq8df.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 09:34:20 +0000
> 
> The only callback paradigm I know of in Emacs is something like process
> sentinels. Do you have something else in mind?

Think about an API that is passed a function, and calls that function
when appropriate, to perform caller-defined processing of the stuff
generated by the API's implementation.  That function is what I
referred to as "callback".

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 09:43:02 +0000
Resent-Message-ID: <handler.64735.B64735.169010536031320 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169010536031320
          (code B ref 64735); Sun, 23 Jul 2023 09:43:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 09:42:40 +0000
Received: from localhost ([127.0.0.1]:38341 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNVbr-000896-Rm
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:42:40 -0400
Received: from mout02.posteo.de ([185.67.36.66]:37367)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNVbq-00088r-22
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 05:42:38 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 65B52240104
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 11:42:32 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690105352; bh=ZrAFA289wcl72Cj2hrJf7GXH0mUzbvs1bRCt432M54c=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=bGdmukiHk+diLRDsPgn2+DBNZfpK6QjVdfkP4tb4wm1hTCSX6PP0sg5tknFSetplP
 pXN0oyMH5TTYdxDYE2L4ipvWh+94XXg9oXJJJP1y1ejYorkrJA6GyyB7cdJtBlV3eq
 SuCbAu7ras+fAQJKDGzv6dx5cCgh7fvj3k095c7spOzrB8H2PQAKcLAVxqy3Uhec26
 m1vjoK8NM7DZU8kpAISscF9FzXi8g1jA2XloK9Yx6jzJsTg39ZQiWEprXanrHmn5Kx
 qrZB9mk/EAizCl2iVMnAFugIg5fdhrJtG+IMyieipFZpm72X+mTvxg1iu2WFc6M8FK
 4SJCycV/yHEgA==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R7yxz5QTyz6tsj;
 Sun, 23 Jul 2023 11:42:31 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83y1j7t19d.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 09:42:45 +0000
Message-ID: <87r0ozq7ze.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> The only callback paradigm I know of in Emacs is something like process
>> sentinels. Do you have something else in mind?
>
> Think about an API that is passed a function, and calls that function
> when appropriate, to perform caller-defined processing of the stuff
> generated by the API's implementation.  That function is what I
> referred to as "callback".

But what is the strategy that should be used to call the CALLBACK?
You clearly had something other than "call as soon as we got another
file name" in mind.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 10:21:02 +0000
Resent-Message-ID: <handler.64735.B64735.16901076363451 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901076363451
          (code B ref 64735); Sun, 23 Jul 2023 10:21:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 10:20:36 +0000
Received: from localhost ([127.0.0.1]:38494 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNWCa-0000ta-52
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 06:20:36 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:40440)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNWCW-0000tM-Nt
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 06:20:34 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNWCQ-00018n-SC; Sun, 23 Jul 2023 06:20:26 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=ELRLkdMoVTbXQqvXXCkSUP9O4geSC2i600VTeWLpWo0=; b=kfcnGO8JJ/4I
 e7QaaWhh9cwyt+6Nzb8hdXL7z62JzqG/KUHSf+2nIz2zFOhI9rZSC9c4uT3VeKE0dxRjDsnFsdtGt
 bgWv80Em3NFMYe3bhq/NceO3Ip3QRWLgJQc+Ub9oeQGDiV/wH+Oq+mXuOh0ACVDz5PAt9Nfz0teOv
 4kYgS6tn/OGBW5kVYomtrkAYWXXBy0APWO/2ti3Gvjwiqx1eBcN30UvRTP0w4nTpiYaiZQMbfGin0
 pbgQKDLOqWRv6b+eefOSA+D4drgYMLpoe1GUuLZ0lCcl2XGLauRP0OpRvvnQTYPoAbORlkkg+QGDF
 n/QtmPp38lWN3qA9q9+IAA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNWC6-0003yp-9h; Sun, 23 Jul 2023 06:20:19 -0400
Date: Sun, 23 Jul 2023 13:20:45 +0300
Message-Id: <83v8ebszcy.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87r0ozq7ze.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 09:42:45 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <878rb944wi.fsf@localhost>
 <83tttx9q4v.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN> <87r0ozq7ze.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 09:42:45 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > Think about an API that is passed a function, and calls that function
> > when appropriate, to perform caller-defined processing of the stuff
> > generated by the API's implementation.  That function is what I
> > referred to as "callback".
> 
> But what is the strategy that should be used to call the CALLBACK?
> You clearly had something other than "call as soon as we got another
> file name" in mind.

It could be "call as soon as we got 100 file names", for example.  The
number can even be a separate parameter passed to the API.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 10:47:01 +0000
Resent-Message-ID: <handler.64735.B64735.16901092046977 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901092046977
          (code B ref 64735); Sun, 23 Jul 2023 10:47:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 10:46:44 +0000
Received: from localhost ([127.0.0.1]:38602 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNWbr-0001oT-Jp
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 06:46:43 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:40321)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNWbp-0001oG-IO
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 06:46:42 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id 79D1C32002E8;
 Sun, 23 Jul 2023 06:46:34 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Sun, 23 Jul 2023 06:46:34 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690109194; x=1690195594; bh=OVh2saTZlwwRL/aJOIcnqc6iHo8cueNIxyt
 Qm0eHLU4=; b=fxLL6paXptYloogOeih4wkjYUat6Qbu5Ek9cm+wMjwZ8vdFvj/A
 eT2PTfLoGqxhTSm0r0nWFoQ8es+OKoV9VTjuf7WR9egPj3KJKg5ZdBdAtGK1Xnxu
 JSWZaJfUGuYmITNpEKFmagpLa2lpBxgxvXK+7I0GwW3+9XDHEXGMzo1WRy+Jqw11
 QjYem72iLG7FDDb8AkxiYYSfmYGwiK1FMWhCpIyghISbRW7R8I7ZithAOknGxxm+
 w8HZF1w9e2StpookLBPYwljKGpUX65FzAA3iscoVi7VKZQiep1VFrHeX6K++vNGc
 V7WOH28+zLBiZHBeAeoUhvYMV0U1NA/wujw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690109194; x=1690195594; bh=OVh2saTZlwwRL/aJOIcnqc6iHo8cueNIxyt
 Qm0eHLU4=; b=bjrfmGuHrbGrhvhiT0/DZWWDta4lFRblQU5tr1JFcrhsJrLJqzK
 V5rRkUIrMhuGiV/BOOZLvxVdgaOPT/08U8jIdPxO0g7DaIZ2BcgW4IT9X81zCQb4
 6I69Yu5TgXg3Jaaom25muvkvu/qASL3aEnyNx9qgqQ32jcWke7HfythmsvJUWBVr
 z+2dRtiI57lX9Kyl/qHGPajFWg4AhSE3RtzoC4sEMHX8XAhCp0TGeoRI/wNl3Paa
 yNPYBzdpKGTzGb7j9rHFgggg4i0YaMOxz0koKFsVluwExc813h5nxG1h1qNhq+pl
 f0JqQJa3BZoSuy4JtSFA/1F79D3mh8uhKKg==
X-ME-Sender: <xms:CQW9ZH9r7rVqsgdQYLgrZ8hti7Henk_zJ6LudVs_MNO0TkDBDC7yQQ>
 <xme:CQW9ZDv7IS6siXnyAOJesQtBOxYu-NY-RjvYaNqRZSGsCUB0RDiAieoD0HW88nX30
 4bKQVutzZm7czBW7pE>
X-ME-Received: <xmr:CQW9ZFBS8-fjOCPOlY89v9NZB8ewF8c7se977lL4zcyTRfIbfAdiOMNvJcDWhlg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigdefudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:CQW9ZDdYMeUUcwzhbQENCX7c-qTsOS04D8H3mFHSETz66ADo4SfOMw>
 <xmx:CQW9ZMO9Oyyq7sAFPEutCe2bOLDbCNU7U1Xr8a7b-8pdBC6U6KMKHw>
 <xmx:CQW9ZFmalwwkuZ4x0dvHHPm41fBZjE4h5kApedUCfHOHHLlj-9T2aw>
 <xmx:CQW9ZBqS7UPBLiq2wVPIAX7InMUozOCsyec3TzofKvmUsq64YJMG_w>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 06:46:32 -0400 (EDT)
Message-ID: <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
Date: Sun, 23 Jul 2023 13:46:30 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83sf9f6wm0.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 08:11, Eli Zaretskii wrote:
> Even better: compute completion-regexp-list so that IGNOREs are
> filtered by file-name-all-completions in the first place.

We don't have lookahead in Emacs regexps, so I'm not sure it's possible 
to construct regexp that says "don't match entries A, B and C".

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 11:18:02 +0000
Resent-Message-ID: <handler.64735.B64735.169011107310690 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169011107310690
          (code B ref 64735); Sun, 23 Jul 2023 11:18:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 11:17:53 +0000
Received: from localhost ([127.0.0.1]:38733 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNX60-0002mM-OM
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:17:53 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:45700)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNX5w-0002m7-D1
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:17:50 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNX5q-0001KP-Pa; Sun, 23 Jul 2023 07:17:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=881VaWWR6/6hmLW60V6Rrk0SLr4RLKmpV1Q6Z3WqHT8=; b=KplbbSoe/fhE
 fm6pqJPDba1OcI83HvAVflMsD6RWHOZ/pe57MV0WwF3zUa+UKgGo16NHIKyypfCxOyffxflqSqSFJ
 KoRODxjdSB7qfkqZOFkBBmFHbEcjFE1XglBbFaCUlD4x0Wp71KOsFoE/0oOkC4zUKXCO5Tu9fciYF
 EdC0jLc9g1P79I3CQLaUmO8bzbUkYrSL3XhZKMCj7VwoRFrS4TQhf32j6rYab27AL8lUWBB217Ozc
 VvFZ6AOYLZoMwJfDpv6eVt4R8ArawTZyEmVV6iqgnq1jLTdcSjAaV1RCQRuZsI/RPlExvwOVQkPmh
 utFdpqCE9Zv8WU3puaMRWw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNX5q-0001Pt-8j; Sun, 23 Jul 2023 07:17:42 -0400
Date: Sun, 23 Jul 2023 14:18:22 +0300
Message-Id: <83sf9eub9d.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 13:46:30 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 23 Jul 2023 13:46:30 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 23/07/2023 08:11, Eli Zaretskii wrote:
> > Even better: compute completion-regexp-list so that IGNOREs are
> > filtered by file-name-all-completions in the first place.
> 
> We don't have lookahead in Emacs regexps, so I'm not sure it's possible 
> to construct regexp that says "don't match entries A, B and C".

Well, maybe just having a way of telling file-name-all-completions to
negate the sense of completion-regexp-list would be enough to make
that happen?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 11:44:01 +0000
Resent-Message-ID: <handler.64735.B64735.169011260023382 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169011260023382
          (code B ref 64735); Sun, 23 Jul 2023 11:44:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 11:43:20 +0000
Received: from localhost ([127.0.0.1]:38844 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNXUd-000654-NP
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:43:20 -0400
Received: from mout02.posteo.de ([185.67.36.66]:58097)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNXUa-00064n-Ll
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:43:18 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 6007D240104
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 13:43:10 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690112590; bh=EprmvCXSJ/iOtbRGNLhnACUSKqCsfdR3S/u16gsSWyo=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=r7ha4COXbDsaW5xacSz2vbExKt1206Xe0gARCR1Xpn6vT/KIWBvSr2xGhH5T4Ny0U
 ffUzVJE7A+OxtKxbYwW/pelPtvefI1mrjGRaThD3sE4OEWhE3pb3fOydg8ReJA+sE0
 YGgF80fe4rLXHg4Fhwo669uxeayRXvdVFjhfkA6eqn49TRFHFZEKIGyWAO5dVErFVv
 Q2u8ewi/Rw2fIX/c78iZ+HeLklVhWz4Yy9JRqIbcCajsoCTzQ0PtRGj4S7jm1dqCcW
 ZfyNXiLUZM/vhk4T/G4SbWQ5E+u49tKCuU1oT6g7QeavJr/Xy4NSJ1yBotNgvaNKbu
 WBPWNVpr0lO7g==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R81d90Vb1z9rym;
 Sun, 23 Jul 2023 13:43:08 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83v8ebszcy.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN>
 <87r0ozq7ze.fsf@localhost> <83v8ebszcy.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 11:43:22 +0000
Message-ID: <87v8ean99h.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> But what is the strategy that should be used to call the CALLBACK?
>> You clearly had something other than "call as soon as we got another
>> file name" in mind.
>
> It could be "call as soon as we got 100 file names", for example.  The
> number can even be a separate parameter passed to the API.

Will consing the filename strings also be delayed until the callback is invoked?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 11:45:02 +0000
Resent-Message-ID: <handler.64735.B64735.169011268623533 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Spencer Baugh <sbaugh@HIDDEN>
Cc: yantar92@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169011268623533
          (code B ref 64735); Sun, 23 Jul 2023 11:45:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 11:44:46 +0000
Received: from localhost ([127.0.0.1]:38848 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNXW2-00067T-5x
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:44:46 -0400
Received: from mout.gmx.net ([212.227.17.22]:50907)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qNXW0-00067F-2G
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:44:44 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1690112670; x=1690717470; i=michael.albinus@HIDDEN;
 bh=1a+NW7ugdQ/3HVhlZJk7ylEVb6Sfu4llkirb+q+P4rY=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=L7fc4sK8BfREX/qB5ZwEyO9WU/9v5K+h7Eri9Va7lt1SGKaqf0Z9iSJH58OmR66Gz5wSSDb
 AjDMo4Ny/d/9+cEW/nL9jaDUG1dh19tBoyUdv9GZxESh5wZCvndLcMTRDqQ32Ei7BKDDdeh/E
 FgQSZJzgs1PdJJ278Q6i/4wtUk9rTD6Sk+atnB9XfyY9NI9zfZjtmKEgi8e6YPIJMjcnbkGXE
 C/p6AGABrK9t3Y2seiTWslHdeDLnysG054o7auD8XCirjuE2hSYGCtu2zGlPE/NHQo9wslFc2
 JZx5jbz2H/dvQ4Dj30lVKPGnSv/fshVB/2qoU+EugswX8OB2HywQ==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MKbg4-1qbv860BCu-00Kzqg; Sun, 23
 Jul 2023 13:44:30 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <ierwmyru0r2.fsf@HIDDEN> (Spencer Baugh's message of
 "Sat, 22 Jul 2023 16:53:05 -0400")
References: <iermszrwqj4.fsf@HIDDEN> <83v8edzb31.fsf@HIDDEN>
 <87r0p1cta3.fsf@HIDDEN> <87pm4ll7ox.fsf@localhost>
 <87a5vpcmc7.fsf@HIDDEN> <878rb9l1f5.fsf@localhost>
 <87zg3pb6yt.fsf@HIDDEN> <83zg3p9s39.fsf@HIDDEN>
 <878rb944wi.fsf@localhost> <83tttx9q4v.fsf@HIDDEN>
 <87pm4lb4fr.fsf@HIDDEN> <83pm4l9n0o.fsf@HIDDEN>
 <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87cz0jj25g.fsf@HIDDEN> <83wmyr7sbq.fsf@HIDDEN>
 <ierwmyru0r2.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 13:44:28 +0200
Message-ID: <875y6azwbn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:RtX3AJPAFvxVmGaea4ktSBDmAWhoCvKfotBeglCWyt8Cl7Tlkbn
 DZ+xaXWrGfpsmRexHmHxFM+yI4s6Ab211WHQFC1BJ3CgwOiB3yGOO/8gdG/eaYXt87/cd97
 /DVpEQOeve8JNKsqxWWK37KJphLhuaRz5nscmuB5u1tKX1ovguN8Qe0C8ivOXfZuIVcrvPf
 owcfxAn2Ezy/C+wGBUYqQ==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:3xh9gDaq5JU=;zjbC969cWaKqfEQ46b3/DnW1fJf
 t5q6oPu22QmvMgYpOUYZO772sEmHYTrdS9EZTuuHIEiS6r4XEjMc394lOqY/UZ2Yr6ZyNM3mX
 Ik3aq+3sK34ITMBzFGEVFJheVgjNte5xKbWgGeSaSx37vKJC/9/cFRQbHNKXky7Ze21ZsD0tM
 1crfntNxTQnqgMMYEIlm98LuLI87RyJI9pZXGEjlRkSXlMi7/s1uSdA3J+al9Y73XVpg8qQ2Q
 RxJtgFMZzYrghjasbrmwoloZ38qsLqTtWBvBWAfsrXFX/vJ9BUQ31nzdrG5YG7FCQVIKmYGN1
 UsVmRA+GdPs5cx8sz3zRftLRHV6OlfI3EI3cBIcBoNQeynEoAhBSHSgdVSUF5VWUJk34ZSGcR
 ROrXyLlzvPiZTCg8+dJOIQh490zo0NcGUTHQnxjYekgd5YEWLQusZVVB+auyPLaMh5lWfPqRO
 d7ph4oC5sBgh1jY9d01XKO0hSIkDKCjwxx5i3xqqVit5TW+haTUW2dnHuR2BspJO7t/f4OeEJ
 2Yjk+geP/X9Xkik91nZn0WguZvwnB3KyoO9QBYCffcP/d+ZAtZAJUbxXauGddGoeTdjMQgjv9
 HIozorDNtuCdIF8cODPH4SbiPYdNxktjNR3sfkd9599enZZRtEvps5ZUTsaVDoMU0X2KOvZXS
 jzCTLa53Gt9ZMyeb3dMayEdsAP/+5vsCEBZ5qz6K9wPmdhbEkbOuVS+8YLTY/DLJTIm2OY9K2
 m6PgdiER44h+Mn0xkB6x6ueoL47WOX7NUw4ICNxoqtk+wR1ZjxrD/++LsiOIASLHUaavVHsG3
 MX1Tx7Vh2gPFRUFzYkmAxMQlt7o/lmjSaLl9WPZj/glPd+4Ah+r6UTCVQN1SY+L2bYeYxDvA/
 bNInyF4/ZqOM377mEd0rbRxCQG2liPCHK45egkRQsS2PqPMs+TEp+K6aYBqUSjdAs1YZJiN2Z
 bQCo6OHF5RI/Z1QlUA6HQZL6iUA=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Spencer Baugh <sbaugh@HIDDEN> writes:

Hi Spencer,

> I mean having Emacs read output from the process and turn them into
> strings while find is still running and walking the directory tree.  So
> the two parts are running in parallel.  This, specifically:

Just as POC, I have modified your function slightly that it runs with
both local and remote directories.

--8<---------------cut here---------------start------------->8---
(defun find-directory-files-recursively (dir regexp &optional include-directories _predicate follow-symlinks)
  (let* (buffered
         result
	 (remote (file-remote-p dir))
	 (file-name-handler-alist (and remote file-name-handler-alist))
         (proc
	  (make-process
           :name "find" :buffer nil
	   :connection-type 'pipe
	   :noquery t
	   :sentinel #'ignore
	   :file-handler remote
           :filter (lambda (proc data)
                     (let ((start 0))
		       (when-let ((end (string-search "\0" data start)))
			 (push (concat buffered (substring data start end)) result)
			 (setq buffered "")
			 (setq start (1+ end))
			 (while-let ((end (string-search "\0" data start)))
                           (push (substring data start end) result)
                           (setq start (1+ end))))
                       (setq buffered (concat buffered (substring data start)))))
	   :command (append
	             (list "find" (file-local-name dir))
	             (if follow-symlinks
		         '("-L")
	               '("!" "(" "-type" "l" "-xtype" "d" ")"))
	             (unless (string-empty-p regexp)
	               "-regex" (concat ".*" regexp ".*"))
	             (unless include-directories
	               '("!" "-type" "d"))
	             '("-print0")
	             ))))
    (while (accept-process-output proc))
    (if remote (mapcar (lambda (file) (concat remote file)) result) result)))
--8<---------------cut here---------------end--------------->8---

This returns on my laptop

--8<---------------cut here---------------start------------->8---
(my-bench 100 "~/src/tramp" "")
(("built-in" . "Elapsed time: 99.177562s (3.403403s in 107 GCs)")
 ("with-find" . "Elapsed time: 83.432360s (2.820053s in 98 GCs)"))

(my-bench 100 "/ssh:remotehost:~/src/tramp" "")
(("built-in" . "Elapsed time: 128.406359s (34.981183s in 1850 GCs)")
 ("with-find" . "Elapsed time: 82.765064s (4.155410s in 163 GCs)"))
--8<---------------cut here---------------end--------------->8---

Of course the other problems still remain. For example, you cannot know
whether on a given host (local or remote) find supports all
arguments. On my NAS, for example, we have

--8<---------------cut here---------------start------------->8---
[~] # find -h
BusyBox v1.01 (2022.10.27-23:57+0000) multi-call binary

Usage: find [PATH...] [EXPRESSION]

Search for files in a directory hierarchy.  The default PATH is
the current directory; default EXPRESSION is '-print'

EXPRESSION may consist of:
	-follow		Dereference symbolic links.
	-name PATTERN	File name (leading directories removed) matches PATTERN.
	-print		Print (default and assumed).

	-type X		Filetype matches X (where X is one of: f,d,l,b,c,...)
	-perm PERMS	Permissions match any of (+NNN); all of (-NNN);
			or exactly (NNN)
	-mtime TIME	Modified time is greater than (+N); less than (-N);
			or exactly (N) days
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 11:51:02 +0000
Resent-Message-ID: <handler.64735.B64735.169011303924635 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169011303924635
          (code B ref 64735); Sun, 23 Jul 2023 11:51:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 11:50:39 +0000
Received: from localhost ([127.0.0.1]:38892 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNXbi-0006PH-Sp
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:50:39 -0400
Received: from mout.gmx.net ([212.227.15.19]:33733)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qNXbg-0006P4-3g
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 07:50:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1690113022; x=1690717822; i=michael.albinus@HIDDEN;
 bh=CDK8rNQ8n+aCQFo8/XW6B1venrAIDbQIy7cmMh/3a+0=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=kl+A/mW29FcsTFfZ3LEKCrhG7LjYOQ0bND4UzD+bFr8BhW+9tSniW+yrVYU1SBhyrWFwJ8J
 cLIwIAEy6wH2G19QLBTHwxzmwapGYqqMJ7/f4XbdtM/vBTKa4D17Te1OY97y1XX0vyjaAOqpm
 JoX9WIj8v4mZN75mOWr7J1/nszqCXXV2nmniNodiYRNjllSFIK2S8Sy0Q2oJvfyApOA2sYA6F
 CkXNGFJVWvL7sAgXq3dKHoLWfODCb4VI6hkjFt5FT5hS+Dovi03W6A/BeA9/vViSbvjrOis0W
 AnK2s9uy4GsRER0dTG9t+speNROh/LRquxp0FjRs1sLABwVyxPsA==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N1OXZ-1pvSkv1igN-012prw; Sun, 23
 Jul 2023 13:50:22 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87pm4jrxra.fsf@localhost> (Ihor Radchenko's message of "Sun, 23
 Jul 2023 05:40:41 +0000")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <87pm4jrxra.fsf@localhost>
Date: Sun, 23 Jul 2023 13:50:20 +0200
Message-ID: <871qgyzw1v.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:Rx3NFDdga11fN1b0+hG/6bcqKTQMxI+bFChq2MWbsUXVROcpMoc
 7aeWfR0vyiuAwsGGTQqfC/4QDHD6bztEi7vrX0+PnlACSfkHQC1hLsFpIycZzb1RXfcIl4E
 ITcMqdhBhlCvrXmbiNBUFUjQbEfhT5totBQUrMsRmjZZKvjTYX+GnGyu1MsSKykht4UAT6M
 PN7V962ScHWSl41yf0w2A==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:JL50sloy2i4=;rCx/XO2hQJfjVu6tAouGt84zNMi
 q5qEEmEUJ2mYvp2f/va2ISUDDoDO9hDMm3M/qhiuQoZQKO1IzpZG8Y9hT6aixZDP4AnFRaVbD
 kAYIvdOTlUqX3EAe40dI7/oH/ysAYR1knBQ41ljGML+UiRWSrkCjTwgN1ddKX8aCOlp99AGYm
 WF8h93FzTjsjy79uDeG1AoG9YhYa14SBUAuo+FYSO+du9HiVpIhqIU0QNTfPYsEJltsW++MpQ
 SzVJNqjalqRRMpiLI/5+Z64GARtYAfZ2qeTJoPT7d1KswJ5spW8HHp+VtXJepeDFrmLlPhXKq
 KNgQPDNQ314ww6BD2z+0V1Paj2H1RLMpi/2x6+x6VfqbTuzYPU+b2jb9f6GMC4H7YLL2RrAT3
 ETaAhSDE1pA0YbMT8PPTsY4+pQ9wtJcm0BCOFWBOdyzGLf21AWbgmTcQgp+jRYhVN/hxjm13I
 bsXuQOjIiHaEIomkz54QP9DmdE57TaW5LG9oa+ZvhGJQABmgql4lZUBkExevwbJZEUP09qbSV
 ECjAIV24CAPDs+H0C7v+uJgdD25Z7WNNF8xjRxRQwh29eJcTOoEsge/+AbikSE/yX+xaZDjOU
 sPHrDJmSWc0hI2m077I6MLPbJ1Q4Wabbau+n2I9FNjeB+eEVNJ8Pu55C07zNcidOAH17qfyH4
 47PklClm+OimUqMcI5/h/WLgnfbUEoskN1B8QkPtzizZiYdhhu1JdaK6gSMvqalQzEvxNtr4I
 a+M5hz4j+PA60CQam16rqP9/swphIocskfy2uygHa3o/ndclC1znawmnpSOPxKN690iRZK5l9
 ne618w29wJ9kxEorQWCYNP6xMDhC4mpEyVMRO6loAniNH/eWAhP+6f2wJrAl81tPd+EWbu2QD
 qYJdKdO8O4GOitJLiBllFWUVxGP3EtDSR33Oc687zDhEpfSDXgjOmnSAEJuzbmweWyeP8NjFK
 MbQ8l8OmETXoNk2OULYxknZQXlA=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

>> On remote host, I can see that `find-lisp-find-files' must use
>> tramp entries in `file-name-handler-alist'. Although, it will likely not
>> be usable then - running GNU find on remote host is going to be
>> unbeatable compared to repetitive TRAMP queries for file listing.
>
> That said, Michael, may you please provide some insight about TRAMP
> directory listing queries. May they be more optimized when we need to
> query recursively rather than per directory?

Tramp is just a stupid library, w/o own intelligence. It offers
alternative implementations for the set of primitive operations listed in
(info "(elisp) Magic File Names")

There's no optimization wrt to bundling several operations into a more
suited remote command.

> GNU find is faster simply because it is running on remote machine itself.
> But AFAIU, if TRAMP could convert repetitive network request for each
> directory into a single request, it would speed things up significantly.

If you want something like this, you must add directory-files-recursively
to that list of primitive operations.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 12:50:01 +0000
Resent-Message-ID: <handler.64735.B64735.16901165668428 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901165668428
          (code B ref 64735); Sun, 23 Jul 2023 12:50:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 12:49:26 +0000
Received: from localhost ([127.0.0.1]:38992 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNYWb-0002Br-Pj
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 08:49:26 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:59200)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNYWX-0002Bb-DY
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 08:49:24 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNYWQ-0003zj-Cc; Sun, 23 Jul 2023 08:49:14 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=uX6QBxVov59s1+6CP6IDoiWIBCUltr7wAD2iiMwPFCY=; b=L6Qd+9W868iz
 gRXWfpzwxoCc3nwk4XipON9CEXxLZWcuYaeq27YaZIrjnDXTA027YqhROJCzGyezc36DZpdnw3AQy
 8I8eAOba4Jrq6PkhVk3ST9wnTUdCHAPNHpKeZDevKhJcMNcJw0lSnb60NjTMhIm/tS/TmeA5O+B9w
 rVqEIcQqYbFayQ3zXEVVlOKvkOt/luMvBUwxkJuRdOTq9nAvN9cXhkEdze11QYEUo+gonB3/rRTfv
 7XKZDo6Xdhiqea4C27mg/pRuZyqATAyQwzy4WRODjrtXrhoW1OgAIHu2sOIyd5CXHgqU+gwvrDabU
 iC2t9uFCJkCaqUQUY3wfrQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNYWJ-0007q6-Ir; Sun, 23 Jul 2023 08:49:07 -0400
Date: Sun, 23 Jul 2023 15:49:48 +0300
Message-Id: <83o7k2u70z.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87v8ean99h.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 11:43:22 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87pm4lb4fr.fsf@HIDDEN>
 <83pm4l9n0o.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN> <83mszp9kl2.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN>
 <87r0ozq7ze.fsf@localhost> <83v8ebszcy.fsf@HIDDEN> <87v8ean99h.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 11:43:22 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> But what is the strategy that should be used to call the CALLBACK?
> >> You clearly had something other than "call as soon as we got another
> >> file name" in mind.
> >
> > It could be "call as soon as we got 100 file names", for example.  The
> > number can even be a separate parameter passed to the API.
> 
> Will consing the filename strings also be delayed until the callback is invoked?

No.  I don't think it's possible (or desirable).  We could keep them
in some malloc'ed buffer, of course, but what's the point?  This would
only be justified if somehow creation of Lisp strings proved to be a
terrible bottleneck, which would leave me mightily surprised.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 12:58:01 +0000
Resent-Message-ID: <handler.64735.B64735.16901170749121 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901170749121
          (code B ref 64735); Sun, 23 Jul 2023 12:58:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 12:57:54 +0000
Received: from localhost ([127.0.0.1]:39001 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNYeo-0002N3-6F
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 08:57:54 -0400
Received: from mout01.posteo.de ([185.67.36.65]:44825)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNYel-0002Mq-MB
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 08:57:53 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 21E1B240029
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 14:57:45 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690117066; bh=cBj0+v06C819vrM2g7+8Pj3Zjd6RLU8eX2iUpx/YHAA=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=YfRvBdoaM4wk+9AnAjptjqJ/nSI5XZX2m39GT8uhnbTLkTQuEaBmAVfnmA+Le5yDO
 /pdYiJ9RmNR/xtag+3w9/Bvj3uds2pXsslwQtSO59zf1A5Vd241WVLtR0niBiwBZI4
 QAU/3HpoHS8wAiN2XiqysyAEGaATYTOxRy5jJMzKmGJKXp30aWWaaJ11LmJUCZIz52
 h1G1GJzf0T3eOQXAUNV9np6FXuCzUfODGyn24ZwIbUPf2bp8C1vPYoGIlu8ySnKgcY
 VTSBdpYo3vDrq0bcNBFG18vQ4/8T6o3aMpk1JypRPv/gQituQ0wGEloIbmfjmbGX1d
 S5reKcwuJkNBg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R83HD2psgz6v2X;
 Sun, 23 Jul 2023 14:57:43 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83o7k2u70z.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87pm4krq2m.fsf@localhost>
 <838rb881ak.fsf@HIDDEN> <87mszornlq.fsf@localhost>
 <83351g7yn5.fsf@HIDDEN> <875y6brrnk.fsf@localhost>
 <83fs5f6opg.fsf@HIDDEN> <87351frqr7.fsf@localhost>
 <831qgzuh5n.fsf@HIDDEN> <87tttvq8df.fsf@localhost>
 <83y1j7t19d.fsf@HIDDEN> <87r0ozq7ze.fsf@localhost>
 <83v8ebszcy.fsf@HIDDEN> <87v8ean99h.fsf@localhost>
 <83o7k2u70z.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 12:57:53 +0000
Message-ID: <87sf9en5ta.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > It could be "call as soon as we got 100 file names", for example.  The
>> > number can even be a separate parameter passed to the API.
>> 
>> Will consing the filename strings also be delayed until the callback is invoked?
>
> No.  I don't think it's possible (or desirable).  We could keep them
> in some malloc'ed buffer, of course, but what's the point?  This would
> only be justified if somehow creation of Lisp strings proved to be a
> terrible bottleneck, which would leave me mightily surprised.

Thanks for the clarification!
Then, would it make sense to have such a callback API more general? (not
just for listing directory files).

For example, the callbacks might be attached to a list variable that
will accumulate the async results. Then, the callbacks will be called on
that list, similar to how process sentinels are called when a chunk of
output is arriving to the process buffer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 13:32:01 +0000
Resent-Message-ID: <handler.64735.B64735.169011910012157 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169011910012157
          (code B ref 64735); Sun, 23 Jul 2023 13:32:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 13:31:40 +0000
Received: from localhost ([127.0.0.1]:39025 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNZBU-0003A1-B8
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 09:31:40 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47542)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNZBR-00039k-2h
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 09:31:38 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNZBL-0001xm-BT; Sun, 23 Jul 2023 09:31:31 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=8WthDV1koexSuSPHEquKSnF5vOF3UGGDXZ8fsypYCgo=; b=LxBi3A775/0U
 otmEzaLmxGg7+Rg7Ubkq9bLwdVrGIhR6EwTavE/BD/J7M0be73U64NnBLfcrDe/CgBiFmqLaKdNdo
 Nmpm3ne0b6RfhUqhTn5uy3hCA3OAcPy0jH+MXqCWaFkxS733NDIcgTglCg0fMbH4VYvCIFtsubkhS
 zthq/3WHrvl2qcbfw75RlDXd/PwisQUAB5kG+DxChEZpDrNGyb9Sb1AuKZ3RILlq9p2SvsfYkTWwo
 V3n5aDKgJfo/K7kB2PC1T+Pzs4V2/ZbZXpi+jyK8QIhxD+i2GovTrJCtibkojTDyt6caiHGJJMTR7
 3TNvyAnLGEojNc4IjdiumA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNZBF-00040i-5N; Sun, 23 Jul 2023 09:31:25 -0400
Date: Sun, 23 Jul 2023 16:32:02 +0300
Message-Id: <83jzuqu52l.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87sf9en5ta.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 12:57:53 +0000)
References: <iermszrwqj4.fsf@HIDDEN> <87jzutb14l.fsf@HIDDEN>
 <83mszp9kl2.fsf@HIDDEN> <ier8rb9ukjq.fsf@HIDDEN>
 <83h6pwa52z.fsf@HIDDEN> <87ilaci637.fsf@HIDDEN>
 <83sf9g88eh.fsf@HIDDEN> <87pm4krq2m.fsf@localhost>
 <838rb881ak.fsf@HIDDEN> <87mszornlq.fsf@localhost>
 <83351g7yn5.fsf@HIDDEN> <875y6brrnk.fsf@localhost>
 <83fs5f6opg.fsf@HIDDEN> <87351frqr7.fsf@localhost>
 <831qgzuh5n.fsf@HIDDEN> <87tttvq8df.fsf@localhost>
 <83y1j7t19d.fsf@HIDDEN> <87r0ozq7ze.fsf@localhost>
 <83v8ebszcy.fsf@HIDDEN> <87v8ean99h.fsf@localhost>
 <83o7k2u70z.fsf@HIDDEN> <87sf9en5ta.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 12:57:53 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> > It could be "call as soon as we got 100 file names", for example.  The
> >> > number can even be a separate parameter passed to the API.
> >> 
> >> Will consing the filename strings also be delayed until the callback is invoked?
> >
> > No.  I don't think it's possible (or desirable).  We could keep them
> > in some malloc'ed buffer, of course, but what's the point?  This would
> > only be justified if somehow creation of Lisp strings proved to be a
> > terrible bottleneck, which would leave me mightily surprised.
> 
> Thanks for the clarification!
> Then, would it make sense to have such a callback API more general? (not
> just for listing directory files).
> 
> For example, the callbacks might be attached to a list variable that
> will accumulate the async results. Then, the callbacks will be called on
> that list, similar to how process sentinels are called when a chunk of
> output is arriving to the process buffer.

Anything's possible, but when a function produces text, like file
names, then the natural thing is either to return them as strings or
to insert them into some buffer.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 13:57:01 +0000
Resent-Message-ID: <handler.64735.B64735.169012059216962 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169012059216962
          (code B ref 64735); Sun, 23 Jul 2023 13:57:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 13:56:32 +0000
Received: from localhost ([127.0.0.1]:41070 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNZZY-0004PW-0R
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 09:56:32 -0400
Received: from mout01.posteo.de ([185.67.36.65]:44633)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNZZU-0004PP-OZ
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 09:56:30 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 3298F240028
 for <64735 <at> debbugs.gnu.org>; Sun, 23 Jul 2023 15:56:23 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690120583; bh=ICVlmVsAbXZFmLBOrdichBsEf/CTjZIapsEi9luil1A=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=naxVmHADHuL0vrLV4lJgfy/VgmmnYa/r6OtNxObsJatan53UTpMwlfoILnMGlBx0R
 t7vMyk4J63xx9fCIdDB2guO0c5lXlu8U1bDi3IDNecBpveuhw+oC4So/pvHO9zh30B
 OdNk56rUWr1G8RlOzAs6uvsGGeWRA93M9RPAdNYBDYWTb4DfLpHPqCMyD3+hNwL1n4
 sbYo8GbUMhwi1UkjviKGjHnTVoqV0LChs7aX/byVQpPc6Ppory7EgI9X433fGn37HW
 WPowW+p3Z0JkAx0p/ITdfqksuTR3F/Yn+wRh4uoYyrErFDwcVKoYCU0WDmDpOp6u/+
 7Mrwlu2MuVgxg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R84Zt05Hvz9s87;
 Sun, 23 Jul 2023 15:56:21 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83jzuqu52l.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN>
 <87r0ozq7ze.fsf@localhost> <83v8ebszcy.fsf@HIDDEN>
 <87v8ean99h.fsf@localhost> <83o7k2u70z.fsf@HIDDEN>
 <87sf9en5ta.fsf@localhost> <83jzuqu52l.fsf@HIDDEN>
Date: Sun, 23 Jul 2023 13:56:35 +0000
Message-ID: <87o7k2n33g.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> Anything's possible, but when a function produces text, like file
> names, then the natural thing is either to return them as strings or
> to insert them into some buffer.

Do you mean to re-use process buffer and process API, but for internal
asynchronous C functions (rather than sub-processes)?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 14:33:03 +0000
Resent-Message-ID: <handler.64735.B64735.169012274820731 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: sbaugh@HIDDEN, rms@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN, michael.albinus@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169012274820731
          (code B ref 64735); Sun, 23 Jul 2023 14:33:03 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 14:32:28 +0000
Received: from localhost ([127.0.0.1]:41120 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNa8J-0005OI-Nk
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 10:32:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:36750)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNa8H-0005O5-4a
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 10:32:27 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNa8A-00040A-SK; Sun, 23 Jul 2023 10:32:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=6kBE+t1hb6wxHid8ElAB4wMt7B/PKiGc0eWJ1JMsptw=; b=BM8BXOSo0gI0
 v9KR2tj+2anlvcK6T27anzBHpiJw77oEL8gee+36eKVr2BXHia/LuF5rs3uTkU9t6HIerK2/kXQw2
 Nf6fRn+JMhzM40LSiATv+VF5DoF9n0nL6Q7LX84c1umwAAGpBsuBNn3C2FYjovbgeBRzhulhAWJ/B
 YM+MfU2PvanS4ffL6jiOx8A6WD5lXXy1BuiCnsOQDUx7eljIA08rQ3w4i1Id6NHfCg6xU96Nh4Di8
 lI6t2qnpDOk8656PKrxOzcmUbImsL3cJNaUvDm7coAq03VXUww3VgstrwbmhOksCoZO2FVbIKrJEA
 PRtj7PqnQHH2IsSKzmYmuw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNa80-0003xc-Ns; Sun, 23 Jul 2023 10:32:09 -0400
Date: Sun, 23 Jul 2023 17:32:48 +0300
Message-Id: <83fs5eu29b.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87o7k2n33g.fsf@localhost> (message from Ihor Radchenko on Sun,
 23 Jul 2023 13:56:35 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <ier8rb9ukjq.fsf@HIDDEN> <83h6pwa52z.fsf@HIDDEN>
 <87ilaci637.fsf@HIDDEN> <83sf9g88eh.fsf@HIDDEN>
 <87pm4krq2m.fsf@localhost> <838rb881ak.fsf@HIDDEN>
 <87mszornlq.fsf@localhost> <83351g7yn5.fsf@HIDDEN>
 <875y6brrnk.fsf@localhost> <83fs5f6opg.fsf@HIDDEN>
 <87351frqr7.fsf@localhost> <831qgzuh5n.fsf@HIDDEN>
 <87tttvq8df.fsf@localhost> <83y1j7t19d.fsf@HIDDEN>
 <87r0ozq7ze.fsf@localhost> <83v8ebszcy.fsf@HIDDEN>
 <87v8ean99h.fsf@localhost> <83o7k2u70z.fsf@HIDDEN>
 <87sf9en5ta.fsf@localhost> <83jzuqu52l.fsf@HIDDEN> <87o7k2n33g.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: sbaugh@HIDDEN, sbaugh@HIDDEN, dmitry@HIDDEN,
>  michael.albinus@HIDDEN, rms@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Sun, 23 Jul 2023 13:56:35 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > Anything's possible, but when a function produces text, like file
> > names, then the natural thing is either to return them as strings or
> > to insert them into some buffer.
> 
> Do you mean to re-use process buffer and process API, but for internal
> asynchronous C functions (rather than sub-processes)?

Not necessarily a process buffer, no.  Just some temporary buffer.  We
already do stuff like that for some C primitives.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 17:47:01 +0000
Resent-Message-ID: <handler.64735.B64735.16901343968960 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901343968960
          (code B ref 64735); Sun, 23 Jul 2023 17:47:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 17:46:36 +0000
Received: from localhost ([127.0.0.1]:41207 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNdAB-0002KR-So
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:46:36 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:44269)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNdA7-0002K8-5V
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:46:34 -0400
Received: from compute2.internal (compute2.nyi.internal [10.202.2.46])
 by mailout.west.internal (Postfix) with ESMTP id 8CEDD320047A;
 Sun, 23 Jul 2023 13:46:24 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute2.internal (MEProxy); Sun, 23 Jul 2023 13:46:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690134384; x=1690220784; bh=FqUSWgxGmq/9h/KPhMqPbHj179uxlDJNrBF
 6nqcmeWE=; b=ZvcYRTWTT78J6NpgMgioyUZDNfdxFrIfAVIvmY4KDa3eK5e+L0j
 q26wp2IYrf2BgSoEL7Fp89RztmV7ygSZeWzRETPtm0VZzNkeKBWMQgLdFRPRdk4H
 MzooqXDxNVUHRePJ1VuyEDVfhdtOD31s3xVdFS5/ckj2KcvkWqquND7rF4lpkkwq
 7YPgHPPZLXcDH8NVt4JBXim4lK9Zk1S1FvGMj4T+tpRCELKpnj1YgP1xZyFcTtcL
 e/coFMxqM9IFtcs7q5QjzNhIDNIDVwZ8URY7lpCp9UzbXIe1CgBzogjhftEjQaMQ
 oi+a0ZCLKW5UMsHp83A93KQIRk5kRa/cnyw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690134384; x=1690220784; bh=FqUSWgxGmq/9h/KPhMqPbHj179uxlDJNrBF
 6nqcmeWE=; b=dh1KSvWPEp5ad+dT7FVHfiisoOPgZriIMXlimij1fJkJw5KGvNc
 EedSrApmLFb46+QXAKdyShsJ5aMcD8JfQSH0khw0TUkdgt3oTr8S6Fojc9fbsn4h
 +fAWrjiHfEpg7pMo8Nie+fCrvVaY4nJnkvubrVrcNkl/h3nUEFQJWxzTdeXJCb9B
 fqD1QzzGqA+4hLJEaVmoWWy3/ogk7kKW4iPp8DP/gPMkr/9xTAW9h+RzmnTl0Bh5
 E+NmatvV5Br/PLY7QwzJzjIKRJgqNVOjXinX7dJkgLIQ+BHNTgwWvRFBqWqpG1WL
 4crYIrJ1l/29S+ZdoqpK9xWm38ZmBBdKfTg==
X-ME-Sender: <xms:b2e9ZClVaTmCrDWe7wQSOotBfPS4_3SdL34lX3TrYmgZw1wAIhcqsA>
 <xme:b2e9ZJ00HmxzBvT6lOEs3b57P9-mJk6QfZu1o-TIS42rgo1jNK5fdQhPrc9EjV-PU
 esieVBASchWtZorWEI>
X-ME-Received: <xmr:b2e9ZArNPcowzuQmZ7XxtCdsiN-TC1fKgijIAKzIcpC9KBpGJkcZ_HRmA3HDRQc>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigdduudeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:b2e9ZGm1opgBur1tW1CnZ8g65V2TTUIox_VtxEaRZ2fSgThS1Ida_A>
 <xmx:b2e9ZA1YU7NOlqHG1l-Sa7gnvgflDu06S8eupemGdp_VIxrcJuFPrA>
 <xmx:b2e9ZNvHei6yD18Hj60paC96VSRWcH0ndwZQ-e_MZJsvdqIEgFy0Ig>
 <xmx:cGe9ZNwtRA2vkev7Hz1kdeWVsJLxoIEfrPcdWxIbAdzjOOrG_tJgrw>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 13:46:22 -0400 (EDT)
Message-ID: <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
Date: Sun, 23 Jul 2023 20:46:19 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83sf9eub9d.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.1 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 14:18, Eli Zaretskii wrote:
>> Date: Sun, 23 Jul 2023 13:46:30 +0300
>> Cc:luangruo@HIDDEN,sbaugh@HIDDEN,yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@HIDDEN>
>>
>> On 23/07/2023 08:11, Eli Zaretskii wrote:
>>> Even better: compute completion-regexp-list so that IGNOREs are
>>> filtered by file-name-all-completions in the first place.
>> We don't have lookahead in Emacs regexps, so I'm not sure it's possible
>> to construct regexp that says "don't match entries A, B and C".
> Well, maybe just having a way of telling file-name-all-completions to
> negate the sense of completion-regexp-list would be enough to make
> that happen?

Some way to do that is certainly possible (e.g. a new option and 
corresponding code, maybe; maybe not), it's just that the person 
implementing it should consider the performance of the resulting solution.

And, ideally, do all the relevant benchmarking when proposing the change.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 17:56:01 +0000
Resent-Message-ID: <handler.64735.B64735.169013495210243 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169013495210243
          (code B ref 64735); Sun, 23 Jul 2023 17:56:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 17:55:52 +0000
Received: from localhost ([127.0.0.1]:41212 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNdJ7-0002f4-0V
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:55:52 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42778)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNdJ2-0002ed-F1
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:55:48 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNdIv-0007OL-Dw; Sun, 23 Jul 2023 13:55:37 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=9loBuXdpOhwiQ4Q5Eh3mjURms7w4P6xpZYnIg0br91Y=; b=ip9d/HnpmSQR
 6grqUFo7E72LVA8tWXeWNejoX9wEqdSpGWA7G5j+m67/6e12d/iB3vWKHtLO8MG26VjRvRjQkCFwy
 /rYMD7jB400jgJXHZAFDUG3EolwHhlIYOL70OphHAm7Nj0uG+I9cXSvgZhlct81G/WZrxnkXBheAT
 7xH4Z+mlR2WJlh7JhVonTNPyYg8Ha1Itj9rLtWKwVSNDAD+MhtvkoJJUkQDBDIpS73VEhIAP/ZU6m
 Fzj2IsB3o25RkH3heSe316pkYtPZWZATr9ruj0+YEKLOY3s4ic3MyJ7YF5vsICK5IQzzpAWTB4o00
 J/GqshgOra+0FNOuSPMnYQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNdIu-0006qx-S6; Sun, 23 Jul 2023 13:55:37 -0400
Date: Sun, 23 Jul 2023 20:56:17 +0300
Message-Id: <83bkg2tsu6.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 20:46:19 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN> 
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 23 Jul 2023 20:46:19 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 23/07/2023 14:18, Eli Zaretskii wrote:
> >> Date: Sun, 23 Jul 2023 13:46:30 +0300
> >> Cc:luangruo@HIDDEN,sbaugh@HIDDEN,yantar92@HIDDEN,
> >>   64735 <at> debbugs.gnu.org
> >> From: Dmitry Gutov<dmitry@HIDDEN>
> >>
> >> On 23/07/2023 08:11, Eli Zaretskii wrote:
> >>> Even better: compute completion-regexp-list so that IGNOREs are
> >>> filtered by file-name-all-completions in the first place.
> >> We don't have lookahead in Emacs regexps, so I'm not sure it's possible
> >> to construct regexp that says "don't match entries A, B and C".
> > Well, maybe just having a way of telling file-name-all-completions to
> > negate the sense of completion-regexp-list would be enough to make
> > that happen?
> 
> Some way to do that is certainly possible (e.g. a new option and 
> corresponding code, maybe; maybe not), it's just that the person 
> implementing it should consider the performance of the resulting solution.

I agree.  However, if we are going to implement filtering of file
names, I don't think it matters where in the pipeline to perform the
filtering.  The advantage of using completion-regexp-list is that the
matching is done in C, so is probably at least a tad faster.

> And, ideally, do all the relevant benchmarking when proposing the change.

Of course.  Although the benchmarks until now already show quite a
variability.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 17:59:02 +0000
Resent-Message-ID: <handler.64735.B64735.169013511710558 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169013511710558
          (code B ref 64735); Sun, 23 Jul 2023 17:59:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 17:58:37 +0000
Received: from localhost ([127.0.0.1]:41217 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNdLo-0002kE-Ki
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:58:36 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:53169)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNdLm-0002k1-Fx
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 13:58:35 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.west.internal (Postfix) with ESMTP id 90C7732002E8;
 Sun, 23 Jul 2023 13:58:28 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Sun, 23 Jul 2023 13:58:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690135108; x=1690221508; bh=yq5fEfKbuJB8fN1QdIXxmmOWKMAnPKDk3xx
 91X4qLt8=; b=buaEjVLROhPLwUFElEehba9yNmMWUo/wtSepVFOEctGBi4OOm4l
 lxu/0Tmzh6byGD4cZmjCdpmT4wlepWrtuAhjGpRpce3Tv6cdqfIdHcZslAdJvA2l
 FoF6dDqAwytxLDGyAYz/sJ5FDhgi4HGFto+7lyw7g5DGfeCP2o6IWgthOgvCmUhv
 tG6VDH4ZpfF2HHdbQiHcm0tYk0uJcBSaducl5JEagtNRAvqYFhQxGtuVFdWiTHsO
 jWc+5hiu+Q3K35rKLjIxEtrTEQEya3SKci1xJdVXCk8p5k78g/NT6tAveHBrnEQI
 hXcoczgC+mLVruB3STwDX0N4aLkYeu+FW/w==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690135108; x=1690221508; bh=yq5fEfKbuJB8fN1QdIXxmmOWKMAnPKDk3xx
 91X4qLt8=; b=PwngWDJTsSyuBzU+4mjnAaf7dxP+Epof66BOlUTmRnaShKvPaQd
 5bm35qAQNOa4pFJr3YJc9c8cUwQDIuhj5lYWXGFuFU7/rm91BAe9i692rk9pJLzW
 Ayz1d7/1pdiBvBGdIVrZbv6MYwAKWPPdLOObLpdFG+d8rqgm9mMQXNDtlscO4uYb
 4gVZ2U0bO2KASYoMy3SHwj53DGh+Uckkuvp/m79hJP6nl2y1JxfBvb9XKvDFv467
 Uf4EbD9plnkdCetz1kuw1qzvgExNVTvJYkKeS5GHX7ClqoegKK3zaGgwrs6JIfa0
 uLojRX4IHucoNe2WVxIoO6kkOcbZW5BLM3w==
X-ME-Sender: <xms:Q2q9ZEp7mHU1XR21mAIoSc7JzHz2yuZIrOZb1tS0XtEEJ94HZkxvVA>
 <xme:Q2q9ZKoW3ywYptSFbbsV387biu5C3cM6Vv2Y2UQAUMmvvVETHwmQIx0SUSmMdZExx
 dqJ571Thx1xTyQIra8>
X-ME-Received: <xmr:Q2q9ZJO6cdEAzd-tZuJXl2E1udhG_upcuTLWccC_JJbAhP6HEMpiRW3kLl-dTcE>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigdduudekucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:Q2q9ZL7ucxnReHZzsKP2W7opEtibVLZzRuD6SgsYMQPb5XJki5whXQ>
 <xmx:Q2q9ZD4fRxZGZSPnuU9bHh2FEfhyDa8bXqB80ZrtG13wivaxAuAr8A>
 <xmx:Q2q9ZLidaHRGHSzEwYqs5pmG-62gl7ORYHr_zg1nUX0bY1kg0lbkNQ>
 <xmx:RGq9ZFkCsvRcBQADp_QZjxq723Yc0lSwtwm3P6_maRuUdIUfs00MdA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 13:58:26 -0400 (EDT)
Message-ID: <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN>
Date: Sun, 23 Jul 2023 20:58:24 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83bkg2tsu6.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 20:56, Eli Zaretskii wrote:
>> Date: Sun, 23 Jul 2023 20:46:19 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> On 23/07/2023 14:18, Eli Zaretskii wrote:
>>>> Date: Sun, 23 Jul 2023 13:46:30 +0300
>>>> Cc:luangruo@HIDDEN,sbaugh@HIDDEN,yantar92@HIDDEN,
>>>>    64735 <at> debbugs.gnu.org
>>>> From: Dmitry Gutov<dmitry@HIDDEN>
>>>>
>>>> On 23/07/2023 08:11, Eli Zaretskii wrote:
>>>>> Even better: compute completion-regexp-list so that IGNOREs are
>>>>> filtered by file-name-all-completions in the first place.
>>>> We don't have lookahead in Emacs regexps, so I'm not sure it's possible
>>>> to construct regexp that says "don't match entries A, B and C".
>>> Well, maybe just having a way of telling file-name-all-completions to
>>> negate the sense of completion-regexp-list would be enough to make
>>> that happen?
>>
>> Some way to do that is certainly possible (e.g. a new option and
>> corresponding code, maybe; maybe not), it's just that the person
>> implementing it should consider the performance of the resulting solution.
> 
> I agree.  However, if we are going to implement filtering of file
> names, I don't think it matters where in the pipeline to perform the
> filtering.

A possible advantage of doing it earlier, is that if filtering happens 
in C code you could do it before allocating Lisp strings, thereby 
lowering the resulting GC pressure at the outset.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 18:21:02 +0000
Resent-Message-ID: <handler.64735.B64735.169013645212823 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169013645212823
          (code B ref 64735); Sun, 23 Jul 2023 18:21:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 18:20:52 +0000
Received: from localhost ([127.0.0.1]:41224 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNdhL-0003Kk-S1
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 14:20:52 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:50558)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNdhG-0003K8-Be
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 14:20:49 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNdhA-00054K-EK; Sun, 23 Jul 2023 14:20:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=3huSKIquAcjLowpeC6+P1Pg0dbYPqwCPUEtw4jbLDSA=; b=mn4AMLItPZok
 B06cZciTNH5Ii00Y8GjHJxEccvc8pTj2izmZcAVCpxvOnGKzirbZ5ewIqhOBrrcIUAi/M8IxiIDST
 1tQwtkpuDid2rM7S3s1ldlnLSlBuVE0pnZ0jcSGy1fm2H9Zj2n/Uzk6MnNFXWfRuLsxb9OfX4W/GU
 JQA1JSb47+QqmKIMMy06nu/36K+gA7YOKow26buYMB+/zCsRJ6Xi0TR21i+QSBZJyqPApJcKrXek0
 MOLCdZcSAqxkudjt3PibZ/KkWKTZvIHohB6qhRgTVRvD8l7UNBEVUptmNrb8U6z7XA3fyn5eCubcq
 uFIWbiRQaDttV8hDMEzWXQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNdh9-0004dY-Tu; Sun, 23 Jul 2023 14:20:40 -0400
Date: Sun, 23 Jul 2023 21:21:19 +0300
Message-Id: <83a5vmtrog.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 20:58:24 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 23 Jul 2023 20:58:24 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> A possible advantage of doing it earlier, is that if filtering happens 
> in C code you could do it before allocating Lisp strings

That's not what happens today.  And it isn't easy to do what you
suggest, since the file names we get from the C APIs need to be
decoded, and that is awkward at best with C strings.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 19:08:01 +0000
Resent-Message-ID: <handler.64735.B64735.169013925017241 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169013925017241
          (code B ref 64735); Sun, 23 Jul 2023 19:08:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 19:07:30 +0000
Received: from localhost ([127.0.0.1]:41237 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNeQT-0004U0-Gn
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:07:29 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:37789)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNeQQ-0004Tm-ID
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:07:28 -0400
Received: from compute2.internal (compute2.nyi.internal [10.202.2.46])
 by mailout.nyi.internal (Postfix) with ESMTP id 286345C00D9;
 Sun, 23 Jul 2023 15:07:21 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute2.internal (MEProxy); Sun, 23 Jul 2023 15:07:21 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690139241; x=1690225641; bh=1Ee063bk7od1pJ2gcn7SesVGEGGjWNmniE9
 Jcu5v7m8=; b=IzYHnh0g9fTS9g8inzsB6sY+yk59Zo/ejiu6Zw0Oi7rBf3adDch
 Kys243WsDsWQJbv+mv52aPUmuuHq4OlvP9xCvbN+jh+HpCZtVndqkRBkTh+ow8zB
 WH+rmVQ9Vh/Wup/yg9cb+1nnk3EoW2aNgA32Jyz9C/gdJ1GyTjcuGs4egsIxpFTV
 1iRKqxSicmjgPDzc7CkE1ByJAt3MysN69VzL6A35PonxzX+zWL+gp02/lpdY/X9T
 DLpm6qIxIw5b/+kqaiCNGWjiO9iIOdh53EgW8Gee5O6tDvlSBw4gHIcwF6nu3Jc5
 /Ozkd3NO4bYz/ejEKc4vu/IfO/YNdijeWhQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690139241; x=1690225641; bh=1Ee063bk7od1pJ2gcn7SesVGEGGjWNmniE9
 Jcu5v7m8=; b=pikfNHvJ/C8Q37IvfGOTOVyrhV9xL9YOinXm/Ck1xov6WnkS/Ol
 yA0xNAA5l11mEgviejbNUpV/Gemt8YtZplEIWbTirFewE1S8XGtrttw68VyZZFzT
 iGBZuuKREaGcoroG3HxUsgP+sC7dlnjUiHq6IHlIhWHyZf3V04wr+1xSIsN3BPh0
 Imkbj51yV7e6AaN1TDNXhYOvlLmGaTm7lSPC13YkiO+HOGVHYnHBa1Ek29W/HXX2
 xGbYpAcv7Hs9W1vYO5Owmp+O9F5BDfc9qSgEG36643LwFn+FZGNXIQq9EdhXbgr3
 +xRinCDJgysz9zFSSCe9kkYq9Rg6sB7cpNw==
X-ME-Sender: <xms:aHq9ZCZrBb-XpiL-zPaZhQbrILw44Nw-aNj6Y_Dw_N2MaRlSfSI1_w>
 <xme:aHq9ZFbfkDbtnHDZXrhmmeg4iJavBqdqxBVzbDE4R3N-UYXVrYbSZW2G25zbjWBr9
 CLsuIz2NnAyIHfNDG0>
X-ME-Received: <xmr:aHq9ZM-z_tI7LYeHq83Mu7feHzmi0dkPnR1vyPJgunzSynQmQXS6GmTFId8x7bw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigddufedvucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:aHq9ZEr1tTXyL5yCdv-CgCxAG60vlqAatFt8W1pgrMn4RX59OEkiog>
 <xmx:aHq9ZNqwVeG13IH8oAYdopHGb_Ch8ZbpdgV3Lm6LY4-vkKkPPBb0Uw>
 <xmx:aHq9ZCTFPoQvFMQNS6jTVoYounQcEtT5CRgSR30d88huum3QfySNaQ>
 <xmx:aXq9ZEUrZciyM6iv1VJq2Pt0FV7CNeIgJboxnYqo3pL0_p-uzkUOMQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 15:07:19 -0400 (EDT)
Message-ID: <1ff691d6-67fe-7aa5-75bc-435438e59152@HIDDEN>
Date: Sun, 23 Jul 2023 22:07:17 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN> <83a5vmtrog.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83a5vmtrog.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 21:21, Eli Zaretskii wrote:
>> Date: Sun, 23 Jul 2023 20:58:24 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> A possible advantage of doing it earlier, is that if filtering happens
>> in C code you could do it before allocating Lisp strings
> 
> That's not what happens today.  And it isn't easy to do what you
> suggest, since the file names we get from the C APIs need to be
> decoded, and that is awkward at best with C strings.

It is what happens today when 'find' is used, though.

Far be it from me to insist, though, but if we indeed reimplemented all 
the good parts of 'find', that would make the new function a suitable 
replacement/improvement, at least on local hosts (instead of it just 
being used as a fallback).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 19:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169014042818931 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169014042818931
          (code B ref 64735); Sun, 23 Jul 2023 19:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 19:27:08 +0000
Received: from localhost ([127.0.0.1]:41248 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNejT-0004vH-Qy
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:27:08 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47780)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNejR-0004un-Nc
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:27:06 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNejM-0004di-BW; Sun, 23 Jul 2023 15:27:00 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=bcrmXRIWQz+FPlz0O3TCToeHfSucAGZnDgu6yX3G6Hc=; b=BFGYpifczX9y
 TsPSJGbb8q67YkMg+uYwrlDNOsGUNhTx7Ay3BZkh4YDZGGsekfu0JqyOPvjwncqSAWyxjn+7n25r0
 N5qmNwPc5zOQ0JhsVNqnOa1QxHXr9oTXp0nBb+Vwr+poS5fsgzeopv33z977aEI6l5vKt92tZYAbU
 uwhNXV/VbVcE/n4QYONQh/sur0Ono/9GWLn8HxKCDRgDnHSmG6dh0ATnlrVvnY6T/AHpgUK5sJCX7
 AD52tISoAGAh5kfeVtDfZ2pIDN4wL3DwU5Lq2Yp6KpbTeywzV5mGDyOxD1dOnNOCAC3J01TC08vJ9
 pmhgOsb/3aSetrfAQ9RR6w==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNejJ-0002Uj-Lz; Sun, 23 Jul 2023 15:26:59 -0400
Date: Sun, 23 Jul 2023 22:27:38 +0300
Message-Id: <837cqqtolx.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <1ff691d6-67fe-7aa5-75bc-435438e59152@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 22:07:17 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN> <83a5vmtrog.fsf@HIDDEN>
 <1ff691d6-67fe-7aa5-75bc-435438e59152@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 23 Jul 2023 22:07:17 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 23/07/2023 21:21, Eli Zaretskii wrote:
> >> Date: Sun, 23 Jul 2023 20:58:24 +0300
> >> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
> >>   64735 <at> debbugs.gnu.org
> >> From: Dmitry Gutov <dmitry@HIDDEN>
> >>
> >> A possible advantage of doing it earlier, is that if filtering happens
> >> in C code you could do it before allocating Lisp strings
> > 
> > That's not what happens today.  And it isn't easy to do what you
> > suggest, since the file names we get from the C APIs need to be
> > decoded, and that is awkward at best with C strings.
> 
> It is what happens today when 'find' is used, though.

No, I was talking about what file-name-all-completions does.

> Far be it from me to insist, though, but if we indeed reimplemented all 
> the good parts of 'find', that would make the new function a suitable 
> replacement/improvement, at least on local hosts (instead of it just 
> being used as a fallback).

The basic problem here is this: the regexp or pattern to filter out
ignorables is specified as a Lisp string, which is in the internal
Emacs representation of characters.  So to compare file names we
receive either from Find or from a C API, we need either to decode the
file names we receive (which in practice means they should be Lisp
strings), or encode the regexp and use its C string payload.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 19:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169014045718974 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169014045718974
          (code B ref 64735); Sun, 23 Jul 2023 19:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 19:27:37 +0000
Received: from localhost ([127.0.0.1]:41252 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNejx-0004vy-9t
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:27:37 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:59089)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNeju-0004vk-M1
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:27:35 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.nyi.internal (Postfix) with ESMTP id 7BA745C0043;
 Sun, 23 Jul 2023 15:27:29 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Sun, 23 Jul 2023 15:27:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690140449; x=1690226849; bh=FbeRyXKYgT9IIX2gOp6H/B6wY2557U1eNIG
 Rf40ubkE=; b=Flt1Uu1OXcY0GgyALc/oLzFONhpD7wviJYWbAQZXsQtAVjOaxTg
 pU6ctedUSXdGuJ0GNko/RbHHwDhVDHz+XfYagKLgkfMvahef+yf2UImJY6W3TGsv
 2+PyogyCu2DoP8GLZhj3A78rIg/Bcgvg9wEbzpCnL4y4k6V76eRdTIDubjxpvajq
 U5HYfH2jb9pHEXxxI8avniyeaDVbPNZdpFhDaG8Zjxmt4SZiIFa+eBfJJEaQKtR8
 LTRipdYvs9fhmWE5EDOn248wAHyTxf33h8OyZcR52TAU6NxdOwnWbplUSmtL0NzA
 C1ZKuCb+Q5Fl6xtTk9dyUUkVTBrTJotacpw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690140449; x=1690226849; bh=FbeRyXKYgT9IIX2gOp6H/B6wY2557U1eNIG
 Rf40ubkE=; b=iPFKlGRxaGkkFiBCuahTf0kxvrLb8gyC7OvlpZ6cvYuDvyIQI3X
 Z76gkiumO0qXLZtyad78onQDBcxdqMcrhywUgMBOOQvaSeD/NESr67C4/WRUjxUb
 aX5KxX+GAOyw3oZ76DPMIivTpaqkxncgRvO7RZRbBWNIKI/GM9e8zYzvtDp4tLui
 X372XFfHEMI5Rbi2VwKRHo+3keq2oKQmmPisKNp8H41ddLZmTsNxtzfxtwtpdOst
 6cijhKTDWo5MkSowIr9TtnBLGdxF0Q6mzyUDpNVlpKeiLL8+vUK+ZIl7K59vTbJk
 ifHSIkdMiUs3vhebvbKUSzHI+u3kKHiKR7A==
X-ME-Sender: <xms:IX-9ZIEXeMOIKmnTOA44Zd5cUTaRx3vBqbMQoZFO0AyKy-DQRonurA>
 <xme:IX-9ZBUEIvD_6gKUOEObRiNEhvuLpop5PegM_33A1V5NdW4eAx4AJ0IdBgz2aVDWv
 QkeXAePwm_tzGaPlIU>
X-ME-Received: <xmr:IX-9ZCLz4r5iQ6RoypR1tPrNS57XSBDT8fXTgfhyn8pLUkzO1duqZkMPVizvHCU>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigddufeeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:IX-9ZKF9TRFoQJFDZszZN4XtjjFmXCF1ptw3PanIossa_fz27JokQw>
 <xmx:IX-9ZOXnjKdlJos62Cxx-ImVV_sa9FPNAtvqB_4C9avgevjv0ssqnw>
 <xmx:IX-9ZNMWWc16-RSGfklP82-VIeJfZwVw9o2PBJyL4BhgszzEI0TB2Q>
 <xmx:IX-9ZNRj0AzlKuvbSug9PBQilVXiLsFUn1ZYoTP-xb6Oq5FwbDKcog>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 15:27:27 -0400 (EDT)
Message-ID: <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
Date: Sun, 23 Jul 2023 22:27:26 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83bkg2tsu6.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 20:56, Eli Zaretskii wrote:
>> And, ideally, do all the relevant benchmarking when proposing the change.
> Of course.  Although the benchmarks until now already show quite a
> variability.

Speaking of your MS Windows results that are unflattering to 'find', it 
might be worth it to do a more varied comparison, to determine the 
OS-specific bottleneck.

Off the top of my head, here are some possibilities:

1. 'find' itself is much slower there. There is room for improvement in 
the port.

2. The process output handling is worse.

3. Something particular to the project being used for the test.

To look into the possibility #1, you can try running the same command in 
the terminal with the output to NUL and comparing the runtime to what's 
reported in the benchmark.

I actually remember, from my time on MS Windows about 10 years ago, that 
some older ports of 'find' and/or 'grep' did have performance problems, 
but IIRC ezwinports contained the improved versions.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 23 Jul 2023 19:45:01 +0000
Resent-Message-ID: <handler.64735.B64735.169014147720493 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169014147720493
          (code B ref 64735); Sun, 23 Jul 2023 19:45:01 +0000
Received: (at 64735) by debbugs.gnu.org; 23 Jul 2023 19:44:37 +0000
Received: from localhost ([127.0.0.1]:41259 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNf0P-0005KT-2b
	for submit <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:44:37 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:42355)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNf0K-0005KA-28
 for 64735 <at> debbugs.gnu.org; Sun, 23 Jul 2023 15:44:35 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 052885C00BF;
 Sun, 23 Jul 2023 15:44:26 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute6.internal (MEProxy); Sun, 23 Jul 2023 15:44:26 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690141466; x=1690227866; bh=AkWN7gpo2ZAb/gBv8GCIY3kbasv77uvb2RK
 T2gVvpQY=; b=h259HOPBWiGLsDZPiomiGJulL0HdMHEhCQFpblSkvy3Xeu7VXUu
 XoKxuTlSnuSsL5jLIDe9UOsNlYptSQ2EZqUzotqgwNHoV05AF1sBvTPfStuYhviW
 96znj/wvQHPGxe1n0nUFDeUR8iOKikd+kaxxz2ritjkIDd6ySfRQdi8rZZbCtEtz
 0aXQASYl3plwdA8yj4A3jsQ9THUQaOHEyZ6EhdrMNWhi1MQ1wWDTgO1o4/naGF1u
 dxqj29ATu31mn/JHIP4F2k/dQfMuGdYgPOCYSU2ghdUJPyoH8U0jAOWtBO87R97L
 7kz7v+wKPyK9w5QStDu3878e52NU9uSsiBw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690141466; x=1690227866; bh=AkWN7gpo2ZAb/gBv8GCIY3kbasv77uvb2RK
 T2gVvpQY=; b=SwUOVS2kD8RASWGsyNzymKyjYF60PHZ20cx0ju7hMreeYfEzk1g
 1TgVJEVqUkPtz+a3iIRFSmNeg7XanRBhj0w28VbnT0q0k7Ca3yJf2jBsn9XZYICD
 JhYItvs6bgIhmNLp0AROcLmEKaNHjncBJfdUWFZ5Msd9hmW1ewpV4AZfF+kqImc0
 ZZ4ZFHx6NGHVMWIuy9MXIRoOo9ip7dagoeGLSCEFwRFVp5vZWGRbPoFt7l56yUCM
 xB67TgLKv5JnDK/7+DYoEspc9WtPSXCMk1+DGSa10Cp9lyrmwai2CndhZPJoiUA5
 19wsoSH7nR6UTFlbbbX+BJQcTr3vKBQ04Ww==
X-ME-Sender: <xms:GYO9ZFVSR0ZlIafqJ0z84kpeBZR8XFiD3-RyO57n4AAC-Meifd3xKw>
 <xme:GYO9ZFmIqOQxvWG_BAd9lQge5Jdaf247x0DLr--AE5E8vGfQUYVgfSfs6f2HiYYEJ
 cRj77ccXs3d72wiDmM>
X-ME-Received: <xmr:GYO9ZBZcrp38zkQoxX7uzMnlt-Pwp7_ItpB-bbKckYcBsp54DnKQZtkgIY3rlqw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheeigddufeelucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:GYO9ZIVlJAdDmN47Tta6WjDzDEsP3OsnEp_k_4gIIXMDBlf8NqzGzw>
 <xmx:GYO9ZPl7lOYUk2kGicCXqgPV1lwVbyrc2Z2nrCzyepFx4LsZxYIpPA>
 <xmx:GYO9ZFcbDU37HrdXKJBLPBjCXTzxvyhp9Uk0Uyyx3dcVwJVVmkpxSg>
 <xmx:GoO9ZNj05vSkTltz-i2QJ4oiRVSdM99TY6eduSIyUZpvKFudnC78tA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 23 Jul 2023 15:44:24 -0400 (EDT)
Message-ID: <2c03fa11-6c97-cbdd-567e-b5dba85b7f77@HIDDEN>
Date: Sun, 23 Jul 2023 22:44:23 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <e2f1e366-71f6-15b2-7515-8bce1cb43dc3@HIDDEN> <83a5vmtrog.fsf@HIDDEN>
 <1ff691d6-67fe-7aa5-75bc-435438e59152@HIDDEN> <837cqqtolx.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <837cqqtolx.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 23/07/2023 22:27, Eli Zaretskii wrote:
>> Far be it from me to insist, though, but if we indeed reimplemented all
>> the good parts of 'find', that would make the new function a suitable
>> replacement/improvement, at least on local hosts (instead of it just
>> being used as a fallback).
> The basic problem here is this: the regexp or pattern to filter out
> ignorables is specified as a Lisp string, which is in the internal
> Emacs representation of characters.  So to compare file names we
> receive either from Find or from a C API, we need either to decode the
> file names we receive (which in practice means they should be Lisp
> strings), or encode the regexp and use its C string payload.

Yes, the latter sounds more fiddly, but it seems to be *the* way toward 
find's performance levels. But only the benchmarks can tell whether the 
hassle is worthwhile.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 07:35:02 +0000
Resent-Message-ID: <handler.64735.B64735.169018410127471 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169018410127471
          (code B ref 64735); Mon, 24 Jul 2023 07:35:02 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 07:35:01 +0000
Received: from localhost ([127.0.0.1]:41650 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNq5t-00078v-8a
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 03:35:01 -0400
Received: from mout01.posteo.de ([185.67.36.65]:57435)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNq5q-00078e-CZ
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 03:34:59 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 4E8F3240029
 for <64735 <at> debbugs.gnu.org>; Mon, 24 Jul 2023 09:34:52 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690184092; bh=OqthtdZWfqnOhu5N5YjqAQkhCfk3lm6674KrvpkvZuk=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=YRqvcU6n4nC4yu8+bwSof9K8XgdwIw2fdRvu/C7rx6RwQC2fmmSv4BpVWHUN7rmOo
 XHnogDHl9tTuDsFcoH7n6FF/tCk7kcNKjGfuU5RF+ssrudVp9SX8u8tV4EFTyfDD/p
 5b4/INDSNo620tdI4ylhYLUhxA+d1PyyHlBbpngJcyRoFvJQW/qJBvQRa2Z7uQP9Pv
 9aU0/Aw+mC+5exnE96XXf+F9ytCdLYdYVMtvjm/6OjLQuUiyu/yrIsaW4AAKEbzt6e
 MbWYJnJ8MgwXAQ6FKCRsqNkhJnk9j8yF95hUfP3cugkCZPV1PLaICQ/KM5rTToUvd2
 Wo1lH3zWRxtFg==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R8X4C011Xz9rxd;
 Mon, 24 Jul 2023 09:34:50 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <871qgyzw1v.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <87pm4jrxra.fsf@localhost>
 <871qgyzw1v.fsf@HIDDEN>
Date: Mon, 24 Jul 2023 07:35:04 +0000
Message-ID: <87mszl3gpj.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

> Tramp is just a stupid library, w/o own intelligence. It offers
> alternative implementations for the set of primitive operations listed in
> (info "(elisp) Magic File Names")

This makes me wonder we can simply add a "find" file handler that will
use find as necessary when GNU find executable is available.

>> GNU find is faster simply because it is running on remote machine itself.
>> But AFAIU, if TRAMP could convert repetitive network request for each
>> directory into a single request, it would speed things up significantly.
>
> If you want something like this, you must add directory-files-recursively
> to that list of primitive operations.

So, it is doable, and not difficult. Good to know.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 08:00:02 +0000
Resent-Message-ID: <handler.64735.B64735.169018559330492 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169018559330492
          (code B ref 64735); Mon, 24 Jul 2023 08:00:02 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 07:59:53 +0000
Received: from localhost ([127.0.0.1]:41688 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNqTx-0007vj-CG
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 03:59:53 -0400
Received: from mout.gmx.net ([212.227.15.18]:49967)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qNqTs-0007vQ-Gt
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 03:59:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1690185570; x=1690790370; i=michael.albinus@HIDDEN;
 bh=NrcYiXhIBO5kFifKga3t0wzZvLSK3OvxqhLKNVE6hPE=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=DjCmnmqur4BrqAE9Z6FmV80ASfjgDOEFuU18JZAu0gh2UP11jpEp5JTj6/gHSno7QEZYrOj
 DF7gZjbbw+n1VzaV22uD01SeXVDpE7wFATl1MZqENLOipC5Ldz1FUwmCOxYP0YwzUeTKr7ERh
 VMFpfTcCUPm7wp6moMX1cdB+bnqEuq1kpPr5e2gWbbST+NJo5QXZMNBhgJ8Glb2fuzx+UtmDJ
 t8eDg5AkHWlO9kxGC4qq8hF5wMVN5ma0wANiXDkBKlx5BoYcwTvygh/bE8UBEsG0mTuEn4ju7
 J3PmNZWbkWgNXj0jTLhMCinV4i5pCW7BX0Rq4hUcEFxJVBN+8fbw==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MryT9-1pajsV1pOX-00nvdL; Mon, 24
 Jul 2023 09:59:30 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87mszl3gpj.fsf@localhost> (Ihor Radchenko's message of "Mon, 24
 Jul 2023 07:35:04 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <87pm4jrxra.fsf@localhost>
 <871qgyzw1v.fsf@HIDDEN> <87mszl3gpj.fsf@localhost>
Date: Mon, 24 Jul 2023 09:59:28 +0200
Message-ID: <871qgxyc2n.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:R2RVGbsVISVb4UN2+3ZNwMXNFzJtLCNuVlwW06WXf1teeOh49lN
 wLZDxygE+Kfa0RhA4PXHVtvv3LX439wtqfDpNoOhh0udSQjYwygK/y/h+GpeLRFoARo4T30
 utGs//jEQSu3kz595t4XUawMzrvBLWqs1AIFjNDWDPLcXjm0M7hDFZNRT5HNz7y5ZAduV4B
 STECC87wnDC8tih+VK+2w==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:K2aHPcvDJ0I=;Lkv1uknEpgucN+RDWWF7irbiKrV
 68OCuu7hCHMZ4Ye0k0UOyJAnfg0tZkcLrnCgBFuEKj4EDj0rZxwInG9JtKRfDS5acb7oy14ZX
 4cBnP2uYYuqNZFaQ+0HC3ydcEE/llEv2PFAwgRb+klMoju1vOAeAnZuxfNOpaqJndg+uTCVlo
 EgaE5vtcOrgeXIvyfdMSQRx90mouG1J4V+La7KO7mqdYqHG6c/sjJBGtSEhalPKZEFSf5+HYB
 knbRxp00s7cw4gNSS7H1NI3K3T3EyejdPjssLWgUk6fBZ7SeoRniuFXd0j39e0rwCrYtCOhQv
 i/kHdmYZMA3E0+r82+U+0NHpteFJe0xnDRNezvXSyz1r/fyk6Ww8iTYcEcVz3jP4wx9ON4kGc
 B+2GawJN0Qk1hk2FYL6qr63OY48tExGLbKB5WwpzvzaeUHMsBkorJlD6hEbJSPRGS9KK+xdsE
 nORRzUNS/u4C+tZnDg1qm1emMzVxnockW4GICh+uGaMPKyPANXqc/bXx7LlxI97LL3ljYCzWb
 /QtZjVGG0EYIPgplSQfbNpXpW+TrYLwp0LuBbLleMdzTkeV+BPW6pcKqugUeg5UOgLAM61VcA
 82B1RhRwkzGMq3cMpYOHbt9qeLdbMLY6rhCZf12fXreSdl9e9KYTdr1ATf4xlEl1A+7Z78hnS
 TBQVSrykUj8EzyzCjkCUiwFYidaFohk7aHS9NCHCIsUT4fJMbUPTvtLk/Y7acG7JArktPWtXh
 XWkbpqaGT2vuD7TT0fcu5USzfSyQHB1Rkaqu0z+VzrQDFM52QNOrUVDdNvAJ6NSvIMlzwx1Tj
 o04uybE8kqyviuW+bZAwmlgj1iwWPmWcSQ1GgT14BohPeZ3ZGBffSSJpBilqySaSrioWGqzsG
 AdEnE22xsH19wjC4aWvPR/QVWP/+0BjULJ/bftUdi8Cf5ZuM4k3zkqTjSJeA7+YliVtMCf+/h
 frFiUA==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

>> Tramp is just a stupid library, w/o own intelligence. It offers
>> alternative implementations for the set of primitive operations listed in
>> (info "(elisp) Magic File Names")
>
> This makes me wonder we can simply add a "find" file handler that will
> use find as necessary when GNU find executable is available.
>
>>> GNU find is faster simply because it is running on remote machine itself.
>>> But AFAIU, if TRAMP could convert repetitive network request for each
>>> directory into a single request, it would speed things up significantly.
>>
>> If you want something like this, you must add directory-files-recursively
>> to that list of primitive operations.
>
> So, it is doable, and not difficult. Good to know.

Technically it isn't difficult. But don't forget:

- We support already ~80 primitive operations.

- A new primitive operation must be handled by all Tramp backends, which
  could require up to 10 different implementations.

- I'm the only Tramp maintainer, for decades.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 08:23:01 +0000
Resent-Message-ID: <handler.64735.B64735.1690186946475 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.1690186946475
          (code B ref 64735); Mon, 24 Jul 2023 08:23:01 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 08:22:26 +0000
Received: from localhost ([127.0.0.1]:41719 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNqpm-00007Z-5b
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 04:22:26 -0400
Received: from mout01.posteo.de ([185.67.36.65]:39523)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qNqpk-00007F-5X
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 04:22:24 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 99C39240029
 for <64735 <at> debbugs.gnu.org>; Mon, 24 Jul 2023 10:22:18 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690186938; bh=t3nN9O44R+GphrJ/Vetgc1UsnNZoruTezJzMr49lUnw=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=TfpRFw6T5pwzqw6aHFmZjqM2IteuI0ekI2vq9SeOWCCeAoILEhEZA9ohNYvqQHJuZ
 hKS0/kuEZbaza9z2XplRhjmOsxA3iKP66ZRRQhMG4TEIhputICymoamEa80nw2MUzC
 PJ5Rw8it/vtdmBd5VJbi5mPYNdlXydQ8eDek3WtWT3q2rKwQ0CfsGmF5k8H+L3oeSH
 7dSaJ7gjaUoFpbVm6uPw/WBJhY3SrczPtm0st1rDakhQDv2H3fI/Py/u2g51Uo3xsM
 2q3OyPmiLGHW1SVC78EeSsK9kb+I8XiWBIgmYEefqxwABvxe7e3+6RDAlQZPY0pUDL
 iRWSl1CFGbFqQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R8Y6x5Mp6z9s6g;
 Mon, 24 Jul 2023 10:22:17 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <871qgxyc2n.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <837cqu35lw.fsf@HIDDEN>
 <87o7k6pmk3.fsf@localhost> <834jly351p.fsf@HIDDEN>
 <87lefapkdx.fsf@localhost> <831qh230h5.fsf@HIDDEN>
 <87wmyu8mi0.fsf@localhost> <83wmyu1l1k.fsf@HIDDEN>
 <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <87pm4jrxra.fsf@localhost>
 <871qgyzw1v.fsf@HIDDEN> <87mszl3gpj.fsf@localhost> <871qgxyc2n.fsf@HIDDEN>
Date: Mon, 24 Jul 2023 08:22:31 +0000
Message-ID: <87bkg13eig.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Michael Albinus <michael.albinus@HIDDEN> writes:

>> So, it is doable, and not difficult. Good to know.
>
> Technically it isn't difficult. But don't forget:
>
> - We support already ~80 primitive operations.
>
> - A new primitive operation must be handled by all Tramp backends, which
>   could require up to 10 different implementations.

Why so? `directory-files-recursively' is already supported by Tramp via
`directory-files'. But at least for some backends
`directory-files-recursively' may be implemented more efficiently. If
other backends do not implement it, `directory-files' will be used.

> - I'm the only Tramp maintainer, for decades.

I hope that the above approach with only some backends implementing such
support will not add too much of maintenance burden.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 09:33:02 +0000
Resent-Message-ID: <handler.64735.B64735.16901911307870 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: Dmitry Gutov <dmitry@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16901911307870
          (code B ref 64735); Mon, 24 Jul 2023 09:33:02 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 09:32:10 +0000
Received: from localhost ([127.0.0.1]:41778 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNrvC-00022o-RY
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 05:32:10 -0400
Received: from mout.gmx.net ([212.227.17.21]:58251)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1qNrv8-00022H-AQ
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 05:32:06 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de;
 s=s31663417; t=1690191109; x=1690795909; i=michael.albinus@HIDDEN;
 bh=4Gh4/eltQqhWdolKvPTExVf3zsQY4Qle5Xzn8RSisoc=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date;
 b=tITyChjwzJIi07SUoNi6srb1bYQt034lUusidH9zz00abb6eyyl+bmKQ+nSzoDGNrhmIRLV
 gqd5gixVWKu7aG8Tos4vbapoTrovs0STpJojFzExHEOQL1+vCdZumD41pg1rVKVuDoP2FzP0+
 wZAXa/CiO/PAVqtvx35fAxY3PntidPoBQcqJoU8UFMXb54MEeeZs5R7HE8lyyI6gtlF4yxRsv
 /MeeVjbzsJLUFeamKWEzu4bCQpNQJBcJYEAkCfjsdUxdRaR8e0/+0LYFRz81TBKQ1vGDhbH/Q
 ArPR3qzjBnvkE2McT0Kz3qrw24IxNTxp6sWExJxHHvWNYdgoy2Hw==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from gandalf.gmx.de ([185.89.39.27]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MdebB-1pp5CG3BxZ-00ZeEG; Mon, 24
 Jul 2023 11:31:49 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
In-Reply-To: <87bkg13eig.fsf@localhost> (Ihor Radchenko's message of "Mon, 24
 Jul 2023 08:22:31 +0000")
References: <iermszrwqj4.fsf@HIDDEN> <87tttypp2e.fsf@localhost>
 <837cqu35lw.fsf@HIDDEN> <87o7k6pmk3.fsf@localhost>
 <834jly351p.fsf@HIDDEN> <87lefapkdx.fsf@localhost>
 <831qh230h5.fsf@HIDDEN> <87wmyu8mi0.fsf@localhost>
 <83wmyu1l1k.fsf@HIDDEN> <87ttty8l73.fsf@localhost>
 <6f55b301-7610-ee4a-996e-fc6a8db33c33@HIDDEN>
 <87y1j9l8u2.fsf@localhost>
 <0c6f4daf-ea8a-0636-1b51-f4640ff39834@HIDDEN>
 <87jzutl76e.fsf@localhost>
 <b725f1c7-58bd-8538-39be-5da01a055424@HIDDEN>
 <87h6pxl6m4.fsf@localhost> <87pm4jrxra.fsf@localhost>
 <871qgyzw1v.fsf@HIDDEN> <87mszl3gpj.fsf@localhost>
 <871qgxyc2n.fsf@HIDDEN> <87bkg13eig.fsf@localhost>
Date: Mon, 24 Jul 2023 11:31:48 +0200
Message-ID: <87v8e9wt8b.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:y75v4HIHR7VRZT1UKzxkxMrAIUQglKuuhVGuKmCAgTtyaoQ7xTq
 1kVaweepLaz4FAejv3ykZTvRniIJ5T5jj6FSV41beBoOuOPYN86ku3XYH1LFy14Tli8XCkE
 QypcIl1JyJPezr6JZ58VVT4f9187zectrYcewZTypR/wVDdWb/96UvaLFi9yh5LN6a5vNuF
 lAaBGisVaOL0U7KowFHmw==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:KH9YYJVTzMw=;8TShp4QnKk6kqxrfHS3AFPTJA/0
 UaNbFi7jwx4am+r4qkyGSTJPMufXfJc6SgxTTMDSTPoVfiTCEgs6Er5yVMwp4L2T2LASHgpHn
 exvvVd7WwH+/EhkRCuG5Xocp+5oVu6SdY1XsFsUAnqIwI0NJZIE5jno1l1zolVKmcbsIsZkvu
 aVdgmF257s94Uu5lkYKqh5ZxbjxQiI4mdpDDwRI6yTgMb24UnKzX4uPz2wfjXA8fP4LljXsmT
 aE4ZPdnaifaArbZirlFrZMFJsnlnEeV5maD1npxw3rftDdg3O/sFVwE67VjaKa9eZh37KC/e+
 2OSEojRnNRogBxcwZso9uFN0ZeoLXiQI0DOg6JVcIrR9rWCBzcZbnLrx8oMuBaOsCf2s9Zj0v
 CZfBo7RHdpnRYl+TdMdPT3kWt8l3DJy2ZpYiVhE3MD+5cL/MjR6wPoN3j+0H5W1icYE+jHYce
 qonEHggovhtSExkCbBMCjUaSBpBV625qlOLhDYmG80iyxhF7tJb5j1KBn0dbWK/E6eFZsNi3+
 2OcZeR7gTC1TIlKFuJYpFanqs7MT3qUdoXa7Z9taeBhfqRyEjQAve0AV+Afrng/wx3v1OmSzO
 4C96osvVpl6QyKEu0DPJXPiddzF+k7P9YrHEDL8fV6zC2/iMCS4bOgzMB6sA1NmS57u/3Vgr9
 vgId/mqtKTa2r1zwQXOwEAbchVRdVxafA97mKvlhKGetIRCa2HKuRsvM7oh23SvfY3snY1Nvc
 e+5wgPxlIvX3Q7wyHnJ/EHr9BlvKq4BQGFXlL/mAUn+J5aWi+t8D094YIxN027+CYe8xwGcTn
 ddpi7yBjNKRRpXuHss7Enk2bPEUDpVS1VFRiZPWvu/+quukMDyNxhgsqvGx8AH1h6X3CBHNnB
 DXmImxbk7SDKI610mzJGw06SNHZUzmfRF+bgdzjBGgVao3g8gG8pzmQgd6IO3sUEaOpkGHN4f
 KwcgmQ==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Ihor Radchenko <yantar92@HIDDEN> writes:

Hi Ihor,

> Why so? `directory-files-recursively' is already supported by Tramp via
> `directory-files'.

It isn't supported by Tramp yet. Tramp hasn't heard ever about.

> But at least for some backends `directory-files-recursively' may be
> implemented more efficiently. If other backends do not implement it,
> `directory-files' will be used.

Of course, and I did propose to add it. I just wanted to avoid an
inflation of proposals for primitive operations being supported by Tramp.

And yes, not all backends need to implement an own version of
`directory-files-recursively'. But this could happen for other primitive
operations, so we must always think about whether it is worth to add a
primitive to Tramp.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 11:21:02 +0000
Resent-Message-ID: <handler.64735.B64735.169019762819533 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169019762819533
          (code B ref 64735); Mon, 24 Jul 2023 11:21:02 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 11:20:28 +0000
Received: from localhost ([127.0.0.1]:41902 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNtc4-00054y-AL
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 07:20:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:51848)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNtc0-00054j-MH
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 07:20:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNtbu-0000Ew-Go; Mon, 24 Jul 2023 07:20:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=FR7LxRVIVf1c4Yu40K/82nWcs2J1WHFdSaBPl4M5EMo=; b=Q++I7mtygtr1
 R94QZt2UvEfTmogOVS0Wgk+rxFlo8N/OL18dTXR/pZJ5lfUu4vydgspd/+8ma471i3d2bLm5VpRAB
 At/fz6DW0LzHsDnKNK1qxgVM8PzbrRumQSpEvhhV0nEi9Z6j7WhCY94aPJLG6/yQWRkhj4ryUWrrn
 1P2Z8XNUeVjnrTmsSGxdoTOjMCAE7A73A+3933ociQ+04t2PGslHEY6je0JIA7ywmffRwLR3q9M2+
 2RHk9ExwuvMPibxegXjsLlfpj6Np2F/QulkfTpJ1/2purjRwwdRkIjoPo6Vr24KxkoVyTYW2xGlzh
 OdhfcXmyph/9dHRHvF4CWQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNtbj-000096-LU; Mon, 24 Jul 2023 07:20:18 -0400
Date: Mon, 24 Jul 2023 14:20:50 +0300
Message-Id: <834jlttv1p.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> (message from
 Dmitry Gutov on Sun, 23 Jul 2023 22:27:26 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 23 Jul 2023 22:27:26 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 23/07/2023 20:56, Eli Zaretskii wrote:
> >> And, ideally, do all the relevant benchmarking when proposing the change.
> > Of course.  Although the benchmarks until now already show quite a
> > variability.
> 
> Speaking of your MS Windows results that are unflattering to 'find', it 
> might be worth it to do a more varied comparison, to determine the 
> OS-specific bottleneck.
> 
> Off the top of my head, here are some possibilities:
> 
> 1. 'find' itself is much slower there. There is room for improvement in 
> the port.

I think it's the filesystem, not the port (which I did myself in this
case).  But I'd welcome similar tests on other Windows systems with
other ports of Find.  Just remember to measure this particular
benchmark, not just Find itself from the shell, as the times are very
different (as I reported up-thread).

> 2. The process output handling is worse.

Not sure what that means.

> 3. Something particular to the project being used for the test.

I don't think I understand this one.

> To look into the possibility #1, you can try running the same command in 
> the terminal with the output to NUL and comparing the runtime to what's 
> reported in the benchmark.

Output to the null device is a bad idea, as (AFAIR) Find is clever
enough to detect that and do nothing.  I run "find | wc" instead, and
already reported that it is much faster.

> I actually remember, from my time on MS Windows about 10 years ago, that 
> some older ports of 'find' and/or 'grep' did have performance problems, 
> but IIRC ezwinports contained the improved versions.

The ezwinports is the version I'm using here.  But maybe someone came
up with a better one: after all, I did my port many years ago (because
the native ports available back then were abysmally slow).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 12:56:01 +0000
Resent-Message-ID: <handler.64735.B64735.169020332526382 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169020332526382
          (code B ref 64735); Mon, 24 Jul 2023 12:56:01 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 12:55:25 +0000
Received: from localhost ([127.0.0.1]:42086 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNv5x-0006rS-CP
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 08:55:25 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:44141)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qNv5u-0006rA-8h
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 08:55:24 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.nyi.internal (Postfix) with ESMTP id 1C2775C0182;
 Mon, 24 Jul 2023 08:55:17 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Mon, 24 Jul 2023 08:55:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690203317; x=1690289717; bh=+UK8rdOiJ+qouJOeYUXXhbJJVrZFhFeylYi
 g/OjKxVU=; b=JVx1O7Uq+n65oMZb6YgbjYlePyHK6/BW08zYocmm6tcN1rOndAR
 xQy+t70v80vtwIO2eCk0w6g4nGVpCommCcGjW9fz1qtAGZu7WllKSutRRE5DpJ0r
 LW1GoHJhv8vJ7TEztPmrlktNDAeIskXQba3zeF7k3T/bW8TBVPOdIxCrGS9UI1VN
 /8yLurUX4RRL09DzLYOolvMoz3oKZlXjESqW7YL3/xB9vQ8psEdE5ap2W2Ho+VOb
 AwwbqpgWh25XyN7okRnTQDNG6VSqUF0XJKD0yK4NyrDsvEGZBNwv1MNtj+ANhVoV
 uqzKLl0qrKJytf3FDt1h+aglu8cFezlaXZA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690203317; x=1690289717; bh=+UK8rdOiJ+qouJOeYUXXhbJJVrZFhFeylYi
 g/OjKxVU=; b=nX+RgStoztPdk+MDbt2OiY3I66SJh0/jZ9zwKhCfL5IBgEiLgfU
 kWtta3PIPHycEqQmP0Hi5WNOf0h/ITLM3JxmG7sClRDrnL5iO1HBBVaj/36T3vP6
 ggf/R5DMv+wFdo94eJYac8jo/0hsUrfDY6Hh8u3jwA9y1hiC027Ek4uGR/G/MXvn
 lMqwR82/vwlW0GG2lIiu6FoHO7RcpMngfoBBQbOESW3WyYiNSYLTN/WnNsTPZuAg
 jzpuLyKE+2nmqzYCGKrv6hAjf8VVCgCOyAAF5hC7OEmFEMPkAutaRKmFWdQG9SJW
 MoSLYoxSfJb/uGNOzNHI+DM4XABulof7Lig==
X-ME-Sender: <xms:tHS-ZLvTYh1hkXMHy4N-0EEWhIzuiBnhnc04nbCavBFC6rtLaWkAyw>
 <xme:tHS-ZMdEqO-MLRRGOmHDwPEZOVxmWPf2Z2zxB8UHvwj0fI_3kel-Yz3bNPzA3365Z
 WBhfTjApFK192_YMXo>
X-ME-Received: <xmr:tHS-ZOxqRCmR7uyfuBZ0L5HVLIJ1Hu7OvXIL5JVSphE9yp-IrUAyve3kKudKbbY>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheekgdehiecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:tHS-ZKP6pIgYynlXjULXdtVCI0Kay16QasasA5DOACv6ND9hmxJquQ>
 <xmx:tHS-ZL_t9At6xRVT1LXX2u4DOFKwF945Cb0OoIfJdE8adWBvQqbDig>
 <xmx:tHS-ZKVw3XNRC7775_E2gXtBZRsYYbV79cdboy38nrV3EDPAW0J_Bg>
 <xmx:tXS-ZOZlAyFFj6wjbACp6EySokauYvk7GVTBhKih5T4p8zsZJsUVhw>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon,
 24 Jul 2023 08:55:15 -0400 (EDT)
Message-ID: <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
Date: Mon, 24 Jul 2023 15:55:13 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <834jlttv1p.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 24/07/2023 14:20, Eli Zaretskii wrote:
>> Date: Sun, 23 Jul 2023 22:27:26 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> On 23/07/2023 20:56, Eli Zaretskii wrote:
>>>> And, ideally, do all the relevant benchmarking when proposing the change.
>>> Of course.  Although the benchmarks until now already show quite a
>>> variability.
>>
>> Speaking of your MS Windows results that are unflattering to 'find', it
>> might be worth it to do a more varied comparison, to determine the
>> OS-specific bottleneck.
>>
>> Off the top of my head, here are some possibilities:
>>
>> 1. 'find' itself is much slower there. There is room for improvement in
>> the port.
> 
> I think it's the filesystem, not the port (which I did myself in this
> case).

But directory-files-recursively goes through the same filesystem, 
doesn't it?

> But I'd welcome similar tests on other Windows systems with
> other ports of Find.  Just remember to measure this particular
> benchmark, not just Find itself from the shell, as the times are very
> different (as I reported up-thread).

Concur.

>> 2. The process output handling is worse.
> 
> Not sure what that means.

Emacs's ability to process the output of a process on the particular 
platform.

You said:

   Btw, the Find command with pipe to some other program, like wc,
   finishes much faster, like 2 to 4 times faster than when it is run
   from find-directory-files-recursively.  That's probably the slowdown
   due to communications with async subprocesses in action.

One thing to try it changing the -with-find implementation to use a 
synchronous call, to compare (e.g. using 'process-file'). And repeat 
these tests on GNU/Linux too.

That would help us gauge the viability of using an asynchronous process 
to get the file listing. But also, if one was just looking into 
reimplementing directory-files-recursively using 'find' (to create an 
endpoint with swappable implementations, for example), 'process-file' is 
a suitable substitute because the original is also currently synchronous.

>> 3. Something particular to the project being used for the test.
> 
> I don't think I understand this one.

This described the possibility where the disparity between the 
implementations' runtimes was due to something unusual in the project 
structure, if you tested different projects between Windows and 
GNU/Linux, making direct comparison less useful. It's the least likely 
cause, but still sometimes a possibility.

>> To look into the possibility #1, you can try running the same command in
>> the terminal with the output to NUL and comparing the runtime to what's
>> reported in the benchmark.
> 
> Output to the null device is a bad idea, as (AFAIR) Find is clever
> enough to detect that and do nothing.  I run "find | wc" instead, and
> already reported that it is much faster.

Now I see it, thanks.

>> I actually remember, from my time on MS Windows about 10 years ago, that
>> some older ports of 'find' and/or 'grep' did have performance problems,
>> but IIRC ezwinports contained the improved versions.
> 
> The ezwinports is the version I'm using here.  But maybe someone came
> up with a better one: after all, I did my port many years ago (because
> the native ports available back then were abysmally slow).

We should also look at the exact numbers. If you say that "| wc" 
invocation is 2-4x faster than what's reported in the benchmark, then it 
takes about 2-4 seconds. Which is still oddly slower than your reported 
numbers for directory-files-recursively.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 24 Jul 2023 13:26:01 +0000
Resent-Message-ID: <handler.64735.B64735.169020515529458 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169020515529458
          (code B ref 64735); Mon, 24 Jul 2023 13:26:01 +0000
Received: (at 64735) by debbugs.gnu.org; 24 Jul 2023 13:25:55 +0000
Received: from localhost ([127.0.0.1]:42167 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qNvZS-0007f3-T2
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 09:25:55 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:36468)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qNvZQ-0007ep-Du
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 09:25:53 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNvZK-0007Rn-AS; Mon, 24 Jul 2023 09:25:46 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=H1lJ48cWW5Ttpe9JSy1LcYxPiI0BF08jN5Bcy/d9fHA=; b=fZikmWSoqQE9
 +WZLdKcjYBf+EgULAO+HnLS41hKKEcmOZEFcQcKbYHS7EEjuiMnLfW+Uy+wuLMUFXWwCW1Wnduvro
 m7aQfKuWESRdNpOU1Tc8ESj6EnMVFo6AqsUMpAJsscYBiou4+DPGlNa1gcRIrAsxPWOji63vR+cE9
 dkbyGPFvj4F35bLZtSDdXO/p0k1oKLq2tdopJLnUuYnMymcWwRY9m1wE9nFxapYkebsS01AR1eCbS
 qvfT4j2IWbWothjagQ8yoF6H8PzOJyaWhAuOuOX0KfRFzKKJJokH1K/pqUbZy288s+c6sDELb35Lb
 mel4fn/DsWjJ5XJKeAQhqw==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qNvZJ-0007dw-BU; Mon, 24 Jul 2023 09:25:45 -0400
Date: Mon, 24 Jul 2023 16:26:27 +0300
Message-Id: <83a5vlsanw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> (message from
 Dmitry Gutov on Mon, 24 Jul 2023 15:55:13 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Mon, 24 Jul 2023 15:55:13 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> >> 1. 'find' itself is much slower there. There is room for improvement in
> >> the port.
> > 
> > I think it's the filesystem, not the port (which I did myself in this
> > case).
> 
> But directory-files-recursively goes through the same filesystem, 
> doesn't it?

It does (more or less; see below).  But I was not trying to explain
why Find is slower than directory-files-recursively, I was trying to
explain why Find on Windows is slower than Find on GNU/Linux.

If you are asking why directory-files-recursively is so much faster on
Windows than Find, then the main factors I can think about are:

  . IPC, at least in how we implement it in Emacs on MS-Windows, via a
    separate thread and OS-level events between them to signal that
    stuff is available for reading, whereas
    directory-files-recursively avoids this overhead completely;
  . Find uses Posix APIs: 'stat', 'chdir', 'readdir' -- which on
    Windows are emulated by wrappers around native APIs.  Moreover,
    Find uses 'char *' for file names, so calling native APIs involves
    transparent conversion to UTF-16 and back, which is what native
    APIs accept and return.  By contrast, Emacs on Windows calls the
    native APIs directly, and converts to UTF-16 from UTF-8, which is
    faster.  (This last point also means that using Find on Windows
    has another grave disadvantage: it cannot fully support non-ASCII
    file names, only those that can be encoded by the current
    single-byte system codepage.)

> >> 2. The process output handling is worse.
> > 
> > Not sure what that means.
> 
> Emacs's ability to process the output of a process on the particular 
> platform.
> 
> You said:
> 
>    Btw, the Find command with pipe to some other program, like wc,
>    finishes much faster, like 2 to 4 times faster than when it is run
>    from find-directory-files-recursively.  That's probably the slowdown
>    due to communications with async subprocesses in action.

I see this slowdown on GNU/Linux as well.

> One thing to try it changing the -with-find implementation to use a 
> synchronous call, to compare (e.g. using 'process-file'). And repeat 
> these tests on GNU/Linux too.

This still uses pipes, albeit without the pselect stuff.

> >> 3. Something particular to the project being used for the test.
> > 
> > I don't think I understand this one.
> 
> This described the possibility where the disparity between the 
> implementations' runtimes was due to something unusual in the project 
> structure, if you tested different projects between Windows and 
> GNU/Linux, making direct comparison less useful. It's the least likely 
> cause, but still sometimes a possibility.

I have on my Windows system a d:/usr/share tree that is very similar
to (albeit somewhat smaller than) a typical /usr/share tree on Posix
systems.  I tried with that as well, and the results were similar.

> > The ezwinports is the version I'm using here.  But maybe someone came
> > up with a better one: after all, I did my port many years ago (because
> > the native ports available back then were abysmally slow).
> 
> We should also look at the exact numbers. If you say that "| wc" 
> invocation is 2-4x faster than what's reported in the benchmark, then it 
> takes about 2-4 seconds. Which is still oddly slower than your reported 
> numbers for directory-files-recursively.

Yes, so there are additional factors at work, at least with this port
of Find.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 25 Jul 2023 02:42:02 +0000
Resent-Message-ID: <handler.64735.B64735.16902528885773 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16902528885773
          (code B ref 64735); Tue, 25 Jul 2023 02:42:02 +0000
Received: (at 64735) by debbugs.gnu.org; 25 Jul 2023 02:41:28 +0000
Received: from localhost ([127.0.0.1]:44431 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qO7zL-0001V2-Pu
	for submit <at> debbugs.gnu.org; Mon, 24 Jul 2023 22:41:28 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:59715)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qO7zG-0001Um-OD
 for 64735 <at> debbugs.gnu.org; Mon, 24 Jul 2023 22:41:26 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id 88AB95C00C4;
 Mon, 24 Jul 2023 22:41:17 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Mon, 24 Jul 2023 22:41:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-type:content-type:date:date:from:from:in-reply-to
 :in-reply-to:message-id:mime-version:references:reply-to:sender
 :subject:subject:to:to; s=fm2; t=1690252877; x=1690339277; bh=Kd
 D5CaufEZfzbvLLgSfE9fmyv5EqLYURgdScJ9kdKTE=; b=nw49x9lOqHPJVGu/0W
 vn4Bg5spnMUpe2mZIYelT6T4PNb9nHIVKpNL11HwHeLl0kq2TLtXEWwexqo66lTZ
 KxSoJ2XpxSZRmh71Rlao+NKChcjw3ebbRfyEwGMjqSOSYChA6GJBCvpThnJn4I2x
 AQLKIt4f0BE8jXZ4gVK/Ct+n5GXUNDzpCe368+dZUYEvyeEj500pXtAv9vW1luKx
 5RehGpk71dM0zPNId7fGrdkxHEL8w77qdjXWIMlWWfl4tUctjmim3RGaQRkq9qmp
 LXCvRIcL8lK8pp6+uzp2F2uY9Q6HODPtOXSujrjCZMt8C5zaaxeMGpXwSvrkRj0I
 RWzA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:content-type:date:date
 :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:sender:subject
 :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm3; t=1690252877; x=1690339277; bh=KdD5CaufEZfzb
 vLLgSfE9fmyv5EqLYURgdScJ9kdKTE=; b=1YfQoGQ04+1oX/FM8HljVelrMPR5G
 /MJmum4B88dNdr7NJAa2WbbbBCYHOh6Ka5B7FX0rGUK3oOKAkhEm3ogF5RWcfi5M
 r/MPI9uva9xc57MqDow0m6tnKu5I7nFC+d9rm3YTSvMSpx4mv4w0GtF7vuZ7eUq4
 4brCsKd10JrVuSJJwOpbYnB1B6vr+DROVl+ViwFCTULWRxOb1zRgp/ilxBUdykoC
 mxCTNNVYWQQgO872AeG1as7D1f6suIt7aWZFS2rJ/G538PCFotXZ4J088ZJtIIHE
 bVmaDuqDqY6vJk3fiYJtZLSxditXZwm+mRRFo66AGUfaXtGgVF57VxzvQ==
X-ME-Sender: <xms:TDa_ZFDAOBtHlTE1Q-R4LImjLap_m0NFq4W1biYqRBQpL4TcQw2xVA>
 <xme:TDa_ZDjep4U2kFJ6rhiSfw23oxyhGW3nNjEp3TCwg5E_LFgMSg_XNT_GPievTQhXI
 tlpsSTcbnZvJ7FDYfg>
X-ME-Received: <xmr:TDa_ZAlGIB0bk6kSJl1sxGZDtc5k1y5qFNqwIl--HlszgfvHl4wTq_E3tUNLxqY>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrheelgdeitdcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpegtkfffgggfuffvvehfhfgjsehmtderredtfeejnecuhfhrohhmpeffmhhithhr
 hicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrthhtvg
 hrnhepheetkefgteeiueejuedvtddvleetffeljeetuddvfeffjeeiheehueetffevieej
 necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepughmih
 htrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:TDa_ZPxT2QhHl74mquHDzRC1wSNl01OchEkU55VQuBQZyVeE0N3SrA>
 <xmx:TDa_ZKTukDZxUuYO8oQb_hrABguy4Gl2gDQwPe2-5Mb-8_C-KTimjQ>
 <xmx:TDa_ZCaQMyrTbTamyVMdVtHd9aMDz6b6yhcbYb1eu8soWVAyGVLDBg>
 <xmx:TTa_ZIeUU8zww7l9ywF1ywGBP53DoEHDKHW6u6PLQg9sv71f4v0ilQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon,
 24 Jul 2023 22:41:15 -0400 (EDT)
Content-Type: multipart/mixed; boundary="------------jKC7NdUtouvK1j69QsODn7eD"
Message-ID: <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
Date: Tue, 25 Jul 2023 05:41:13 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83a5vlsanw.fsf@HIDDEN>
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

This is a multi-part message in MIME format.
--------------jKC7NdUtouvK1j69QsODn7eD
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 24/07/2023 16:26, Eli Zaretskii wrote:
>> Date: Mon, 24 Jul 2023 15:55:13 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>>> 1. 'find' itself is much slower there. There is room for improvement in
>>>> the port.
>>>
>>> I think it's the filesystem, not the port (which I did myself in this
>>> case).
>>
>> But directory-files-recursively goes through the same filesystem,
>> doesn't it?
> 
> It does (more or less; see below).  But I was not trying to explain
> why Find is slower than directory-files-recursively, I was trying to
> explain why Find on Windows is slower than Find on GNU/Linux.

Understood. But we probably don't need to worry about the differences 
between platforms as much as about choosing the best option for each 
platform (or not choosing the worst, at least). So I'm more interested 
about how the find-based solution is more than 4x slower than the 
built-in one on MS Windows.

> If you are asking why directory-files-recursively is so much faster on
> Windows than Find, then the main factors I can think about are:
> 
>    . IPC, at least in how we implement it in Emacs on MS-Windows, via a
>      separate thread and OS-level events between them to signal that
>      stuff is available for reading, whereas
>      directory-files-recursively avoids this overhead completely;
>    . Find uses Posix APIs: 'stat', 'chdir', 'readdir' -- which on
>      Windows are emulated by wrappers around native APIs.  Moreover,
>      Find uses 'char *' for file names, so calling native APIs involves
>      transparent conversion to UTF-16 and back, which is what native
>      APIs accept and return.  By contrast, Emacs on Windows calls the
>      native APIs directly, and converts to UTF-16 from UTF-8, which is
>      faster.  (This last point also means that using Find on Windows
>      has another grave disadvantage: it cannot fully support non-ASCII
>      file names, only those that can be encoded by the current
>      single-byte system codepage.)

I seem to remember that Wine, which also does a similar dance of 
translating library and system calls, is often very close to the native 
performance for many programs. So this could be a problem, but 
necessarily a significant one.

Although text encoding conversion seems like a prime suspect, if the 
problem is here.

>>>> 2. The process output handling is worse.
>>>
>>> Not sure what that means.
>>
>> Emacs's ability to process the output of a process on the particular
>> platform.
>>
>> You said:
>>
>>     Btw, the Find command with pipe to some other program, like wc,
>>     finishes much faster, like 2 to 4 times faster than when it is run
>>     from find-directory-files-recursively.  That's probably the slowdown
>>     due to communications with async subprocesses in action.
> 
> I see this slowdown on GNU/Linux as well.
> 
>> One thing to try it changing the -with-find implementation to use a
>> synchronous call, to compare (e.g. using 'process-file'). And repeat
>> these tests on GNU/Linux too.
> 
> This still uses pipes, albeit without the pselect stuff.

I'm attaching an extended benchmark, one that includes a "synchronous" 
implementation as well. Please give it a spin as well.

Here (GNU/Linux) the reported numbers look like this:

 > (my-bench 1 default-directory "")

(("built-in" . "Elapsed time: 1.601649s (0.709108s in 22 GCs)")
  ("with-find" . "Elapsed time: 1.792383s (1.135869s in 38 GCs)")
  ("with-find-p" . "Elapsed time: 1.248543s (0.682827s in 20 GCs)")
  ("with-find-sync" . "Elapsed time: 0.922291s (0.343497s in 10 GCs)"))
--------------jKC7NdUtouvK1j69QsODn7eD
Content-Type: text/x-emacs-lisp; charset=UTF-8; name="find-bench.el"
Content-Disposition: attachment; filename="find-bench.el"
Content-Transfer-Encoding: base64

KGRlZnVuIGZpbmQtZGlyZWN0b3J5LWZpbGVzLXJlY3Vyc2l2ZWx5IChkaXIgcmVnZXhwICZv
cHRpb25hbCBpbmNsdWRlLWRpcmVjdG9yaWVzIF9wIGZvbGxvdy1zeW1saW5rcykKICAoY2wt
YXNzZXJ0IChudWxsIF9wKSB0ICJmaW5kLWRpcmVjdG9yeS1maWxlcy1yZWN1cnNpdmVseSBj
YW4ndCBhY2NlcHQgYXJiaXRyYXJ5IHByZWRpY2F0ZXMiKQogICh3aXRoLXRlbXAtYnVmZmVy
CiAgICAoc2V0cSBjYXNlLWZvbGQtc2VhcmNoIG5pbCkKICAgIChjZCBkaXIpCiAgICAobGV0
KiAoKGNvbW1hbmQKCSAgICAoYXBwZW5kCgkgICAgIChsaXN0ICJmaW5kIiAoZmlsZS1sb2Nh
bC1uYW1lIGRpcikpCgkgICAgIChpZiBmb2xsb3ctc3ltbGlua3MKCQkgJygiLUwiKQoJICAg
ICAgICcoIiEiICIoIiAiLXR5cGUiICJsIiAiLXh0eXBlIiAiZCIgIikiKSkKCSAgICAgKHVu
bGVzcyAoc3RyaW5nLWVtcHR5LXAgcmVnZXhwKQoJICAgICAgIChsaXN0ICItcmVnZXgiIChj
b25jYXQgIi4qIiByZWdleHAgIi4qIikpKQoJICAgICAodW5sZXNzIGluY2x1ZGUtZGlyZWN0
b3JpZXMKCSAgICAgICAnKCIhIiAiLXR5cGUiICJkIikpCgkgICAgICcoIi1wcmludDAiKQoJ
ICAgICApKQoJICAgKHJlbW90ZSAoZmlsZS1yZW1vdGUtcCBkaXIpKQoJICAgKHByb2MKCSAg
ICAoaWYgcmVtb3RlCgkJKGxldCAoKHByb2MgKGFwcGx5ICMnc3RhcnQtZmlsZS1wcm9jZXNz
CgkJCQkgICAiZmluZCIgKGN1cnJlbnQtYnVmZmVyKSBjb21tYW5kKSkpCgkJICAoc2V0LXBy
b2Nlc3Mtc2VudGluZWwgcHJvYyAobGFtYmRhIChfcHJvYyBfc3RhdGUpKSkKCQkgIChzZXQt
cHJvY2Vzcy1xdWVyeS1vbi1leGl0LWZsYWcgcHJvYyBuaWwpCgkJICBwcm9jKQoJICAgICAg
KG1ha2UtcHJvY2VzcyA6bmFtZSAiZmluZCIgOmJ1ZmZlciAoY3VycmVudC1idWZmZXIpCgkJ
CSAgICA6Y29ubmVjdGlvbi10eXBlICdwaXBlCgkJCSAgICA6bm9xdWVyeSB0CgkJCSAgICA6
c2VudGluZWwgKGxhbWJkYSAoX3Byb2MgX3N0YXRlKSkKCQkJICAgIDpjb21tYW5kIGNvbW1h
bmQpKSkpCiAgICAgICh3aGlsZSAoYWNjZXB0LXByb2Nlc3Mtb3V0cHV0IHByb2MpKQogICAg
ICAobGV0ICgoc3RhcnQgKGdvdG8tY2hhciAocG9pbnQtbWluKSkpIHJldCkKCSh3aGlsZSAo
c2VhcmNoLWZvcndhcmQgIlwwIiBuaWwgdCkKCSAgKHB1c2ggKGNvbmNhdCByZW1vdGUgKGJ1
ZmZlci1zdWJzdHJpbmctbm8tcHJvcGVydGllcyBzdGFydCAoMS0gKHBvaW50KSkpKSByZXQp
CgkgIChzZXRxIHN0YXJ0IChwb2ludCkpKQoJcmV0KSkpKQoKKGRlZnVuIGZpbmQtZGlyZWN0
b3J5LWZpbGVzLXJlY3Vyc2l2ZWx5LTIgKGRpciByZWdleHAgJm9wdGlvbmFsIGluY2x1ZGUt
ZGlyZWN0b3JpZXMgX3AgZm9sbG93LXN5bWxpbmtzKQogIChjbC1hc3NlcnQgKG51bGwgX3Ap
IHQgImZpbmQtZGlyZWN0b3J5LWZpbGVzLXJlY3Vyc2l2ZWx5IGNhbid0IGFjY2VwdCBhcmJp
dHJhcnkgcHJlZGljYXRlcyIpCiAgKGNsLWFzc2VydCAobm90IChmaWxlLXJlbW90ZS1wIGRp
cikpKQogIChsZXQqIChidWZmZXJlZAogICAgICAgICByZXN1bHQKICAgICAgICAgKHByb2MK
CSAgKG1ha2UtcHJvY2VzcwogICAgICAgICAgIDpuYW1lICJmaW5kIiA6YnVmZmVyIG5pbAoJ
ICAgOmNvbm5lY3Rpb24tdHlwZSAncGlwZQoJICAgOm5vcXVlcnkgdAoJICAgOnNlbnRpbmVs
IChsYW1iZGEgKF9wcm9jIF9zdGF0ZSkpCiAgICAgICAgICAgOmZpbHRlciAobGFtYmRhIChw
cm9jIGRhdGEpCiAgICAgICAgICAgICAgICAgICAgIChsZXQgKChzdGFydCAwKSkKICAgICAg
ICAgICAgICAgICAgICAgICAod2hlbi1sZXQgKGVuZCAoc3RyaW5nLXNlYXJjaCAiXDAiIGRh
dGEgc3RhcnQpKQogICAgICAgICAgICAgICAgICAgICAgICAgKHB1c2ggKGNvbmNhdCBidWZm
ZXJlZCAoc3Vic3RyaW5nIGRhdGEgc3RhcnQgZW5kKSkgcmVzdWx0KQogICAgICAgICAgICAg
ICAgICAgICAgICAgKHNldHEgYnVmZmVyZWQgIiIpCiAgICAgICAgICAgICAgICAgICAgICAg
ICAoc2V0cSBzdGFydCAoMSsgZW5kKSkKICAgICAgICAgICAgICAgICAgICAgICAgICh3aGls
ZS1sZXQgKChlbmQgKHN0cmluZy1zZWFyY2ggIlwwIiBkYXRhIHN0YXJ0KSkpCiAgICAgICAg
ICAgICAgICAgICAgICAgICAgIChwdXNoIChzdWJzdHJpbmcgZGF0YSBzdGFydCBlbmQpIHJl
c3VsdCkKICAgICAgICAgICAgICAgICAgICAgICAgICAgKHNldHEgc3RhcnQgKDErIGVuZCkp
KSkKICAgICAgICAgICAgICAgICAgICAgICAoc2V0cSBidWZmZXJlZCAoY29uY2F0IGJ1ZmZl
cmVkIChzdWJzdHJpbmcgZGF0YSBzdGFydCkpKSkpCgkgICA6Y29tbWFuZCAoYXBwZW5kCgkg
ICAgICAgICAgICAgKGxpc3QgImZpbmQiIChmaWxlLWxvY2FsLW5hbWUgZGlyKSkKCSAgICAg
ICAgICAgICAoaWYgZm9sbG93LXN5bWxpbmtzCgkJICAgICAgICAgJygiLUwiKQoJICAgICAg
ICAgICAgICAgJygiISIgIigiICItdHlwZSIgImwiICIteHR5cGUiICJkIiAiKSIpKQoJICAg
ICAgICAgICAgICh1bmxlc3MgKHN0cmluZy1lbXB0eS1wIHJlZ2V4cCkKCSAgICAgICAgICAg
ICAgIChsaXN0ICItcmVnZXgiIChjb25jYXQgIi4qIiByZWdleHAgIi4qIikpKQoJICAgICAg
ICAgICAgICh1bmxlc3MgaW5jbHVkZS1kaXJlY3RvcmllcwoJICAgICAgICAgICAgICAgJygi
ISIgIi10eXBlIiAiZCIpKQoJICAgICAgICAgICAgICcoIi1wcmludDAiKQoJICAgICAgICAg
ICAgICkpKSkKICAgICh3aGlsZSAoYWNjZXB0LXByb2Nlc3Mtb3V0cHV0IHByb2MpKQogICAg
cmVzdWx0KSkKCihkZWZ1biBmaW5kLWRpcmVjdG9yeS1maWxlcy1yZWN1cnNpdmVseS0zIChk
aXIgcmVnZXhwICZvcHRpb25hbCBpbmNsdWRlLWRpcmVjdG9yaWVzIF9wIGZvbGxvdy1zeW1s
aW5rcykKICAoY2wtYXNzZXJ0IChudWxsIF9wKSB0ICJmaW5kLWRpcmVjdG9yeS1maWxlcy1y
ZWN1cnNpdmVseSBjYW4ndCBhY2NlcHQgYXJiaXRyYXJ5IHByZWRpY2F0ZXMiKQogIChjbC1h
c3NlcnQgKG5vdCAoZmlsZS1yZW1vdGUtcCBkaXIpKSkKICAobGV0ICgoYXJncyBgKCwoZmls
ZS1sb2NhbC1uYW1lIGRpcikKCSAgICAgICAgLEAoaWYgZm9sbG93LXN5bWxpbmtzCgkJICAg
ICAgJygiLUwiKQoJICAgICAgICAgICAgJygiISIgIigiICItdHlwZSIgImwiICIteHR5cGUi
ICJkIiAiKSIpKQoJICAgICAgICAsQCh1bmxlc3MgKHN0cmluZy1lbXB0eS1wIHJlZ2V4cCkK
CSAgICAgICAgICAgIChsaXN0ICItcmVnZXgiIChjb25jYXQgIi4qIiByZWdleHAgIi4qIikp
KQoJICAgICAgICAsQCh1bmxlc3MgaW5jbHVkZS1kaXJlY3RvcmllcwoJICAgICAgICAgICAg
JygiISIgIi10eXBlIiAiZCIpKQoJICAgICAgICAiLXByaW50MCIpKSkKICAgICh3aXRoLXRl
bXAtYnVmZmVyCiAgICAgIChsZXQgKChzdGF0dXMgKGFwcGx5ICMncHJvY2Vzcy1maWxlCiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICJmaW5kIgogICAgICAgICAgICAgICAgICAgICAg
ICAgICBuaWwKICAgICAgICAgICAgICAgICAgICAgICAgICAgdAogICAgICAgICAgICAgICAg
ICAgICAgICAgICBuaWwKICAgICAgICAgICAgICAgICAgICAgICAgICAgYXJncykpCiAgICAg
ICAgICAgIChwdCAocG9pbnQtbWluKSkKICAgICAgICAgICAgcmVzKQogICAgICAgICh1bmxl
c3MgKHplcm9wIHN0YXR1cykKICAgICAgICAgIChlcnJvciAiTGlzdGluZyBmYWlsZWQiKSkK
ICAgICAgICAoZ290by1jaGFyIChwb2ludC1taW4pKQogICAgICAgICh3aGlsZSAoc2VhcmNo
LWZvcndhcmQgIlwwIiBuaWwgdCkKICAgICAgICAgIChwdXNoIChidWZmZXItc3Vic3RyaW5n
LW5vLXByb3BlcnRpZXMgcHQgKDEtIChwb2ludCkpKQogICAgICAgICAgICAgICAgcmVzKQog
ICAgICAgICAgKHNldHEgcHQgKHBvaW50KSkpCiAgICAgICAgcmVzKSkpKQoKKGRlZnVuIG15
LWJlbmNoIChjb3VudCBwYXRoIHJlZ2V4cCkKICAoc2V0cSBwYXRoIChleHBhbmQtZmlsZS1u
YW1lIHBhdGgpKQogIDs7IChsZXQgKChvbGQgKGRpcmVjdG9yeS1maWxlcy1yZWN1cnNpdmVs
eSBwYXRoIHJlZ2V4cCkpCiAgOzsgICAgICAgKG5ldyAoZmluZC1kaXJlY3RvcnktZmlsZXMt
cmVjdXJzaXZlbHktMyBwYXRoIHJlZ2V4cCkpKQogIDs7ICAgKGRvbGlzdCAocGF0aCBvbGQp
CiAgOzsgICAgICh1bmxlc3MgKG1lbWJlciBwYXRoIG5ldykgKGVycm9yICIhICVzIG5vdCBp
biIgcGF0aCkpKQogIDs7ICAgKGRvbGlzdCAocGF0aCBuZXcpCiAgOzsgICAgICh1bmxlc3Mg
KG1lbWJlciBwYXRoIG9sZCkgKGVycm9yICIhISAlcyBub3QgaW4iIHBhdGgpKSkpCiAgKGxp
c3QKICAgKGNvbnMgImJ1aWx0LWluIiAoYmVuY2htYXJrIGNvdW50IChsaXN0ICdkaXJlY3Rv
cnktZmlsZXMtcmVjdXJzaXZlbHkgcGF0aCByZWdleHApKSkKICAgKGNvbnMgIndpdGgtZmlu
ZCIgKGJlbmNobWFyayBjb3VudCAobGlzdCAnZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJz
aXZlbHkgcGF0aCByZWdleHApKSkKICAgKGNvbnMgIndpdGgtZmluZC1wIiAoYmVuY2htYXJr
IGNvdW50IChsaXN0ICdmaW5kLWRpcmVjdG9yeS1maWxlcy1yZWN1cnNpdmVseS0yIHBhdGgg
cmVnZXhwKSkpCiAgIChjb25zICJ3aXRoLWZpbmQtc3luYyIgKGJlbmNobWFyayBjb3VudCAo
bGlzdCAnZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMyBwYXRoIHJlZ2V4cCkp
KSkpCg==

--------------jKC7NdUtouvK1j69QsODn7eD--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 25 Jul 2023 08:22:01 +0000
Resent-Message-ID: <handler.64735.B64735.169027331810633 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169027331810633
          (code B ref 64735); Tue, 25 Jul 2023 08:22:01 +0000
Received: (at 64735) by debbugs.gnu.org; 25 Jul 2023 08:21:58 +0000
Received: from localhost ([127.0.0.1]:44610 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qODIs-0002lR-1c
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 04:21:58 -0400
Received: from mout01.posteo.de ([185.67.36.65]:41269)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qODIq-0002lE-FL
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 04:21:57 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id CE7F424002A
 for <64735 <at> debbugs.gnu.org>; Tue, 25 Jul 2023 10:21:50 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690273310; bh=tKB26pFHgJY4X6yxRUui3D+FBB7mc/QFxj66gJ1hNVY=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=bTGwfG85sho6USLiIg03rNXUVIZyanqU4V5H2/f4em4Kymv6otlHehLKLPb2uKgU+
 g/XqYKQvPZwuhNN2Ix9LhZ78b2REuI7YQOS0Q8KQdwxK8dWQ7fYHrwLd5qJRIrJ/CU
 9eh7fJQrQbhWjFHZEMtQiGmRIxZmdxB+HCuovlH5GQOVu6Lur0lmBXfeMRfDq1XMTs
 7KGXm9asq+cQWDGUykTOM2qDMwYX8Td+F9o8BYra7HGVB1A+M4u6ntLN2BxtHyLuhI
 pnaTvV1B7A3u+5dm58VYR5twBxZqSHSqTVUN9yDnGPsrYHy8zouWUDkO7/TM4i9hfr
 Ft7ez6u3T8F5w==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R993x74d2z9s2b;
 Tue, 25 Jul 2023 10:21:49 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
Date: Tue, 25 Jul 2023 08:22:03 +0000
Message-ID: <87351cs8no.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Dmitry Gutov <dmitry@HIDDEN> writes:

> I'm attaching an extended benchmark, one that includes a "synchronous" 
> implementation as well. Please give it a spin as well.

GNU/Linux SSD

(my-bench 10 "/usr/src/linux/" "")

(("built-in" . "Elapsed time: 7.034326s (3.598539s in 14 GCs)")
 ("built-in no filename handler alist" . "Elapsed time: 5.907194s (3.698456s in 15 GCs)")
 ("with-find" . "Elapsed time: 6.078056s (4.052791s in 16 GCs)")
 ("with-find-p" . "Elapsed time: 4.496762s (2.739565s in 11 GCs)")
 ("with-find-sync" . "Elapsed time: 3.702760s (1.715160s in 7 GCs)"))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 25 Jul 2023 18:42:01 +0000
Resent-Message-ID: <handler.64735.B64735.16903104912728 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16903104912728
          (code B ref 64735); Tue, 25 Jul 2023 18:42:01 +0000
Received: (at 64735) by debbugs.gnu.org; 25 Jul 2023 18:41:31 +0000
Received: from localhost ([127.0.0.1]:46953 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOMyR-0000hw-6i
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 14:41:31 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:41030)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qOMyP-0000hk-A8
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 14:41:30 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOMyJ-00012k-BS; Tue, 25 Jul 2023 14:41:23 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=zOgpaUojH+o3WAWSjrNnxm2jFWE0iIVrGL5NIq2ALGA=; b=ETeIdlPEpLQ8
 ZzcrVeUyjj1YVdtoRitBso4m4JYd6uWcRok218IyM9wQrYZlAWlteEWR6/qmj3dKpGThbNONzvR8o
 ERxElQ0uox+ZXprA91TIuRLuS+ug35up9rxZDy4VYqpgqhjVIywLgYJupoHw4JTTV/WVtFIS+JRE2
 by+rEIbjZNUduT9YUed5VpbGTaZ6SeboftTVVePvYFJpjYWoVYE4PezSpe9oNbbw1J1PaSZuL7nJ/
 QbIXD6FQ6kkSOoSJgxCsLl+Ht+6yZzHh9JE62nX3VmcjvbOSdf5iUPNn0hk/QGnIxC32BabQNdkmK
 q1ogdLtCP45WpyFoo04IIA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOMyI-0004rK-Fn; Tue, 25 Jul 2023 14:41:22 -0400
Date: Tue, 25 Jul 2023 21:42:07 +0300
Message-Id: <83351bq1ds.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> (message from
 Dmitry Gutov on Tue, 25 Jul 2023 05:41:13 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 25 Jul 2023 05:41:13 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> >> One thing to try it changing the -with-find implementation to use a
> >> synchronous call, to compare (e.g. using 'process-file'). And repeat
> >> these tests on GNU/Linux too.
> > 
> > This still uses pipes, albeit without the pselect stuff.
> 
> I'm attaching an extended benchmark, one that includes a "synchronous" 
> implementation as well. Please give it a spin as well.
> 
> Here (GNU/Linux) the reported numbers look like this:
> 
>  > (my-bench 1 default-directory "")
> 
> (("built-in" . "Elapsed time: 1.601649s (0.709108s in 22 GCs)")
>   ("with-find" . "Elapsed time: 1.792383s (1.135869s in 38 GCs)")
>   ("with-find-p" . "Elapsed time: 1.248543s (0.682827s in 20 GCs)")
>   ("with-find-sync" . "Elapsed time: 0.922291s (0.343497s in 10 GCs)"))

Almost no change on Windows:

  (("built-in" . "Elapsed time: 1.218750s (0.078125s in 5 GCs)")
   ("with-find" . "Elapsed time: 8.984375s (0.109375s in 7 GCs)")
   ("with-find-p" . "Elapsed time: 8.718750s (0.046875s in 3 GCs)")
   ("with-find-sync" . "Elapsed time: 8.921875s (0.046875s in 3 GCs)"))

I'm beginning to suspect the implementation of pipes (and IPC in
general).  How else can such slowdown be explained?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: sbaugh@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 25 Jul 2023 19:17:02 +0000
Resent-Message-ID: <handler.64735.B64735.16903125886182 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16903125886182
          (code B ref 64735); Tue, 25 Jul 2023 19:17:02 +0000
Received: (at 64735) by debbugs.gnu.org; 25 Jul 2023 19:16:28 +0000
Received: from localhost ([127.0.0.1]:46993 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qONWF-0001be-L8
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 15:16:27 -0400
Received: from s.wrqvtbkv.outbound-mail.sendgrid.net ([149.72.123.24]:45740)
 by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from
 <bounces+21787432-b76c-64735=debbugs.gnu.org@HIDDEN>)
 id 1qONWD-0001bQ-PI
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 15:16:26 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=catern.com;
 h=from:subject:in-reply-to:references:mime-version:to:cc:content-type:
 content-transfer-encoding:cc:content-type:from:subject:to;
 s=s1; bh=IBR4/bLNhmMuVWLT8kELUMGk3qDwcpDbnzCKfKDjpnc=;
 b=Oi6sgk3f8o5xl4ny2SNda8LXfuTPOYqHkAdJ9CfoZwFjPEWeQpMsrGfvrkRWUsiIfeFq
 30wE1kM5nxJ9vFT1fFlx4SQvvNml2kD0VLraQjVq7yD1XqXD1Oe4kko6Nq2kchk15vT3Qj
 MLxtZ1SJtd9NZyA8MgBEmkEbZzj1WaURQ1rPnjQ+vr7pmBYXRyHEAPsZMA3bm5g/bBFnqZ
 W7an40y63uE6dkwoBTdV2TgseTg+4ZrODFB1MT4uS+r/QNQKV/O+0Uuxl/uBonJWOq0dwp
 qzZwkY5HpT+dK/hHSgUBa5E9kcxITJTWYgGkE5PyLbtpRFh3W5rVlpk1YLNSDbSA==
Received: by filterdrecv-d7bbbc8bf-ftxgm with SMTP id
 filterdrecv-d7bbbc8bf-ftxgm-1-64C01F84-38
 2023-07-25 19:16:20.292981975 +0000 UTC m=+6550593.487377294
Received: from earth.catern.com (unknown) by geopod-ismtpd-canary-0 (SG)
 with ESMTP id VmMM01OkQD2enD_ee9hN2g
 Tue, 25 Jul 2023 19:16:20.261 +0000 (UTC)
X-Comment: SPF check N/A for local connections - client-ip=::1; helo=localhost;
 envelope-from=sbaugh@HIDDEN; receiver=gutov.dev 
Received: from localhost (localhost [IPv6:::1])
 by earth.catern.com (Postfix) with ESMTPSA id B92F860171;
 Tue, 25 Jul 2023 15:16:19 -0400 (EDT)
From: sbaugh@HIDDEN
In-Reply-To: <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> (Dmitry Gutov's
 message of "Tue, 25 Jul 2023 05:41:13 +0300")
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
Date: Tue, 25 Jul 2023 19:16:20 +0000 (UTC)
Message-ID: <877cqniyyk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
X-SG-EID: ZgbRq7gjGrt0q/Pjvxk7wM0yQFRdOkTJAtEbkjCkHbJBtyc/2Had/AF8pYFPW7yam57WgVZDCYyakw+TpVDVX8szhc1PBagwIzxv6ScccyHq9TPxXj4qzq/dfKFVQXqNYGms7xDZf0gQ12Bhwv/YP3SgUFrLrpRJkMTJzm9YGIn/tKEbM2XIbAik9MWN5NfyDVL9haS8cz80tYxWa110gA==
X-Entity-ID: d/0VcHixlS0t7iB1YKCv4Q==
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Score: 1.2 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview: Dmitry Gutov <dmitry@HIDDEN> writes: >> (my-bench 1
 default-directory
 "") > > (("built-in" . "Elapsed time: 1.601649s (0.709108s in 22 GCs)") >
 ("with-find" . "Elapsed time: 1.792383s (1.135869s in [...] 
 Content analysis details:   (1.2 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in
 bl.spamcop.net
 [Blocked - see <https://www.spamcop.net/bl.shtml?149.72.123.24>]
 -0.0 RCVD_IN_MSPIKE_H2      RBL: Average reputation (+2)
 [149.72.123.24 listed in wl.mailspike.net]
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 0.0 UNPARSEABLE_RELAY      Informational: message has unparseable relay
 lines -0.0 T_SCC_BODY_TEXT_LINE   No description available.
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.2 (/)

Dmitry Gutov <dmitry@HIDDEN> writes:
>> (my-bench 1 default-directory "")
>
> (("built-in" . "Elapsed time: 1.601649s (0.709108s in 22 GCs)")
>  ("with-find" . "Elapsed time: 1.792383s (1.135869s in 38 GCs)")
>  ("with-find-p" . "Elapsed time: 1.248543s (0.682827s in 20 GCs)")
>  ("with-find-sync" . "Elapsed time: 0.922291s (0.343497s in 10 GCs)"))

Tangent, but:

Ugh, wow, call-process really is a lot faster than make-process.  I see
now why people disliked my idea of replacing call-process with something
based on make-process, this is a big difference...

There's zero reason it has to be so slow... maybe I should try to make a
better make-process API and implementation which is actually fast.
(without worrying about being constrained by compatibility with
something that's already dog-slow)

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 01:52:02 +0000
Resent-Message-ID: <handler.64735.B64735.169033628715478 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169033628715478
          (code B ref 64735); Wed, 26 Jul 2023 01:52:02 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 01:51:27 +0000
Received: from localhost ([127.0.0.1]:47245 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOTgV-00041a-3H
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 21:51:27 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:57915)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qOTgT-00041K-77
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 21:51:25 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id DA7CE32007E8;
 Tue, 25 Jul 2023 21:51:18 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute5.internal (MEProxy); Tue, 25 Jul 2023 21:51:19 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690336278; x=1690422678; bh=LjT85s9eYxb/9vf3gzkWEl6ej7yi2/gbGhI
 PdMab67Y=; b=mmErWlp8QxXlYm2PPav8POsTd55cqx4sf/29z2EsLLwiJquNESY
 eFBiOLHnMANubeUdoG1jruqPRM5t1dLEV3Vdj5ThAs9S3/HRrn5TvybAC3Uvl2os
 +aVhdkRvbCAecsRN3U3ZNhWygmEVUYEH/XTjuSx910fSkqLqyJ1siUXfNqirexp4
 +uOCFeyTZroMjSS3J1aOcqB1Yn17pFeJDHqMBryPL+5Ag6t2GsvocRnnHkmqmdNC
 xhduNIjFDXzddpaf/3K/0VZ2CBtsQCwb9Ffm2DRqVHIz1IEicF1nFfcC+2mnhvak
 7SlzcuygQWdgWVgB4Ra/h7CfYjpxIHlKXfA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690336278; x=1690422678; bh=LjT85s9eYxb/9vf3gzkWEl6ej7yi2/gbGhI
 PdMab67Y=; b=VztAE3mPMlWLZajzNCDgoiqvYd4J51omnP+SJWuqg+/5OLOhqPb
 UctflD3tSz9fEjiKerTSfOZi55kF2IwItma/uWrdXAAWmHRsv/Usc/3fTuC7Uw8a
 dwrE/BcbC2MGpk1LQEYXF2K6kNBGNKkcq5W8d913BbnmdELm+WpSu66QBgDAUDYi
 lo9pUfdkXkHk8vNljRICG31YBlmcriXyVbTCYFEARTDvn+9J6vmetLkn19RG85Ax
 86Qx74NgDeXKrnlz19STJIuA+sofN+hOBVbmm0085lsEJnTLDevEqVMp91TssEEj
 8aPKMGqEKxV2fF82ppyojaVEjQ8/ZqRtvSA==
X-ME-Sender: <xms:FnzAZAJXqDST6BpQn9ytSdcFfYqq73NreuZQIxZ2qihJVRtDvd8xBg>
 <xme:FnzAZAL0puMViViAbIBhvOeGdw2Gg7Z8wZBDMV--gfk4CF8it0Gwk_daGSA590bar
 5L85kATNropiMFYpAM>
X-ME-Received: <xmr:FnzAZAsJiJL86dUe1i_V_wn9XNSjfTZjJWym9o68pcb9XpCzMgcKhqTjZmFQoD0>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedriedugdeglecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:FnzAZNYnE8HjswovkcoVX5rB9uYjYnHL9vvIX0uqcY5wUcJUo3faeA>
 <xmx:FnzAZHb-ffTBMLvAB-8D6E7qS2sG_4vtRWN1olLgQJPkya204rM7EQ>
 <xmx:FnzAZJD48bXt8EHC8PIC70YJQ5enC0P9RcasHsNtiLQtwzAwofiD4w>
 <xmx:FnzAZDHgCgAitm64VpI0zTOEdVzlR5WxNqkLDymOMvR8gHVTcRQeXQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 25 Jul 2023 21:51:16 -0400 (EDT)
Message-ID: <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
Date: Wed, 26 Jul 2023 04:51:15 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <87351cs8no.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 25/07/2023 11:22, Ihor Radchenko wrote:
> Dmitry Gutov<dmitry@HIDDEN>  writes:
> 
>> I'm attaching an extended benchmark, one that includes a "synchronous"
>> implementation as well. Please give it a spin as well.
> GNU/Linux SSD
> 
> (my-bench 10 "/usr/src/linux/" "")
> 
> (("built-in" . "Elapsed time: 7.034326s (3.598539s in 14 GCs)")
>   ("built-in no filename handler alist" . "Elapsed time: 5.907194s (3.698456s in 15 GCs)")
>   ("with-find" . "Elapsed time: 6.078056s (4.052791s in 16 GCs)")
>   ("with-find-p" . "Elapsed time: 4.496762s (2.739565s in 11 GCs)")
>   ("with-find-sync" . "Elapsed time: 3.702760s (1.715160s in 7 GCs)"))

Thanks, for the extra data point in particular. Easy to see how it 
compares to the most efficient use of 'find', right (on GNU/Linix, at 
least)?

It's also something to note that, GC-wise, numbers 1 and 2 are not the 
worst: the time must be spent somewhere else.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 01:57:01 +0000
Resent-Message-ID: <handler.64735.B64735.169033659215936 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169033659215936
          (code B ref 64735); Wed, 26 Jul 2023 01:57:01 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 01:56:32 +0000
Received: from localhost ([127.0.0.1]:47250 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOTlP-00048x-Qs
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 21:56:32 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:38135)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qOTlO-00048l-2I
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 21:56:30 -0400
Received: from compute2.internal (compute2.nyi.internal [10.202.2.46])
 by mailout.west.internal (Postfix) with ESMTP id D2DA732008C0;
 Tue, 25 Jul 2023 21:56:23 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute2.internal (MEProxy); Tue, 25 Jul 2023 21:56:24 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690336583; x=1690422983; bh=Hlobxmo9o5+6CYqNyUY7Jmyztu8RkRGhtjr
 sFcEFgwk=; b=fzjTWdkQpzSzDghfy3iudzekjoVZ3TUwKMSS2suPCK/daaMWwB5
 7yBJW3O6z7VwvwSx+k3CiQHzmHJeEgIBtqmMSaWO3hvS+DVsTTUaGuC2eugLUh1h
 9S5qXpFyqJDj6KDgR+pswMF54LTk2eJnSp5kVy+Vu6fhr8cjwhyQLNLQn0uv33gC
 CtIMS/YimZG6hrM2Qx5JjGgyNkYsbsRdJPPfYdyD8E37rHuOCqNLpwGIj8pZu14J
 PWnAMQkwyJFzs9iVVZYNgyyV9Hac+PkLiebxbbI585QuP9bUqGMQ+S/JvIW/457r
 AsDTdQONvtu/VDJrOp766mmb9YRVRRcqE0g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690336583; x=1690422983; bh=Hlobxmo9o5+6CYqNyUY7Jmyztu8RkRGhtjr
 sFcEFgwk=; b=CTxSgj9swJ0AurQZ2caWn0O3EWnxcq3g9prj8P4CPIVsF/SvWfS
 hPipstfdZMI4jWzVvTtYqaRYcSQ9OfeaQvmCyez4hFNOcI9iIrxtBLXsncCA5107
 jiIcOEKnIobLZ6FBpj513PNCl+9oiYIM0kpCqKZTZTezwH1zp4HeSl+J8tkixgMx
 ztWEU7T5IIZRlQxSXd8C+3jg1csdRmSucCRq3q3rKg693+KcPRsZsqZw/S/3o3hK
 4fvNUiVHF+56KzWuBkmFmusLiemXB2NA9w4C9miS2gLCT0Fh/fvvp/XoZ0zGdxIV
 OgESjWFRZR0kBp/BCwmd3ZwyTi2BFFmjN6A==
X-ME-Sender: <xms:R33AZC1PrMbpUCtQKHTmJRaQBqXtOiilSTyrjgd5cUAQQzwla28VnQ>
 <xme:R33AZFFA8rHvbgH8mI-EpuJqL57IybvACoj5RNLcgm1iNYWYqsqcBGxDjYAD94kMT
 iaS4ucJJJzYtCb5R28>
X-ME-Received: <xmr:R33AZK6A-unrcbcPj3-KN-JLt6N1f8u1a-UVnp7SFOtkJT1_ig0_spZ2g5RxVW8>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedriedugdehudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:R33AZD2KkHh67FFWFuu0SMSxdek4OZbmaqQ3J8MxbKfVQo8SitH4TA>
 <xmx:R33AZFFuPcSx5oxZtd_Dtos-gBMUmmH2EYwlbkoCIo0GarsptcTMxw>
 <xmx:R33AZM9a1zAvyNCst_H-FZFKPVMmtQkYRLJIA5vcToVjpfunuLgEog>
 <xmx:R33AZFCrvKPPl8Q06xu8fV-sFu-sDmqr56NkDPXXJ_b0vV9Ho3-zWg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 25 Jul 2023 21:56:22 -0400 (EDT)
Message-ID: <491724c3-534b-a498-8e48-c8d94531875c@HIDDEN>
Date: Wed, 26 Jul 2023 04:56:20 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <83351bq1ds.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83351bq1ds.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 25/07/2023 21:42, Eli Zaretskii wrote:
> Almost no change on Windows:
> 
>    (("built-in" . "Elapsed time: 1.218750s (0.078125s in 5 GCs)")
>     ("with-find" . "Elapsed time: 8.984375s (0.109375s in 7 GCs)")
>     ("with-find-p" . "Elapsed time: 8.718750s (0.046875s in 3 GCs)")
>     ("with-find-sync" . "Elapsed time: 8.921875s (0.046875s in 3 GCs)"))
> 
> I'm beginning to suspect the implementation of pipes (and IPC in
> general).  How else can such slowdown be explained?

Seems so (I'm not well-versed in the lower level details, alas).

Your other idea (spending time in text conversion) also sounds 
plausible, but I don't know whether this much overhead can be explained 
by it. And don't we have to convert any process's output to our internal 
encoding anyway, on any platform?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 02:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169033845718920 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169033845718920
          (code B ref 64735); Wed, 26 Jul 2023 02:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 02:27:37 +0000
Received: from localhost ([127.0.0.1]:47260 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOUFU-0004v6-Lz
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:27:36 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42178)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qOUFS-0004ut-H1
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:27:35 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOUFM-0005Jf-Tl; Tue, 25 Jul 2023 22:27:28 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=B/5atKyJeUF44bRLmMU4vqV3Rl6keHwhGJZKHqekt94=; b=Aoq/mhwcFQCJ
 rHs20NRF0ch2ITQzerVB5kLwW5mPWFyZIlQrxWS7Mo4IrLPDTRdA3KSboOwCsr9dfb/Il9KAIW5Ey
 jFZlFV2Z6+wEG0JLs3oA2vhVk0xhMQQuyJyQu9+dgx6w6afb8pRsCkWJBhgJBvJQe1sVNV2B9+Phz
 EmgP696UkS5XIDZ7EMmjOF963ruSgVr86qPMWuX2agpbXmvk7lpQDXSIM0dl2sJITItPHlDqIOdIZ
 QPlH2CO61BYZYlVGlwShAGRqVaoqf7uI8AieOenWx5LT0MJ+pIqwDUpQYa+lSNmXLaG6gGjYeOCLU
 gk7g6VMlWVXIUVHDmsa/bA==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOUFM-0007Qm-Cd; Tue, 25 Jul 2023 22:27:28 -0400
Date: Wed, 26 Jul 2023 05:28:15 +0300
Message-Id: <83zg3jo18g.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <491724c3-534b-a498-8e48-c8d94531875c@HIDDEN> (message from
 Dmitry Gutov on Wed, 26 Jul 2023 04:56:20 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <83351bq1ds.fsf@HIDDEN>
 <491724c3-534b-a498-8e48-c8d94531875c@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 26 Jul 2023 04:56:20 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> Your other idea (spending time in text conversion) also sounds 
> plausible, but I don't know whether this much overhead can be explained 
> by it. And don't we have to convert any process's output to our internal 
> encoding anyway, on any platform?

We do, but you-all probably run your tests on a system where the
external encoding is UTF-8, right?  That is much faster.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 02:29:02 +0000
Resent-Message-ID: <handler.64735.B64735.169033851519020 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: sbaugh@HIDDEN
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169033851519020
          (code B ref 64735); Wed, 26 Jul 2023 02:29:02 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 02:28:35 +0000
Received: from localhost ([127.0.0.1]:47265 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOUGR-0004wi-2b
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:28:35 -0400
Received: from wnew4-smtp.messagingengine.com ([64.147.123.18]:34069)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qOUGP-0004wT-Ay
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:28:33 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailnew.west.internal (Postfix) with ESMTP id E7F932B0016E;
 Tue, 25 Jul 2023 22:28:26 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute4.internal (MEProxy); Tue, 25 Jul 2023 22:28:27 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690338506; x=1690345706; bh=74xdhRfrhO14sFCCEbwcAaJMdU3fv/eJbHD
 ie3m3Nng=; b=f0o+1Whq+8VKbm+kPc95nPEJCc4ZatfJ6ZyV0Pba678BFiBfk1D
 sxiclAYFCyn76whDaP37DSWTIvh6bJE8K3N//dX1IRLn+GPDVnmmQnhTJNhP4gsu
 eMYBY9MRJa9xJCx6nuYo6JbJIe875y9R+in9KG6iKA0kYifljeF4btijmZpoag40
 hHj1+MWbFwsgcXST7VD2ZUdi1DLN0RoANmBBgjC8ohRU/dlul+dkGVMORdH6Q102
 80GWWg+CPPo+quhb/eH2bzrIQ3w7S/sMcpwUw7/ylQQFtpk/wlGK6nFU0CN+oxOV
 lb/JtKnKnqnRY841l+MS3a9/O7daR/h6CCg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690338506; x=1690345706; bh=74xdhRfrhO14sFCCEbwcAaJMdU3fv/eJbHD
 ie3m3Nng=; b=ARXXeCKp7e1hGaLdcusSKL+4LEeNrTqqcOiKjvMV12tOe316Ns1
 pwJ3HJOIyrKh2xOnYfoY8WFtex0Lo94sN1nCkVk078JwA0+WftVY3fNf5rxwj20w
 8oJYSneoGQwQX6M85l+e1KUKTFUOpOlvy9BIb0gKIO4W2M18k+EB1EPO3lDC6DR9
 WqWiQROQsprpn8FwS17mcCHAA6BmDALJA/pYhwGkKZ2Q4/u4aQSDjZOGgz5NEVNC
 G4Hg9mduRh0LupbwAuRpbQbIufLhyeS+Fw0oipqFst7o2hdO5NVMsHwVBzk8vr28
 ytLyCYUfltfFFb/Pi5X28NT1EZSUN/Ai2hw==
X-ME-Sender: <xms:wITAZFsIbwY4PNutBkUQqwDCOFiSPS46_uIVnuTCtGF-4Rd7jwHHpA>
 <xme:wITAZOdsomScLaeKPkqZgxr09M_i6v1Z6eP9uFipWT1_Z-Vla3zo-H2XdKGm9bg0v
 m3DLYSJWDuhca--Ypg>
X-ME-Received: <xmr:wITAZIxAVHWpcMirWdkgGx5Y-aOneus_AjQzdylhPo3tm9s2KSx3BlmRu9NAJPo>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedriedugdehjecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:wITAZMOv0NSX_-GjlylDSfuBDFOAYz_6wJp7UY0eeNv12WOMx4ZVrg>
 <xmx:wITAZF_My3irSKZLTAytpdf8szf_7U98SWaVKNq9q_BTtRIDXUlJCA>
 <xmx:wITAZMX63dJqiF-mLgK80apOT85Rqmz0Xqw9jUou6ZvL-eswFDUO-A>
 <xmx:yoTAZKK1uoCJaJ7m1rkSHY8N7m1Bv2Yvw7LmV7W7FQTYRckUSRZ5f_ms0ZQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 25 Jul 2023 22:28:14 -0400 (EDT)
Message-ID: <7163164f-28a3-13b0-ffa8-eb1fc8b0b569@HIDDEN>
Date: Wed, 26 Jul 2023 05:28:13 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <877cqniyyk.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <877cqniyyk.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 25/07/2023 22:16, sbaugh@HIDDEN wrote:
> Dmitry Gutov <dmitry@HIDDEN> writes:
>>> (my-bench 1 default-directory "")
>>
>> (("built-in" . "Elapsed time: 1.601649s (0.709108s in 22 GCs)")
>>   ("with-find" . "Elapsed time: 1.792383s (1.135869s in 38 GCs)")
>>   ("with-find-p" . "Elapsed time: 1.248543s (0.682827s in 20 GCs)")
>>   ("with-find-sync" . "Elapsed time: 0.922291s (0.343497s in 10 GCs)"))
> 
> Tangent, but:
> 
> Ugh, wow, call-process really is a lot faster than make-process.  I see
> now why people disliked my idea of replacing call-process with something
> based on make-process, this is a big difference...

More like forewarned. Do we want to exchange 25% of performance for 
extra reactivity? We might. But we'd probably put that behind a pref and 
have to maintain two implementations.

> There's zero reason it has to be so slow... maybe I should try to make a
> better make-process API and implementation which is actually fast.
> (without worrying about being constrained by compatibility with
> something that's already dog-slow)

I don't know if the API itself is at fault. The first step should be to 
investigate which part of the current one is actually slow, I think.

But then, of course, if improved performance really requires a change in 
the API, we can switch to some new one too (which having to maintain at 
least two implementations for a number of years).

BTW, looking at the difference between the with-find-* approaches' 
performance, it seems like most of it comes down to GC.

Any chance we're doing extra copying of strings even when we don't have 
to, or some inefficient copying -- compared to the sync implementation? 
E.g. we could use the "fast" approach at least when the :filter is not 
specified (which is the case in the first impl, "with-find"). The manual 
says this:

   The default filter simply outputs directly to the process buffer.

Perhaps it's worth looking at.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 02:36:02 +0000
Resent-Message-ID: <handler.64735.B64735.169033892719718 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169033892719718
          (code B ref 64735); Wed, 26 Jul 2023 02:36:02 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 02:35:27 +0000
Received: from localhost ([127.0.0.1]:47276 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOUN4-00057x-BI
	for submit <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:35:27 -0400
Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:50853)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qOUN1-00057h-Ua
 for 64735 <at> debbugs.gnu.org; Tue, 25 Jul 2023 22:35:24 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.west.internal (Postfix) with ESMTP id CD7C53200940;
 Tue, 25 Jul 2023 22:35:17 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Tue, 25 Jul 2023 22:35:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690338917; x=1690425317; bh=K2hPmBI5ORt1P1I9b12hUA1B568rm3T9q0y
 z9cpe+X8=; b=hqOVxsaevI7VjZBjyMluqipQ43mvmjmvYQFmxvL1sKIzhhApGCK
 YvPXJeAbzG9wT9Wxk9VHpBLiJLV6iPTqIFRIpRxo81GW7bF+gjMSgvffAcTI1mf3
 5Lc2Nh/ikntnsiVxPmPQL2ECaYvai1uxZb2SjOtpotmwo1er9S0/v7WbQham1U+Y
 2JcNwCnrNxCMCPvOpsZvrLe1jnqF6XTbTXe7/7hs5NEuiNgEBxRMijmlSTRbY5lX
 5ntBurE0qSszYyI8mW5BBLiyx9oFL2by3oJEQNJrBCehB99Deo3gKue5ZVidkgNJ
 846PtxljZeGIAsV+VunDuwCpVylPJDf4VPg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690338917; x=1690425317; bh=K2hPmBI5ORt1P1I9b12hUA1B568rm3T9q0y
 z9cpe+X8=; b=v0jIIhIi/Gbg1RCOs10eN+mls5i3QjtsFWmOXSRlbBceS8FclZn
 g+llICkQmtOpfsPAWmNivIGb9jmQswq2pKFByjp2X93ki60QK8GbYzFUnWAcWTx5
 UefiAgfeaoTx8GnFnXKCwcu7r8Y4uE7BxNccQ7V2hbAbUGRDn2ySGkVTk2m8rJK8
 iJGm868PydrsLXC5RabX01bnHvhXP1rptqWYDkLYtO9DfcJAQd8Jzi9I+hDQiY7y
 eIUbGCkmQm+PRzA67C6Y5U9Y1aR4wdhzEwIpoWKrMdN8CXr726XpySbBQWIILxpP
 W20j8jGBkHTO61zR3IAEkTXoE1/qSFM1Jlg==
X-ME-Sender: <xms:ZIbAZPcWvsxIiaGQ90GVFlGdbq-h1X2zIC0gmRLqWfEkSIVrOCEAvQ>
 <xme:ZIbAZFPNO5s9AqP_6A8lU3lV2u4fcs3hGNWI0Ec4Psf2yMYdz6I5HiFI-dlJnwymM
 uJxMEfNNBtiXu-MxpE>
X-ME-Received: <xmr:ZIbAZIhW0wwDkD-nf0LU78gT1M0-WNy2RYJJHRCJQ_7l17gm3RidfEVGthS4Vs8>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedriedugdehlecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:ZIbAZA90_U8SgFAQEKeVWZPhE8Vh1JsIq3tRcbJR6nli3I_3ZNGgEA>
 <xmx:ZIbAZLuoRwHnXMPWi3L9Mrt4D5MhvGtFM_NhS1qviecxbEldIUkW2A>
 <xmx:ZIbAZPF1ZSTA7mAKKsW7h6DrFGRV4zKVDsj1TGRGG_1SWzRhv2Mr9Q>
 <xmx:ZYbAZLJI0lVu8yg35YGFpOthT6ILxNscs5CYQejiUe7gDfvCKpv2-Q>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 25 Jul 2023 22:35:14 -0400 (EDT)
Message-ID: <910b6a80-687e-9ffb-03fe-263dfdc10031@HIDDEN>
Date: Wed, 26 Jul 2023 05:35:13 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <83351bq1ds.fsf@HIDDEN>
 <491724c3-534b-a498-8e48-c8d94531875c@HIDDEN> <83zg3jo18g.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83zg3jo18g.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 26/07/2023 05:28, Eli Zaretskii wrote:
>> Date: Wed, 26 Jul 2023 04:56:20 +0300
>> Cc:luangruo@HIDDEN,sbaugh@HIDDEN,yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@HIDDEN>
>>
>> Your other idea (spending time in text conversion) also sounds
>> plausible, but I don't know whether this much overhead can be explained
>> by it. And don't we have to convert any process's output to our internal
>> encoding anyway, on any platform?
> We do, but you-all probably run your tests on a system where the
> external encoding is UTF-8, right?  That is much faster.

I do. I suppose that transcoding can/uses the short-circuit approach, 
avoiding extra copying when the memory representations match.

It should be possible to measure the encoding's overhead by checking how 
big the output is, testing our code on a smaller string, and 
multiplying. Or, more roughly, by piping it to "iconv -f Windows-1251 -t 
UTF-8" and measuring how long it will take to finish (if our encoding 
takes longer, that could point to an optimization opportunity as well).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 26 Jul 2023 09:10:01 +0000
Resent-Message-ID: <handler.64735.B64735.169036256926393 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169036256926393
          (code B ref 64735); Wed, 26 Jul 2023 09:10:01 +0000
Received: (at 64735) by debbugs.gnu.org; 26 Jul 2023 09:09:29 +0000
Received: from localhost ([127.0.0.1]:47556 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOaWO-0006rX-3T
	for submit <at> debbugs.gnu.org; Wed, 26 Jul 2023 05:09:28 -0400
Received: from mout02.posteo.de ([185.67.36.66]:40895)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qOaWI-0006qf-3P
 for 64735 <at> debbugs.gnu.org; Wed, 26 Jul 2023 05:09:23 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id CC654240105
 for <64735 <at> debbugs.gnu.org>; Wed, 26 Jul 2023 11:09:15 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690362555; bh=dfRxu3WjvbG3Z1ab4VqywW3Zm8wPksFVH7XOINpcQmQ=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=h+j0SfJCpqNj4RdF0Kg9nAuvk/5Bv04nuVjuTbbr5Sgp2Imzj7zUzFhQKq4qbm4Ex
 WZL6z3ydQLtNn0x8dUaDTtwkrrXudrIvbi1293n0QOSOfhjnCjgd7uV82/IQcykGKi
 KJtw2oA4VsX7orOi6Ub9Q4YspFaNQavkxy7i4h5O7LmcrUiC18WKZuil+ky9xUFaAd
 GZNLIcScYXywjD8bZJil9YRpOXAKkRWslNVS1b/Ht/aXZ/bkPxrHKCVoZPv1UVJze5
 mAGm6YQdXrHchbzuq/i9aWdWDaVDLXugY3zsTuPpgAlstWTqnnbAKafupq0YFv43P6
 vZODBNltCkn2Q==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4R9p4B4gBkz6tn4;
 Wed, 26 Jul 2023 11:09:14 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
Date: Wed, 26 Jul 2023 09:09:28 +0000
Message-ID: <878rb3m43b.fsf@localhost>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

--=-=-=
Content-Type: text/plain

Dmitry Gutov <dmitry@HIDDEN> writes:

>> (my-bench 10 "/usr/src/linux/" "")
>> 
>> (("built-in" . "Elapsed time: 7.034326s (3.598539s in 14 GCs)")
>>   ("built-in no filename handler alist" . "Elapsed time: 5.907194s (3.698456s in 15 GCs)")
>>   ("with-find" . "Elapsed time: 6.078056s (4.052791s in 16 GCs)")
>>   ("with-find-p" . "Elapsed time: 4.496762s (2.739565s in 11 GCs)")
>>   ("with-find-sync" . "Elapsed time: 3.702760s (1.715160s in 7 GCs)"))
>
> Thanks, for the extra data point in particular. Easy to see how it 
> compares to the most efficient use of 'find', right (on GNU/Linix, at 
> least)?
>
> It's also something to note that, GC-wise, numbers 1 and 2 are not the 
> worst: the time must be spent somewhere else.

Indeed. I did more detailed analysis in
https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/

Main contributors in the lisp versions are (in the order from most
significant to less significant) (1) file name handlers; (2) regexp
matching of the file names; (3) nconc calls in the current
`directory-files-recursively' implementation.

I have modified `directory-files-recursively' to avoid O(N^2) `nconc'
calls + bypassing regexp matches when REGEXP is nil.

Here are the results (using the attached modified version of your
benchmark file):

(my-bench 10 "/usr/src/linux/" "")
(("built-in" . "Elapsed time: 7.285597s (3.853368s in 6 GCs)")
 ("built-in no filename handler alist" . "Elapsed time: 5.855019s (3.760662s in 6 GCs)")
 ("built-in non-recursive no filename handler alist" . "Elapsed time: 5.817639s (4.326945s in 7 GCs)")
 ("built-in non-recursive no filename handler alist + skip re-match" . "Elapsed time: 2.708306s (1.871665s in 3 GCs)")
 ("with-find" . "Elapsed time: 6.082200s (4.262830s in 7 GCs)")
 ("with-find-p" . "Elapsed time: 4.325503s (3.058647s in 5 GCs)")
 ("with-find-sync" . "Elapsed time: 3.267648s (1.903655s in 3 GCs)"))

 (let ((gc-cons-threshold most-positive-fixnum))
   (my-bench 10 "/usr/src/linux/" ""))
(("built-in" . "Elapsed time: 2.754473s")
 ("built-in no filename handler alist" . "Elapsed time: 1.322443s")
 ("built-in non-recursive no filename handler alist" . "Elapsed time: 1.235044s")
 ("built-in non-recursive no filename handler alist + skip re-match" . "Elapsed time: 0.750275s")
 ("with-find" . "Elapsed time: 1.438510s")
 ("with-find-p" . "Elapsed time: 1.200876s")
 ("with-find-sync" . "Elapsed time: 1.349755s"))

If we forget about GC, Elisp version can get fairly close to GNU find.
And if we do not perform regexp matching (which makes sense when the
REGEXP is ""), Elisp version is faster.


--=-=-=
Content-Type: text/plain
Content-Disposition: inline; filename=find-bench.el

;; -*- lexical-binding: t; -*-

(defun find-directory-files-recursively (dir regexp &optional include-directories _p follow-symlinks)
  (cl-assert (null _p) t "find-directory-files-recursively can't accept arbitrary predicates")
  (with-temp-buffer
    (setq case-fold-search nil)
    (cd dir)
    (let* ((command
	    (append
	     (list "find" (file-local-name dir))
	     (if follow-symlinks
		 '("-L")
	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
	     (unless (string-empty-p regexp)
	       (list "-regex" (concat ".*" regexp ".*")))
	     (unless include-directories
	       '("!" "-type" "d"))
	     '("-print0")
	     ))
	   (remote (file-remote-p dir))
	   (proc
	    (if remote
		(let ((proc (apply #'start-file-process
				   "find" (current-buffer) command)))
		  (set-process-sentinel proc (lambda (_proc _state)))
		  (set-process-query-on-exit-flag proc nil)
		  proc)
	      (make-process :name "find" :buffer (current-buffer)
			    :connection-type 'pipe
			    :noquery t
			    :sentinel (lambda (_proc _state))
			    :command command))))
      (while (accept-process-output proc))
      (let ((start (goto-char (point-min))) ret)
	(while (search-forward "\0" nil t)
	  (push (concat remote (buffer-substring-no-properties start (1- (point)))) ret)
	  (setq start (point)))
	ret))))

(defun find-directory-files-recursively-2 (dir regexp &optional include-directories _p follow-symlinks)
  (cl-assert (null _p) t "find-directory-files-recursively can't accept arbitrary predicates")
  (cl-assert (not (file-remote-p dir)))
  (let* (buffered
         result
         (proc
	  (make-process
           :name "find" :buffer nil
	   :connection-type 'pipe
	   :noquery t
	   :sentinel (lambda (_proc _state))
           :filter (lambda (proc data)
                     (let ((start 0))
                       (when-let (end (string-search "\0" data start))
                         (push (concat buffered (substring data start end)) result)
                         (setq buffered "")
                         (setq start (1+ end))
                         (while-let ((end (string-search "\0" data start)))
                           (push (substring data start end) result)
                           (setq start (1+ end))))
                       (setq buffered (concat buffered (substring data start)))))
	   :command (append
	             (list "find" (file-local-name dir))
	             (if follow-symlinks
		         '("-L")
	               '("!" "(" "-type" "l" "-xtype" "d" ")"))
	             (unless (string-empty-p regexp)
	               (list "-regex" (concat ".*" regexp ".*")))
	             (unless include-directories
	               '("!" "-type" "d"))
	             '("-print0")
	             ))))
    (while (accept-process-output proc))
    result))

(defun find-directory-files-recursively-3 (dir regexp &optional include-directories _p follow-symlinks)
  (cl-assert (null _p) t "find-directory-files-recursively can't accept arbitrary predicates")
  (cl-assert (not (file-remote-p dir)))
  (let ((args `(,(file-local-name dir)
	        ,@(if follow-symlinks
		      '("-L")
	            '("!" "(" "-type" "l" "-xtype" "d" ")"))
	        ,@(unless (string-empty-p regexp)
	            (list "-regex" (concat ".*" regexp ".*")))
	        ,@(unless include-directories
	            '("!" "-type" "d"))
	        "-print0")))
    (with-temp-buffer
      (let ((status (apply #'process-file
                           "find"
                           nil
                           t
                           nil
                           args))
            (pt (point-min))
            res)
        (unless (zerop status)
          (error "Listing failed"))
        (goto-char (point-min))
        (while (search-forward "\0" nil t)
          (push (buffer-substring-no-properties pt (1- (point)))
                res)
          (setq pt (point)))
        res))))

(defun directory-files-recursively-strip-nconc
    (dir regexp
	 &optional include-directories predicate
	 follow-symlinks)
  "Return list of all files under directory DIR whose names match REGEXP.
This function works recursively.  Files are returned in \"depth
first\" order, and files from each directory are sorted in
alphabetical order.  Each file name appears in the returned list
in its absolute form.

By default, the returned list excludes directories, but if
optional argument INCLUDE-DIRECTORIES is non-nil, they are
included.

PREDICATE can be either nil (which means that all subdirectories
of DIR are descended into), t (which means that subdirectories that
can't be read are ignored), or a function (which is called with
the name of each subdirectory, and should return non-nil if the
subdirectory is to be descended into).

If FOLLOW-SYMLINKS is non-nil, symbolic links that point to
directories are followed.  Note that this can lead to infinite
recursion."
  (let* ((result nil)
	 (dirs (list dir))
         (dir (directory-file-name dir))
	 ;; When DIR is "/", remote file names like "/method:" could
	 ;; also be offered.  We shall suppress them.
	 (tramp-mode (and tramp-mode (file-remote-p (expand-file-name dir)))))
    (while (setq dir (pop dirs))
      (dolist (file (file-name-all-completions "" dir))
	(unless (member file '("./" "../"))
	  (if (directory-name-p file)
	      (let* ((leaf (substring file 0 (1- (length file))))
		     (full-file (concat dir "/" leaf)))
		;; Don't follow symlinks to other directories.
		(when (and (or (not (file-symlink-p full-file))
			       follow-symlinks)
			   ;; Allow filtering subdirectories.
			   (or (eq predicate nil)
			       (eq predicate t)
			       (funcall predicate full-file)))
                  (push full-file dirs))
		(when (and include-directories
			   (string-match regexp leaf))
		  (setq result (nconc result (list full-file)))))
	    (when (and regexp (string-match regexp file))
	      (push (concat dir "/" file) result))))))
    (sort result #'string<)))

(defun my-bench (count path regexp)
  (setq path (expand-file-name path))
  ;; (let ((old (directory-files-recursively path regexp))
  ;;       (new (find-directory-files-recursively-3 path regexp)))
  ;;   (dolist (path old)
  ;;     (unless (member path new) (error "! %s not in" path)))
  ;;   (dolist (path new)
  ;;     (unless (member path old) (error "!! %s not in" path))))
  (list
   (cons "built-in" (benchmark count (list 'directory-files-recursively path regexp)))
   (cons "built-in no filename handler alist" (let (file-name-handler-alist) (benchmark count (list 'directory-files-recursively path regexp))))
   (cons "built-in non-recursive no filename handler alist" (let (file-name-handler-alist) (benchmark count (list 'directory-files-recursively-strip-nconc path regexp))))
   (cons "built-in non-recursive no filename handler alist + skip re-match" (let (file-name-handler-alist) (benchmark count (list 'directory-files-recursively-strip-nconc path nil))))
   (cons "with-find" (benchmark count (list 'find-directory-files-recursively path regexp)))
   (cons "with-find-p" (benchmark count (list 'find-directory-files-recursively-2 path regexp)))
   (cons "with-find-sync" (benchmark count (list 'find-directory-files-recursively-3 path regexp)))))

(provide 'find-bench)

--=-=-=
Content-Type: text/plain


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 00:42:01 +0000
Resent-Message-ID: <handler.64735.B64735.169041850423705 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, Eli Zaretskii <eliz@HIDDEN>, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169041850423705
          (code B ref 64735); Thu, 27 Jul 2023 00:42:01 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 00:41:44 +0000
Received: from localhost ([127.0.0.1]:40346 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOp4a-0006AG-8l
	for submit <at> debbugs.gnu.org; Wed, 26 Jul 2023 20:41:44 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:57969)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qOp4X-0006A3-Aw
 for 64735 <at> debbugs.gnu.org; Wed, 26 Jul 2023 20:41:42 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.west.internal (Postfix) with ESMTP id ADF0332007E8;
 Wed, 26 Jul 2023 20:41:34 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Wed, 26 Jul 2023 20:41:35 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690418494; x=1690504894; bh=ySSA/l4LP/FOYqC9lH92WgOaNrzVFPoeleu
 zEfCWzU8=; b=V7xeKrzYYuyM+v0dzAsopQQoH5F8g2yxXDCMKPZxXRmysbGcdsq
 503guhJVqtapQ2EpxYXj6DyYbi2ZAgFtMrhqKhOd0S/U7OyWqylfLLxzwl0xkfgh
 xhS9EFbqcVotOray3uLn1VF2ehqXuE2/wyoOHeBU1+6o24gxZFifeE5RckAoMGYT
 ZAZkKXDRc84khWtTKzipWSEgmry86ozmBH41mApAZXovt/tm3LtCaKNSx3oStUHz
 RigcoW7k42whJYat/S2q8ZAgYHL6dc66Q++mpFyftXEmEC3I9HQ1fwym5Oz7kzR0
 itTonfrNssKTSUD3QS+mYHP4LbwkXSusHZw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690418494; x=1690504894; bh=ySSA/l4LP/FOYqC9lH92WgOaNrzVFPoeleu
 zEfCWzU8=; b=OWDzB9Fj1vHq4elpHc/hlg7vMvGF56o9KnrXCFxpedxA+h/SYTW
 alO8zo/BaGnduo9NByaPOKY+HjkeyHn8ni6JVukGR8bITqtSk4td6nZYHIhQtcux
 pBMIDjKT3F8YXyOgDLoL5BXIWHYjCxnPY++c40LKdSARbvqMIaXL307y6/+80QKM
 rwZt0F41EySsBCJJI1ouQSG8DWHuDxckNYWHjkIrans4V6OCCqzibCi7T0yTmRQb
 8AkmJx4dCyzYxiDcuDkwk0GZIC/MDbOB2Z6Yv+weG+Ojl6wqcZWXL3FO1yC6xO48
 c5xzh2ZYBo0cfe9K6kh6tdaiGeHQ3L940SA==
X-ME-Sender: <xms:Pb3BZJCoXgkP7-Pz5mHTaNkY7d7DaMsHpClXOHDxR1-b3pc-A_pzhg>
 <xme:Pb3BZHhBdZVVKEQwIZmnCC4mT8c2pCoviMsuqnrARwpYFjayaFHwTYzSyChTjrZMj
 As07OLK0_iWZt2qpnY>
X-ME-Received: <xmr:Pb3BZEkIv13_fs2HGHDucJSKntsWcPz9D9GNWGFc3NB3VJaz3fCp78jUGDJmq40>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrieefgdefiecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeehleeflefhheeujeduteejveeggeejjefhleeljeettdfhjedthfejgfeivdeu
 leenucffohhmrghinhephihhvghtihhlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd
 enucfrrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:Pb3BZDyJ98F-MHiF2x3FO5ddNtHqppc0R75FFIK42Xu9rks60IT5EA>
 <xmx:Pb3BZOQaRcOXTN08yADSVuAziXsFu_Hrrbzsd_5w6vFAVke9_DyV_w>
 <xmx:Pb3BZGbnqz36LCkCj-nkv9sB209P4mN8W8P-oUQq4gUgnN4MKE-yxA>
 <xmx:Pr3BZMeTR2miPyYq3Oc19prcgeXw2J6A31lE7Eifh_sBCcyUslrkHg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed,
 26 Jul 2023 20:41:32 -0400 (EDT)
Message-ID: <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
Date: Thu, 27 Jul 2023 03:41:29 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN> <878rb3m43b.fsf@localhost>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <878rb3m43b.fsf@localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 26/07/2023 12:09, Ihor Radchenko wrote:
>> It's also something to note that, GC-wise, numbers 1 and 2 are not the
>> worst: the time must be spent somewhere else.
> Indeed. I did more detailed analysis in
> https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
> 
> Main contributors in the lisp versions are (in the order from most
> significant to less significant) (1) file name handlers; (2) regexp
> matching of the file names; (3) nconc calls in the current
> `directory-files-recursively' implementation.
> 
> I have modified `directory-files-recursively' to avoid O(N^2) `nconc'
> calls + bypassing regexp matches when REGEXP is nil.

Sounds good. I haven't examined the diff closely, but it sounds like an 
improvement that can be applied irrespective of how this discussion ends.

Skipping regexp matching entirely, though, will make this benchmark 
farther removed from real-life usage: this thread started from being 
able to handle multiple ignore entries when listing files (e.g. in a 
project). So any solution for that (whether we use it on all or just 
some platforms) needs to be able to handle those. And it doesn't seem 
like directory-files-recursively has any alternative solution for that 
other than calling string-match on every found file.

> Here are the results (using the attached modified version of your
> benchmark file):
> 
> (my-bench 10 "/usr/src/linux/" "")
> (("built-in" . "Elapsed time: 7.285597s (3.853368s in 6 GCs)")
>   ("built-in no filename handler alist" . "Elapsed time: 5.855019s (3.760662s in 6 GCs)")
>   ("built-in non-recursive no filename handler alist" . "Elapsed time: 5.817639s (4.326945s in 7 GCs)")
>   ("built-in non-recursive no filename handler alist + skip re-match" . "Elapsed time: 2.708306s (1.871665s in 3 GCs)")
>   ("with-find" . "Elapsed time: 6.082200s (4.262830s in 7 GCs)")
>   ("with-find-p" . "Elapsed time: 4.325503s (3.058647s in 5 GCs)")
>   ("with-find-sync" . "Elapsed time: 3.267648s (1.903655s in 3 GCs)"))

Nice.

>   (let ((gc-cons-threshold most-positive-fixnum))
>     (my-bench 10 "/usr/src/linux/" ""))
> (("built-in" . "Elapsed time: 2.754473s")
>   ("built-in no filename handler alist" . "Elapsed time: 1.322443s")
>   ("built-in non-recursive no filename handler alist" . "Elapsed time: 1.235044s")
>   ("built-in non-recursive no filename handler alist + skip re-match" . "Elapsed time: 0.750275s")
>   ("with-find" . "Elapsed time: 1.438510s")
>   ("with-find-p" . "Elapsed time: 1.200876s")
>   ("with-find-sync" . "Elapsed time: 1.349755s"))
> 
> If we forget about GC, Elisp version can get fairly close to GNU find.
> And if we do not perform regexp matching (which makes sense when the
> REGEXP is ""), Elisp version is faster.

We can't really forget about GC, though.

But the above numbers make me hopeful about the async-parallel solution, 
implying that the parallelization really can help (and offset whatever 
latency we lose on pselect), as soon as we determine the source of extra 
consing and decide what to do about it.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 05:22:01 +0000
Resent-Message-ID: <handler.64735.B64735.169043531819282 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169043531819282
          (code B ref 64735); Thu, 27 Jul 2023 05:22:01 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 05:21:58 +0000
Received: from localhost ([127.0.0.1]:40515 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOtRk-00050u-61
	for submit <at> debbugs.gnu.org; Thu, 27 Jul 2023 01:21:58 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:55860)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qOtRa-00050P-19
 for 64735 <at> debbugs.gnu.org; Thu, 27 Jul 2023 01:21:54 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOtRS-0003GL-Qb; Thu, 27 Jul 2023 01:21:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=Em312YoA8zr106LqmGsn/fJfxHg7M4cadCjL2P8DwXk=; b=ZMEqN/1IAyur
 LkgqKB7Uf+WCadyvjpLJZtC7jdNhbGE2zBMfMxjgWYTeTpuY/dl4qLiu3CSqBqgrPQHy1RZyvJGcR
 gG84Xd+e4wHu55yVnrLTXHW+9RCgKq1o/zCBkbYXQNm45B/3k+y61vpEb/uIJ7d76tBMEqP6Hh2vg
 8yH++UZd9oZWbZAWXtAMo3dZOXCRSAMKwcQ/YcINHOypd676pIp07/BdRb4/cXnCTHh+g8DVlpmmp
 x60My5MLUebGo96IJKk093ZlQDMOXHCcVqrZL+SiDRTE6eOW0c8rEpDeZCsW2OA9z5L6Ht7tOczH/
 shpxAV+4PGE21W4nyuxbAQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOtRR-0001kO-QM; Thu, 27 Jul 2023 01:21:38 -0400
Date: Thu, 27 Jul 2023 08:22:27 +0300
Message-Id: <83v8e6lyi4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> (message from
 Dmitry Gutov on Thu, 27 Jul 2023 03:41:29 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN> <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Thu, 27 Jul 2023 03:41:29 +0300
> Cc: Eli Zaretskii <eliz@HIDDEN>, luangruo@HIDDEN, sbaugh@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > I have modified `directory-files-recursively' to avoid O(N^2) `nconc'
> > calls + bypassing regexp matches when REGEXP is nil.
> 
> Sounds good. I haven't examined the diff closely, but it sounds like an 
> improvement that can be applied irrespective of how this discussion ends.

That change should be submitted as a separate issue and discussed in
detail before we decide we can make it.

> Skipping regexp matching entirely, though, will make this benchmark 
> farther removed from real-life usage: this thread started from being 
> able to handle multiple ignore entries when listing files (e.g. in a 
> project).

Agreed.  From my POV, that variant's purpose was only to show how much
time is spent in matching file names against some include or exclude
list.

> So any solution for that (whether we use it on all or just 
> some platforms) needs to be able to handle those. And it doesn't seem 
> like directory-files-recursively has any alternative solution for that 
> other than calling string-match on every found file.

There's a possibility of pushing this filtering into
file-name-all-completions, but I'm not sure that will be faster.  We
should try that and measure the results, I think.

> > If we forget about GC, Elisp version can get fairly close to GNU find.
> > And if we do not perform regexp matching (which makes sense when the
> > REGEXP is ""), Elisp version is faster.
> 
> We can't really forget about GC, though.

But we could temporarily lift the threshold while this function runs,
if that leads to significant savings.

> But the above numbers make me hopeful about the async-parallel solution, 
> implying that the parallelization really can help (and offset whatever 
> latency we lose on pselect), as soon as we determine the source of extra 
> consing and decide what to do about it.

Isn't it clear that additional consing comes from the fact that we
first insert the Find's output into a buffer or produce a string from
it, and then chop that into individual file names?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 08:21:02 +0000
Resent-Message-ID: <handler.64735.B64735.16904460515439 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, Dmitry Gutov <dmitry@HIDDEN>, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16904460515439
          (code B ref 64735); Thu, 27 Jul 2023 08:21:02 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 08:20:51 +0000
Received: from localhost ([127.0.0.1]:40684 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOwEt-0001Pf-5i
	for submit <at> debbugs.gnu.org; Thu, 27 Jul 2023 04:20:51 -0400
Received: from mout01.posteo.de ([185.67.36.65]:39391)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qOwEr-0001PS-FP
 for 64735 <at> debbugs.gnu.org; Thu, 27 Jul 2023 04:20:50 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 8A7C4240029
 for <64735 <at> debbugs.gnu.org>; Thu, 27 Jul 2023 10:20:43 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690446043; bh=8CK4frDJIpaTtKrKYiSTXrB+ElVpBcM6E1a0mmb6o1Q=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=ealKtDjJUROjtxIE65nvxGkX0PytgtR2EbRtkqybupTbEWkAXuNQVasRQiY+9nFRW
 +VPXxHaX6Ai7sxii/5KdlMhAdBcNj1TfCeqoChGHwIBq4Ot+ACZzuMML6lgVRSltRK
 +jBXXsKEa3vlCfSnGmEuMDnlv1BsLHgfz2GVGi2boxO6xan0ZUCatFuhCNk0QQXH8z
 QfJFwCnA7Vi+62x3Rlwe9JjNsoEvYFeQnJCUF3PnW+e9LjAmFYQQeKqw77kEDo9ytv
 abRInWx0iGu5R0h55oj3qfYw8T6RDr8mQykKqkWvzpIo0ODKA/dVqtlv/yS7hpWF6p
 ZyHWycvM6b+Rw==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RBNxk3V4Tz6tvc;
 Thu, 27 Jul 2023 10:20:42 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83v8e6lyi4.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
Date: Thu, 27 Jul 2023 08:20:55 +0000
Message-ID: <87lef17ok8.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

>> > I have modified `directory-files-recursively' to avoid O(N^2) `nconc'
>> > calls + bypassing regexp matches when REGEXP is nil.
>> 
>> Sounds good. I haven't examined the diff closely, but it sounds like an 
>> improvement that can be applied irrespective of how this discussion ends.
>
> That change should be submitted as a separate issue and discussed in
> detail before we decide we can make it.

I will look into it. This was mostly a quick and dirty rewrite without
paying too match attention to file order in the result.

>> Skipping regexp matching entirely, though, will make this benchmark 
>> farther removed from real-life usage: this thread started from being 
>> able to handle multiple ignore entries when listing files (e.g. in a 
>> project).
>
> Agreed.  From my POV, that variant's purpose was only to show how much
> time is spent in matching file names against some include or exclude
> list.

Yes and no.

It is not uncommon to query _all_ the files in directory and something
as simple as

(when (and (not (member regexp '("" ".*"))) (string-match regexp file))...)

can give considerable speedup.

Might be worth adding such optimization.

>> So any solution for that (whether we use it on all or just 
>> some platforms) needs to be able to handle those. And it doesn't seem 
>> like directory-files-recursively has any alternative solution for that 
>> other than calling string-match on every found file.
>
> There's a possibility of pushing this filtering into
> file-name-all-completions, but I'm not sure that will be faster.  We
> should try that and measure the results, I think.

Isn't `file-name-all-completions' more limited and cannot accept
arbitrary regexp?

>> We can't really forget about GC, though.
>
> But we could temporarily lift the threshold while this function runs,
> if that leads to significant savings.

Yup. Also, GC times and frequencies will vary across different Emacs
sessions. So, we may not want to rely on it when comparing the
benchmarks from different people.

>> But the above numbers make me hopeful about the async-parallel solution, 
>> implying that the parallelization really can help (and offset whatever 
>> latency we lose on pselect), as soon as we determine the source of extra 
>> consing and decide what to do about it.
>
> Isn't it clear that additional consing comes from the fact that we
> first insert the Find's output into a buffer or produce a string from
> it, and then chop that into individual file names?

To add to it, I also tried to implement a version of
`directory-files-recursively' that first inserts all the files in buffer
and then filters them using `re-search-forward' instead of calling
`string-match' on every file name string.
That ended up being slower compared to the current `string-match' approach.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 08:48:01 +0000
Resent-Message-ID: <handler.64735.B64735.16904476367938 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Ihor Radchenko <yantar92@HIDDEN>
Cc: luangruo@HIDDEN, dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16904476367938
          (code B ref 64735); Thu, 27 Jul 2023 08:48:01 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 08:47:16 +0000
Received: from localhost ([127.0.0.1]:40717 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOweR-00023y-UJ
	for submit <at> debbugs.gnu.org; Thu, 27 Jul 2023 04:47:16 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:53756)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qOweP-00023i-6p
 for 64735 <at> debbugs.gnu.org; Thu, 27 Jul 2023 04:47:14 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOweJ-00064Z-8j; Thu, 27 Jul 2023 04:47:07 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=FGidqSqyf3NQoMOuBTjJkJ1H2ZvLfT/qCzbWv9CTy9g=; b=YHi74+EpbofA
 X/ZZe1fH7BGp53f0FOFpoRkl1tv0noSjQJgUJBM12id2RJTzt3o0sQ2K5NiCV1Dgc8JHjlEttxaAv
 vbFXteHQSMNpAkT1aVSXl+m0tPKpqfi9hyeNu8JuqADRyV6ABCZYoC8cj/WE3mk2ueneSFZ+EFKr/
 UBTltMmkv/dSTT+HiUBy9jjdaQE9EgvsgnmmMIK4amkQwq1x7No7492ZwEa1lLej4F6KNbQ53QGY2
 stXuFPGRbKYyo/o1Z5GKXOuGBlcmtLDeev6r0eZXLNesDL6hY1fyJmquXlpB55xKr58kPAFS9ole/
 +pefz/1jjGzPITJkFBZIxQ==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qOweI-0004Zi-OT; Thu, 27 Jul 2023 04:47:07 -0400
Date: Thu, 27 Jul 2023 11:47:55 +0300
Message-Id: <83edktn3k4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <87lef17ok8.fsf@localhost> (message from Ihor Radchenko on Thu,
 27 Jul 2023 08:20:55 +0000)
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
 <87lef17ok8.fsf@localhost>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Ihor Radchenko <yantar92@HIDDEN>
> Cc: Dmitry Gutov <dmitry@HIDDEN>, luangruo@HIDDEN,
>  sbaugh@HIDDEN, 64735 <at> debbugs.gnu.org
> Date: Thu, 27 Jul 2023 08:20:55 +0000
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> >> Skipping regexp matching entirely, though, will make this benchmark 
> >> farther removed from real-life usage: this thread started from being 
> >> able to handle multiple ignore entries when listing files (e.g. in a 
> >> project).
> >
> > Agreed.  From my POV, that variant's purpose was only to show how much
> > time is spent in matching file names against some include or exclude
> > list.
> 
> Yes and no.
> 
> It is not uncommon to query _all_ the files in directory and something
> as simple as
> 
> (when (and (not (member regexp '("" ".*"))) (string-match regexp file))...)
> 
> can give considerable speedup.

I don't understand what you are saying.  The current code already
checks PREDICATE for being nil, and if it is, does nothing about
filtering.

And if this is about testing REGEXP for being a trivial one, adding
such a test to the existing code is trivial, and hardly justifies an
objection to what I wrote.

> > There's a possibility of pushing this filtering into
> > file-name-all-completions, but I'm not sure that will be faster.  We
> > should try that and measure the results, I think.
> 
> Isn't `file-name-all-completions' more limited and cannot accept
> arbitrary regexp?

No, see completion-regexp-list.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Ihor Radchenko <yantar92@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 09:29:02 +0000
Resent-Message-ID: <handler.64735.B64735.169045008912216 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, dmitry@HIDDEN, 64735 <at> debbugs.gnu.org, sbaugh@HIDDEN
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169045008912216
          (code B ref 64735); Thu, 27 Jul 2023 09:29:02 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 09:28:09 +0000
Received: from localhost ([127.0.0.1]:40761 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qOxI0-0003Ay-MX
	for submit <at> debbugs.gnu.org; Thu, 27 Jul 2023 05:28:09 -0400
Received: from mout01.posteo.de ([185.67.36.65]:34351)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <yantar92@HIDDEN>) id 1qOxHy-0003AT-ED
 for 64735 <at> debbugs.gnu.org; Thu, 27 Jul 2023 05:28:07 -0400
Received: from submission (posteo.de [185.67.36.169]) 
 by mout01.posteo.de (Postfix) with ESMTPS id 5357B240028
 for <64735 <at> debbugs.gnu.org>; Thu, 27 Jul 2023 11:27:59 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1690450080; bh=Ytd8VeSAvDPI0pZncAHkopW37IDlkzXaSTuFZBDLBSA=;
 h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From;
 b=UZIc4HoR+pQI+aMOoSRzZ9ejnXSvibXMSBwegGokQiXpH6jx0/ghEu73ZGnnainhK
 sSR7XSArREeddIioxcbp1X2l0YuMbzeylTan32ZivAJIVG9n5PFdPZdf08JHmPyXaN
 u7naM8/PGZPU2xstIauSJMOycl7lXVKI0QSBYZYdPApDsFTY5pEJ41HbuSNDidLTiZ
 ctTz3jVMoLxN7Pr8Je5iQxP/ExDDEi8EnzSs4GlSimRGFvuiZYZta/6yoSwH6z8HWc
 F9bpnJRn3qc75IfXMEGj9RFD+8J5LqhITfwOTbGHiyLx/qJRPbJAEb8+ZARKHG08hV
 QSPV1h9T43iSQ==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4RBQRL3pxJz9rxQ;
 Thu, 27 Jul 2023 11:27:58 +0200 (CEST)
From: Ihor Radchenko <yantar92@HIDDEN>
In-Reply-To: <83edktn3k4.fsf@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
 <87lef17ok8.fsf@localhost> <83edktn3k4.fsf@HIDDEN>
Date: Thu, 27 Jul 2023 09:28:11 +0000
Message-ID: <87a5vh7lg4.fsf@localhost>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Eli Zaretskii <eliz@HIDDEN> writes:

> And if this is about testing REGEXP for being a trivial one, adding
> such a test to the existing code is trivial, and hardly justifies an
> objection to what I wrote.

I was replying to your interpretations on why I included "no-regexp"
test.

I agree that we should not use this test as comparison with GNU find.
But I also wanted to say that adding the trivial REGEXP test will be
useful. Especially because it is easy. Should I prepare a patch?

>> Isn't `file-name-all-completions' more limited and cannot accept
>> arbitrary regexp?
>
> No, see completion-regexp-list.

That would be equivalent to forcing `include-directories' being t.

In any case, even if we ignore INCLUDE-DIRECTORIES, there is no gain:

(my-bench 10 "/usr/src/linux/" "")

(("built-in non-recursive no filename handler alist" . "Elapsed time: 5.780714s (4.352086s in 6 GCs)")
 ("built-in non-recursive no filename handler alist + completion-regexp-list" . "Elapsed time: 5.739315s (4.359772s in 6 GCs)"))

 (let ((gc-cons-threshold most-positive-fixnum))
   (my-bench 10 "/usr/src/linux/" ""))
   
(("built-in non-recursive no filename handler alist" . "Elapsed time: 1.267828s")
("built-in non-recursive no filename handler alist + completion-regexp-list" . "Elapsed time: 1.275656s"))

In the test, I removed all the `string-match' calls and instead let-bound (completion-regexp-list (list regexp))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 27 Jul 2023 13:32:01 +0000
Resent-Message-ID: <handler.64735.B64735.169046467115342 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169046467115342
          (code B ref 64735); Thu, 27 Jul 2023 13:32:01 +0000
Received: (at 64735) by debbugs.gnu.org; 27 Jul 2023 13:31:11 +0000
Received: from localhost ([127.0.0.1]:40957 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qP15D-0003zO-85
	for submit <at> debbugs.gnu.org; Thu, 27 Jul 2023 09:31:11 -0400
Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:47203)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qP15A-0003zA-54
 for 64735 <at> debbugs.gnu.org; Thu, 27 Jul 2023 09:31:10 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.west.internal (Postfix) with ESMTP id 4C05D32008FD;
 Thu, 27 Jul 2023 09:31:01 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Thu, 27 Jul 2023 09:31:01 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690464660; x=1690551060; bh=waeVfQIki+HZTOaa0uly1FtCecdYCc2KDOT
 pFnDrxV4=; b=Fu6XNhQYD8DQZv2Rrc53t+xhHmsXy/dr72czO3ZrOQ9qoSHxcbv
 vlMcZQLk65YhehhBjarDnat/Z7JexZRcTpW31QW/SxxutizG4H/uMQmWaKOKkx04
 23wpgkEErDNp0/UN7o9UHDxgnWYtVAP2ZkDFKtdwd24Jje/akV7iecL9FJ+ARWFE
 ozU46BWyNDGnnVN9QdgnXNR0cry+0uRdsF2r98kk+gXCTE0AVmHqM/raiJoXlonA
 Bm48FY6uA2ec16hsbSq+NcoswvB8szNjcvCNf3dtY1IKARxkbbb8ThG1YJQuoerr
 NST6E0kqgJxLG8MzAIhcTHVoEDE3zIbfcNA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690464660; x=1690551060; bh=waeVfQIki+HZTOaa0uly1FtCecdYCc2KDOT
 pFnDrxV4=; b=uhtOiyAutyrsDZ0qrAkGgXec43QBKknbbyaLiddqmDkiVaep+By
 b/HzlNt09C5buv3E5ejuaSTxEpPLjb2kO3xigbif2AQuz7/nxM2GlR8V6PkknGvf
 wNjRHCTL7IdkLSMCJJFMZamXN0gyStlE2epEv0/eGwvoinFhgiBBOrVgN464gllF
 /7kU5zs/lqMr+zuQDZHfqyUzLEwv1wSGIS/LbKeFfhjoE3lKVRLF64NONoBGneWX
 qTyZ4koCyZ7c9Vqe+WdYvBK35qSsTzqcVSAcofH6FJzDJtJ1jJRwXot/DAwTcoxK
 UymW93WD+zRhIAGC86H8kX+8mkkZt9xdxaQ==
X-ME-Sender: <xms:lHHCZFvV3vnfhQDhMW3kU8XVj_-DfGeTQvGveFSeL8TnNjRjSd9AlQ>
 <xme:lHHCZOe1anl5OdT5p9spdsjhinQaAd9TqoHot_heb7NDpYasVuI2GmVFzvDjyU3jV
 X6or2bBcYbBg0wbi9s>
X-ME-Received: <xmr:lHHCZIxVMYxmLpjfVH1HPG_3-tI21orEYwWOsxTNgpjN4EU0XjXhN0oSIOhU_4Q>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrieeggdeflecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:lHHCZMOg0I2Sb5JJhhedW5LzwY33bMZuqUZQioBgLHHP4IY7LmiBVQ>
 <xmx:lHHCZF9pS1xY1h8aOdEldaJaDSo_E8GZvxEx-FujDAEHBbxqy4NqCg>
 <xmx:lHHCZMXTQROsZcirsy-4GjbT_zv8atkCEA2ecFTa0qG7A67lixJg3A>
 <xmx:lHHCZAYNJToI0_1foWtsJcx21ZYMPmF3l8Nb-ZDePtef-S5NxvmOZQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 27 Jul 2023 09:30:58 -0400 (EDT)
Message-ID: <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
Date: Thu, 27 Jul 2023 16:30:56 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN> <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83v8e6lyi4.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 27/07/2023 08:22, Eli Zaretskii wrote:
>> Date: Thu, 27 Jul 2023 03:41:29 +0300
>> Cc: Eli Zaretskii <eliz@HIDDEN>, luangruo@HIDDEN, sbaugh@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>> I have modified `directory-files-recursively' to avoid O(N^2) `nconc'
>>> calls + bypassing regexp matches when REGEXP is nil.
>>
>> Sounds good. I haven't examined the diff closely, but it sounds like an
>> improvement that can be applied irrespective of how this discussion ends.
> 
> That change should be submitted as a separate issue and discussed in
> detail before we decide we can make it.

Sure.

>>> If we forget about GC, Elisp version can get fairly close to GNU find.
>>> And if we do not perform regexp matching (which makes sense when the
>>> REGEXP is ""), Elisp version is faster.
>>
>> We can't really forget about GC, though.
> 
> But we could temporarily lift the threshold while this function runs,
> if that leads to significant savings.

I mean, everything's doable, but if we do this for this function, why 
not others? Most long-running code would see an improvement from that 
kind of change (the 'find'-based solutions too).

IIRC the main drawback is running out of memory in extreme conditions or 
on low-memory platforms/devices. It's not like this feature is 
particularly protected from this.

>> But the above numbers make me hopeful about the async-parallel solution,
>> implying that the parallelization really can help (and offset whatever
>> latency we lose on pselect), as soon as we determine the source of extra
>> consing and decide what to do about it.
> 
> Isn't it clear that additional consing comes from the fact that we
> first insert the Find's output into a buffer or produce a string from
> it, and then chop that into individual file names?

But we do that in all 'find'-based solutions: the synchronous one takes 
buffer text and chops it into strings. The first asynchronous does the 
same. The other ("with-find-p") works from a process filter, chopping up 
strings that get passed to it.

But the amount of time spent in GC is different, with most of the 
difference in performance attributable to it: if we subtract time spent 
in GC, the runtimes are approximately equal.

I can imagine that the filter-based approach necessarily creates more 
strings (to pass to the filter function). Maybe we could increase those 
strings' size (thus reducing the number) by increasing the read buffer 
size? I haven't found a relevant variable, though.

Or if there was some other callback that runs after the next chunk of 
output arrives from the process, we could parse it from the buffer. But 
the insertion into the buffer would need to be made efficient 
(apparently internal-default-process-filter currently uses the same 
sequence of strings as the other filters for input, with the same amount 
of consing).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 29 Jul 2023 00:13:01 +0000
Resent-Message-ID: <handler.64735.B64735.169058956722993 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169058956722993
          (code B ref 64735); Sat, 29 Jul 2023 00:13:01 +0000
Received: (at 64735) by debbugs.gnu.org; 29 Jul 2023 00:12:47 +0000
Received: from localhost ([127.0.0.1]:46293 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qPXZe-0005ym-V9
	for submit <at> debbugs.gnu.org; Fri, 28 Jul 2023 20:12:47 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:60679)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qPXZc-0005yZ-LE
 for 64735 <at> debbugs.gnu.org; Fri, 28 Jul 2023 20:12:45 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 7F2A75C00CF;
 Fri, 28 Jul 2023 20:12:39 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute1.internal (MEProxy); Fri, 28 Jul 2023 20:12:39 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690589559; x=1690675959; bh=kDAy/oC8PTT3tibtcSafhP6m7ELCJpuOacX
 162MYb58=; b=ZuGxXyaG6Dv3YzuwVPbxRtuxdzofsnn8W9cyhGWh7RwvLVJw+/A
 2+n5nly/4Ag/VclJoEU1uf3X9EAE7OKci6WiQJ65wqdS3/QN2ZDWZgGXdjKEb7LZ
 KYaL0I6nVZKzNdiNSrYRiJtXw6ZhVybN+hwmT4ssrC0sPOJ1hYDGyRjEma5B6QE1
 Gq9LD20rxk1FYlLUCLAEAsiQUwA4UrsY6vAVmI4hjx4VbpPCfv9ZNZx1w+is/9Hr
 aDm45+QAoD2WEpWVFigpmvWvrtnfi60gb0x1PONcLW4mnwJcWHEqPLj1+FXvlT/l
 OovEHp9F8jgrvMb7nkiZTDTEoLTI0c7WfgA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690589559; x=1690675959; bh=kDAy/oC8PTT3tibtcSafhP6m7ELCJpuOacX
 162MYb58=; b=NzWC8Lh00Qrblh9/livnDpRMs50pFyYQXlKBGtE9JzmrdG8NO6I
 O/GB3oULeR9JybJiYXn+dxof/SuJLpe1dawD3SWrIBV844HbbV7og4OPBXQrU2jl
 hdE+Rtqudm1a7b63wSGD2JsK38r2amFdqXsfAp11t0nejw5FOmpNYtiwwAWh2op2
 rRdLcjiaZNXJBa6Ne5aXNtVHnIaS7srX0eobHoNlDS+BOs2dnjFc7g0Cxl5CA2Gz
 XzwEPFbu7uTYpHTh1Cp2tWS28uo6QZ6DuiFP98iBwsiuQ6GV5uQZe3JvqPc13HLv
 4NURigtoSQ1+2urLFBL8WkB15WDWt6ojyfw==
X-ME-Sender: <xms:dlnEZCHkll446RUyZgP323ZlD5OYgELky48fXDZqPUThcdEIPz1y9g>
 <xme:dlnEZDXklgu24TyMF4Olx_9cc7oF8X7ANqYRlnh1eqlKtQf-P7ohjzQHfOwXC_YTx
 CI-Qhk_R7BRi5hJRfY>
X-ME-Received: <xmr:dlnEZMIgg84QFHMAK6YhNvu27zk6bRGQHR0MdNyD1012pZ89GTYt9o_RKU6bkhw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrieejgdefudcutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuhffvvehfjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeejgeeivdefgffhudetueevjefgfeelleetvddtfeevkeehveffueeuhfehjeff
 hfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:dlnEZMGq_nCBj17H4vI4kRvRywiXOJu9yUsWcaLzhCFCRRyoBo6M5g>
 <xmx:dlnEZIXeB-7K0yCJEyVW67x0roURLa4BZuPHkeeETKkQqyuFUktYww>
 <xmx:dlnEZPPYx2Qc6zNBYn5-66LwZJaHXzUCWbyHCUTOp1k3ePxZ-VLrpg>
 <xmx:d1nEZPRJSUuROj1prQLzfwlYGTAsS6jfCMj6qN1InRjCFi8gJ1-MyA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri,
 28 Jul 2023 20:12:36 -0400 (EDT)
Message-ID: <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
Date: Sat, 29 Jul 2023 03:12:34 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
From: Dmitry Gutov <dmitry@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN> <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
In-Reply-To: <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 27/07/2023 16:30, Dmitry Gutov wrote:
> I can imagine that the filter-based approach necessarily creates more 
> strings (to pass to the filter function). Maybe we could increase those 
> strings' size (thus reducing the number) by increasing the read buffer 
> size?

To go further along this route, first of all, I verified that the input 
strings are (almost) all the same length: 4096. And they are parsed into 
strings with length 50-100 characters, meaning the number of "junk" 
objects due to the process-filter approach probably shouldn't matter too 
much, given that the number of strings returned is 40-80x more.

But then I ran these tests with different values of 
read-process-output-max, which exactly increased those strings' size, 
proportionally reducing their number. The results were:

 > (my-bench-rpom 1 default-directory "")

=>

(("with-find-p 4096" . "Elapsed time: 0.945478s (0.474680s in 6 GCs)")
  ("with-find-p 40960" . "Elapsed time: 0.760727s (0.395379s in 5 GCs)")
("with-find-p 409600" . "Elapsed time: 0.729757s (0.394881s in 5 GCs)"))

where

(defun my-bench-rpom (count path regexp)
   (setq path (expand-file-name path))
   (list
    (cons "with-find-p 4096"
          (let ((read-process-output-max 4096))
            (benchmark count (list 'find-directory-files-recursively-2 
path regexp))))
    (cons "with-find-p 40960"
          (let ((read-process-output-max 40960))
            (benchmark count (list 'find-directory-files-recursively-2 
path regexp))))
    (cons "with-find-p 409600"
          (let ((read-process-output-max 409600))
            (benchmark count (list 'find-directory-files-recursively-2 
path regexp))))))

...with the last iteration showing consistently the same or better 
performance than the "sync" version I benchmarked previously.

What does that mean for us? The number of strings in the heap is 
reduced, but not by much (again, the result is a list with 43x more 
elements). The combined memory taken up by these intermediate strings to 
be garbage-collected, is the same.

It seems like per-chunk overhead is non-trivial, and affects GC somehow 
(but not in a way that just any string would).

In this test, by default, the output produces ~6000 strings and passes 
them to the filter function. Meaning, read_and_dispose_of_process_output 
is called about 6000 times, producing the overhead of roughly 0.2s. 
Something in there must be producing extra work for the GC.

This line seems suspect:

        list3 (outstream, make_lisp_proc (p), text),

Creates 3 conses and one Lisp object (tagged pointer). But maybe I'm 
missing something bigger.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 29 Jul 2023 06:15:02 +0000
Resent-Message-ID: <handler.64735.B64735.169061127430080 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169061127430080
          (code B ref 64735); Sat, 29 Jul 2023 06:15:02 +0000
Received: (at 64735) by debbugs.gnu.org; 29 Jul 2023 06:14:34 +0000
Received: from localhost ([127.0.0.1]:46377 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qPdDl-0007p5-Ob
	for submit <at> debbugs.gnu.org; Sat, 29 Jul 2023 02:14:34 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:60452)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qPdDj-0007oq-Vj
 for 64735 <at> debbugs.gnu.org; Sat, 29 Jul 2023 02:14:33 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qPdDd-0007PE-2o; Sat, 29 Jul 2023 02:14:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=SKQqTmH/Rt0eZcq836DttsASeUiyw+hd9kw/kzzepCc=; b=lnDPcJx+vkIw
 8evJXHSkO1q+WakRkdOut/NfjSUy86218Oe6ht9sV3kj/fDDfXT7juSWS6ouCSZwptNICQOR+nUiu
 wYSpB/pvw89mMhE2uF/NJL3vhrw3V98p+U4PU5Umyn0zeK/iXSI4lgOR/ByuygAdNrt60s76XjLGe
 bpBvot4DPRadVWx/V7j+imWO4GTfj5xq6AmnGtt14YOdHZ4au96WBHhZ2xPdoIZIuRZCjaZtV2hjl
 mi44EUNGAqo0NbiXHpEl6ByxOnNEkYBSndo2IZV6YGpHJXQaXiQZ0Pr4oU0ozC4MERtwmGLORDtfp
 vr6PFpidv/iH9N1OVocF7g==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qPdDS-0006vw-Qs; Sat, 29 Jul 2023 02:14:18 -0400
Date: Sat, 29 Jul 2023 09:15:09 +0300
Message-Id: <83pm4bi6qa.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN> (message from
 Dmitry Gutov on Sat, 29 Jul 2023 03:12:34 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN> <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN> <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN> <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN> <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN> <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN> <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN> <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN> <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sat, 29 Jul 2023 03:12:34 +0300
> From: Dmitry Gutov <dmitry@HIDDEN>
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> 
> It seems like per-chunk overhead is non-trivial, and affects GC somehow 
> (but not in a way that just any string would).
> 
> In this test, by default, the output produces ~6000 strings and passes 
> them to the filter function. Meaning, read_and_dispose_of_process_output 
> is called about 6000 times, producing the overhead of roughly 0.2s. 
> Something in there must be producing extra work for the GC.
> 
> This line seems suspect:
> 
>         list3 (outstream, make_lisp_proc (p), text),
> 
> Creates 3 conses and one Lisp object (tagged pointer). But maybe I'm 
> missing something bigger.

I don't understand what puzzles you here.  You need to make your
descriptions more clear to allow others to follow your logic.  You use
terms you never explain: "junk objects", "number of strings in the
heap", "per-chunk overhead" (what is "chunk"?), which is a no-no when
explaining complex technical stuff to others.

If I read what you wrote superficially, without delving into the
details (which I can't understand), you are saying that the overall
amount of consing is roughly the same.  This is consistent with the
fact that the GC times change only very little.  So I don't think I
see, on this level, what puzzles you in this picture.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 30 Jul 2023 01:37:01 +0000
Resent-Message-ID: <handler.64735.B64735.16906809624695 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16906809624695
          (code B ref 64735); Sun, 30 Jul 2023 01:37:01 +0000
Received: (at 64735) by debbugs.gnu.org; 30 Jul 2023 01:36:02 +0000
Received: from localhost ([127.0.0.1]:49110 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qPvLl-0001DU-Kb
	for submit <at> debbugs.gnu.org; Sat, 29 Jul 2023 21:36:02 -0400
Received: from out2-smtp.messagingengine.com ([66.111.4.26]:44143)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qPvLi-0001D9-Mf
 for 64735 <at> debbugs.gnu.org; Sat, 29 Jul 2023 21:35:59 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.nyi.internal (Postfix) with ESMTP id 747DF5C009A;
 Sat, 29 Jul 2023 21:35:53 -0400 (EDT)
Received: from mailfrontend1 ([10.202.2.162])
 by compute3.internal (MEProxy); Sat, 29 Jul 2023 21:35:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm2; t=
 1690680953; x=1690767353; bh=YgxU7i11/XCn0vDsz5JhY3gGnIHKv6/VfGQ
 6I0trznE=; b=VvKi7r4z/9L5coIKMDCVXvr2+rVOq3hH1Fa/5+L/9blEMwDS9b6
 nkv5yZBBufAn1t1kNY9NiNzkGFKXcaQAEz9YuMpbAWg+vpt+9IEdYXTdXQv3ALv5
 Z7xM1M4b5h9PTeqf2i88/hfTIKh1XCxpx/fDLEy/vltlfIu8cnU7F++BzRi2aE25
 tBr/p8BzhcPUUOpDi4ZxEcyLUPHRcW2e1b2iRvWLkZCjQzou5HRmFtOzzP3PdRyR
 ed58C+TlvctmE/MKX2R21dZjHfCguUdLFj01ZAhehrOtGthJnQ0xns6jek8QwuvN
 65i4j2PZWkKmXvkIKjeCbx+AXa07SnC6oZA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1690680953; x=1690767353; bh=YgxU7i11/XCn0vDsz5JhY3gGnIHKv6/VfGQ
 6I0trznE=; b=yn7ctk9vTJeIXvLKYh7VfMfLSWHGPu9Wu/NqkvziZuacn2bGchO
 LoL236dS7gzG295xQiGCmLugTK7yiII4yvlw5RQe1o2Rx7/d9k5DdcpBSzmCN9zX
 U7HZjlws/vOwDsW/mGw5nSzd7jEtYIPayxJ6oe0rJ7ZCdsdt5TtEXY/YKeKBPd1J
 n+G8BPtU19ecm8o4XpCJWhcbWG8wUVBoDTKZEx9JDS8rlr69f9W48pje2vuk/yQB
 zvXlPkBw0k85gA6C5IDRArWQYb2f5K/sAqcpnBhgHUOG/sVElaB/UP5t/iein7wF
 utEk1IcQuG+1F58No+BfSKzAb1mSd27nJNg==
X-ME-Sender: <xms:eb7FZDtbJO4O1H1bh7rU4icrFzvrJ6jLb-2d76aK5_LsN0n1Dgq6Cg>
 <xme:eb7FZEf2y5sJxKotlRUWgJ4y_KybVcVn7SenGnjzq0JYPLbRF-0in6LiCOAilHWbY
 jJ8KtTlv18MA42PiS0>
X-ME-Received: <xmr:eb7FZGz-7ydrGwB2eg5MkZgFAFWzaHzaRJoxh1FyCzwR9g1spYP9FdYkhRTUe2g>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrieelgdegjecutefuodetggdotefrodftvf
 curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
 uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
 fjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeevledv
 veenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:eb7FZCPINxi_N4pebyZyQ0kQ6i6P3jsJEHwxpaoc7ohsw8XMXjE09w>
 <xmx:eb7FZD9tF4TfPAJBFGYD45CRzUqwVipC1Uj88qbjWGFK2jyCUCrszw>
 <xmx:eb7FZCW6GiORjrPvx_nlV61l17kI17u7tJIOphN7QgTlN8Dk5a4Nqg>
 <xmx:eb7FZGZfEOjI2W3kiyQ6ltuA-DcETIaCk6YG_MirtPgTCeFmwSvptA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat,
 29 Jul 2023 21:35:51 -0400 (EDT)
Message-ID: <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
Date: Sun, 30 Jul 2023 04:35:49 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83pm4bi6qa.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -0.8 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.8 (-)

On 29/07/2023 09:15, Eli Zaretskii wrote:
>> Date: Sat, 29 Jul 2023 03:12:34 +0300
>> From: Dmitry Gutov <dmitry@HIDDEN>
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>>
>> It seems like per-chunk overhead is non-trivial, and affects GC somehow
>> (but not in a way that just any string would).
>>
>> In this test, by default, the output produces ~6000 strings and passes
>> them to the filter function. Meaning, read_and_dispose_of_process_output
>> is called about 6000 times, producing the overhead of roughly 0.2s.
>> Something in there must be producing extra work for the GC.
>>
>> This line seems suspect:
>>
>>          list3 (outstream, make_lisp_proc (p), text),
>>
>> Creates 3 conses and one Lisp object (tagged pointer). But maybe I'm
>> missing something bigger.
> 
> I don't understand what puzzles you here.  You need to make your
> descriptions more clear to allow others to follow your logic.  You use
> terms you never explain: "junk objects", "number of strings in the
> heap", "per-chunk overhead" (what is "chunk"?), which is a no-no when
> explaining complex technical stuff to others.

In this context, junks objects are objects that will need to be 
collected by garbage collector very soon because they are just a 
byproduct of a function's execution (but aren't used in the return 
value, for example). The more of them a function creates, the more work 
it will be, supposedly, for the GC.

Heap is perhaps the wrong term (given that C has its own notion of 
heap), but I meant the memory managed by the Lisp runtime.

And chunks are the buffered strings that get passed to the process 
filter. Chunks of the process' output. By default, these chunks are 4096 
characters long, but the comparisons tweak that value by 10x and 100x.

> If I read what you wrote superficially, without delving into the
> details (which I can't understand), you are saying that the overall
> amount of consing is roughly the same.

What is "amount of consing"? Is it just the number of objects? Or does 
their size (e.g. string length) affect GC pressure as well?

> This is consistent with the
> fact that the GC times change only very little.  So I don't think I
> see, on this level, what puzzles you in this picture.

Now that you pointed that out, the picture is just more puzzling. While 
0.1s in GC is not insignificant (it's 10% of the whole runtime), it does 
seem to have been more of a fluke, and on average the fluctuations in GC 
time are smaller.

Here's an extended comparison:

(("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)")
  ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)")
  ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)")
  ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)")
  ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)")
  ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)")
("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)")
("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)")
("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)")
("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)")
("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)"
))

What was puzzling for me, overall, is that if we take "with-find 409600" 
(the fastest among the asynchronous runs without parallelism) and 
"with-find-sync", the difference in GC time (which is repeatable), 
0.66s, almost covers all the difference in performance. And as for 
"with-find-p 409600", it would come out on top! Which it did in Ihor's 
tests when GC was disabled.

But where does the extra GC time come from? Is it from extra consing in 
the asynchronous call's case? If it is, it's not from all the chunked 
strings, apparently, given that increasing max string's size (and 
decreasing their number by 2x-6x, according to my logging) doesn't 
affect the reported GC time much.

Could the extra time spent in GC just come from the fact that it's given 
more opportunities to run, maybe? call_process stays entirely in C, 
whereas make-process, with its asynchronous approach, goes between C and 
Lisp even time it receives input. The report above might indicate so: 
with-find-p have ~20 garbage collection cycles, whereas with-find-sync - 
only ~10. Or could there be some other source of consing, unrelated to 
the process output string, and how finely they are sliced?

Changing process-adaptive-read-buffering to nil didn't have any effect here.

If we get back to increasing read-process-output-max, which does help 
(apparently due to reducing the number we switch between reading from 
the process and doing... whatever else), the sweet spot seems to be 
1048576, which is my system's maximum value. Anything higher - and the 
perf goes back to worse -- I'm guessing something somewhere resets the 
value to default? Not sure why it doesn't clip to the maximum allowed, 
though.

Anyway, it would be helpful to be able to decide on as high as possible 
value without manually reading from /proc/sys/fs/pipe-max-size. And what 
of other OSes?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 31 Jul 2023 11:39:02 +0000
Resent-Message-ID: <handler.64735.B64735.169080350810070 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169080350810070
          (code B ref 64735); Mon, 31 Jul 2023 11:39:02 +0000
Received: (at 64735) by debbugs.gnu.org; 31 Jul 2023 11:38:28 +0000
Received: from localhost ([127.0.0.1]:51935 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qQREJ-0002cL-P6
	for submit <at> debbugs.gnu.org; Mon, 31 Jul 2023 07:38:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:36488)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qQREH-0002c9-8I
 for 64735 <at> debbugs.gnu.org; Mon, 31 Jul 2023 07:38:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qQREA-00043j-MV; Mon, 31 Jul 2023 07:38:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=e3t9o9cqVCQ4loUKEOavC7d3QlIDyiAJPPlZ64FunRM=; b=Zo7XCgOthxeh
 1Y3ZfXbGPvOttiM/m1fvCB0mlxT7PzMZiKXvcochspr+2Xf8Os2EYzlFQB5P6qnkHLlDDGbk3uT5L
 xlmGdMlPAHI+1aF/taopwAFYSJ1xpCj6GiPj7bTEtLatFNSYaJtMuSvBcETsxI61whgE6HGoJdu0X
 1DkWfF5JsJ+7/smra/QfsXZShxC0w0vjiuhe1TwUIdY4pxZ8ubr815O/O2uXjF/kSZoFwLkbhAjm+
 Gd9RDwYlRjs6zX8d8suECn+ncsjMNaPj6AqWoYSfeIWh48Ii3sbt/F+Y1h6Tt7VI23drkb9yevu5w
 gC13C55q1IeVGLFSUcIz9g==;
Received: from [87.69.77.57] (helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qQRE9-0006ZH-UE; Mon, 31 Jul 2023 07:38:18 -0400
Date: Mon, 31 Jul 2023 14:38:18 +0300
Message-Id: <83bkfs2tw5.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN> (message from
 Dmitry Gutov on Sun, 30 Jul 2023 04:35:49 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@HIDDEN>
 <87wmyupvlw.fsf@localhost>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN> <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 30 Jul 2023 04:35:49 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> In this context, junks objects are objects that will need to be 
> collected by garbage collector very soon because they are just a 
> byproduct of a function's execution (but aren't used in the return 
> value, for example). The more of them a function creates, the more work 
> it will be, supposedly, for the GC.
> 
> Heap is perhaps the wrong term (given that C has its own notion of 
> heap), but I meant the memory managed by the Lisp runtime.
> 
> And chunks are the buffered strings that get passed to the process 
> filter. Chunks of the process' output. By default, these chunks are 4096 
> characters long, but the comparisons tweak that value by 10x and 100x.

If the subprocess output is inserted into a buffer, its effect on the
GC will be different.  (Not sure if this is relevant to the issue at
hand, as I lost track of the many variants of the function that were
presented.)

> > If I read what you wrote superficially, without delving into the
> > details (which I can't understand), you are saying that the overall
> > amount of consing is roughly the same.
> 
> What is "amount of consing"? Is it just the number of objects? Or does 
> their size (e.g. string length) affect GC pressure as well?

In general, both, since we have 2 GC thresholds, and GC is actually
done when both are exceeded.  So the effect will also depend on how
much Lisp memory is already allocated in the Emacs process where these
benchmarks are run.

> > This is consistent with the
> > fact that the GC times change only very little.  So I don't think I
> > see, on this level, what puzzles you in this picture.
> 
> Now that you pointed that out, the picture is just more puzzling. While 
> 0.1s in GC is not insignificant (it's 10% of the whole runtime), it does 
> seem to have been more of a fluke, and on average the fluctuations in GC 
> time are smaller.
> 
> Here's an extended comparison:
> 
> (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)")
>   ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)")
>   ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)")
>   ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)")
>   ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)")
>   ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)")
> ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)")
> ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)")
> ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)")
> ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)")
> ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)"
> ))
> 
> What was puzzling for me, overall, is that if we take "with-find 409600" 
> (the fastest among the asynchronous runs without parallelism) and 
> "with-find-sync", the difference in GC time (which is repeatable), 
> 0.66s, almost covers all the difference in performance. And as for 
> "with-find-p 409600", it would come out on top! Which it did in Ihor's 
> tests when GC was disabled.
> 
> But where does the extra GC time come from? Is it from extra consing in 
> the asynchronous call's case? If it is, it's not from all the chunked 
> strings, apparently, given that increasing max string's size (and 
> decreasing their number by 2x-6x, according to my logging) doesn't 
> affect the reported GC time much.
> 
> Could the extra time spent in GC just come from the fact that it's given 
> more opportunities to run, maybe? call_process stays entirely in C, 
> whereas make-process, with its asynchronous approach, goes between C and 
> Lisp even time it receives input. The report above might indicate so: 
> with-find-p have ~20 garbage collection cycles, whereas with-find-sync - 
> only ~10. Or could there be some other source of consing, unrelated to 
> the process output string, and how finely they are sliced?

These questions can only be answered by dumping the values of the 2 GC
thresholds and of consing_until_gc for each GC cycle.  It could be
that we are consing more Lisp memory, or it could be that one of the
implementations provides fewer opportunities for Emacs to call
maybe_gc.  Or it could be some combination of the two.

> If we get back to increasing read-process-output-max, which does help 
> (apparently due to reducing the number we switch between reading from 
> the process and doing... whatever else), the sweet spot seems to be 
> 1048576, which is my system's maximum value. Anything higher - and the 
> perf goes back to worse -- I'm guessing something somewhere resets the 
> value to default? Not sure why it doesn't clip to the maximum allowed, 
> though.
> 
> Anyway, it would be helpful to be able to decide on as high as possible 
> value without manually reading from /proc/sys/fs/pipe-max-size. And what 
> of other OSes?

Is this with pipes or with PTYs?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 08 Sep 2023 00:54:02 +0000
Resent-Message-ID: <handler.64735.B64735.16941344319761 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16941344319761
          (code B ref 64735); Fri, 08 Sep 2023 00:54:02 +0000
Received: (at 64735) by debbugs.gnu.org; 8 Sep 2023 00:53:51 +0000
Received: from localhost ([127.0.0.1]:41775 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qePks-0002XM-Vx
	for submit <at> debbugs.gnu.org; Thu, 07 Sep 2023 20:53:51 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:60991)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qePkq-0002X6-DE
 for 64735 <at> debbugs.gnu.org; Thu, 07 Sep 2023 20:53:49 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 43CA95C01AE;
 Thu,  7 Sep 2023 20:53:41 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute6.internal (MEProxy); Thu, 07 Sep 2023 20:53:41 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694134421; x=1694220821; bh=vEWUKj+DkYzcQEJei0wqPj0y5tPhO96g1TN
 Ff1lUa+k=; b=Zq4KNiWFWfKpHCq0oQfGGLYra1GUWY1LWvxjEleQk0PWG9bBJ3Z
 0++jG6rPKIga1a0VK/0A/zUJ+Yyeg+aJLb7lnz2kBDEh9xE5zheHKBDMjj7ZGJt1
 x/9ALLYeSxU58IQGAQw1aj2PGcmutmNRP/4somdRBh6q2rDSy9q3AIA0K9P2m88m
 Xb8aCEVLcVBxUTsb5SctrqDQQiSHcE3wEhfbxDdXmoUGTF+d/LkcIl0Hb5/Czb8y
 BxAF8p0sQla6ru1qjlNdJX6QITjRLSW6YPwBuVU2BkF4Df1nCZw1GtLz/RAPOnuO
 tsp6EefpGeZpyASn/akoRXRO8E8k4ZWM2Yg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694134421; x=1694220821; bh=vEWUKj+DkYzcQEJei0wqPj0y5tPhO96g1TN
 Ff1lUa+k=; b=hefmK20OOl0Bamt4J7mgijNrbm9/k7bGllggAAQZgM8kHmuphcQ
 gSrvu94LCQ7seAyMKtut3rmZZFz+XgTh59uNqQLRoXfwMz5YZAWfW/otKzIADleG
 qc8WhGK7EWZdCDGghT0RBPQvTYDRkbxwflfRt3w6D8ndQkfmUmR8nUROaURQ5vur
 w2IESNe7VDPaHyc2lFsEPyBjuWrwaIGMFXtwAqWYQikhg2AzmMIbet4mYGibI2uz
 2wEn4ixy9hWJXAS3/igV65j7sT4wa/rjsB5l1R9+BG/ugWK35WOfHgBh9jj1uoq9
 /D6f37BkiVvF60EJ1HhqL/6xGv5+slL5c8g==
X-ME-Sender: <xms:lHD6ZEsB-JEzmEYCxPpMsMFfktoRWxbmMjdoMwFFdveWYKH5iZDYng>
 <xme:lHD6ZBcL3Hoe8c8vc5TON4tkXqwVLU9OmG7HcmKgNiv94Sskwo44Gof65m2cdQOGT
 uGuX66-AvqfCgeQjLs>
X-ME-Received: <xmr:lHD6ZPy6OySssv8t2NR5HWN4tuoT-ui_RsYgUdB4-KP88DKJRsXj_FUEnePk7sk>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudehiedggeduucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhephfeutdekveeggeetteekfeejffegudduudfhueevleeftdffffeggeeivddv
 jeelnecuffhomhgrihhnpehgnhhurdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenuc
 frrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:lHD6ZHNxoEQsRZl6yXdv5Al60kdfHzn8itIvwzFI-EkB5C4kS6yH6g>
 <xmx:lHD6ZE9kheqsOOqY3O_jLqAwKbq5zRuawC138YUYH-FQGoMgqRUoVg>
 <xmx:lHD6ZPVLpS7uNaWRVlmkaJ8SLYr8Z-splI3DwCJlk2cKFyDG-bCAyQ>
 <xmx:lXD6ZDZbjbitJg-pKKQxuqApwiL9PbWFumloX3XQf9CKAbdmgavadA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu,
 7 Sep 2023 20:53:39 -0400 (EDT)
Message-ID: <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
Date: Fri, 8 Sep 2023 03:53:37 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83bkfs2tw5.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

Let's try to investigate this some more, if we can.

On 31/07/2023 14:38, Eli Zaretskii wrote:
>> Date: Sun, 30 Jul 2023 04:35:49 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> In this context, junks objects are objects that will need to be
>> collected by garbage collector very soon because they are just a
>> byproduct of a function's execution (but aren't used in the return
>> value, for example). The more of them a function creates, the more work
>> it will be, supposedly, for the GC.
>>
>> Heap is perhaps the wrong term (given that C has its own notion of
>> heap), but I meant the memory managed by the Lisp runtime.
>>
>> And chunks are the buffered strings that get passed to the process
>> filter. Chunks of the process' output. By default, these chunks are 4096
>> characters long, but the comparisons tweak that value by 10x and 100x.
> 
> If the subprocess output is inserted into a buffer, its effect on the
> GC will be different.  (Not sure if this is relevant to the issue at
> hand, as I lost track of the many variants of the function that were
> presented.)

Yes, one of the variants inserts into the buffer (one that uses a 
synchronous process call and also, coincidentally, spends less time in 
GC), and the asynchronous work from a process filter.

>>> If I read what you wrote superficially, without delving into the
>>> details (which I can't understand), you are saying that the overall
>>> amount of consing is roughly the same.
>>
>> What is "amount of consing"? Is it just the number of objects? Or does
>> their size (e.g. string length) affect GC pressure as well?
> 
> In general, both, since we have 2 GC thresholds, and GC is actually
> done when both are exceeded.  So the effect will also depend on how
> much Lisp memory is already allocated in the Emacs process where these
> benchmarks are run.

All right.

>>> This is consistent with the
>>> fact that the GC times change only very little.  So I don't think I
>>> see, on this level, what puzzles you in this picture.
>>
>> Now that you pointed that out, the picture is just more puzzling. While
>> 0.1s in GC is not insignificant (it's 10% of the whole runtime), it does
>> seem to have been more of a fluke, and on average the fluctuations in GC
>> time are smaller.
>>
>> Here's an extended comparison:
>>
>> (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)")
>>    ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)")
>>    ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)")
>>    ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)")
>>    ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)")
>>    ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)")
>> ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)")
>> ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)")
>> ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)")
>> ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)")
>> ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)"
>> ))
>>
>> What was puzzling for me, overall, is that if we take "with-find 409600"
>> (the fastest among the asynchronous runs without parallelism) and
>> "with-find-sync", the difference in GC time (which is repeatable),
>> 0.66s, almost covers all the difference in performance. And as for
>> "with-find-p 409600", it would come out on top! Which it did in Ihor's
>> tests when GC was disabled.
>>
>> But where does the extra GC time come from? Is it from extra consing in
>> the asynchronous call's case? If it is, it's not from all the chunked
>> strings, apparently, given that increasing max string's size (and
>> decreasing their number by 2x-6x, according to my logging) doesn't
>> affect the reported GC time much.
>>
>> Could the extra time spent in GC just come from the fact that it's given
>> more opportunities to run, maybe? call_process stays entirely in C,
>> whereas make-process, with its asynchronous approach, goes between C and
>> Lisp even time it receives input. The report above might indicate so:
>> with-find-p have ~20 garbage collection cycles, whereas with-find-sync -
>> only ~10. Or could there be some other source of consing, unrelated to
>> the process output string, and how finely they are sliced?
> 
> These questions can only be answered by dumping the values of the 2 GC
> thresholds and of consing_until_gc for each GC cycle.  It could be
> that we are consing more Lisp memory, or it could be that one of the
> implementations provides fewer opportunities for Emacs to call
> maybe_gc.  Or it could be some combination of the two.

Do you think the outputs of 
https://elpa.gnu.org/packages/emacs-gc-stats.html could help?

Otherwise, I suppose I need to add some fprintf's somewhere. Would the 
beginning of maybe_gc inside lisp.h be a good place for that?

>> If we get back to increasing read-process-output-max, which does help
>> (apparently due to reducing the number we switch between reading from
>> the process and doing... whatever else), the sweet spot seems to be
>> 1048576, which is my system's maximum value. Anything higher - and the
>> perf goes back to worse -- I'm guessing something somewhere resets the
>> value to default? Not sure why it doesn't clip to the maximum allowed,
>> though.
>>
>> Anyway, it would be helpful to be able to decide on as high as possible
>> value without manually reading from /proc/sys/fs/pipe-max-size. And what
>> of other OSes?
> 
> Is this with pipes or with PTYs?

All examples which use make-process call it with :connection-type 'pipe.

The one that calls process-file (the "synchronous" impl) also probably 
does, but I don't see that in the docstring.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 08 Sep 2023 06:37:01 +0000
Resent-Message-ID: <handler.64735.B64735.169415498411898 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169415498411898
          (code B ref 64735); Fri, 08 Sep 2023 06:37:01 +0000
Received: (at 64735) by debbugs.gnu.org; 8 Sep 2023 06:36:24 +0000
Received: from localhost ([127.0.0.1]:42030 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qeV6N-00035q-HJ
	for submit <at> debbugs.gnu.org; Fri, 08 Sep 2023 02:36:24 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:37808)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qeV6J-00035X-AY
 for 64735 <at> debbugs.gnu.org; Fri, 08 Sep 2023 02:36:21 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qeV6B-0007SI-7z; Fri, 08 Sep 2023 02:36:11 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=rlsXkYvfXeSdQet6A3eWOCfH2P3C49eUVv3ZL7mxz/0=; b=Szbrau3yHD2L
 OmDsJZmh7ebyYqOrRUkQN1zunnAZdgkkSD0Vejw15meINIrNwb8LWHP4gk0Kzl1UKqbeWoUMd5Q1J
 0Hc7Gqwi3rh0ya0pBUNBj8wxMx3dxVPwZOxFPIZprvDw5NtgekBhFrm/rxAzpLlvlhtPhuwMJ+oJv
 K3cJkYiXjPFi7Bjfn39awRHBobOhIOvIF/2Xl7ybo9+W1b8IWDbGwAZgW1/Vuzj6WExmPf9SdvoC5
 wmo7oZcTTXK7PBBIgW1rmjiQEJ0wRP6McchSFnfUp72U6js7Jo7whu4cBOT74DBMhZy7VHVNWGXsq
 +hJ4CQP09McT/kSujXSxRw==;
Date: Fri, 08 Sep 2023 09:35:45 +0300
Message-Id: <83il8lxjcu.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN> (message from
 Dmitry Gutov on Fri, 8 Sep 2023 03:53:37 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <5c4d9bea-3eb9-b262-138a-4ea0cb203436@HIDDEN>
 <87tttypp2e.fsf@localhost> <ierfs5iwlx3.fsf@HIDDEN>
 <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 8 Sep 2023 03:53:37 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> >> (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)")
> >>    ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)")
> >>    ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)")
> >>    ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)")
> >>    ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)")
> >>    ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)")
> >> ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)")
> >> ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)")
> >> ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)")
> >> ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)")
> >> ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)"
> >> ))
> >>
> >> What was puzzling for me, overall, is that if we take "with-find 409600"
> >> (the fastest among the asynchronous runs without parallelism) and
> >> "with-find-sync", the difference in GC time (which is repeatable),
> >> 0.66s, almost covers all the difference in performance. And as for
> >> "with-find-p 409600", it would come out on top! Which it did in Ihor's
> >> tests when GC was disabled.
> >>
> >> But where does the extra GC time come from? Is it from extra consing in
> >> the asynchronous call's case? If it is, it's not from all the chunked
> >> strings, apparently, given that increasing max string's size (and
> >> decreasing their number by 2x-6x, according to my logging) doesn't
> >> affect the reported GC time much.
> >>
> >> Could the extra time spent in GC just come from the fact that it's given
> >> more opportunities to run, maybe? call_process stays entirely in C,
> >> whereas make-process, with its asynchronous approach, goes between C and
> >> Lisp even time it receives input. The report above might indicate so:
> >> with-find-p have ~20 garbage collection cycles, whereas with-find-sync -
> >> only ~10. Or could there be some other source of consing, unrelated to
> >> the process output string, and how finely they are sliced?
> > 
> > These questions can only be answered by dumping the values of the 2 GC
> > thresholds and of consing_until_gc for each GC cycle.  It could be
> > that we are consing more Lisp memory, or it could be that one of the
> > implementations provides fewer opportunities for Emacs to call
> > maybe_gc.  Or it could be some combination of the two.
> 
> Do you think the outputs of 
> https://elpa.gnu.org/packages/emacs-gc-stats.html could help?

I think you'd need to expose consing_until_gc to Lisp, and then you
can collect the data from Lisp.

> Otherwise, I suppose I need to add some fprintf's somewhere. Would the 
> beginning of maybe_gc inside lisp.h be a good place for that?

I can only recommend the fprintf method if doing this from Lisp is
impossible for some reason.

> >> If we get back to increasing read-process-output-max, which does help
> >> (apparently due to reducing the number we switch between reading from
> >> the process and doing... whatever else), the sweet spot seems to be
> >> 1048576, which is my system's maximum value. Anything higher - and the
> >> perf goes back to worse -- I'm guessing something somewhere resets the
> >> value to default? Not sure why it doesn't clip to the maximum allowed,
> >> though.
> >>
> >> Anyway, it would be helpful to be able to decide on as high as possible
> >> value without manually reading from /proc/sys/fs/pipe-max-size. And what
> >> of other OSes?
> > 
> > Is this with pipes or with PTYs?
> 
> All examples which use make-process call it with :connection-type 'pipe.
> 
> The one that calls process-file (the "synchronous" impl) also probably 
> does, but I don't see that in the docstring.

Yes, call-process uses pipes.  So finding the optimum boils down to
running various scenarios.  It is also possible that the optimum will
be different on different systems, btw.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 10 Sep 2023 01:31:02 +0000
Resent-Message-ID: <handler.64735.B64735.169430944211934 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169430944211934
          (code B ref 64735); Sun, 10 Sep 2023 01:31:02 +0000
Received: (at 64735) by debbugs.gnu.org; 10 Sep 2023 01:30:42 +0000
Received: from localhost ([127.0.0.1]:48660 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qf9He-00036P-18
	for submit <at> debbugs.gnu.org; Sat, 09 Sep 2023 21:30:42 -0400
Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:59513)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qf9HZ-000368-Ah
 for 64735 <at> debbugs.gnu.org; Sat, 09 Sep 2023 21:30:40 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.west.internal (Postfix) with ESMTP id F2DF1320085B;
 Sat,  9 Sep 2023 21:30:27 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute3.internal (MEProxy); Sat, 09 Sep 2023 21:30:28 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694309427; x=1694395827; bh=cLJDhPyxPDrxRcQ1y2gfepKp3soinuXDF3w
 olOn9Ax0=; b=dXKbGqIif99K2SUjRssTAJom3y/Z7qJCu97wj5bhSEmeYU+yPwf
 SP//ePN5HoqD19JVHzF5oLlZWv76EIyVZnibE2YjzdC9VoIgVCmN5qgoCzAbI+8s
 g2LQ1Iqt8OwxPD20jtX8BytDRMOoehGBni5rlHv537pNpec3xdS+C8VStNJIEoFw
 3a8qcMr5i6YMHHuuKxYgbTceOS5KBzOvaaFaMSi6aRU7hrWp9b/tjhTtex4cDkmL
 +7Q7OG9x35k5N4HmhigwtyyeOOn5Uox5efqJFt03HMSXAaiMN/C9ilTyF2eSDNV6
 JLZDffQDVYgY9b+bwmKmksEx8hG4Cr+JvZg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694309427; x=1694395827; bh=cLJDhPyxPDrxRcQ1y2gfepKp3soinuXDF3w
 olOn9Ax0=; b=zPc36bjldH1Cdk9++EGP3pQBlrrpqxFJgsLaOg6JQ0K7gxLqPTh
 Eexs6XWkpyM5KnHnWpLyko9t/e3v6kQcJE/rruG/QnfumIPjKFX/Z7+jmyg3gZhz
 CDDbM4+ThysQAB7e2JjLgbejWQom1nN7TeMCRn/f3wLf6FwLdA1saFb01RlK32qv
 aQI8xzdEiFePMv1ksMO6H/mWljeJyYcx5cHroJUd3BcP9H+RUUzYZQ0YVrTC/5Jf
 644n7VPHgtiqsw4eLKvAITtaE8WKmUS7eKL+TApMKnZiZbo+nfGUirEarmaeWl6k
 efl1//x84fFn8vbEmUgDSUF8693dLxCAXIQ==
X-ME-Sender: <xms:Mxz9ZNWQbHo8zh5cllnb-ykHDAbRNMzBePwpHJPHHGPoxehSTPqQzw>
 <xme:Mxz9ZNkkqyy9DOQt_7dCXOfsj6A6rRXB7p9WOYyog1cyJjLHN_gxxxhtHBvEtsaYb
 MxFygpGfdJcLT3Bc2M>
X-ME-Received: <xmr:Mxz9ZJb-ZeW_qa9JTe2i_sYLvYf3KSwSMJ9qwZaI0JeqITw20IcXX-dVewfPvcM>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeitddggeefucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhephfeutdekveeggeetteekfeejffegudduudfhueevleeftdffffeggeeivddv
 jeelnecuffhomhgrihhnpehgnhhurdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenuc
 frrghrrghmpehmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:Mxz9ZAWWU1pWa0AEAaEd-6qczIjOTMJ8m70kIbUrw1AwoHOW3-22tg>
 <xmx:Mxz9ZHkiP22_7zvxhnYy3OBmZitfOHM5eRnE5IMeMZnEcXe-kBpxMQ>
 <xmx:Mxz9ZNf_THOAB0dVGbNcya4v3WXEwYGwsU1ckIW05ZnfyNUu72g39g>
 <xmx:Mxz9ZFjpjhjyFzXL97GzhwMc7CSqSfGv_HKQ5QesJqG7ncgYj6b9xQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat,
 9 Sep 2023 21:30:25 -0400 (EDT)
Message-ID: <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
Date: Sun, 10 Sep 2023 04:30:24 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83il8lxjcu.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 08/09/2023 09:35, Eli Zaretskii wrote:
>>> These questions can only be answered by dumping the values of the 2 GC
>>> thresholds and of consing_until_gc for each GC cycle.  It could be
>>> that we are consing more Lisp memory, or it could be that one of the
>>> implementations provides fewer opportunities for Emacs to call
>>> maybe_gc.  Or it could be some combination of the two.
>> Do you think the outputs of
>> https://elpa.gnu.org/packages/emacs-gc-stats.html  could help?
> I think you'd need to expose consing_until_gc to Lisp, and then you
> can collect the data from Lisp.

I can expose it to Lisp and print all three from post-gc-hook, but the 
result just looks like this:

gc-pct 0.1 gc-thr 800000 cugc 4611686018427387903

Perhaps I need to add a hook which runs at the beginning of GC? Or of 
maybe_gc even?

Alternatively, (memory-use-counts) seems to retain some counters which 
don't get erased during garbage collection.

>> All examples which use make-process call it with :connection-type 'pipe.
>>
>> The one that calls process-file (the "synchronous" impl) also probably
>> does, but I don't see that in the docstring.
> Yes, call-process uses pipes.  So finding the optimum boils down to
> running various scenarios.  It is also possible that the optimum will
> be different on different systems, btw.

Sure, but I'd like to improve the state of affairs in at least the main one.

And as for MS Windows, IIRC all find-based solution are currently slow 
equally, so we're unlikely to make things worse there anyway.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 10 Sep 2023 05:35:02 +0000
Resent-Message-ID: <handler.64735.B64735.16943240423264 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16943240423264
          (code B ref 64735); Sun, 10 Sep 2023 05:35:02 +0000
Received: (at 64735) by debbugs.gnu.org; 10 Sep 2023 05:34:02 +0000
Received: from localhost ([127.0.0.1]:48783 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfD58-0000qZ-7o
	for submit <at> debbugs.gnu.org; Sun, 10 Sep 2023 01:34:02 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:40588)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qfD54-0000q4-T3
 for 64735 <at> debbugs.gnu.org; Sun, 10 Sep 2023 01:34:00 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qfD4v-0003Nz-IN; Sun, 10 Sep 2023 01:33:49 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=B5RnSeg5gnFEVm4o4/zT3TKzHS+gtBC+WlH+nPO64a4=; b=blgjFS5FZIEW
 7eUO9KoUjpQc3PmNMTBxDD8JJ/KLoDbeOtZFFp7w8tTe8Jjp60SVEYzep9W7KSVr4PBLvFqABlGrJ
 AabZbT/zZti8SUDxV4i8Fn7TiTsKAHqp/yHI4HhLCSZ0f8rj0k2ECD7TUc6nT0Me+vVWokIzGOtlr
 sjAOGANg7KkPvJGs01L90ZST18x7Tgqt8tmDC0eHQYI+3oVMFJhPyw9FhHkRKLCrPHJDrvZ1zfZlw
 mEPVvkVhrhSoCtIwCXywPtFIqvzUWbCfjG2/QMRPzfCNVWYig25f61nF8AWiIDQI7Srf/tBRUvO03
 P5qYrRHngC8GBhTthom6sA==;
Date: Sun, 10 Sep 2023 08:33:33 +0300
Message-Id: <8334zmtwwi.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN> (message from
 Dmitry Gutov on Sun, 10 Sep 2023 04:30:24 +0300)
References: <iermszrwqj4.fsf@HIDDEN> <87tttypp2e.fsf@localhost>
 <ierfs5iwlx3.fsf@HIDDEN> <87r0p030w0.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN> <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Sun, 10 Sep 2023 04:30:24 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> On 08/09/2023 09:35, Eli Zaretskii wrote:
> > I think you'd need to expose consing_until_gc to Lisp, and then you
> > can collect the data from Lisp.
> 
> I can expose it to Lisp and print all three from post-gc-hook, but the 
> result just looks like this:
> 
> gc-pct 0.1 gc-thr 800000 cugc 4611686018427387903
> 
> Perhaps I need to add a hook which runs at the beginning of GC? Or of 
> maybe_gc even?

You could record its value in a local variable at the entry to
garbage_collect, and the expose that value to Lisp.

> Alternatively, (memory-use-counts) seems to retain some counters which 
> don't get erased during garbage collection.

Maybe using those will be good enough, indeed.

> And as for MS Windows, IIRC all find-based solution are currently slow 
> equally, so we're unlikely to make things worse there anyway.

I was actually thinking about *BSD and macOS.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 11 Sep 2023 00:04:01 +0000
Resent-Message-ID: <handler.64735.B64735.16943905938808 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16943905938808
          (code B ref 64735); Mon, 11 Sep 2023 00:04:01 +0000
Received: (at 64735) by debbugs.gnu.org; 11 Sep 2023 00:03:13 +0000
Received: from localhost ([127.0.0.1]:51546 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfUOW-0002Hz-L2
	for submit <at> debbugs.gnu.org; Sun, 10 Sep 2023 20:03:13 -0400
Received: from out3-smtp.messagingengine.com ([66.111.4.27]:42585)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qfUOR-0002Hj-LQ
 for 64735 <at> debbugs.gnu.org; Sun, 10 Sep 2023 20:03:10 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 801EA5C0067;
 Sun, 10 Sep 2023 20:02:58 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute6.internal (MEProxy); Sun, 10 Sep 2023 20:02:58 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-type:content-type:date:date:from:from:in-reply-to
 :in-reply-to:message-id:mime-version:references:reply-to:sender
 :subject:subject:to:to; s=fm3; t=1694390578; x=1694476978; bh=fL
 aeDQZpvI2i8RppxU+bKtKe1kdbtfqOkUEZdEflxMk=; b=rwqJdtXVmdI+kt+I+H
 7FZ2dGe68GkUjA5uywEoXFwC2T/skhew4EuuV77lH7JWwDVNlkBI40uLMCiwsJGo
 hWz/iqLFaopwgaZxdAnnuuUezEBZMJtn8oUN0qMOd3RhgXHlw5xtv95MHnC8CqlE
 RSVLvp7VahwfpHK+g5zh+ZxlZXXqi9r3u41OB8yrnQnQy3IiLqelQUGZ0shBjo3p
 IwnjBxhLNEcc2+oWhWCu5l/4e4hvKD/InX2BlBpdgT+/y/alzu6J8vZ6kbKQUAmC
 IXi9RZA7zDa6R5Fax7CY2lRVDKUBYQQCJdPPt3ZICwaC1+SYYrFyyy5axNZEWT6t
 Ga0g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:content-type:date:date
 :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:sender:subject
 :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm1; t=1694390578; x=1694476978; bh=fLaeDQZpvI2i8
 RppxU+bKtKe1kdbtfqOkUEZdEflxMk=; b=JSys0Nq1YX2/n65YFOHX3m4xl3iQC
 dT0AbL3ZKkfOpZSoJ64YyayLCHM3Z3InFlh9Pp7uOjtZVU0//4i/3f+cchL1JsoJ
 1sf8YKdLn0fphwWkPK78oiY5YbH16K3dxeaj4AtlqtZYg695hjUym1VR+WjTwM0g
 q0kvurBHieI/K2pa1SYuSvZyLwDsK9EV1PcUv14r71XK0Gwfx7cpE+8b/6dseMoH
 UAkyEy5nsW/VxhWM2uEqvcW/iabH+ndXukVqXGoq9Ph7ULlXKGHuUDlSFxLm2HoQ
 BUCVUjVg3xHCURJH8Th0zcO8gM4yHIxBug0mOb2n6RwdtBlqPv8f7fc/g==
X-ME-Sender: <xms:MVn-ZIHtGrSzxVxTNWrT1NrA-ZjN3bPUKfq7atRgdczeQINbzMn2kg>
 <xme:MVn-ZBUiu_mT0UA4U2J1g6LhAyBilvW8dnNYnvJoC-M2dNozboAA6p1xSuYa_-oPS
 o-HKV4ZzIC1c5ByDEI>
X-ME-Received: <xmr:MVn-ZCK5CA5-ZE7TZL1DHXAHqvexjUWNK78P6ev775XwZwiyzICePdmHxSBuNSU>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeifedgvdelucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurheptgfkffggfgfuvfevfhfhjgesmhdtreertdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeehteekgfetieeujeeuvddtvdelteffleejteduvdefffejieehheeuteffveei
 jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:MVn-ZKHLRjx_Ibfc1W49BXSHEpVIjqWxrmX9afbGiBA7v9ev-IzXTQ>
 <xmx:MVn-ZOUdx9f-sFdnFBja6aSOEzmnb8FUHnz3VIWGo2mJkHJ0MKUyvg>
 <xmx:MVn-ZNMUpmq6yzvtGu8dhxE5nO8HbUouBWID5TgZQbqT_KDsDSS69A>
 <xmx:Mln-ZNQhf9tPvYfzGSqkxlrJ5yDmFMYtO7DKOFLqi3PQmkNYAvAj7A>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun,
 10 Sep 2023 20:02:56 -0400 (EDT)
Content-Type: multipart/mixed; boundary="------------mo9wiY0Ph8bca03YUu6cB50g"
Message-ID: <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
Date: Mon, 11 Sep 2023 03:02:55 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <8334zmtwwi.fsf@HIDDEN>
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

This is a multi-part message in MIME format.
--------------mo9wiY0Ph8bca03YUu6cB50g
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 10/09/2023 08:33, Eli Zaretskii wrote:
>> Date: Sun, 10 Sep 2023 04:30:24 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> On 08/09/2023 09:35, Eli Zaretskii wrote:
>>> I think you'd need to expose consing_until_gc to Lisp, and then you
>>> can collect the data from Lisp.
>>
>> I can expose it to Lisp and print all three from post-gc-hook, but the
>> result just looks like this:
>>
>> gc-pct 0.1 gc-thr 800000 cugc 4611686018427387903
>>
>> Perhaps I need to add a hook which runs at the beginning of GC? Or of
>> maybe_gc even?
> 
> You could record its value in a local variable at the entry to
> garbage_collect, and the expose that value to Lisp.

That also doesn't seem to give much, given that the condition for 
entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait 
until it's down to 0, then garbage-collect. What we could perhaps do is 
add another hook (or a printer) at the beginning of maybe_gc, but either 
would result in lots and lots of output.

>> Alternatively, (memory-use-counts) seems to retain some counters which
>> don't get erased during garbage collection.
> 
> Maybe using those will be good enough, indeed.

I added this instrumentation:

(defvar last-mem-counts '(0 0 0 0 0 0 0))

(defun gc-record-after ()
   (let* ((counts (memory-use-counts))
          (diff (cl-map 'list
                        (lambda (old new) (- new old))
                        last-mem-counts counts)))
     (setq last-mem-counts counts)
     (message "counts diff %s" diff)))

(add-hook 'post-gc-hook #'gc-record-after)

so that after each garbage collection we print the differences in all 
counters (CONSES FLOATS VECTOR-CELLS SYMBOLS STRING-CHARS INTERVALS 
STRINGS).

And a message call when the process finishes.

And made those recordings during the benchmark runs of two different 
listing methods (one using make-process, another using process-file) to 
list all files in a large directory (there are ~200000 files there).

The make-process one I also ran with a different (large) value of 
read-process-output-max. Results attached.

What's in there? First of all, for find-directory-files-recursively-3, 
there are 0 garbage collections between the beginning of the function 
and when we start parsing the output (no GCs while the process is 
writing to the buffer synchronously). I guess inserting output in a 
buffer doesn't increase consing, so there's nothing to GC?

Next: for find-directory-files-recursively-2, the process only finishes 
at the end, when all GC cycles are done for. I suppose that also means 
we block the process's output while Lisp is running, and also that 
whatever GC events occur might coincide with the chunks of output coming 
from the process, and however many of them turn out to be in total.

So there is also a second recording for 
find-directory-files-recursively-2 with read-process-output-max=409600. 
It does improve the performance significantly (and reduce the number of 
GC pauses). I guess what I'm still not clear on, is whether the number 
of GC pauses is fewer because of less consing (the only column that 
looks significantly different is the 3rd: VECTOR-CELLS), or because the 
process finishes faster due to larger buffers, which itself causes fewer 
calls to maybe_gc.

And, of course, what else could be done to reduce the time spent in GC 
in the asynchronous case.
--------------mo9wiY0Ph8bca03YUu6cB50g
Content-Type: text/plain; charset=UTF-8; name="gcs2.txt"
Content-Disposition: attachment; filename="gcs2.txt"
Content-Transfer-Encoding: base64

ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMgoKVXNlcyBtYWtlLXByb2Nlc3Mg
YW5kIDpmaWx0ZXIgdG8gcGFyc2UgdGhlIG91dHB1dApjb25jdXJyZW50bHkgd2l0aCB0aGUg
cHJvY2Vzcy4KCldpdGggKHJlYWQtcHJvY2Vzcy1vdXRwdXQtbWF4IDQwOTYpOgoKc3RhcnQg
bm93CmNvdW50cyBkaWZmICg3NTg0MCAxMyAzMTE3NyA2MCAzNDM0NDMgMzQ5NiA0NzQ4KQpj
b3VudHMgZGlmZiAoNDE5NDYgMSA0NjAgMCAxMjI2NDk0IDAgODQyNSkKY291bnRzIGRpZmYg
KDQzMTY1IDEgNDUwIDAgMTI4NDIxNCAwIDg5NTEpCmNvdW50cyBkaWZmICg0MzUxMyAxIDM2
NCAwIDEzNDMzMTYgMCAxMDEyNSkKY291bnRzIGRpZmYgKDQzMjAwIDEgMzg0IDAgMTQ3OTA0
OCAwIDk3NjYpCmNvdW50cyBkaWZmICg0NjIyMCAxIDQyOCAwIDE1Mjg4NjMgMCAxMDI0MikK
Y291bnRzIGRpZmYgKDQzMTI1IDEgNDYyIDAgMTc2NzA2OCAwIDg3OTApCmNvdW50cyBkaWZm
ICg0OTExOCAxIDQ1OCAwIDE3MjMyNzEgMCAxMDgzMikKY291bnRzIGRpZmYgKDUzMTU2IDEg
NTcyIDAgMTc4OTkxOSAwIDEwNzc0KQpjb3VudHMgZGlmZiAoNTc3NTUgMSA1NDggMCAxNzgz
Mjg2IDAgMTI2MDApCmNvdW50cyBkaWZmICg2MjE3MSAxIDU1NCAwIDE3OTUyMTYgMCAxMzk5
NSkKY291bnRzIGRpZmYgKDYyMDIwIDEgNTUwIDAgMTk2MzI1NSAwIDEzOTk2KQpjb3VudHMg
ZGlmZiAoNTQ1NTkgMSA2MTYgMCAyMzg3MzA4IDAgMTA3MDApCmNvdW50cyBkaWZmICg1NjQy
OCAxIDYzNCAwIDI1MTMyMTkgMCAxMTA5NSkKY291bnRzIGRpZmYgKDYyNjExIDEgNjU4IDAg
MjUxMDc1NiAwIDEyODY0KQpjb3VudHMgZGlmZiAoNjc1NjAgMSA3MDggMCAyNTc0MzEyIDAg
MTM4OTkpCmNvdW50cyBkaWZmICg3ODE1NCAxIDkyOCAwIDI1NzIyNzMgMCAxNDcxNCkKY291
bnRzIGRpZmYgKDg2Nzk0IDEgOTc2IDAgMjUyMDkxNSAwIDE3MDA0KQpjb3VudHMgZGlmZiAo
NzgxMTIgMSA4NzQgMCAyOTQzNTQ4IDAgMTUzNjcpCmNvdW50cyBkaWZmICg3OTQ0MyAxIDg5
NCAwIDMxMzg5NDggMCAxNTU1OSkKY291bnRzIGRpZmYgKDgxODYxIDEgOTg0IDAgMzM0Mzc2
NCAwIDE1MjYwKQpjb3VudHMgZGlmZiAoODc3MjQgMSAxMDMwIDAgMzQzMDk2OSAwIDE2NjUw
KQpjb3VudHMgZGlmZiAoODg1MzIgMSA5MDIgMCAzNTkxMDUyIDAgMTg0ODcpCmNvdW50cyBk
aWZmICg5MjA4MyAxIDk1MiAwIDM3NjkyOTAgMCAxOTA2NSkKPGZpbmlzaGVkXG4+CkVsYXBz
ZWQgdGltZTogMS4zNDQ0MjJzICgwLjc0NzEyNnMgaW4gMjQgR0NzKQoKQW5kIGhlcmUncyB3
aXRoIChyZWFkLXByb2Nlc3Mtb3V0cHV0LW1heCA0MDk2MDApOgoKc3RhcnQgbm93CmNvdW50
cyBkaWZmICg1Nzk2NyAxIDQwNDAgMSA5ODE5MTIgMTA2IDc3MzEpCmNvdW50cyBkaWZmICgz
MjA3NSAxIDIwIDAgMTkxOTA5NiAwIDEwNTYwKQpjb3VudHMgZGlmZiAoNDM0MzEgMSAxOCAw
IDIyNTkzMTQgMCAxNDM3MSkKY291bnRzIGRpZmYgKDQ2MzM1IDEgMTggMCAyNDI2MjkwIDAg
MTUzMzkpCmNvdW50cyBkaWZmICgzMTg3MiAxIDE4IDAgMjQ0NzYzOSAwIDEwNTE4KQpjb3Vu
dHMgZGlmZiAoNDY1MjcgMSAxOCAwIDIzMjgwNDIgMCAxNTQwMykKY291bnRzIGRpZmYgKDQy
NDY4IDEgMTggMCAyMDk5OTc2IDAgMTQwNTApCmNvdW50cyBkaWZmICg0ODY0OCAxIDE4IDAg
MjMwMjcxMyAwIDE2MTEwKQpjb3VudHMgZGlmZiAoNTA0MDQgMSAyMCAwIDMyNjA5MjEgMCAx
NjY2OSkKY291bnRzIGRpZmYgKDQwMTQ3IDEgMjAgMCAzMjY0NDYzIDAgMTMyNTEpCmNvdW50
cyBkaWZmICg0ODExOCAxIDIwIDAgMzI2MTcyNSAwIDE1OTA4KQpjb3VudHMgZGlmZiAoNjA3
MzIgMSAyODIgMCAyNzkxMDAzIDAgMTY3ODUpCmNvdW50cyBkaWZmICg3MTMyOSAxIDUwNiAw
IDI3NjIyMzcgMCAxNzQ4NykKY291bnRzIGRpZmYgKDYxNDU1IDEgMzQyIDAgMzE5Mjc3MSAw
IDE2MjcxKQpjb3VudHMgZGlmZiAoNDkwMzUgMSAzMCAwIDM2NjM3MTUgMCAxNjA4NSkKY291
bnRzIGRpZmYgKDU4NjUxIDEgMjM2IDAgMzc4Mzg4OCAwIDE2NjgzKQpjb3VudHMgZGlmZiAo
NTcxMzIgMSAyNCAwIDQ1NTc2ODggMCAxODg2MikKY291bnRzIGRpZmYgKDcxMzE5IDEgMjQg
MCA0NzY5ODkxIDAgMjM1OTEpCjxmaW5pc2hlZFxuPgpFbGFwc2VkIHRpbWU6IDAuODkwNzEw
cyAoMC41NDY0ODZzIGluIDE4IEdDcykK
--------------mo9wiY0Ph8bca03YUu6cB50g
Content-Type: text/plain; charset=UTF-8; name="gcs3.txt"
Content-Disposition: attachment; filename="gcs3.txt"
Content-Transfer-Encoding: base64

ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMwoKVXNlcyBwcm9jZXNzLWZpbGUs
IHBhcnNlcyB0aGUgYnVmZmVyCndpdGggc2VhcmNoLWZvcndhcmQgYXQgdGhlIGVuZC4KCnN0
YXJ0IG5vdwo8cHJvY2VzcyBmaW5pc2hlZCwgbm93IHBhcnNpbmc+CmNvdW50cyBkaWZmICg2
Mjc3MSA1IDI2NjI5IDYzIDQ1ODIxMSAzMjIzIDgwMzgpCmNvdW50cyBkaWZmICgxNzA0NSAx
IDEyIDAgMTI4ODE1MyAwIDE2OTQ5KQpjb3VudHMgZGlmZiAoMTgzMDEgMSAxMiAwIDE0MzIx
NjUgMCAxODIwNSkKY291bnRzIGRpZmYgKDE3NjQzIDEgMTIgMCAxNzE2Mjk0IDAgMTc1NDcp
CmNvdW50cyBkaWZmICgyMTkxNyAxIDEyIDAgMTcyNjQ2MiAwIDIxODIxKQpjb3VudHMgZGlm
ZiAoMjU4ODggMSAxMiAwIDE3NzczNzEgMCAyNTc5MikKY291bnRzIGRpZmYgKDIxNzQzIDEg
MTIgMCAyMzQ1MTQzIDAgMjE2NDcpCmNvdW50cyBkaWZmICgyNDAzNSAxIDEyIDAgMjU2MTQ5
MSAwIDIzOTM5KQpjb3VudHMgZGlmZiAoMzAwMjggMSAxMiAwIDI1OTMwNjkgMCAyOTkzMikK
Y291bnRzIGRpZmYgKDI5NjI3IDEgMTIgMCAzMDQxMzA3IDAgMjk1MzEpCmNvdW50cyBkaWZm
ICgzMDE0MCAxIDEyIDAgMzQ3OTIwOSAwIDMwMDQ0KQpjb3VudHMgZGlmZiAoMzUxODEgMSAx
MiAwIDM2OTA0ODAgMCAzNTA4NSkKRWxhcHNlZCB0aW1lOiAwLjk0MzA5MHMgKDAuMzUxNzk5
cyBpbiAxMiBHQ3MpCg==

--------------mo9wiY0Ph8bca03YUu6cB50g--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 11 Sep 2023 11:58:02 +0000
Resent-Message-ID: <handler.64735.B64735.16944334629578 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16944334629578
          (code B ref 64735); Mon, 11 Sep 2023 11:58:02 +0000
Received: (at 64735) by debbugs.gnu.org; 11 Sep 2023 11:57:42 +0000
Received: from localhost ([127.0.0.1]:52273 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qffXx-0002UP-Ka
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:57:42 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:43040)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qffXu-0002U9-Id
 for 64735 <at> debbugs.gnu.org; Mon, 11 Sep 2023 07:57:40 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qffXk-0004cx-5q; Mon, 11 Sep 2023 07:57:28 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=ApsTiErd2jgr2dnSg7gSl250EvRb98aN/jZKQ2h1of8=; b=UE7elJ/Jjkdl
 5X2B2ZvL0opu+nsl9Zxwl0T9oEoO7fI8Ke9PraQkw2MWMruj8HSEAq/57pIT2bsGCww06y/r0s11U
 IK0BfXCG/nPX8A1QTg5et86IUg9NA10qFJlAXrYFKsMfi6/Vak92JQ5vjF1HAt0zk/rmBPdCNzUAK
 rYgFc5H/ElKrmO+9Hx90QxG9ZwnexQjMKkm/ip7jTupL3JTytlg36/hsRwBYhcgLWnfi878vgxr6L
 rf3vHs1S6CltmFkGQoPoRCP4HFKHXNDb24JIY+Y2uLU/362rQDxEfsyjYV23HLN4vV9vlN3tpX5v3
 Xm+POh5xKWqI+onV57/DGg==;
Date: Mon, 11 Sep 2023 14:57:10 +0300
Message-Id: <83tts0rkh5.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN> (message from
 Dmitry Gutov on Mon, 11 Sep 2023 03:02:55 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <fde3f1c3-628f-591d-4c52-87104b87c10a@HIDDEN>
 <83sf9f6wm0.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN> <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Mon, 11 Sep 2023 03:02:55 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > You could record its value in a local variable at the entry to
> > garbage_collect, and the expose that value to Lisp.
> 
> That also doesn't seem to give much, given that the condition for 
> entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait 
> until it's down to 0, then garbage-collect.

No, we don't wait until it's zero, we perform GC on the first
opportunity that we _notice_ that it crossed zero.  So examining how
negative is the value of consing_until_gc when GC is actually
performed could tell us whether we checked the threshold with high
enough frequency, and comparing these values between different runs
could tell us whether the shorter time spend in GC means really less
garbage or less frequent checks for the need to GC.

> What's in there? First of all, for find-directory-files-recursively-3, 
> there are 0 garbage collections between the beginning of the function 
> and when we start parsing the output (no GCs while the process is 
> writing to the buffer synchronously). I guess inserting output in a 
> buffer doesn't increase consing, so there's nothing to GC?

No, we just don't count increasing size of buffer text in the "consing
since GC" counter.  Basically, buffer text is never "garbage", except
when a buffer is killed.

> Next: for find-directory-files-recursively-2, the process only finishes 
> at the end, when all GC cycles are done for. I suppose that also means 
> we block the process's output while Lisp is running, and also that 
> whatever GC events occur might coincide with the chunks of output coming 
> from the process, and however many of them turn out to be in total.

We don't block the process when GC runs.  We do stop reading from the
process, so if and when the pipe fills, the OS will block the process.

> So there is also a second recording for 
> find-directory-files-recursively-2 with read-process-output-max=409600. 
> It does improve the performance significantly (and reduce the number of 
> GC pauses). I guess what I'm still not clear on, is whether the number 
> of GC pauses is fewer because of less consing (the only column that 
> looks significantly different is the 3rd: VECTOR-CELLS), or because the 
> process finishes faster due to larger buffers, which itself causes fewer 
> calls to maybe_gc.

I think the latter.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 11 Sep 2023 23:08:01 +0000
Resent-Message-ID: <handler.64735.B64735.16944736279702 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16944736279702
          (code B ref 64735); Mon, 11 Sep 2023 23:08:01 +0000
Received: (at 64735) by debbugs.gnu.org; 11 Sep 2023 23:07:07 +0000
Received: from localhost ([127.0.0.1]:55087 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qfpzm-0002WQ-F8
	for submit <at> debbugs.gnu.org; Mon, 11 Sep 2023 19:07:07 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:59347)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qfpzi-0002Vt-Os
 for 64735 <at> debbugs.gnu.org; Mon, 11 Sep 2023 19:07:04 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id 51A0A5C00C3;
 Mon, 11 Sep 2023 19:06:53 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Mon, 11 Sep 2023 19:06:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-type:content-type:date:date:from:from:in-reply-to
 :in-reply-to:message-id:mime-version:references:reply-to:sender
 :subject:subject:to:to; s=fm3; t=1694473613; x=1694560013; bh=0F
 hjaCLYH0J+BiznP34bJ0U1c7h84T2s2gXbd3fQUA0=; b=EQB0CXLV84MNmvrHWA
 P8M2DtSRAJbL36478imu7qFYttgDXebCq+papZYPQVgHqOS9x2CBpXZPGz1+QfHn
 rHAG4v1hafj3Q6Ig3SyiVPZn69uW8VCFCxW9+gC+5BcY/jgRboh188zliSchVuIt
 Qxnj70V5eG6reFBAaUKpz2Zh/DFoJvPKwOhY6kFNJuugiBTi/is/MSI2S92H9sLy
 hpKLeeEpxnicKCraM9RBDje9VMPUOWoKmvabjNiVUJrE73K+42h29fOx0l8HKcGc
 10VOw9+ZNqZkgYzFUUrOK2kEzlCWzr5pe0Xk9+ICExcvvIftCfywf9DngxDHxnEz
 ENtg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:content-type:date:date
 :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:sender:subject
 :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm1; t=1694473613; x=1694560013; bh=0FhjaCLYH0J+B
 iznP34bJ0U1c7h84T2s2gXbd3fQUA0=; b=EfHaOo8SH2cS4tiE2YeR9C0jgaYG/
 EnPlQ8hWL26KTGflOndIh3/P8473razPBYM91orNjqX8hWTYlgd9eDN7eXM9FbDH
 mZ+OxbBLQqC8AzmeUSSdJsZIcETU/XRgppB0/i19jFx3h0UXC3eHQElLfUg1Aehd
 CH1Sdt0noTs1n7yDmRNrZ7/QkNDpmi4hYmxlSmkOlDgUCGG132bmXyEOCXVFWTL4
 kp+U30Gg9XryKco1cpuzVpZB7JSMCwEtjOSGT+U2avNoL3tFls1iukibD3Aobgwx
 Wu64Q4j056KH1b0umSgZA2QBlktUYKmECnYO5FLBN97s8cpESuyJfNhWg==
X-ME-Sender: <xms:jJ3_ZEtIl2AVBLG9ncZ9JE31ezrTL-cBrdB2uxIPmk62LJv3SKmHaA>
 <xme:jJ3_ZBdGdwJYxr3XjlqyWEZ1AVczMXH1KbEX7Wq4YdpWoQfkPNSvthqbEbG3ROTaI
 b2pX1Nui3CVLtjAUUM>
X-ME-Received: <xmr:jJ3_ZPzZVbyFTybKsjN_xsJtKn-uSA2fJxWjCThrexSQN1uSZ4ppvEh_Ifqhbh8>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeihedgudejucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurheptgfkffggfgfuvfevfhfhjgesmhdtreertdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeehteekgfetieeujeeuvddtvdelteffleejteduvdefffejieehheeuteffveei
 jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:jZ3_ZHOkmROmp37frTj1p2-GAPTOqZ6WJmZKL33YBCZ_lbiMzBOfGA>
 <xmx:jZ3_ZE8bcvLcf0NGmASesW_v96hVdjysdmPj9RRMasy1kUbfEl8I6g>
 <xmx:jZ3_ZPU22e7nNduE4jYv0YMXqIEpMHHAxTEvE_dP0TC6BvxIm-71sg>
 <xmx:jZ3_ZDamM7WPmYrAsk5T0TW1CkABaNfbgJ1HfcBSEV778uRyFXHS3Q>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon,
 11 Sep 2023 19:06:51 -0400 (EDT)
Content-Type: multipart/mixed; boundary="------------I3MK6f5qXVUuZUbk3Nf06WpH"
Message-ID: <14b26ff0-9d9b-3600-bf6e-133db4031622@HIDDEN>
Date: Tue, 12 Sep 2023 02:06:50 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83tts0rkh5.fsf@HIDDEN>
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

This is a multi-part message in MIME format.
--------------I3MK6f5qXVUuZUbk3Nf06WpH
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 11/09/2023 14:57, Eli Zaretskii wrote:
>> Date: Mon, 11 Sep 2023 03:02:55 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>> You could record its value in a local variable at the entry to
>>> garbage_collect, and the expose that value to Lisp.
>>
>> That also doesn't seem to give much, given that the condition for
>> entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait
>> until it's down to 0, then garbage-collect.
> 
> No, we don't wait until it's zero, we perform GC on the first
> opportunity that we _notice_ that it crossed zero.  So examining how
> negative is the value of consing_until_gc when GC is actually
> performed could tell us whether we checked the threshold with high
> enough frequency, and comparing these values between different runs
> could tell us whether the shorter time spend in GC means really less
> garbage or less frequent checks for the need to GC.

Good point, I'm attaching the same outputs with "last value of 
consing_until_gc" added to every line.

There are some pretty low values in the "read-process-output-max 409600" 
part of the experiment, which probably means runtime staying in C 
accumulating the output into the (now larger) buffer? Not sure.

>> What's in there? First of all, for find-directory-files-recursively-3,
>> there are 0 garbage collections between the beginning of the function
>> and when we start parsing the output (no GCs while the process is
>> writing to the buffer synchronously). I guess inserting output in a
>> buffer doesn't increase consing, so there's nothing to GC?
> 
> No, we just don't count increasing size of buffer text in the "consing
> since GC" counter.  Basically, buffer text is never "garbage", except
> when a buffer is killed.

That makes sense. Perhaps it hints at a faster design for calling 
process asynchronously as well (more on another experiment later).

>> Next: for find-directory-files-recursively-2, the process only finishes
>> at the end, when all GC cycles are done for. I suppose that also means
>> we block the process's output while Lisp is running, and also that
>> whatever GC events occur might coincide with the chunks of output coming
>> from the process, and however many of them turn out to be in total.
> 
> We don't block the process when GC runs.  We do stop reading from the
> process, so if and when the pipe fills, the OS will block the process.

Right. But the effect is almost the same, including the potential 
side-effect that (IIUC) when a process is waiting like that, it's just 
suspended and not rushing ahead using the CPU/disk/etc resources to the 
max. That's an orthogonal train of thought, sorry.
--------------I3MK6f5qXVUuZUbk3Nf06WpH
Content-Type: text/plain; charset=UTF-8; name="gcs2b.txt"
Content-Disposition: attachment; filename="gcs2b.txt"
Content-Transfer-Encoding: base64

ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMgoKVXNlcyBtYWtlLXByb2Nlc3Mg
YW5kIDpmaWx0ZXIgdG8gcGFyc2UgdGhlIG91dHB1dApjb25jdXJyZW50bHkgd2l0aCB0aGUg
cHJvY2Vzcy4KCldpdGggKHJlYWQtcHJvY2Vzcy1vdXRwdXQtbWF4IDQwOTYpOgoKc3RhcnQg
bm93CmN1Z2MgLTI1NjAgY291bnRzIGRpZmYgKDQyMzQ1IDMgMTQwOTcgMiA0MDg1ODMgMTk2
NiA0NDc5KQpjdWdjIC00NDQ5IGNvdW50cyBkaWZmICgyNDU5OSAxIDM0MiAwIDg4MzI0NyAw
IDYyNjYpCmN1Z2MgLTEwMCBjb3VudHMgZGlmZiAoMjQwNzAgMSAzNTQgMCA5NzczODcgMCA2
MDA5KQpjdWdjIC0xMTYgY291bnRzIGRpZmYgKDI3MjY2IDEgMjc4IDAgOTQwNzIzIDAgNzQ4
NSkKY3VnYyAtOTUgY291bnRzIGRpZmYgKDI3NDg2IDEgMjcwIDAgMTAxNDU5MSAwIDc1ODYp
CmN1Z2MgLTExNyBjb3VudHMgZGlmZiAoMjcxNTcgMSAyOTQgMCAxMTIxMDY1IDAgNzMyOSkK
Y3VnYyAtMTQ2IGNvdW50cyBkaWZmICgyODIzMyAxIDMxNiAwIDExODU1MjcgMCA3NTYyKQpj
dWdjIC0xNDMgY291bnRzIGRpZmYgKDMwNTk3IDEgMzU0IDAgMTIxNzMyMCAwIDgxNDcpCmN1
Z2MgLTQ4MDcgY291bnRzIGRpZmYgKDI1OTI1IDEgMzgwIDAgMTQ3NDYxOCAwIDY0MDcpCmN1
Z2MgLTEyNyBjb3VudHMgZGlmZiAoMzMzNDQgMSAzNjggMCAxMzQxNDUzIDAgODk2NSkKY3Vn
YyAtMTc3IGNvdW50cyBkaWZmICgzNDc4NSAxIDQ3OCAwIDE0MzQ0MzIgMCA4ODQyKQpjdWdj
IC0yODAxIGNvdW50cyBkaWZmICgzNzA2OSAxIDQ2NCAwIDE0Nzc4MjUgMCA5Njc1KQpjdWdj
IC0yMyBjb3VudHMgZGlmZiAoNDA4MTcgMSA0NDggMCAxNDc4NDQ1IDAgMTA5OTkpCmN1Z2Mg
LTEyMTUgY291bnRzIGRpZmYgKDQ0NTI2IDEgNTAwIDAgMTUwMzYwNCAwIDExOTY0KQpjdWdj
IC00MTg5IGNvdW50cyBkaWZmICg0MjMwNSAxIDQ2OCAwIDE3MDE5ODkgMCAxMTM1NCkKY3Vn
YyAtNDcxNSBjb3VudHMgZGlmZiAoMzY2NDQgMSA1MzIgMCAyMDM2Nzc4IDAgOTA4MikKY3Vn
YyAtODUgY291bnRzIGRpZmYgKDM4MjM0IDEgNTQyIDAgMjEzMTc1NiAwIDk1MzUpCmN1Z2Mg
LTg2MSBjb3VudHMgZGlmZiAoNDE2MzIgMSA1NzggMCAyMTg4MTg2IDAgMTA0NzQpCmN1Z2Mg
LTExNyBjb3VudHMgZGlmZiAoNDYwMjkgMSA1ODAgMCAyMjExNjg1IDAgMTE5MjEpCmN1Z2Mg
LTM4IGNvdW50cyBkaWZmICg1MDM1MyAxIDcyOCAwIDIyODAzODggMCAxMjU2OCkKY3VnYyAt
MjUzNyBjb3VudHMgZGlmZiAoNTcxNjggMSA4ODggMCAyMjg2MzgxIDAgMTM5NzQpCmN1Z2Mg
LTM2NzYgY291bnRzIGRpZmYgKDYxNTcwIDEgOTI0IDAgMjM0MTQwMiAwIDE1MjQ2KQpjdWdj
IC0xNzQgY291bnRzIGRpZmYgKDU2NTA0IDEgOTI0IDAgMjY4OTMwMCAwIDEzNTAyKQpjdWdj
IC0xMDAxIGNvdW50cyBkaWZmICg1NzA2NiAxIDg0MiAwIDI4NTUwMjggMCAxNDA5OCkKY3Vn
YyAtMTQ2IGNvdW50cyBkaWZmICg1NzcxNiAxIDkxNiAwIDMwNjMyMzggMCAxMzg5MSkKY3Vn
YyAtMTQ4IGNvdW50cyBkaWZmICg2Mjg2OCAxIDk4MiAwIDMxMzkxMTEgMCAxNTI0NCkKY3Vn
YyAtMTczMCBjb3VudHMgZGlmZiAoNjQ4MDkgMSA4NTYgMCAzMjgzODU1IDAgMTY1MzUpCmN1
Z2MgLTE2MiBjb3VudHMgZGlmZiAoNjkxODMgMSA4NzAgMCAzMzk0MDMxIDAgMTc5MDIpCjxm
aW5pc2hlZFxuPgp0b3RhbCBjaHVua3MgNjY1MgpFbGFwc2VkIHRpbWU6IDEuMjMzMDE2cyAo
MC42Njg4MTlzIGluIDI4IEdDcykKCkFuZCBoZXJlJ3Mgd2l0aCAocmVhZC1wcm9jZXNzLW91
dHB1dC1tYXggNDA5NjAwKToKCnN0YXJ0IG5vdwpjdWdjIC0xMiBjb3VudHMgZGlmZiAoNTkx
NjAgNSAyMjU0NyAxMTYgMTU1NDM0IDIwNDYgMjEwMykKY3VnYyAtMTU0MDAxIGNvdW50cyBk
aWZmICgxODY3MSAxIDE2IDAgMTAzNDUzOCAwIDYxNzIpCmN1Z2MgLTEwMCBjb3VudHMgZGlm
ZiAoMjAyNTAgMSAxNCAwIDEwMDM5NjYgMCA2NzA4KQpjdWdjIC0xOTAyOTQgY291bnRzIGRp
ZmYgKDE5NjIzIDEgMTYgMCAxMjQ0NDQxIDAgNjQ4OSkKY3VnYyAtNTggY291bnRzIGRpZmYg
KDI2MTYwIDEgMTQgMCAxMDE1MTI4IDAgODY3OCkKY3VnYyAtMjkzMDY3IGNvdW50cyBkaWZm
ICgyMjczNyAxIDE2IDAgMTQyNjg3NCAwIDc1MjcpCmN1Z2MgLTkyIGNvdW50cyBkaWZmICgy
ODMwOCAxIDE0IDAgMTE2MDIxMyAwIDkzOTQpCmN1Z2MgLTI1IGNvdW50cyBkaWZmICgyMTYy
MCAxIDE2IDAgMTUzNTY4NiAwIDcxNTMpCmN1Z2MgLTIxIGNvdW50cyBkaWZmICgyMzI1MSAx
IDE2IDAgMTU1NDcyMCAwIDc2OTgpCmN1Z2MgLTE0MyBjb3VudHMgZGlmZiAoMjk5ODggMSAx
NiAwIDE0NjI2MzkgMCA5OTQzKQpjdWdjIC0xMTcgY291bnRzIGRpZmYgKDI4ODI3IDEgMTYg
MCAxNjIyNTYyIDAgOTU1NikKY3VnYyAtMjYgY291bnRzIGRpZmYgKDMzOTU5IDEgMTYgMCAx
NjA2ODE1IDAgMTEyNjYpCmN1Z2MgLTE3IGNvdW50cyBkaWZmICgzNzQ3NiAxIDE2IDAgMTYz
OTg1MyAwIDEyNDM5KQpjdWdjIC0yNTA5OTIgY291bnRzIGRpZmYgKDMxMzQ1IDEgMTggMCAy
MDgxNjYzIDAgMTAzODMpCmN1Z2MgLTI4OTE0MiBjb3VudHMgZGlmZiAoMjk5MDQgMSAxOCAw
IDI0NDg0MTAgMCA5OTAxKQpjdWdjIC0yOTAyMjcgY291bnRzIGRpZmYgKDMwNjc1IDEgMTgg
MCAyNDQ4MTU2IDAgMTAxNTkpCmN1Z2MgLTI2NDMxNSBjb3VudHMgZGlmZiAoMzU0MTggMSAx
OCAwIDI0NDY1MDggMCAxMTc0MSkKY3VnYyAtMzIgY291bnRzIGRpZmYgKDQxNzQxIDEgMTgg
MCAyMzQzOTAwIDAgMTM4NDcpCmN1Z2MgLTIyMDEgY291bnRzIGRpZmYgKDQ0NTIzIDEgMTEy
IDAgMjQ3ODMxMCAwIDE0MjM5KQpjdWdjIC0xNTY3MyBjb3VudHMgZGlmZiAoNDk2MjIgMSAx
NzAgMCAyNTI4MjIxIDAgMTU1OTIpCmN1Z2MgLTQwMjY3IGNvdW50cyBkaWZmICg0MTk5MCAx
IDU4IDAgMjk3MjAxNSAwIDEzNjkzKQpjdWdjIC0xNTkgY291bnRzIGRpZmYgKDQxMDEwIDEg
MjIgMCAzMTc3OTk0IDAgMTM1ODApCmN1Z2MgLTQyIGNvdW50cyBkaWZmICg0NzYwMiAxIDE1
NiAwIDMyNTk4MzMgMCAxNTAwOSkKY3VnYyAtMzU4ODg0IGNvdW50cyBkaWZmICg0Mzc0MCAx
IDM0IDAgMzY4NzE0NSAwIDE0NDM2KQpjdWdjIC0yMiBjb3VudHMgZGlmZiAoNTU1OTggMSAy
MCAwIDM0OTQxOTAgMCAxODQ1NCkKY3VnYyAtMTI3MCBjb3VudHMgZGlmZiAoNjAxMjggMSAx
OTAgMCAzNjgzNDYxIDAgMTg5ODApCjxmaW5pc2hlZFxuPgp0b3RhbCBjaHVua3MgMjczCkVs
YXBzZWQgdGltZTogMC45MzI2MjVzICgwLjYwODcxM3MgaW4gMjYgR0NzKQo=
--------------I3MK6f5qXVUuZUbk3Nf06WpH
Content-Type: text/plain; charset=UTF-8; name="gcs3b.txt"
Content-Disposition: attachment; filename="gcs3b.txt"
Content-Transfer-Encoding: base64

ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMwoKVXNlcyBwcm9jZXNzLWZpbGUs
IHBhcnNlcyB0aGUgYnVmZmVyCndpdGggc2VhcmNoLWZvcndhcmQgYXQgdGhlIGVuZC4KCnN0
YXJ0IG5vdwo8cHJvY2VzcyBmaW5pc2hlZCwgbm93IHBhcnNpbmc+CmN1Z2MgLTEyOSBjb3Vu
dHMgZGlmZiAoMTc2NjcgMSAxNTY1IDEgNzc5MDgxIDkzIDEwNDY1KQpjdWdjIC0xMzkgY291
bnRzIGRpZmYgKDEyNzg5IDEgMTIgMCA5MTIzNjQgMCAxMjY5NikKY3VnYyAtODQgY291bnRz
IGRpZmYgKDEzNDk2IDEgMTIgMCAxMDI4MDYwIDAgMTM0MDMpCmN1Z2MgLTMzIGNvdW50cyBk
aWZmICgxNDExMiAxIDEyIDAgMTE2ODUyMiAwIDE0MDE5KQpjdWdjIC0xNTMgY291bnRzIGRp
ZmYgKDE0MzU0IDEgMTIgMCAxMzQ3MjQxIDAgMTQyNjEpCmN1Z2MgLTcyIGNvdW50cyBkaWZm
ICgxNzAwNSAxIDEyIDAgMTQwMTA3NSAwIDE2OTEyKQpjdWdjIC0xNyBjb3VudHMgZGlmZiAo
MjA4MTAgMSAxMiAwIDE0MDMzOTYgMCAyMDcxNykKY3VnYyAtOTQgY291bnRzIGRpZmYgKDE4
NTE2IDEgMTIgMCAxNzkyNTA4IDAgMTg0MjMpCmN1Z2MgLTEyMCBjb3VudHMgZGlmZiAoMTc5
ODEgMSAxMiAwIDIxMDg0NTggMCAxNzg4OCkKY3VnYyAtNTAgY291bnRzIGRpZmYgKDIyMDkw
IDEgMTIgMCAyMTY5ODM1IDAgMjE5OTcpCmN1Z2MgLTEzNiBjb3VudHMgZGlmZiAoMjY3NDkg
MSAxMiAwIDIyMzEwMzcgMCAyNjY1NikKY3VnYyAtOTMgY291bnRzIGRpZmYgKDI1MzAwIDEg
MTIgMCAyNjg3ODQzIDAgMjUyMDcpCmN1Z2MgLTcyIGNvdW50cyBkaWZmICgyNjE2NSAxIDEy
IDAgMzA0NjE0MCAwIDI2MDcyKQpjdWdjIC0xNDIgY291bnRzIGRpZmYgKDMwOTY4IDEgMTIg
MCAzMjA1MzA2IDAgMzA4NzUpCkVsYXBzZWQgdGltZTogMC45MzgxODBzICgwLjMxNDYzMHMg
aW4gMTQgR0NzKQo=

--------------I3MK6f5qXVUuZUbk3Nf06WpH--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 11:41:02 +0000
Resent-Message-ID: <handler.64735.B64735.169451881629027 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169451881629027
          (code B ref 64735); Tue, 12 Sep 2023 11:41:02 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 11:40:16 +0000
Received: from localhost ([127.0.0.1]:56326 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg1kV-0007Xu-9h
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 07:40:16 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:39480)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qg1kQ-0007XG-9q
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 07:40:05 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qg1k6-0002fT-AA; Tue, 12 Sep 2023 07:39:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=wcFFnMb+m4sTdfLtKnVv9R8+ktdQFlm0Gd/TEQvajQM=; b=k9NmyED/BhFq
 10y5eqsGvz+8VW0iC4mFvmqq1Xida7XpRQdq1rb9DS4X9mE/XtiWIt/Kp54OAJVkn5qIKrF/k/e02
 lJB2OHKKZiZsI4f+HbCqE4vW1WMaWjLDPEADnbM24rqWqE7snLIR3BHaX9TvGPc2RiPuagc5bQ6Ir
 L2K3nA9L7LBjkOrv2dQx4las2wzaxpgCi5NOvimg7Q3SV8OKiuxCMmH30I6tohZLBghr1HEecoV/j
 +jZKzCJRAlo/z2bIEBLN5bC/5GRJzcZ2LlDgGKvtdLGjrizTRjfMcHAzs2Rq145JAY13wE0ExnM+x
 iTN5ZzX1DdSHfoJzjdA71g==;
Date: Tue, 12 Sep 2023 14:39:24 +0300
Message-Id: <83ledbpqmr.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <14b26ff0-9d9b-3600-bf6e-133db4031622@HIDDEN> (message from
 Dmitry Gutov on Tue, 12 Sep 2023 02:06:50 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN> <14b26ff0-9d9b-3600-bf6e-133db4031622@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 12 Sep 2023 02:06:50 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > No, we don't wait until it's zero, we perform GC on the first
> > opportunity that we _notice_ that it crossed zero.  So examining how
> > negative is the value of consing_until_gc when GC is actually
> > performed could tell us whether we checked the threshold with high
> > enough frequency, and comparing these values between different runs
> > could tell us whether the shorter time spend in GC means really less
> > garbage or less frequent checks for the need to GC.
> 
> Good point, I'm attaching the same outputs with "last value of 
> consing_until_gc" added to every line.
> 
> There are some pretty low values in the "read-process-output-max 409600" 
> part of the experiment, which probably means runtime staying in C 
> accumulating the output into the (now larger) buffer? Not sure.

No, I think this means we really miss some GC opportunities, and we
cons quite a lot more strings between GC cycles due to that.  I guess
this happens because we somehow cons many strings in code that doesn't
call maybe_gc or something.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 13:12:01 +0000
Resent-Message-ID: <handler.64735.B64735.169452427716729 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169452427716729
          (code B ref 64735); Tue, 12 Sep 2023 13:12:01 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 13:11:17 +0000
Received: from localhost ([127.0.0.1]:56473 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg3Ai-0004Ll-Ok
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 09:11:17 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:59111)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qg3Ah-0004LY-5Y
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 09:11:16 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.nyi.internal (Postfix) with ESMTP id 5C18B5C0107;
 Tue, 12 Sep 2023 09:11:05 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute3.internal (MEProxy); Tue, 12 Sep 2023 09:11:05 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694524265; x=1694610665; bh=9qMf1VTEcKVk+StAaAG1jAFIdhxK42vLIjq
 czYrcMj4=; b=gXgmiyO246hmw/gVzKmY0WT8ljXdavcl+xd5cGuGlbuZE2ECUhx
 OLg3eHnz5b1+ozBsJwdZ0WXhsEbK+UPwLgUWFeeIGI5IVr07HaczPPdm0f6NvjU8
 n7TQG5KnDhV0G4SIyKk7MmL0vCdvS1MZr1cthTUa5WjMdFFuUF7y1b2MDcFfg9+v
 aE0OBWHnMxsB9q6zIP3CVpWaDZD772V93ipVdl2M8CSK8Pj3Y81cqeJItg2m1S7q
 cGW/fRZeRcdOvorH+WWsJCCjtda6ejhVlSwx7zQKi/P5nywHO4ZtMpUBXKWYcJIK
 5mECrqjN/KwZqyjkZgfV3tv48GwnnxW569A==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694524265; x=1694610665; bh=9qMf1VTEcKVk+StAaAG1jAFIdhxK42vLIjq
 czYrcMj4=; b=moZM/cie22ToghgRysBm0c8ZUbjOO1maZ67blmN8CeUQ/M3bxTh
 UQ6djtLBAnB8zJTYH8VAUQDW+nDSKRcD3EaegU5i7f+5flG7AIfS6sYGWltKQMO7
 FrEOnQnxE7QZYJFKc24NRCuMJU82UytplekNWjVxZ7kiAHlkHNuSXW1Tz4JvlJM9
 BwAkBjZUrgghGpi/+/JrM8qxZNo6jCq0BMQrXFm+jqnnUdOfCVRhFZkfVvO5K1/9
 DoplGS+wqszxY45D87B/qfPooaz1hAJdu+gHLiRDvq58Qzj8qerGmaM3v2v+LuuK
 lJ/qkcZFZLlB9Yga1xsjcTsRiLds741tx5g==
X-ME-Sender: <xms:aGMAZZZWnKIIIezTkrER4a6DSRVqi2PkVVogZY6Sbzr1AG6Ko_nTqw>
 <xme:aGMAZQa-meAXpjh0aGyhzk2ACkt-SkZ5bEY3UbvjYRQZuSNQ7DUWpw4FKDvVYoKgl
 RImOu-KjBF3cqhj3ZE>
X-ME-Received: <xmr:aGMAZb-kFxsfI7Yybwb9D54-rP6HOm8LFSfzQuoyVRQL_DXY78LVVBkoDE5FNgI>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgiedtucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:aGMAZXorDI4JbnrsVU0oIUNG3o-ubl9JiiXdCUafERSG9d0l1D1v8g>
 <xmx:aGMAZUrj-YZrCvA-YWK5P5GiC6pg_wQmazeRm4k3VO-xhR-ntIvGUA>
 <xmx:aGMAZdTeuC-ODbNl0q98GibrxNMGzy99_3mRc6jCKQ4B8VbEqrQH9g>
 <xmx:aWMAZfXKhr-sZkRy3I5q54UR9U-gw2lzVAkcaitX1vobTVY1UEWx_A>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 12 Sep 2023 09:11:03 -0400 (EDT)
Message-ID: <f157382c-4941-75ff-0937-930d3021d059@HIDDEN>
Date: Tue, 12 Sep 2023 16:11:01 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <14b26ff0-9d9b-3600-bf6e-133db4031622@HIDDEN>
 <83ledbpqmr.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83ledbpqmr.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 12/09/2023 14:39, Eli Zaretskii wrote:
>> Date: Tue, 12 Sep 2023 02:06:50 +0300
>> Cc:luangruo@HIDDEN,sbaugh@HIDDEN,yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@HIDDEN>
>>
>>> No, we don't wait until it's zero, we perform GC on the first
>>> opportunity that we_notice_  that it crossed zero.  So examining how
>>> negative is the value of consing_until_gc when GC is actually
>>> performed could tell us whether we checked the threshold with high
>>> enough frequency, and comparing these values between different runs
>>> could tell us whether the shorter time spend in GC means really less
>>> garbage or less frequent checks for the need to GC.
>> Good point, I'm attaching the same outputs with "last value of
>> consing_until_gc" added to every line.
>>
>> There are some pretty low values in the "read-process-output-max 409600"
>> part of the experiment, which probably means runtime staying in C
>> accumulating the output into the (now larger) buffer? Not sure.
> No, I think this means we really miss some GC opportunities, and we
> cons quite a lot more strings between GC cycles due to that.

Or possibly same number of strings but longer ones?

> I guess
> this happens because we somehow cons many strings in code that doesn't
> call maybe_gc or something.

Yes, staying in some C code that doesn't call maybe_gc for a while.

I think we're describing the same thing, only I was doing that from the 
positive side (less frequent GCs = better performance in this scenario), 
and you from the negative one (less frequent GCs = more chances for an 
OOM to happen in some related but different scenario).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 14:25:01 +0000
Resent-Message-ID: <handler.64735.B64735.169452865028222 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169452865028222
          (code B ref 64735); Tue, 12 Sep 2023 14:25:01 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 14:24:10 +0000
Received: from localhost ([127.0.0.1]:59705 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg4JG-0007L7-1b
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 10:24:10 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:45243)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qg4JD-0007Kt-Co
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 10:24:08 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 632F25C028B;
 Tue, 12 Sep 2023 10:23:57 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute6.internal (MEProxy); Tue, 12 Sep 2023 10:23:57 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694528637; x=1694615037; bh=QqKv4EB9n4HHUgnDi8ixSAF6v1Yo+A/OUTO
 pnI8ETVg=; b=iv6rbAJMZ+6wQrUVJui0ruD2UGNZO1XnpaBrW+/f37XDGq8YZsW
 D/CKSJVO7Zae7ZxUzTVPt7dlb3hzDroyJg94EskN8uKZupTwQwDRI9gh7oN4xqhT
 AwMRSaC3FTSmsokKAQWQzA7ImLNWX8bHtMnvTpvFiNlELAlkP7f2FsCZAs8h6wst
 MaE6K8oack3WnfDQQlaIP6/qAubT8wadiGYzlwIzj3Yyf4dHglpKuXiJUAR2tS2D
 lL30YogJ0VxSLCt/CNJDovbu/JOqJgSj5laxf42mxllPIFeb5Pv8CwgIKYKDDf8t
 Dss/aB83Qa6Lfma9Spmrb/NHOBCorLiM7jQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694528637; x=1694615037; bh=QqKv4EB9n4HHUgnDi8ixSAF6v1Yo+A/OUTO
 pnI8ETVg=; b=JECeLGxyo/l8XSSy2tujwMEjfNqCCfbsvg+7AakIUptJE9kpNEh
 Cz5tpaVx0JxZuNMj7BdUyQ4hApDW2VED7iBYg4VkXBPRF2jl3XJ/UMPd5/BhC+dr
 BHfGV1vqT0SpHb0ITXMNetX4afeAKXvIPJb3KQ2ZIbtg8bDkL8WdrxAf45zrnVyM
 lHAqqZItk1ZUt8Q38K5EQhePwAJltJSrPpT+WIOsqtUWFPYSEMeBBAQgW9+x2WfW
 nqdTpzYuapoTmK7gWPpzj7w5yZ3ZWTTl/9ou9chG2WGjRyA9OL5KOe3T1g7R85zX
 uIcxSFy3qJ1FhMXxWwDI+yf5hy26HaCynIg==
X-ME-Sender: <xms:fXQAZShaSr8HT3UPR3Qk5-1wo8MOTZGEhyAYpsi1RIvMMuwaNUgejg>
 <xme:fXQAZTBvflQrvq0GQzsx2Evc99w4yStQDlwwN_U8KlocfM9DFPz9OG6NT-98UYyFS
 _YlV5a2K__Ptdw3bCY>
X-ME-Received: <xmr:fXQAZaHPWqkmmRivGivmm4MKJzkB1SVZQFc6kSEWWBE7bSzqj0o5cdkS5I3vTvo>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgjeehucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:fXQAZbR5hP0S0f4XehlQkkG7QFaOBReGnhDE1-MJf4hmLjgd2Xh_2A>
 <xmx:fXQAZfxN83x-hQD7tKXrO7eIJ2dZLQjerGwZcACXVfcddjbzQcxhsw>
 <xmx:fXQAZZ4437inoggqoU049fzS0lNr85ssAQ4hmccDxa7ybZY5NNoSvg>
 <xmx:fXQAZf_Tbq_Rt0Aqr4SrsB4-3vR195kvEbDVF3TOHZThcl2d-9pCTg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 12 Sep 2023 10:23:55 -0400 (EDT)
Message-ID: <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
Date: Tue, 12 Sep 2023 17:23:53 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83tts0rkh5.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 11/09/2023 14:57, Eli Zaretskii wrote:
>> So there is also a second recording for
>> find-directory-files-recursively-2 with read-process-output-max=409600.
>> It does improve the performance significantly (and reduce the number of
>> GC pauses). I guess what I'm still not clear on, is whether the number
>> of GC pauses is fewer because of less consing (the only column that
>> looks significantly different is the 3rd: VECTOR-CELLS), or because the
>> process finishes faster due to larger buffers, which itself causes fewer
>> calls to maybe_gc.
> I think the latter.

It might be both.

To try to analyze how large might per-chunk overhead be (CPU and GC-wise 
combined), I first implemented the same function in yet another way that 
doesn't use :filter (so that the default filter is used). But still 
asynchronously, with parsing happening concurrently to the process:

(defun find-directory-files-recursively-5 (dir regexp &optional 
include-directories _p follow-symlinks)
   (cl-assert (null _p) t "find-directory-files-recursively can't accept 
arbitrary predicates")
   (with-temp-buffer
     (setq case-fold-search nil)
     (cd dir)
     (let* ((command
	    (append
	     (list "find" (file-local-name dir))
	     (if follow-symlinks
		 '("-L")
	       '("!" "(" "-type" "l" "-xtype" "d" ")"))
	     (unless (string-empty-p regexp)
	       (list "-regex" (concat ".*" regexp ".*")))
	     (unless include-directories
	       '("!" "-type" "d"))
	     '("-print0")
	     ))
	   (remote (file-remote-p dir))
	   (proc
	    (if remote
		(let ((proc (apply #'start-file-process
				   "find" (current-buffer) command)))
		  (set-process-sentinel proc (lambda (_proc _state)))
		  (set-process-query-on-exit-flag proc nil)
		  proc)
	      (make-process :name "find" :buffer (current-buffer)
			    :connection-type 'pipe
			    :noquery t
			    :sentinel (lambda (_proc _state))
			    :command command)))
            start ret)
       (setq start (point-min))
       (while (accept-process-output proc)
         (goto-char start)
         (while (search-forward "\0" nil t)
	  (push (buffer-substring-no-properties start (1- (point))) ret)
	  (setq start (point))))
       ret)))

This method already improved the performance somewhat (compared to 
find-directory-files-recursively-2), but not too much. So I tried these 
next two steps:

- Dropping most of the setup in read_and_dispose_of_process_output 
(which creates some consing too) and calling 
Finternal_default_process_filter directly (call_filter_directly.diff), 
when it is the filter to be used anyway.

- Going around that function entirely, skipping the creation of a Lisp 
string (CHARS -> TEXT) and inserting into the buffer directly (when the 
filter is set to the default, of course). Copied and adapted some code 
from 'call_process' for that (read_and_insert_process_output.diff).

Neither are intended as complete proposals, but here are some 
comparisons. Note that either of these patches could only help the 
implementations that don't set up process filter (the naive first one, 
and the new parallel number 5 above).

For testing, I used two different repo checkouts that are large enough 
to not finish too quickly: gecko-dev and torvalds-linux.

master

| Function                                         | gecko-dev | linux |
| find-directory-files-recursively                 |      1.69 |  0.41 |
| find-directory-files-recursively-2               |      1.16 |  0.28 |
| find-directory-files-recursively-3               |      0.92 |  0.23 |
| find-directory-files-recursively-5               |      1.07 |  0.26 |
| find-directory-files-recursively (rpom 409600)   |      1.42 |  0.35 |
| find-directory-files-recursively-2 (rpom 409600) |      0.90 |  0.25 |
| find-directory-files-recursively-5 (rpom 409600) |      0.89 |  0.24 |

call_filter_directly.diff (basically, not much difference)

| Function                                         | gecko-dev | linux |
| find-directory-files-recursively                 |      1.64 |  0.38 |
| find-directory-files-recursively-5               |      1.05 |  0.26 |
| find-directory-files-recursively (rpom 409600)   |      1.42 |  0.36 |
| find-directory-files-recursively-5 (rpom 409600) |      0.91 |  0.25 |

read_and_insert_process_output.diff (noticeable differences)

| Function                                         | gecko-dev | linux |
| find-directory-files-recursively                 |      1.30 |  0.34 |
| find-directory-files-recursively-5               |      1.03 |  0.25 |
| find-directory-files-recursively (rpom 409600)   |      1.20 |  0.35 |
| find-directory-files-recursively-5 (rpom 409600) | (!!) 0.72 |  0.21 |

So it seems like we have at least two potential ways to implement an 
asynchronous file listing routine that is as fast or faster than the 
synchronous one (if only thanks to starting the parsing in parallel).

Combining the last patch together with using the very large value of 
read-process-output-max seems to yield the most benefit, but I'm not 
sure if it's appropriate to just raise that value in our code, though.

Thoughts?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 14:27:01 +0000
Resent-Message-ID: <handler.64735.B64735.169452880128470 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169452880128470
          (code B ref 64735); Tue, 12 Sep 2023 14:27:01 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 14:26:41 +0000
Received: from localhost ([127.0.0.1]:59719 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg4Lg-0007P7-Hx
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 10:26:41 -0400
Received: from out4-smtp.messagingengine.com ([66.111.4.28]:58487)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qg4Lf-0007Ot-11
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 10:26:39 -0400
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
 by mailout.nyi.internal (Postfix) with ESMTP id 34AD95C0145;
 Tue, 12 Sep 2023 10:26:29 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute1.internal (MEProxy); Tue, 12 Sep 2023 10:26:29 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-type:content-type:date:date:from:from:in-reply-to
 :in-reply-to:message-id:mime-version:references:reply-to:sender
 :subject:subject:to:to; s=fm3; t=1694528789; x=1694615189; bh=kb
 nA+MxBx0YBIlUbx+xdq3+WRUhtmwdEJkf0eiGO1I8=; b=mirJCNK2fVrZnIxqbh
 CSDZExhedn3+0Lcre+qEbWbjO7sbM3L6FR4/CJV1W89DrRaO5EzlzcrheVWXQBEp
 KlZ7aA1lTpSB+qfdOIOJqGWq5nACoubDehzwlE+hpjaqKwHSGEgAZe2JAIf3lnrr
 BplNRC2Vv0w/GTJpyG08GGcZFToHa6wXTf9UqGfxCoJsDVPHgg3qUzd9wjswLgTm
 7TaGcKE4C3RgDchBSvc1evseTCi6Ma77CWRAetif5GP94QVQ5uWWwdt91LALL/0T
 N5rnd+YiecqgInm9VzHlb0OmcuGTxzBRXoeacZekUGz/IVw5ujV7d6Z23NHK8MGs
 FVEg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-type:content-type:date:date
 :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to
 :message-id:mime-version:references:reply-to:sender:subject
 :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm1; t=1694528789; x=1694615189; bh=kbnA+MxBx0YBI
 lUbx+xdq3+WRUhtmwdEJkf0eiGO1I8=; b=TSx/Spgn5YVuc1tAaugnwsWK5yioH
 knRCDDp6gmfrog8/qAlk3HG66SSJPDtqcR6IO75GjbjwgFcEmt6p+mn/hJWL7QYS
 643bPw2bejp+EkMndE2A2xQxnqPjFqvYXd7QNhK8FBxzAyuf/MlueAd+h9BIphsZ
 3RvWMTqOXxgTX80uBKE/cIMykijgneOM/sTFZB0pm780FppbJQGwurOd6M7pys29
 zXIXbUGBCP0ea9d8AdZa4xUY1n1WZkcub//dtwtTyJZ1q/kdp+z5dZB7NDSHggWU
 fi4HH0NvGKFDRZ52sxAyHmOoHue84GG5mbwfOi2LRs0TZ6vdvW/1IgprQ==
X-ME-Sender: <xms:FXUAZbU2TQEUA4ujJ713719hStz0pMCHeQx-llrVgFfBjylVLWZXyw>
 <xme:FXUAZTkF2ZmEm8UfZjDTv9iTWk8z5u6PBa64z1hbY_paowXYI5aAj_oPJysN1WX47
 WrSkzsW1OwmVrSkgBA>
X-ME-Received: <xmr:FXUAZXbHBLJNfY4fBCW0jKcfBqS1ED6hlhQvJbkDmoO8Du7mUEvo5T1D1D5mF4A>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgjeeiucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurheptgfkffggfgfuhffvvehfjgesmhdtreertdefjeenucfhrhhomhepffhmihht
 rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth
 gvrhhnpeeiheegkeetgffghefhgeeiveeuudegvdeuteffhfettdelleehkeffledvuddt
 leenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh
 hithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:FXUAZWVd6JCMNRqNnVzIYKskgFKbx_PilmjmdQwOVNqgiJek9fqcLQ>
 <xmx:FXUAZVlGOGK3T9UMEHUe7S4G8UUkNU4sLgsSBZ6KPdUBBHeZE5jZbg>
 <xmx:FXUAZTeGHOrqArrHe-GlEYbXMonOEmSUa2uzEs8y5gAtHWCSqavkRQ>
 <xmx:FXUAZbjbpFl-GrK9X5laaExaTMxCDPbe9K1DZJFahw6MOFBTbgp_Qg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 12 Sep 2023 10:26:27 -0400 (EDT)
Content-Type: multipart/mixed; boundary="------------pNrGtcmBObRwwOXzSA0eJQDE"
Message-ID: <e38f060a-b55e-86c7-73e5-630ec76013d5@HIDDEN>
Date: Tue, 12 Sep 2023 17:26:26 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
From: Dmitry Gutov <dmitry@HIDDEN>
References: <iermszrwqj4.fsf@HIDDEN> <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
In-Reply-To: <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

This is a multi-part message in MIME format.
--------------pNrGtcmBObRwwOXzSA0eJQDE
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 12/09/2023 17:23, Dmitry Gutov wrote:
> Neither are intended as complete proposals, but here are some 
> comparisons. Note that either of these patches could only help the 
> implementations that don't set up process filter (the naive first one, 
> and the new parallel number 5 above).

Sorry, forgot to attach the patches.
--------------pNrGtcmBObRwwOXzSA0eJQDE
Content-Type: text/x-patch; charset=UTF-8; name="call_filter_directly.diff"
Content-Disposition: attachment; filename="call_filter_directly.diff"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy9wcm9jZXNzLmMgYi9zcmMvcHJvY2Vzcy5jCmluZGV4IDA4Y2I4
MTBlYzEzLi5iZGJlOGQ5NjA2NCAxMDA2NDQKLS0tIGEvc3JjL3Byb2Nlc3MuYworKysgYi9z
cmMvcHJvY2Vzcy5jCkBAIC02MjI3LDcgKzYyMjcsMTUgQEAgcmVhZF9wcm9jZXNzX291dHB1
dCAoTGlzcF9PYmplY3QgcHJvYywgaW50IGNoYW5uZWwpCiAgICAgIGZyaWVuZHMgZG9uJ3Qg
ZXhwZWN0IGN1cnJlbnQtYnVmZmVyIHRvIGJlIGNoYW5nZWQgZnJvbSB1bmRlciB0aGVtLiAg
Ki8KICAgcmVjb3JkX3Vud2luZF9jdXJyZW50X2J1ZmZlciAoKTsKIAotICByZWFkX2FuZF9k
aXNwb3NlX29mX3Byb2Nlc3Nfb3V0cHV0IChwLCBjaGFycywgbmJ5dGVzLCBjb2RpbmcpOwor
ICBpZiAocC0+ZmlsdGVyID09IFFpbnRlcm5hbF9kZWZhdWx0X3Byb2Nlc3NfZmlsdGVyKQor
ICAgIHsKKyAgICAgIExpc3BfT2JqZWN0IHRleHQ7CisgICAgICBkZWNvZGVfY29kaW5nX2Nf
c3RyaW5nIChjb2RpbmcsICh1bnNpZ25lZCBjaGFyICopIGNoYXJzLCBuYnl0ZXMsIFF0KTsK
KyAgICAgIHRleHQgPSBjb2RpbmctPmRzdF9vYmplY3Q7CisgICAgICBGaW50ZXJuYWxfZGVm
YXVsdF9wcm9jZXNzX2ZpbHRlciAocHJvYywgdGV4dCk7CisgICAgfQorICBlbHNlCisgICAg
cmVhZF9hbmRfZGlzcG9zZV9vZl9wcm9jZXNzX291dHB1dCAocCwgY2hhcnMsIG5ieXRlcywg
Y29kaW5nKTsKIAogICAvKiBIYW5kbGluZyB0aGUgcHJvY2VzcyBvdXRwdXQgc2hvdWxkIG5v
dCBkZWFjdGl2YXRlIHRoZSBtYXJrLiAgKi8KICAgVmRlYWN0aXZhdGVfbWFyayA9IG9kZWFj
dGl2YXRlOwo=
--------------pNrGtcmBObRwwOXzSA0eJQDE
Content-Type: text/x-patch; charset=UTF-8;
 name="read_and_insert_process_output.diff"
Content-Disposition: attachment;
 filename="read_and_insert_process_output.diff"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL3NyYy9wcm9jZXNzLmMgYi9zcmMvcHJvY2Vzcy5jCmluZGV4IDA4Y2I4
MTBlYzEzLi41ZGI1NjY5MmZlMSAxMDA2NDQKLS0tIGEvc3JjL3Byb2Nlc3MuYworKysgYi9z
cmMvcHJvY2Vzcy5jCkBAIC02MTEyLDYgKzYxMTIsMTEgQEAgcmVhZF9hbmRfZGlzcG9zZV9v
Zl9wcm9jZXNzX291dHB1dCAoc3RydWN0IExpc3BfUHJvY2VzcyAqcCwgY2hhciAqY2hhcnMs
CiAJCQkJICAgIHNzaXplX3QgbmJ5dGVzLAogCQkJCSAgICBzdHJ1Y3QgY29kaW5nX3N5c3Rl
bSAqY29kaW5nKTsKIAorc3RhdGljIHZvaWQKK3JlYWRfYW5kX2luc2VydF9wcm9jZXNzX291
dHB1dCAoc3RydWN0IExpc3BfUHJvY2VzcyAqcCwgY2hhciAqYnVmLAorCQkJCSAgICBzc2l6
ZV90IG5yZWFkLAorCQkJCXN0cnVjdCBjb2Rpbmdfc3lzdGVtICpwcm9jZXNzX2NvZGluZyk7
CisKIC8qIFJlYWQgcGVuZGluZyBvdXRwdXQgZnJvbSB0aGUgcHJvY2VzcyBjaGFubmVsLAog
ICAgc3RhcnRpbmcgd2l0aCBvdXIgYnVmZmVyZWQtYWhlYWQgY2hhcmFjdGVyIGlmIHdlIGhh
dmUgb25lLgogICAgWWllbGQgbnVtYmVyIG9mIGRlY29kZWQgY2hhcmFjdGVycyByZWFkLApA
QCAtNjIyNyw3ICs2MjMyLDEwIEBAIHJlYWRfcHJvY2Vzc19vdXRwdXQgKExpc3BfT2JqZWN0
IHByb2MsIGludCBjaGFubmVsKQogICAgICBmcmllbmRzIGRvbid0IGV4cGVjdCBjdXJyZW50
LWJ1ZmZlciB0byBiZSBjaGFuZ2VkIGZyb20gdW5kZXIgdGhlbS4gICovCiAgIHJlY29yZF91
bndpbmRfY3VycmVudF9idWZmZXIgKCk7CiAKLSAgcmVhZF9hbmRfZGlzcG9zZV9vZl9wcm9j
ZXNzX291dHB1dCAocCwgY2hhcnMsIG5ieXRlcywgY29kaW5nKTsKKyAgaWYgKHAtPmZpbHRl
ciA9PSBRaW50ZXJuYWxfZGVmYXVsdF9wcm9jZXNzX2ZpbHRlcikKKyAgICByZWFkX2FuZF9p
bnNlcnRfcHJvY2Vzc19vdXRwdXQgKHAsIGNoYXJzLCBuYnl0ZXMsIGNvZGluZyk7CisgIGVs
c2UKKyAgICByZWFkX2FuZF9kaXNwb3NlX29mX3Byb2Nlc3Nfb3V0cHV0IChwLCBjaGFycywg
bmJ5dGVzLCBjb2RpbmcpOwogCiAgIC8qIEhhbmRsaW5nIHRoZSBwcm9jZXNzIG91dHB1dCBz
aG91bGQgbm90IGRlYWN0aXZhdGUgdGhlIG1hcmsuICAqLwogICBWZGVhY3RpdmF0ZV9tYXJr
ID0gb2RlYWN0aXZhdGU7CkBAIC02MjM2LDYgKzYyNDQsNDYgQEAgcmVhZF9wcm9jZXNzX291
dHB1dCAoTGlzcF9PYmplY3QgcHJvYywgaW50IGNoYW5uZWwpCiAgIHJldHVybiBuYnl0ZXM7
CiB9CiAKK3N0YXRpYyB2b2lkIHJlYWRfYW5kX2luc2VydF9wcm9jZXNzX291dHB1dCAoc3Ry
dWN0IExpc3BfUHJvY2VzcyAqcCwgY2hhciAqYnVmLAorCQkJCSAgICBzc2l6ZV90IG5yZWFk
LAorCQkJCSAgICBzdHJ1Y3QgY29kaW5nX3N5c3RlbSAqcHJvY2Vzc19jb2RpbmcpCit7Cisg
IGlmICghbnJlYWQgfHwgTklMUCAocC0+YnVmZmVyKSB8fCAhQlVGRkVSX0xJVkVfUCAoWEJV
RkZFUiAocC0+YnVmZmVyKSkpCisgICAgOworICBlbHNlIGlmIChOSUxQIChCVkFSIChYQlVG
RkVSKHAtPmJ1ZmZlciksIGVuYWJsZV9tdWx0aWJ5dGVfY2hhcmFjdGVycykpCisJICAgJiYg
ISBDT0RJTkdfTUFZX1JFUVVJUkVfREVDT0RJTkcgKHByb2Nlc3NfY29kaW5nKSkKKyAgICB7
CisgICAgICBpbnNlcnRfMV9ib3RoIChidWYsIG5yZWFkLCBucmVhZCwgMCwgMCwgMCk7Cisg
ICAgICBzaWduYWxfYWZ0ZXJfY2hhbmdlIChQVCAtIG5yZWFkLCAwLCBucmVhZCk7CisgICAg
fQorICBlbHNlCisgICAgewkJCS8qIFdlIGhhdmUgdG8gZGVjb2RlIHRoZSBpbnB1dC4gICov
CisgICAgICBMaXNwX09iamVjdCBjdXJidWY7CisgICAgICBpbnQgY2FycnlvdmVyID0gMDsK
KyAgICAgIHNwZWNwZGxfcmVmIGNvdW50MSA9IFNQRUNQRExfSU5ERVggKCk7CisKKyAgICAg
IFhTRVRCVUZGRVIgKGN1cmJ1ZiwgY3VycmVudF9idWZmZXIpOworICAgICAgLyogV2UgY2Fu
bm90IGFsbG93IGFmdGVyLWNoYW5nZS1mdW5jdGlvbnMgYmUgcnVuCisJIGR1cmluZyBkZWNv
ZGluZywgYmVjYXVzZSB0aGF0IG1pZ2h0IG1vZGlmeSB0aGUKKwkgYnVmZmVyLCB3aGlsZSB3
ZSByZWx5IG9uIHByb2Nlc3NfY29kaW5nLnByb2R1Y2VkIHRvCisJIGZhaXRoZnVsbHkgcmVm
bGVjdCBpbnNlcnRlZCB0ZXh0IHVudGlsIHdlCisJIFRFTVBfU0VUX1BUX0JPVEggYmVsb3cu
ICAqLworICAgICAgc3BlY2JpbmQgKFFpbmhpYml0X21vZGlmaWNhdGlvbl9ob29rcywgUXQp
OworICAgICAgZGVjb2RlX2NvZGluZ19jX3N0cmluZyAocHJvY2Vzc19jb2RpbmcsCisJCQkg
ICAgICAodW5zaWduZWQgY2hhciAqKSBidWYsIG5yZWFkLCBjdXJidWYpOworICAgICAgdW5i
aW5kX3RvIChjb3VudDEsIFFuaWwpOworCisgICAgICBURU1QX1NFVF9QVF9CT1RIIChQVCAr
IHByb2Nlc3NfY29kaW5nLT5wcm9kdWNlZF9jaGFyLAorCQkJUFRfQllURSArIHByb2Nlc3Nf
Y29kaW5nLT5wcm9kdWNlZCk7CisgICAgICBzaWduYWxfYWZ0ZXJfY2hhbmdlIChQVCAtIHBy
b2Nlc3NfY29kaW5nLT5wcm9kdWNlZF9jaGFyLAorCQkJICAgMCwgcHJvY2Vzc19jb2Rpbmct
PnByb2R1Y2VkX2NoYXIpOworICAgICAgY2FycnlvdmVyID0gcHJvY2Vzc19jb2RpbmctPmNh
cnJ5b3Zlcl9ieXRlczsKKyAgICAgIGlmIChjYXJyeW92ZXIgPiAwKQorCW1lbWNweSAoYnVm
LCBwcm9jZXNzX2NvZGluZy0+Y2FycnlvdmVyLAorCQlwcm9jZXNzX2NvZGluZy0+Y2Fycnlv
dmVyX2J5dGVzKTsKKyAgICB9Cit9CisKIHN0YXRpYyB2b2lkCiByZWFkX2FuZF9kaXNwb3Nl
X29mX3Byb2Nlc3Nfb3V0cHV0IChzdHJ1Y3QgTGlzcF9Qcm9jZXNzICpwLCBjaGFyICpjaGFy
cywKIAkJCQkgICAgc3NpemVfdCBuYnl0ZXMsCg==

--------------pNrGtcmBObRwwOXzSA0eJQDE--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 16:34:02 +0000
Resent-Message-ID: <handler.64735.B64735.169453639219569 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169453639219569
          (code B ref 64735); Tue, 12 Sep 2023 16:34:02 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 16:33:12 +0000
Received: from localhost ([127.0.0.1]:59864 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg6K7-00055Z-VZ
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 12:33:12 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:52164)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qg6K3-00055I-MB
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 12:33:10 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qg6Js-0007xZ-9R; Tue, 12 Sep 2023 12:32:56 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=VZwx7DhGquSA4ytEqqiIcDZjXhsS4uUJ1h0/YBNoRXM=; b=Tsc1X9aST/DO
 QIAxsu/o3H/W4q3JkVNIL7Ci38gMm/VYHYJ3XA0a9rT8RsCcIxO5TtIYB54Zuggfuy0b6oUvqwnr0
 JuLec/i6QKwQdEE6p/DpSgW13BsVZ+CWCpQe/+byCkoifOoqW8juKkfIKvjWdezLwWne7/8EUE746
 /wrMbwzKDHsPVDr+CuosAe4dnWb+xbTKGI2wC5YxRhKD3Xoxt6DSwaBX2jcykB/fpDN/4xoQ2wGIo
 XwwtuO4r4Qvq4UZDcxDzHWgoDVQwsv+e+xIjXCOnW4RQhOxQnWVkRf4NzByaMOJhggiRcMDx3bvT/
 a3RWAbFIzRpk48Z+NFpslQ==;
Date: Tue, 12 Sep 2023 19:32:41 +0300
Message-Id: <831qf3pd1y.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN> (message from
 Dmitry Gutov on Tue, 12 Sep 2023 17:23:53 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <c8eab12f-1978-7385-c142-c590642d35f0@HIDDEN>
 <83sf9eub9d.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN> <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 12 Sep 2023 17:23:53 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> - Dropping most of the setup in read_and_dispose_of_process_output 
> (which creates some consing too) and calling 
> Finternal_default_process_filter directly (call_filter_directly.diff), 
> when it is the filter to be used anyway.
> 
> - Going around that function entirely, skipping the creation of a Lisp 
> string (CHARS -> TEXT) and inserting into the buffer directly (when the 
> filter is set to the default, of course). Copied and adapted some code 
> from 'call_process' for that (read_and_insert_process_output.diff).
> 
> Neither are intended as complete proposals, but here are some 
> comparisons. Note that either of these patches could only help the 
> implementations that don't set up process filter (the naive first one, 
> and the new parallel number 5 above).
> 
> For testing, I used two different repo checkouts that are large enough 
> to not finish too quickly: gecko-dev and torvalds-linux.
> 
> master
> 
> | Function                                         | gecko-dev | linux |
> | find-directory-files-recursively                 |      1.69 |  0.41 |
> | find-directory-files-recursively-2               |      1.16 |  0.28 |
> | find-directory-files-recursively-3               |      0.92 |  0.23 |
> | find-directory-files-recursively-5               |      1.07 |  0.26 |
> | find-directory-files-recursively (rpom 409600)   |      1.42 |  0.35 |
> | find-directory-files-recursively-2 (rpom 409600) |      0.90 |  0.25 |
> | find-directory-files-recursively-5 (rpom 409600) |      0.89 |  0.24 |
> 
> call_filter_directly.diff (basically, not much difference)
> 
> | Function                                         | gecko-dev | linux |
> | find-directory-files-recursively                 |      1.64 |  0.38 |
> | find-directory-files-recursively-5               |      1.05 |  0.26 |
> | find-directory-files-recursively (rpom 409600)   |      1.42 |  0.36 |
> | find-directory-files-recursively-5 (rpom 409600) |      0.91 |  0.25 |
> 
> read_and_insert_process_output.diff (noticeable differences)
> 
> | Function                                         | gecko-dev | linux |
> | find-directory-files-recursively                 |      1.30 |  0.34 |
> | find-directory-files-recursively-5               |      1.03 |  0.25 |
> | find-directory-files-recursively (rpom 409600)   |      1.20 |  0.35 |
> | find-directory-files-recursively-5 (rpom 409600) | (!!) 0.72 |  0.21 |
> 
> So it seems like we have at least two potential ways to implement an 
> asynchronous file listing routine that is as fast or faster than the 
> synchronous one (if only thanks to starting the parsing in parallel).
> 
> Combining the last patch together with using the very large value of 
> read-process-output-max seems to yield the most benefit, but I'm not 
> sure if it's appropriate to just raise that value in our code, though.
> 
> Thoughts?

I'm not sure what exactly is here to think about.  Removing portions
of read_and_insert_process_output, or bypassing it entirely, is not
going to fly, because AFAIU it basically means we don't decode text,
which can only work with plain ASCII file names, and/or don't move the
markers in the process buffer, which also cannot be avoided.  If you
want to conclude that inserting the process's output into a buffer
without consing Lisp strings is faster (which I'm not sure, see below,
but it could be true), then we could try extending
internal-default-process-filter (or writing a new filter function
similar to it) so that it inserts the stuff into the gap and then uses
decode_coding_gap, which converts inserted bytes in-place -- that, at
least, will be correct and will avoid consing intermediate temporary
strings from the process output, then decoding them, then inserting
them.  Other than that, the -2 and -3 variants are very close
runners-up of -5, so maybe I'm missing something, but I see no reason
be too excited here?  I mean, 0.89 vs 0.92? really?

About inserting into the buffer: what we do is insert into the gap,
and when the gap becomes full, we enlarge it.  Enlarging the gap
involves: (a) enlarging the chunk of memory allocated to buffer text
(which might mean we ask the OS for more memory), and (b) moving the
characters after the gap to the right to free space for inserting more
stuff.  This is pretty fast, but still, with a large pipe buffer and a
lot of output, we do this many times, so it could add up to something
pretty tangible.  It's hard to me to tell whether this is
significantly faster than consing strings and inserting them, only
measurements can tell.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 18:49:02 +0000
Resent-Message-ID: <handler.64735.B64735.169454453611311 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169454453611311
          (code B ref 64735); Tue, 12 Sep 2023 18:49:02 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 18:48:56 +0000
Received: from localhost ([127.0.0.1]:60054 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg8RT-0002wM-9H
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 14:48:55 -0400
Received: from out5-smtp.messagingengine.com ([66.111.4.29]:58265)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qg8RQ-0002vo-7B
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 14:48:54 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.nyi.internal (Postfix) with ESMTP id 48F855C0186;
 Tue, 12 Sep 2023 14:48:42 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Tue, 12 Sep 2023 14:48:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694544522; x=1694630922; bh=5y1qtQbgVfNrNKwFJo2jBPpTbQGgQXAYB5Q
 OS8leH/U=; b=HWKaG07aW/f1tgLXn2Ug6+SwKe4ltw4b4sIbXA25OLFw60gIKhG
 76jSeIRaHhuDymhoin2oPcSOaHJEEYyMrhv7unJLD4a6uUjicfRqZV7zxYvhavJM
 0e1yc/28xMRJEhHlwIo68yt0cmuwcTa1wRaqA4AeYXZFMy7MMAac+Kzr5shBs7Lu
 c+43so436oAtjeDziTJHmEDUTp12G0aef591An84mD6V7jJT/ovZ/HA1Eo05Kcef
 3eo8YxY59AYhs9b0kh4uugHnqBuRNFqY5oSooCnXEixfU0tRRlRJBimbGv7JoHA3
 crEhPCFTNO1nYUH3bIKvQ5tgKIIG+lQ0GNw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694544522; x=1694630922; bh=5y1qtQbgVfNrNKwFJo2jBPpTbQGgQXAYB5Q
 OS8leH/U=; b=U14jQBM+z728FKFDRsZvSrKwU+N3/J3SmgmFi3Z+1aBX+F/yPGG
 lq20G/0/LvobW54VJbS2gt32R5lYUPViqQC90UBFZcvtbIzOXkrkl4RxJ2IjWJWO
 1/zeb/6vHhC0l+VP3ViAYT/R+Iblhh/A3uPHvq6BRKe326+CxSYISKrFqEKqirt5
 0nmDS8tRhXm3Hnb2OOrWsrYxbvINhTm4lkOUOPuOr2PKrQ2WHgGQ9FtWMuG7WgNB
 IWYG28st632nGt4s9zXxrP3oOKQF82cTWuWwmFT+/Jsm/Lhs3c64rGVQUfUpm4pk
 mrzUStNNzJNWkUxmyMq8U7Wec0zaJTWOVgg==
X-ME-Sender: <xms:ibIAZRmzDP1aJG8n-QYI9roYjjNI9Yh_QlXNANpW1NfyUK5ISwuOWQ>
 <xme:ibIAZc3TUjgFhq1wKip7BjftGomEFUpaUxYcyJuAEOHTSOkmx1By9tscmM4SNRwUk
 IMcEOqzCCJnohSw-eY>
X-ME-Received: <xmr:ibIAZXq06mOxyDMFf8uNkX3Zs0iprWRVsr7sNkuzQmaTA0d0Qj4cxkzyiM7k1m0>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgudeftdcutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm
 ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg
 htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev
 ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe
 gumhhithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:ibIAZRngb9oxyGphMXXfxI9VOwatkAJ3pgtBl0x_-fMTXTorcfYBLQ>
 <xmx:ibIAZf2kSAZsVunFrNyGNWZ0-tDvCXtcfT3E_mtwzOkJOFHU5jvcfg>
 <xmx:ibIAZQsqTPGkqSzNdvaDQ9gT5MUKI5fhzpWS_eCtc2sXETQrDBygqw>
 <xmx:irIAZQymLTdxnikXCIymByTJ9dkX5amclxgclGuXr36fAtJS0p7DOA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 12 Sep 2023 14:48:40 -0400 (EDT)
Message-ID: <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
Date: Tue, 12 Sep 2023 21:48:37 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <831qf3pd1y.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 12/09/2023 19:32, Eli Zaretskii wrote:
>> Date: Tue, 12 Sep 2023 17:23:53 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>> - Dropping most of the setup in read_and_dispose_of_process_output
>> (which creates some consing too) and calling
>> Finternal_default_process_filter directly (call_filter_directly.diff),
>> when it is the filter to be used anyway.
>>
>> - Going around that function entirely, skipping the creation of a Lisp
>> string (CHARS -> TEXT) and inserting into the buffer directly (when the
>> filter is set to the default, of course). Copied and adapted some code
>> from 'call_process' for that (read_and_insert_process_output.diff).
>>
>> Neither are intended as complete proposals, but here are some
>> comparisons. Note that either of these patches could only help the
>> implementations that don't set up process filter (the naive first one,
>> and the new parallel number 5 above).
>>
>> For testing, I used two different repo checkouts that are large enough
>> to not finish too quickly: gecko-dev and torvalds-linux.
>>
>> master
>>
>> | Function                                         | gecko-dev | linux |
>> | find-directory-files-recursively                 |      1.69 |  0.41 |
>> | find-directory-files-recursively-2               |      1.16 |  0.28 |
>> | find-directory-files-recursively-3               |      0.92 |  0.23 |
>> | find-directory-files-recursively-5               |      1.07 |  0.26 |
>> | find-directory-files-recursively (rpom 409600)   |      1.42 |  0.35 |
>> | find-directory-files-recursively-2 (rpom 409600) |      0.90 |  0.25 |
>> | find-directory-files-recursively-5 (rpom 409600) |      0.89 |  0.24 |
>>
>> call_filter_directly.diff (basically, not much difference)
>>
>> | Function                                         | gecko-dev | linux |
>> | find-directory-files-recursively                 |      1.64 |  0.38 |
>> | find-directory-files-recursively-5               |      1.05 |  0.26 |
>> | find-directory-files-recursively (rpom 409600)   |      1.42 |  0.36 |
>> | find-directory-files-recursively-5 (rpom 409600) |      0.91 |  0.25 |
>>
>> read_and_insert_process_output.diff (noticeable differences)
>>
>> | Function                                         | gecko-dev | linux |
>> | find-directory-files-recursively                 |      1.30 |  0.34 |
>> | find-directory-files-recursively-5               |      1.03 |  0.25 |
>> | find-directory-files-recursively (rpom 409600)   |      1.20 |  0.35 |
>> | find-directory-files-recursively-5 (rpom 409600) | (!!) 0.72 |  0.21 |
>>
>> So it seems like we have at least two potential ways to implement an
>> asynchronous file listing routine that is as fast or faster than the
>> synchronous one (if only thanks to starting the parsing in parallel).
>>
>> Combining the last patch together with using the very large value of
>> read-process-output-max seems to yield the most benefit, but I'm not
>> sure if it's appropriate to just raise that value in our code, though.
>>
>> Thoughts?
> 
> I'm not sure what exactly is here to think about.  Removing portions
> of read_and_insert_process_output, or bypassing it entirely, is not
> going to fly, because AFAIU it basically means we don't decode text,
> which can only work with plain ASCII file names, and/or don't move the
> markers in the process buffer, which also cannot be avoided.

That one was really a test to see whether the extra handling added any 
meaningful consing to affect GC. Removing it didn't make a difference, 
table number 2, so no.

> If you
> want to conclude that inserting the process's output into a buffer
> without consing Lisp strings is faster (which I'm not sure, see below,
> but it could be true),

That's what my tests seem to show, see table 3 (the last one).

> then we could try extending
> internal-default-process-filter (or writing a new filter function
> similar to it) so that it inserts the stuff into the gap and then uses
> decode_coding_gap,

Can that work at all? By the time internal-default-process-filter is 
called, we have already turned the string from char* into Lisp_Object 
text, which we then pass to it. So consing has already happened, IIUC.

> which converts inserted bytes in-place -- that, at
> least, will be correct and will avoid consing intermediate temporary
> strings from the process output, then decoding them, then inserting
> them.  Other than that, the -2 and -3 variants are very close
> runners-up of -5, so maybe I'm missing something, but I see no reason
> be too excited here?  I mean, 0.89 vs 0.92? really?

The important part is not 0.89 vs 0.92 (that would be meaningless 
indeed), but that we have an _asyncronous_ implementation of the feature 
that works as fast as the existing synchronous one (or faster! if we 
also bind read-process-output-max to a large value, the time is 0.72).

The possible applications for that range from simple (printing progress 
bar while the scan is happening) to more advanced (launching a 
concurrent process where we pipe the received file names concurrently to 
'xargs grep'), including visuals (xref buffer which shows the 
intermediate search results right away, updating them gradually, all 
without blocking the UI).

> About inserting into the buffer: what we do is insert into the gap,
> and when the gap becomes full, we enlarge it.  Enlarging the gap
> involves: (a) enlarging the chunk of memory allocated to buffer text
> (which might mean we ask the OS for more memory), and (b) moving the
> characters after the gap to the right to free space for inserting more
> stuff.  This is pretty fast, but still, with a large pipe buffer and a
> lot of output, we do this many times, so it could add up to something
> pretty tangible.  It's hard to me to tell whether this is
> significantly faster than consing strings and inserting them, only
> measurements can tell.

See the benchmark tables and the POC patch in my previous email. Using a 
better filter function would be ideal, but it seems like that's not 
going to fit the current design. Happy to be proven wrong, though.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 19:37:01 +0000
Resent-Message-ID: <handler.64735.B64735.169454736725958 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169454736725958
          (code B ref 64735); Tue, 12 Sep 2023 19:37:01 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 19:36:07 +0000
Received: from localhost ([127.0.0.1]:60110 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg9B9-0006kc-7S
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 15:36:07 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:42098)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qg9B6-0006k6-B4
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 15:36:05 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qg9Av-0007Vo-9J; Tue, 12 Sep 2023 15:35:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=R9F4/GmMtcUGFgKQiY8I7hTyYDm96X+ZJs5UYdgq0I8=; b=Bk4ek8Bg4dC4
 RRyJgQlY7Kso0cBjzliB/rG+0tv4lBZvqzsXbeccKVuoRWjIQsg0WE03Aenfe+Egi2NkKB/z1yobF
 RImbY+Sr7m+c+bVaLY2MJwKZzx0dMT2J2/34DwduSSkthv3hiFGisvv4HSYGsNZxSvBlk2cjIOj0H
 pnyp8qwjBheplAgYZMlXjpStMJ9FSw4zAkwsmMs04zmu2bf1tQtCXwk6qDIC7jOlhc0rk8ajgMj8V
 AaoZH4f2lGpaUJ6QPKanvHnpVywG3wIke1EiNBtnCrxvzq+jeaFLghX30cHQsbph7Uv/k8aHk8HGm
 UQWgRmDdxnKEk6rfBcOY3w==;
Date: Tue, 12 Sep 2023 22:35:37 +0300
Message-Id: <83sf7jnq0m.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN> (message from
 Dmitry Gutov on Tue, 12 Sep 2023 21:48:37 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <2d844a34-857d-3d59-b897-73372baac480@HIDDEN>
 <83bkg2tsu6.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN> <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 12 Sep 2023 21:48:37 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > then we could try extending
> > internal-default-process-filter (or writing a new filter function
> > similar to it) so that it inserts the stuff into the gap and then uses
> > decode_coding_gap,
> 
> Can that work at all? By the time internal-default-process-filter is 
> called, we have already turned the string from char* into Lisp_Object 
> text, which we then pass to it. So consing has already happened, IIUC.

That's why I said "or writing a new filter function".
read_and_dispose_of_process_output will have to call this new filter
differently, passing it the raw text read from the subprocess, where
read_and_dispose_of_process_output current first decodes the text and
produces a Lisp string from it.  Then the filter would need to do
something similar to what insert-file-contents does: insert the raw
input into the gap, then call decode_coding_gap to decode that
in-place.

> > which converts inserted bytes in-place -- that, at
> > least, will be correct and will avoid consing intermediate temporary
> > strings from the process output, then decoding them, then inserting
> > them.  Other than that, the -2 and -3 variants are very close
> > runners-up of -5, so maybe I'm missing something, but I see no reason
> > be too excited here?  I mean, 0.89 vs 0.92? really?
> 
> The important part is not 0.89 vs 0.92 (that would be meaningless 
> indeed), but that we have an _asyncronous_ implementation of the feature 
> that works as fast as the existing synchronous one (or faster! if we 
> also bind read-process-output-max to a large value, the time is 0.72).
> 
> The possible applications for that range from simple (printing progress 
> bar while the scan is happening) to more advanced (launching a 
> concurrent process where we pipe the received file names concurrently to 
> 'xargs grep'), including visuals (xref buffer which shows the 
> intermediate search results right away, updating them gradually, all 
> without blocking the UI).

Hold your horses.  Emacs only reads output from sub-processes when
it's idle.  So printing a progress bar (which makes Emacs not idle)
with the asynchronous implementation is basically the same as having
the synchronous implementation call some callback from time to time
(which will then show the progress).

As for piping to another process, this is best handled by using a
shell pipe, without passing stuff through Emacs.  And even if you do
need to pass it through Emacs, you could do the same with the
synchronous implementation -- only the "xargs" part needs to be
asynchronous, the part that reads file names does not.  Right?

Please note: I'm not saying that the asynchronous implementation is
not interesting.  It might even have advantages in some specific use
cases.  So it is good to have it.  It just isn't a breakthrough,
that's all.  And if we want to use it in production, we should
probably work on adding that special default filter which inserts and
decodes directly into the buffer, because that will probably lower the
GC pressure and thus has hope of being faster.  Or even replace the
default filter implementation with that new one.

> > About inserting into the buffer: what we do is insert into the gap,
> > and when the gap becomes full, we enlarge it.  Enlarging the gap
> > involves: (a) enlarging the chunk of memory allocated to buffer text
> > (which might mean we ask the OS for more memory), and (b) moving the
> > characters after the gap to the right to free space for inserting more
> > stuff.  This is pretty fast, but still, with a large pipe buffer and a
> > lot of output, we do this many times, so it could add up to something
> > pretty tangible.  It's hard to me to tell whether this is
> > significantly faster than consing strings and inserting them, only
> > measurements can tell.
> 
> See the benchmark tables and the POC patch in my previous email. Using a 
> better filter function would be ideal, but it seems like that's not 
> going to fit the current design. Happy to be proven wrong, though.

I see no reason why reading subprocess output couldn't use the same
technique as insert-file-contents does.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 12 Sep 2023 20:29:02 +0000
Resent-Message-ID: <handler.64735.B64735.16945504878781 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16945504878781
          (code B ref 64735); Tue, 12 Sep 2023 20:29:02 +0000
Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 20:28:07 +0000
Received: from localhost ([127.0.0.1]:60141 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qg9zS-0002HZ-Pj
	for submit <at> debbugs.gnu.org; Tue, 12 Sep 2023 16:28:07 -0400
Received: from out5-smtp.messagingengine.com ([66.111.4.29]:59149)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qg9zP-0002H3-E2
 for 64735 <at> debbugs.gnu.org; Tue, 12 Sep 2023 16:28:05 -0400
Received: from compute3.internal (compute3.nyi.internal [10.202.2.43])
 by mailout.nyi.internal (Postfix) with ESMTP id A32465C01E8;
 Tue, 12 Sep 2023 16:27:52 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute3.internal (MEProxy); Tue, 12 Sep 2023 16:27:52 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694550472; x=1694636872; bh=GpU9UjNNZ2rCNxsVAsYpUrwlWi5xiIiqvyH
 gHpFpMzY=; b=de3WkuPF7jTAzFqET5DYh7rVgda2jXwi0vZWMACNX0gYIuKlBKI
 1BgJ6aKJNFy8mjre5dvoLb1GkiINlRZxWbbK3QHZ8r72KZiZCgvDHuw6etfaDH/t
 uepMZqc6Ho/9WnP6t4CBQmuF8DTMuZDe1rm20YkheWSNYYaxiqtPjDNOi9ujvPB/
 Afr5+WKrhmHxQeYh0kkawSMi3RZNYAx/uSIgVPTHWMjlQXH+pGr6+VUV1S1L1RR8
 hmxFENpWBf2eKg9gsELARiS8PUs4ZEITIUG0IQo1l9tu7y/4mCR9m71cokBEt8xv
 lFl8cEDuHOETCGssiLxtReCj2e6bk2wE9OQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=
 1694550472; x=1694636872; bh=GpU9UjNNZ2rCNxsVAsYpUrwlWi5xiIiqvyH
 gHpFpMzY=; b=z3Gyux7C8NG2xZLIHaTEOdgwJYAJNJVp2fCF3oIxtH0CKHFcHRl
 pyai1QAI/YLyt7qsuTa7TwMZe9XUCUtHDdUu6FWewJzNMXv5h9fcVfR4gub90Km1
 L3Vw3468stMtsGrdHL6z/JO9uYY45FHkkj+P40bxRnuw8zYjZ8lpXlfoc09gfMZ8
 Z2fAWb3mqI94+amX3Zy6FshlV9hkGneHjXM6Ntmdzz8jWgT4qEO2wdl+ageC8SAB
 0yd7SvrblPSiENtUsFJYvUWLW/xI+XbNjrrk8RyEx/FYQdHc6ZMQpEg5+QbzpIZF
 koqFs8vdQru/sR0f0trhaEKZ5FI7jZEf1Pw==
X-ME-Sender: <xms:yMkAZYBvmMPLofYHck7SuufQ8UJoYRFI_09BrzVGV8wFQTKnYwva9A>
 <xme:yMkAZaiVCk7XjUvDavzrX30IQY3ptTLuCA10XPTbBqVB5gNykvb3k17SoRr_Evo47
 eiV7kgE0oKHBTrc0V4>
X-ME-Received: <xmr:yMkAZbkAN1u-QvDVS7C0MdYsYJwQ94cE50H6r_t-2fG5WsS7TyIJL9jR8Rvn_Sc>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgudehtdcutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm
 ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg
 htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev
 ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe
 gumhhithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:yMkAZezwgJ6-zp2a5sUgeP_TAD8qYjy1YM_xeEfqUOpeg2beHIAiYg>
 <xmx:yMkAZdRgLlMnY2um4-BXa7ft1hVw_h_TJJ2u4_ue9HowhpRlgo6G7A>
 <xmx:yMkAZZZOFLKaOEHyOw3qTEJ4K1bZ78UsGEr_OxgoCuWmWNqlfsiG7A>
 <xmx:yMkAZfeww_xjE_UbYyilTAlFPfGFyeJ5ytHz12VkcwLkCx1JmBRsqA>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue,
 12 Sep 2023 16:27:51 -0400 (EDT)
Message-ID: <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
Date: Tue, 12 Sep 2023 23:27:49 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83sf7jnq0m.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 12/09/2023 22:35, Eli Zaretskii wrote:
>> Date: Tue, 12 Sep 2023 21:48:37 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>> then we could try extending
>>> internal-default-process-filter (or writing a new filter function
>>> similar to it) so that it inserts the stuff into the gap and then uses
>>> decode_coding_gap,
>>
>> Can that work at all? By the time internal-default-process-filter is
>> called, we have already turned the string from char* into Lisp_Object
>> text, which we then pass to it. So consing has already happened, IIUC.
> 
> That's why I said "or writing a new filter function".
> read_and_dispose_of_process_output will have to call this new filter
> differently, passing it the raw text read from the subprocess, where
> read_and_dispose_of_process_output current first decodes the text and
> produces a Lisp string from it.  Then the filter would need to do
> something similar to what insert-file-contents does: insert the raw
> input into the gap, then call decode_coding_gap to decode that
> in-place.

Does the patch from my last patch-bearing email look similar enough to 
what you're describing?

The one called read_and_insert_process_output.diff

The result there, though, is that a "filter" (in the sense that 
make-process uses that term) is not used at all.

>>> which converts inserted bytes in-place -- that, at
>>> least, will be correct and will avoid consing intermediate temporary
>>> strings from the process output, then decoding them, then inserting
>>> them.  Other than that, the -2 and -3 variants are very close
>>> runners-up of -5, so maybe I'm missing something, but I see no reason
>>> be too excited here?  I mean, 0.89 vs 0.92? really?
>>
>> The important part is not 0.89 vs 0.92 (that would be meaningless
>> indeed), but that we have an _asyncronous_ implementation of the feature
>> that works as fast as the existing synchronous one (or faster! if we
>> also bind read-process-output-max to a large value, the time is 0.72).
>>
>> The possible applications for that range from simple (printing progress
>> bar while the scan is happening) to more advanced (launching a
>> concurrent process where we pipe the received file names concurrently to
>> 'xargs grep'), including visuals (xref buffer which shows the
>> intermediate search results right away, updating them gradually, all
>> without blocking the UI).
> 
> Hold your horses.  Emacs only reads output from sub-processes when
> it's idle.  So printing a progress bar (which makes Emacs not idle)
> with the asynchronous implementation is basically the same as having
> the synchronous implementation call some callback from time to time
> (which will then show the progress).

Obviously there is more work to be done, including further desgin and 
benchmarking. But unlike before, at least the starting performance 
(before further features are added) is not worse.

Note that the variant -5 is somewhat limited since it doesn't use a 
filter - that means that no callbacks a issued while the output is 
arriving, meaning that if it's taken as base, whatever refreshes would 
have to be initiated from somewhere else. E.g. from a timer.

> As for piping to another process, this is best handled by using a
> shell pipe, without passing stuff through Emacs.  And even if you do
> need to pass it through Emacs, you could do the same with the
> synchronous implementation -- only the "xargs" part needs to be
> asynchronous, the part that reads file names does not.  Right?

Yes and no: if both steps are asynchronous, the final output window 
could be displayed right away, rather than waiting for the first step 
(or both) to be finished. Which can be a meaningful improvement for some 
(and still is an upside of 'M-x rgrep').

> Please note: I'm not saying that the asynchronous implementation is
> not interesting.  It might even have advantages in some specific use
> cases.  So it is good to have it.  It just isn't a breakthrough,
> that's all.

Not a breakthrough, of course, just a lower-level insight (hopefully).

I do think it would be meaningful to manage to reduce the runtime of a 
real-life program (which includes other work) by 10-20% solely by 
reducing GC pressure in a generic facility like process output handling.

> And if we want to use it in production, we should
> probably work on adding that special default filter which inserts and
> decodes directly into the buffer, because that will probably lower the
> GC pressure and thus has hope of being faster.  Or even replace the
> default filter implementation with that new one.

But a filter must be a Lisp function, which can't help but accept only 
Lisp strings (not C string) as argument. Isn't that right?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 11:40:02 +0000
Resent-Message-ID: <handler.64735.B64735.169460515120356 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169460515120356
          (code B ref 64735); Wed, 13 Sep 2023 11:40:02 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 11:39:11 +0000
Received: from localhost ([127.0.0.1]:60954 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgOD8-0005IF-BL
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 07:39:10 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:43088)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qgOD6-0005I2-0v
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 07:39:09 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qgOCu-0008Af-KX; Wed, 13 Sep 2023 07:38:56 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=OOjaOQ+avF9SXZYieipNqFMRzI2rBjiRDZy4dy6P2eg=; b=ojck29XUVHmF
 e+sw+h+nysdFLqIXKpWlq7eMxDOLIQk7PvdeZdmO7dmGE3w4mR+8zLvnAQSJpCDe5JQIiZFEyN9pV
 l426RwV/Salel2pYNG5XvhXjUQZr4rxqpTbyx4fJ14ZjEV5yTdEtb3Z+XmvaXVyzoW/2lYr5JGOf0
 qPDUDoeKtt50GMNzrHBCpm6O1VLGdHcAmLzIoQMdx6TNfYJkGk0QJUBRs76+2yqS4Yyka+sLo//ET
 HUpupbW29uDs+U1LNRe4L1M/qlWQ5/xsE1sZZQkxfhWDuXg72k73z5DLkFWz1AQ99mao256+FiJCx
 b7Q6LUwr8/8erxNhinv7Dw==;
Date: Wed, 13 Sep 2023 14:38:43 +0300
Message-Id: <83ledanvzw.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN> (message from
 Dmitry Gutov on Tue, 12 Sep 2023 23:27:49 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <83bd4246-ac41-90ec-1df3-02d0bd59ca44@HIDDEN>
 <834jlttv1p.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN> <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 12 Sep 2023 23:27:49 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > That's why I said "or writing a new filter function".
> > read_and_dispose_of_process_output will have to call this new filter
> > differently, passing it the raw text read from the subprocess, where
> > read_and_dispose_of_process_output current first decodes the text and
> > produces a Lisp string from it.  Then the filter would need to do
> > something similar to what insert-file-contents does: insert the raw
> > input into the gap, then call decode_coding_gap to decode that
> > in-place.
> 
> Does the patch from my last patch-bearing email look similar enough to 
> what you're describing?
> 
> The one called read_and_insert_process_output.diff

No, not entirely: it still produces a Lisp string when decoding is
needed, and then inserts that string into the buffer.

Did you look at what insert-file-contents does?  If not I suggest to
have a look, starting from this comment:

  /* Here, we don't do code conversion in the loop.  It is done by
     decode_coding_gap after all data are read into the buffer.  */

and ending here:

  if (CODING_MAY_REQUIRE_DECODING (&coding)
      && (inserted > 0 || CODING_REQUIRE_FLUSHING (&coding)))
    {
      /* Now we have all the new bytes at the beginning of the gap,
         but `decode_coding_gap` can't have them at the beginning of the gap,
         so we need to move them.  */
      memmove (GAP_END_ADDR - inserted, GPT_ADDR, inserted);
      decode_coding_gap (&coding, inserted);
      inserted = coding.produced_char;
      coding_system = CODING_ID_NAME (coding.id);
    }
  else if (inserted > 0)
    {
      /* Make the text read part of the buffer.  */
      eassert (NILP (BVAR (current_buffer, enable_multibyte_characters)));
      insert_from_gap_1 (inserted, inserted, false);

      invalidate_buffer_caches (current_buffer, PT, PT + inserted);
      adjust_after_insert (PT, PT_BYTE, PT + inserted, PT_BYTE + inserted,
			   inserted);
    }

> The result there, though, is that a "filter" (in the sense that 
> make-process uses that term) is not used at all.

Sure, but in this case we don't need any filtering.  It's basically
the same idea as internal-default-process-filter: we just need to
insert the process output into a buffer, and optionally decode it.

> > And if we want to use it in production, we should
> > probably work on adding that special default filter which inserts and
> > decodes directly into the buffer, because that will probably lower the
> > GC pressure and thus has hope of being faster.  Or even replace the
> > default filter implementation with that new one.
> 
> But a filter must be a Lisp function, which can't help but accept only 
> Lisp strings (not C string) as argument. Isn't that right?

We can provide a special filter identified by a symbol.  Such a filter
will not be Lisp-callable, it will exist for the cases where we need
to insert the output into the process buffer.  Any Lisp callback could
then access the process output as the text of that buffer, no Lisp
strings needed.  I thought this was a worthy goal; apologies if I
misunderstood.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 14:29:04 +0000
Resent-Message-ID: <handler.64735.B64735.16946152935453 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16946152935453
          (code B ref 64735); Wed, 13 Sep 2023 14:29:04 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 14:28:13 +0000
Received: from localhost ([127.0.0.1]:35535 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgQqi-0001Pr-Ro
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 10:28:13 -0400
Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:55661)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qgQqb-0001P9-5Y
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 10:28:12 -0400
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
 by mailout.west.internal (Postfix) with ESMTP id C4A3332007BE;
 Wed, 13 Sep 2023 10:27:53 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute4.internal (MEProxy); Wed, 13 Sep 2023 10:27:54 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694615273; x=1694701673; bh=qw2CUkx912C0EhU21xm0F0yGgOIkoPNpXaq
 0dSd83CU=; b=gdYalQe+frSoSrAKBZoRc6hx11c7rEThP5nn9LASD57AjVT1tda
 2+3ktvA+S6RlwvXsxvcNe/DJSFa5kPO+fbgmlGnRkPQuqfJ61oLzJNzkCyx+g7e4
 kx1FLKf5TE9YkXqJING17bTVFdKPxxM4hyPbV/f1INeXSuSSRLBOi4QlPfJmipLy
 dDLsqYGWzsfp+WqbyxG4raHHrccGOIBei18G8MASOK9at1a/BYPj5u/HoybPYh/J
 fH0Gw/dleI4D9lmibsk7UxErry8X4TuTr81VmjEs70k2wTprnUNo7iCUauDV1bdE
 Ij6x3cbJosj/RFDHKkowaFP3nxBMjLMcPlg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=
 1694615273; x=1694701673; bh=qw2CUkx912C0EhU21xm0F0yGgOIkoPNpXaq
 0dSd83CU=; b=Sz8bHqbZoWoVobpfkxe8AEylesT6As0o+iHqKhr0GbB7rlzt5yi
 M/OEw3ISn+x0R2kder2XAve0l5gYbthg/SQaapoZvRXohLCV5WKTf2ZQLDbjBxxQ
 ZKFIAv2naipmpkiKGP+k3OiVv7k5rAkAp8+MfEu2V4fHOa8xcDmkB1WNtySsUQ7K
 nyStORqb6pAAEdvb8RHTQrE/KPnqGaRHkuy7blIAilKxzNZQaDF2dx0Cygmz8rpU
 ATLkcGEqkohM/Xn2cRVhUXHRUhWzq7BGdnKFNMwgBbY40jTobIryBoSRrx8qKIE3
 6FFBhZE3Ua1zKBpNtDtNfZmaEqm6nfjfZAQ==
X-ME-Sender: <xms:6MYBZeQ7ZWScsPsfo4h7QFMkhoYKlKAQ_53nyoUKbpJIQtuYpn2XfQ>
 <xme:6MYBZTx0D-nDkDhwazXK8PaygeqRGDnZWY-WwazKe1NNNKX8L9oNloPNlwqorZgVs
 nidxOWf1nEKxdl68LE>
X-ME-Received: <xmr:6MYBZb29skSOr8GQhZrZMrFmP1eVKEbFYzjf8igf3fFPyNMOR6bgeVDveCFeIaE>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeikedgjeejucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeffmhhi
 thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth
 htvghrnhepiefgteevheevveffheeltdeukeeiieekueefgedugfefgefhudelgfefveel
 vdevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug
 hmihhtrhihsehguhhtohhvrdguvghv
X-ME-Proxy: <xmx:6cYBZaADsuvmlKrlrsibUd6arx__vTyOejC0cs7BSZb16D5mPfuVJw>
 <xmx:6cYBZXgP0Z1Y2ZeAyStaAG6Ipa4PqydzXzs-JgsZKkCJFC5O8RbxVA>
 <xmx:6cYBZWoCcNnfwob944naC1zDadiq4vhTOOLfT2x-N8kCKEpc2zdZdw>
 <xmx:6cYBZUuL6q1w75Lt0AqNAsO-l-ZnjHFm6PrOR8GXgi4rPtbskID6Wg>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed,
 13 Sep 2023 10:27:51 -0400 (EDT)
Message-ID: <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
Date: Wed, 13 Sep 2023 17:27:49 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83ledanvzw.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 13/09/2023 14:38, Eli Zaretskii wrote:
>> Date: Tue, 12 Sep 2023 23:27:49 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>> That's why I said "or writing a new filter function".
>>> read_and_dispose_of_process_output will have to call this new filter
>>> differently, passing it the raw text read from the subprocess, where
>>> read_and_dispose_of_process_output current first decodes the text and
>>> produces a Lisp string from it.  Then the filter would need to do
>>> something similar to what insert-file-contents does: insert the raw
>>> input into the gap, then call decode_coding_gap to decode that
>>> in-place.
>>
>> Does the patch from my last patch-bearing email look similar enough to
>> what you're describing?
>>
>> The one called read_and_insert_process_output.diff
> 
> No, not entirely: it still produces a Lisp string when decoding is
> needed, and then inserts that string into the buffer.

Are you sure? IIUC the fact that is passes 'curbuf' as the last argument 
to 'decode_coding_c_string' means that decoding happens inside the 
buffer. This has been my explanation for the performance improvement anyway.

If it still generated a Lisp string, I think that would mean that we 
could save the general shape of internal-default-process-filter and just 
improve its implementation for the same measured benefit.

> Did you look at what insert-file-contents does?  If not I suggest to
> have a look, starting from this comment:
> 
>    /* Here, we don't do code conversion in the loop.  It is done by
>       decode_coding_gap after all data are read into the buffer.  */
> 
> and ending here:
> 
>    if (CODING_MAY_REQUIRE_DECODING (&coding)
>        && (inserted > 0 || CODING_REQUIRE_FLUSHING (&coding)))
>      {
>        /* Now we have all the new bytes at the beginning of the gap,
>           but `decode_coding_gap` can't have them at the beginning of the gap,
>           so we need to move them.  */
>        memmove (GAP_END_ADDR - inserted, GPT_ADDR, inserted);
>        decode_coding_gap (&coding, inserted);
>        inserted = coding.produced_char;
>        coding_system = CODING_ID_NAME (coding.id);
>      }
>    else if (inserted > 0)
>      {
>        /* Make the text read part of the buffer.  */
>        eassert (NILP (BVAR (current_buffer, enable_multibyte_characters)));
>        insert_from_gap_1 (inserted, inserted, false);
> 
>        invalidate_buffer_caches (current_buffer, PT, PT + inserted);
>        adjust_after_insert (PT, PT_BYTE, PT + inserted, PT_BYTE + inserted,
> 			   inserted);
>      }

That does look different. I'm not sure how long it would take me to 
adapt this code (if you have an alternative patch to suggest right away, 
please go ahead), but if this method turns out to be faster, it sounds 
like we could improve the performance of 'call_process' the same way. 
That would be a win-win.

>> The result there, though, is that a "filter" (in the sense that
>> make-process uses that term) is not used at all.
> 
> Sure, but in this case we don't need any filtering.  It's basically
> the same idea as internal-default-process-filter: we just need to
> insert the process output into a buffer, and optionally decode it.

Pretty much. But that raises the question of what to do with the 
existing function internal-default-process-filter.

Looking around, it doesn't seem to be used with advice (a good thing: 
the proposed change would break that), but it is called directly in some 
packages like magit-blame, org-assistant, with-editor, wisi, sweeprolog, 
etc. I suppose we'd just keep it around unchanged.

>>> And if we want to use it in production, we should
>>> probably work on adding that special default filter which inserts and
>>> decodes directly into the buffer, because that will probably lower the
>>> GC pressure and thus has hope of being faster.  Or even replace the
>>> default filter implementation with that new one.
>>
>> But a filter must be a Lisp function, which can't help but accept only
>> Lisp strings (not C string) as argument. Isn't that right?
> 
> We can provide a special filter identified by a symbol.  Such a filter
> will not be Lisp-callable, it will exist for the cases where we need
> to insert the output into the process buffer.

The would be the safest alternative. OTOH, this way we'd pass up on the 
opportunity to make all existing asynchronous processes without custom 
filters, a little bit faster in one fell swoop.

> Any Lisp callback could
> then access the process output as the text of that buffer, no Lisp
> strings needed.  I thought this was a worthy goal; apologies if I
> misunderstood.

Sorry, I was just quibbling about the terminology, to make sure we are 
on the same page on what is being proposed. If the patch and evidence 
look good to people, that is. And I'd like to explore that improvement 
venue to the max.

But note that it has limitations as well (e.g. filter is the only way to 
get in-process callbacks from the process, and avoiding it for best 
performance will require external callback such as timers), so if 
someone has any better ideas how to improve GC time to a comparable 
extent but keep design unchanged, that's also welcome.

Should we also discuss increasing the default of 
read-process-output-max? Even increasing it 10x (not necessarily 100x) 
creates a noticeable difference, especially combined with the proposed 
change.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 15:09:02 +0000
Resent-Message-ID: <handler.64735.B64735.169461769520288 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169461769520288
          (code B ref 64735); Wed, 13 Sep 2023 15:09:02 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 15:08:15 +0000
Received: from localhost ([127.0.0.1]:35647 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgRTT-0005H9-A0
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 11:08:15 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:56124)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qgRTQ-0005Gq-MI
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 11:08:14 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qgRTF-0005tL-8O; Wed, 13 Sep 2023 11:08:01 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=P9+tH0UISD2MLZiMZ3tAEzZoaPa7jA2EjoJzp7FNr/Y=; b=WYWJL3LF6GnT
 VCDXFBve8y/JJN1UL5CFoIT9qElM22BosGeEbsMlIKyCUdGmebjk1MLnXITWSC8avRT+PUJzp5Mak
 qwzivg3YN0esahY7e6DZ5ubpRGQt4qB4mv0zmC98DQ/1+X0hd7zR99Q/qbV69qtF/R5wNUjZCq+9T
 CM6+ndYLJXFrhJo0K50aZdFsDGH+ei1xrE1gmRfcyNzN6f2ALM4kRgeSZxEcxQhFT9fwXVuHpd8d3
 od+Nt3Xwao9AgRP5XNVaeLX/+RUlrJIdfaEdP5a0jEw/JbRslaysyJ2qk2oIFGtWDcUjqqodMskYI
 bN3+T2oWoftS1M7P7IhlZw==;
Date: Wed, 13 Sep 2023 18:07:48 +0300
Message-Id: <83r0n2m7qz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN> (message from
 Dmitry Gutov on Wed, 13 Sep 2023 17:27:49 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@HIDDEN>
 <83a5vlsanw.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN> <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 13 Sep 2023 17:27:49 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> >> Does the patch from my last patch-bearing email look similar enough to
> >> what you're describing?
> >>
> >> The one called read_and_insert_process_output.diff
> > 
> > No, not entirely: it still produces a Lisp string when decoding is
> > needed, and then inserts that string into the buffer.
> 
> Are you sure? IIUC the fact that is passes 'curbuf' as the last argument 
> to 'decode_coding_c_string' means that decoding happens inside the 
> buffer. This has been my explanation for the performance improvement anyway.

Yes, you are right, sorry.

> > Sure, but in this case we don't need any filtering.  It's basically
> > the same idea as internal-default-process-filter: we just need to
> > insert the process output into a buffer, and optionally decode it.
> 
> Pretty much. But that raises the question of what to do with the 
> existing function internal-default-process-filter.

Nothing.  It will remain as the default filter.

> Looking around, it doesn't seem to be used with advice (a good thing: 
> the proposed change would break that), but it is called directly in some 
> packages like magit-blame, org-assistant, with-editor, wisi, sweeprolog, 
> etc. I suppose we'd just keep it around unchanged.

Yes.

> > We can provide a special filter identified by a symbol.  Such a filter
> > will not be Lisp-callable, it will exist for the cases where we need
> > to insert the output into the process buffer.
> 
> The would be the safest alternative. OTOH, this way we'd pass up on the 
> opportunity to make all existing asynchronous processes without custom 
> filters, a little bit faster in one fell swoop.

We could change the ones we care about, though.

> Should we also discuss increasing the default of 
> read-process-output-max? Even increasing it 10x (not necessarily 100x) 
> creates a noticeable difference, especially combined with the proposed 
> change.

That should be limited to specific cases where we expect to see a lot
of stuff coming from the subprocess.  We could also discuss changing
the default value, but that would require measurements in as many
cases as we can afford.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 17:28:02 +0000
Resent-Message-ID: <handler.64735.B64735.169462604922309 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169462604922309
          (code B ref 64735); Wed, 13 Sep 2023 17:28:02 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 17:27:29 +0000
Received: from localhost ([127.0.0.1]:35891 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgTeC-0005nk-Qo
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 13:27:29 -0400
Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:45405)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qgTe9-0005nW-FP
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 13:27:27 -0400
Received: from compute6.internal (compute6.nyi.internal [10.202.2.47])
 by mailout.west.internal (Postfix) with ESMTP id DB7DF3200989;
 Wed, 13 Sep 2023 13:27:13 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute6.internal (MEProxy); Wed, 13 Sep 2023 13:27:14 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694626033; x=1694712433; bh=VDCkBYEU3Ebxlnylz5EUpmpYkukUVkNSGuj
 0ISVOiwk=; b=pg/+pvH8NcfvYCEDSx9sWJzy6L4LLntKqM41m8wyAHsrrKEsLYS
 vTOLIDFZEeGk7o6CdzsaJHj6pD1LNSElqEACMgjDZqG50VtGB2v3UDczT53R8vjg
 s7wB/kKDYOfWPfm94VIzL8HywPtGDgAzU0F2tHnmOz0QB/NxevrQRdVGq2u/OWnq
 4zhJRWo+ACrzh12zrL+SXh+ShrshUBtmOHYGdTrJyH4SypTZHTwlkOTrUBEMbWWv
 UwK9qifyerpq6Ux0gYEO2dXJzpw/6wIl/oincYjFedD6bwB6htUu0g3JlP8zaS18
 KLvXHJk6Llsqqny+/yFTllPwjdu1aMG7Zyw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=
 1694626033; x=1694712433; bh=VDCkBYEU3Ebxlnylz5EUpmpYkukUVkNSGuj
 0ISVOiwk=; b=b+Rrx2dFfS4hBpqLkjPfcE5RNnodJe2Y43NJAwfuo+R6hdEfXm3
 wN8fxpj/tD84o+LiozB/Y0VnYfErBG6WyjALiy4a0J+DvgEkRxa9l+7rhtHxqZVM
 MMgcg4sPTEQDS+95sibneUjORFuXksJvMdNYBHUy3vesvWdkbH7DB36hKPy3Ha2w
 QOff9T5jF40dtMlLo51kHR7enwm3Ha5neESreKkWIWQvnSIHEWHQASnmxdZQO/ek
 XWTX0kKNYJHA4nYmOVx0Qaukk9K2giDA2q6UDRKFxU6qgXpsBHJY2yMcqUWuDPNp
 q2lqDSykXA0THwZtddBqCoe2At+iwrUvWeg==
X-ME-Sender: <xms:8fABZdk72gfkc29K7dPOyakkOfrApHo3392Glyihzw2bFY6eqToDHg>
 <xme:8fABZY2NsguHzoyN6GKLvgD-mnF6kU_GsXcIppAvz3n0MDHiyUNFHtCtdJ21p9SUm
 caYrQja0oI7TpjR7GI>
X-ME-Received: <xmr:8fABZTpLjyFBF2MMcoh3HwoG-N_WbojzZGvVZOLvqDbSXRu3roSx90lCDeGKlkw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeikedguddufecutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm
 ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg
 htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev
 ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe
 gumhhithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:8fABZdmP9PHEubJ73eyFT-0fMEGktkOD7A_AgwxBWBTSa_x-rYSg7g>
 <xmx:8fABZb35rcRaDrsX0sTkRHq7fXakk-uUepHuX7Mm0CeUOOH8a1CoXA>
 <xmx:8fABZcu_la1HUzwIzRitWfXFMr_bOzDZqsz9DgPa247hDBLM1eCnTQ>
 <xmx:8fABZcwzbBX-8OSDZoIFoH2BKW2WMUIXwjXlWV42upj8VUjJQzkIig>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed,
 13 Sep 2023 13:27:11 -0400 (EDT)
Message-ID: <26afa109-9ba3-78a3-0e68-7585ae8e3a19@HIDDEN>
Date: Wed, 13 Sep 2023 20:27:09 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN>
 <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
 <83r0n2m7qz.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83r0n2m7qz.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 13/09/2023 18:07, Eli Zaretskii wrote:
>> Date: Wed, 13 Sep 2023 17:27:49 +0300
>> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>>   64735 <at> debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@HIDDEN>
>>
>>>> Does the patch from my last patch-bearing email look similar enough to
>>>> what you're describing?
>>>>
>>>> The one called read_and_insert_process_output.diff
>>>
>>> No, not entirely: it still produces a Lisp string when decoding is
>>> needed, and then inserts that string into the buffer.
>>
>> Are you sure? IIUC the fact that is passes 'curbuf' as the last argument
>> to 'decode_coding_c_string' means that decoding happens inside the
>> buffer. This has been my explanation for the performance improvement anyway.
> 
> Yes, you are right, sorry.

So we're not going to try the gap-based approach? Okay.

>>> Sure, but in this case we don't need any filtering.  It's basically
>>> the same idea as internal-default-process-filter: we just need to
>>> insert the process output into a buffer, and optionally decode it.
>>
>> Pretty much. But that raises the question of what to do with the
>> existing function internal-default-process-filter.
> 
> Nothing.  It will remain as the default filter.

Okay, if you are sure.

>>> We can provide a special filter identified by a symbol.  Such a filter
>>> will not be Lisp-callable, it will exist for the cases where we need
>>> to insert the output into the process buffer.
>>
>> The would be the safest alternative. OTOH, this way we'd pass up on the
>> opportunity to make all existing asynchronous processes without custom
>> filters, a little bit faster in one fell swoop.
> 
> We could change the ones we care about, though.

Which ones do we care about? I've found a bunch of 'make-process' calls 
without :filter specified (flymake backends, ). Do we upgrade them all?

The difference is likely not critical in most of them, but the change 
would likely result in small reduction of GC pressure in the 
corresponding Emacs sessions.

We'll also need to version-guard the ones that are in ELPA.

We don't touch the implementations of functions like start-file-process, 
right?

What about the callers of functions like 
start-file-process-shell-command who want to take advantage of the 
improvement? Are we okay with them all having to call 
(set-process-filter proc 'buffer) on the returned process value?

>> Should we also discuss increasing the default of
>> read-process-output-max? Even increasing it 10x (not necessarily 100x)
>> creates a noticeable difference, especially combined with the proposed
>> change.
> 
> That should be limited to specific cases where we expect to see a lot
> of stuff coming from the subprocess.

So it would be okay to bump it in particular functions? Okay.

> We could also discuss changing
> the default value, but that would require measurements in as many
> cases as we can afford.

If you have some particular scenarios in mind, and what to look out for, 
I could test them out at least on one platform.

I'm not sure what negatives to test for, though. Raising the limit 10x 
is unlikely to lead to an OOM, but I guess some processes could grow 
higher latency?..

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 19:33:01 +0000
Resent-Message-ID: <handler.64735.B64735.169463355322760 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169463355322760
          (code B ref 64735); Wed, 13 Sep 2023 19:33:01 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 19:32:33 +0000
Received: from localhost ([127.0.0.1]:36083 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgVbE-0005v1-GQ
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 15:32:33 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:47812)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qgVbA-0005um-0J
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 15:32:31 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qgVay-0005eK-2F; Wed, 13 Sep 2023 15:32:16 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date:
 mime-version; bh=z8u8XtDibBmHLeS/Myf7pIOYA7LEmdU1n04fZ78sMR8=; b=E4I7XGiZc84y
 4ylu6wx/ATF1g+TLl8O+bg8kRgfFwZL2Aiskf+rTCal/7rE1j9ydFKPpn5dv4oovuIZ3XFHfaxHWb
 un35wOyU8v8DleuGJE5/tXjo/5SSQ+F1LqVhuwV0eM8+FuxAXn7Am45rCoIt3rPBxzJSVlVdXx6Nv
 57xGZjeMwjQjUho1668Qa1EwV/HF5rRn59CYkK3m0jf8f7YmYZjSmJWt8fqN+WCD1l5YS1IqMejI1
 Bnxm4SHeD7UQ1CLVO2z2I+E7nO09VceUf2Jz7Wwc05kqoWfm1mnj2M1trkoBUD0CUGhkcPH4tw1mc
 inuUkJIGBTiy0AEVUSHpxg==;
Date: Wed, 13 Sep 2023 22:32:03 +0300
Message-Id: <83il8dna30.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <26afa109-9ba3-78a3-0e68-7585ae8e3a19@HIDDEN> (message from
 Dmitry Gutov on Wed, 13 Sep 2023 20:27:09 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <69a98e2a-5816-d36b-9d04-8609291333cd@HIDDEN>
 <87351cs8no.fsf@localhost>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN>
 <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
 <83r0n2m7qz.fsf@HIDDEN> <26afa109-9ba3-78a3-0e68-7585ae8e3a19@HIDDEN>
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 13 Sep 2023 20:27:09 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> >> Are you sure? IIUC the fact that is passes 'curbuf' as the last argument
> >> to 'decode_coding_c_string' means that decoding happens inside the
> >> buffer. This has been my explanation for the performance improvement anyway.
> > 
> > Yes, you are right, sorry.
> 
> So we're not going to try the gap-based approach? Okay.

decode_coding_c_string does that internally.

> >> The would be the safest alternative. OTOH, this way we'd pass up on the
> >> opportunity to make all existing asynchronous processes without custom
> >> filters, a little bit faster in one fell swoop.
> > 
> > We could change the ones we care about, though.
> 
> Which ones do we care about? I've found a bunch of 'make-process' calls 
> without :filter specified (flymake backends, ). Do we upgrade them all?
> 
> The difference is likely not critical in most of them, but the change 
> would likely result in small reduction of GC pressure in the 
> corresponding Emacs sessions.
> 
> We'll also need to version-guard the ones that are in ELPA.
> 
> We don't touch the implementations of functions like start-file-process, 
> right?
> 
> What about the callers of functions like 
> start-file-process-shell-command who want to take advantage of the 
> improvement? Are we okay with them all having to call 
> (set-process-filter proc 'buffer) on the returned process value?

I think these questions are slightly premature.  We should first have
the implementation of that filter, and then look for candidates that
could benefit from it.  My tendency is to change only callers which
are in many cases expected to get a lot of stuff from a subprocess, so
shell buffers are probably out.  But we could discuss that later.

> > We could also discuss changing
> > the default value, but that would require measurements in as many
> > cases as we can afford.
> 
> If you have some particular scenarios in mind, and what to look out for, 
> I could test them out at least on one platform.

Didn't think about that enough to have scenarios.

> I'm not sure what negatives to test for, though. Raising the limit 10x 
> is unlikely to lead to an OOM, but I guess some processes could grow 
> higher latency?..

With a large buffer and small subprocess output we will ask the OS for
a large memory increment for no good reason.  Then the following GC
will want to compact the gap, which means it will be slower.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Dmitry Gutov <dmitry@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 13 Sep 2023 20:39:01 +0000
Resent-Message-ID: <handler.64735.B64735.169463753729451 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.169463753729451
          (code B ref 64735); Wed, 13 Sep 2023 20:39:01 +0000
Received: (at 64735) by debbugs.gnu.org; 13 Sep 2023 20:38:57 +0000
Received: from localhost ([127.0.0.1]:36183 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgWdQ-0007et-Se
	for submit <at> debbugs.gnu.org; Wed, 13 Sep 2023 16:38:57 -0400
Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:57285)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dmitry@HIDDEN>) id 1qgWdJ-0007ea-Ku
 for 64735 <at> debbugs.gnu.org; Wed, 13 Sep 2023 16:38:51 -0400
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
 by mailout.west.internal (Postfix) with ESMTP id 3C2D832009A4;
 Wed, 13 Sep 2023 16:38:34 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute5.internal (MEProxy); Wed, 13 Sep 2023 16:38:34 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc
 :cc:content-transfer-encoding:content-type:content-type:date
 :date:from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to; s=fm3; t=
 1694637513; x=1694723913; bh=sELkUu/ROWCusXj+gqW6rAA6N72rSwsQEOe
 VjtxuYlQ=; b=hO5uJVW2YRocSDOQrqWSaNxW8PvO84YbuLiSBz5dDyyF24X/I48
 aO/i4bg6WXC4GPCT821FwrqrrGawaYGG0741XCz1A139pOllBNvWOzYo/8QmPt/l
 FTFq+AvVnCkzD1ICQpL++7USCOTwQSdAz6gKcAutkoqsocdDWrDjhoYxLx6zTJEW
 rpH/uUuoccsQopp5iBMZAhyV6rx5dnWYjEQ5Kx196PWC8RqFGiCk6y3c2nfM8Ovi
 LSg/3gkBZRN1g6Q8wi/ubjnN60nutlnf2vsRgbLrAVgyDsau96vfGb8pHXcqCPYR
 dQnWUdfd1us2WAFCdNbSDe1kjPgOA6Rirnw==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:cc:content-transfer-encoding
 :content-type:content-type:date:date:feedback-id:feedback-id
 :from:from:in-reply-to:in-reply-to:message-id:mime-version
 :references:reply-to:sender:subject:subject:to:to:x-me-proxy
 :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=
 1694637513; x=1694723913; bh=sELkUu/ROWCusXj+gqW6rAA6N72rSwsQEOe
 VjtxuYlQ=; b=MWbwc3sdT2+wEXP1oj7L85dkyFDomklnwXR2zTqz4cLeCfyltZ0
 MFU7AnFbYc4z5bcBDImEHx+OgD/zJUe7Ycxzzi8/Jf4yoNFdfRmipp7x35c6azhy
 daQFCktp39T61ep8ZZCT+EFpibPIh3mEaTDTFvzU+LVG0cBOy2gv6aipkm7/+PaL
 XqDLSG8p439wYyKUFYB24tl+BKEfs2ArGmQXJgE07y9zkqRAPrhSBdyxoEJQXmLz
 zTx9B6q7cIiDcBEjZUg+TquwlTYdZKayomDTAPOq8DuMn7RyL4sMtDscI6K7LX29
 bekbJnBoz7/5730Gut30yUFsO3ymWowiUwA==
X-ME-Sender: <xms:yB0CZdjb0rccsMy9i7NsO1LAe6nEoI8DLEPmqTWlo4IDFCbNtj-ZAg>
 <xme:yB0CZSDHYflUKZqY3CfNG4GjHOOcBuJEk9nnqSecoBJo3Zi3xctm6o6CHilk8UNAj
 -zsxi0DqWQ_763C1U8>
X-ME-Received: <xmr:yB0CZdHgaFBTtMBUixvHTGjHFBJjnKDAJw8oLJVr3Zl1EwUJ98XzwmiUf3N313k>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeikedgudehvdcutefuodetggdotefrod
 ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh
 necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd
 enucfjughrpefkffggfgfuvfevfhfhjggtgfesthekredttdefjeenucfhrhhomhepffhm
 ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg
 htthgvrhhnpefhffehleejffegffeugefhkeektdffgfehjedvgeejtedtudehueffgffg
 feejheenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe
 gumhhithhrhiesghhuthhovhdruggvvh
X-ME-Proxy: <xmx:yB0CZSTrMjq76mgZk76BtwqvguMhCWQY08sOWPofEdDh4P027VhHGQ>
 <xmx:yB0CZaw6ZMAFpjJ6oEQ8FC5iGQkHhZ0YHQSn48lCs8QU6C_FGY1fBw>
 <xmx:yB0CZY4f2rYjmnU2hfn8oakPKe8fioz6HzBF1zSYTVqwxyYXizfZbg>
 <xmx:yR0CZe-o5bKMcYKmVrJHvsiqSfTqEasrl8W8q1kmTxM_RMY2YhMTzQ>
Feedback-ID: i0e71465a:Fastmail
Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed,
 13 Sep 2023 16:38:31 -0400 (EDT)
Message-ID: <e46f996a-0bed-8796-23d7-c9c8b7d428be@HIDDEN>
Date: Wed, 13 Sep 2023 23:38:29 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Content-Language: en-US
References: <iermszrwqj4.fsf@HIDDEN>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN>
 <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
 <83r0n2m7qz.fsf@HIDDEN>
 <26afa109-9ba3-78a3-0e68-7585ae8e3a19@HIDDEN>
 <83il8dna30.fsf@HIDDEN>
From: Dmitry Gutov <dmitry@HIDDEN>
In-Reply-To: <83il8dna30.fsf@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.2 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.2 (---)

On 13/09/2023 22:32, Eli Zaretskii wrote:

>>> We could change the ones we care about, though.
>>
>> Which ones do we care about? I've found a bunch of 'make-process' calls
>> without :filter specified (flymake backends, ). Do we upgrade them all?
>>
>> The difference is likely not critical in most of them, but the change
>> would likely result in small reduction of GC pressure in the
>> corresponding Emacs sessions.
>>
>> We'll also need to version-guard the ones that are in ELPA.
>>
>> We don't touch the implementations of functions like start-file-process,
>> right?
>>
>> What about the callers of functions like
>> start-file-process-shell-command who want to take advantage of the
>> improvement? Are we okay with them all having to call
>> (set-process-filter proc 'buffer) on the returned process value?
> 
> I think these questions are slightly premature.  We should first have
> the implementation of that filter, and then look for candidates that
> could benefit from it.

The implementation in that patch looks almost complete to me, unless you 
have any further comments. The main difference would be the change in 
the dispatch comparison from

   if (p->filter == Qinternal_default_process_filter)

to

   if (p->filter == Qbuffer)

, I think. Of course I can re-submit the amended patch, if you like.

Regarding documentation, though. How will we describe that new value?

The process filter is described like this in the manual:

    This function gives PROCESS the filter function FILTER.  If FILTER
      is ‘nil’, it gives the process the default filter, which inserts
      the process output into the process buffer.  If FILTER is ‘t’,
      Emacs stops accepting output from the process, unless it’s a
      network server process that listens for incoming connections.

What can we add?

   If FILTER is ‘buffer’, it works like the default one, only a bit faster.

?

> My tendency is to change only callers which
> are in many cases expected to get a lot of stuff from a subprocess, so
> shell buffers are probably out.  But we could discuss that later.

When I'm thinking of start-file-process-shell-command, I have in mind 
project--files-in-directory, which currently uses 
process-file-shell-command. Though I suppose most cases would be more 
easily converted to use make-process (like xref-matches-in-files uses 
process-file for launching a shell pipeline already).

I was also thinking about Flymake backends because those work in the 
background. The outputs are usually small, but can easily grow in rare 
cases, without particular limit. Flymake also runs in the background, 
meaning whatever extra work it has to do (or especially GC pressure), 
affects the delays when editing.

>>> We could also discuss changing
>>> the default value, but that would require measurements in as many
>>> cases as we can afford.
>>
>> If you have some particular scenarios in mind, and what to look out for,
>> I could test them out at least on one platform.
> 
> Didn't think about that enough to have scenarios.
> 
>> I'm not sure what negatives to test for, though. Raising the limit 10x
>> is unlikely to lead to an OOM, but I guess some processes could grow
>> higher latency?..
> 
> With a large buffer and small subprocess output we will ask the OS for
> a large memory increment for no good reason.  Then the following GC
> will want to compact the gap, which means it will be slower.

I wonder what scenario that might become apparent in. Launching many 
small processes at once? Can't think of a realistic test case.

Anyway, if you prefer to put off the discussion about changing the 
default, that's fine by me. Or split into a separate bug.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 14 Sep 2023 05:42:01 +0000
Resent-Message-ID: <handler.64735.B64735.16946701006684 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 64735
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Dmitry Gutov <dmitry@HIDDEN>
Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN, 64735 <at> debbugs.gnu.org
Received: via spool by 64735-submit <at> debbugs.gnu.org id=B64735.16946701006684
          (code B ref 64735); Thu, 14 Sep 2023 05:42:01 +0000
Received: (at 64735) by debbugs.gnu.org; 14 Sep 2023 05:41:40 +0000
Received: from localhost ([127.0.0.1]:36572 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1qgf6h-0001jj-UP
	for submit <at> debbugs.gnu.org; Thu, 14 Sep 2023 01:41:40 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:56714)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1qgf6f-0001jU-7c
 for 64735 <at> debbugs.gnu.org; Thu, 14 Sep 2023 01:41:39 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1qgf6S-00083R-UQ; Thu, 14 Sep 2023 01:41:24 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From:
 Date; bh=CFuYYF71FoRm+7xntSM48ncTkuKifQE0ZqqGJ+bJEd8=; b=YwNZsL35mCFkOshehrsN
 Xxqua8iG1fvsCx4nGJe6ptj8d8g+Xvo0rjMrq9VY+PppgZywmsXL+UwUBfO7nB5t84oxg9l/sDd1R
 KrN7C3NsYX3CU4xObxVjyzDWgJpemf2aAUyXvQ5oeQq+l0ObX+r+/011+zjiNm87FCYDGzvo6BOwp
 tHc/T1J4CIHbuSl8cvtfCFnBG8fQkwf1RYWNHRVfpIq0BNmhWGx+XhDbRHBqjSRexzYyOsltbB7nk
 cM/YYNJqq1e6ud++5mbSo34hm16n3hgLHYDOJNOwBMQXvrvDHSoYmkgMkd/6h7guEnNmdefCYMMNA
 NZGSCEyNi/FaGw==;
Date: Thu, 14 Sep 2023 08:41:11 +0300
Message-Id: <83bke5mhvs.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-Reply-To: <e46f996a-0bed-8796-23d7-c9c8b7d428be@HIDDEN> (message from
 Dmitry Gutov on Wed, 13 Sep 2023 23:38:29 +0300)
References: <iermszrwqj4.fsf@HIDDEN>
 <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@HIDDEN>
 <878rb3m43b.fsf@localhost>
 <eba1c6f1-6ad6-d7eb-b43d-a4173622ce99@HIDDEN>
 <83v8e6lyi4.fsf@HIDDEN>
 <35f8b664-0241-9f96-1aa0-20ca51b2d34c@HIDDEN>
 <59c30342-a7e0-d83b-a128-0faae4cbd633@HIDDEN>
 <83pm4bi6qa.fsf@HIDDEN>
 <d319bd54-3f0d-a205-3412-ef3961074416@HIDDEN>
 <83bkfs2tw5.fsf@HIDDEN>
 <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@HIDDEN>
 <83il8lxjcu.fsf@HIDDEN>
 <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@HIDDEN>
 <8334zmtwwi.fsf@HIDDEN>
 <e480d2fb-0912-c9d6-66c8-d11e07482545@HIDDEN>
 <83tts0rkh5.fsf@HIDDEN>
 <e6dd2d5c-9a88-6e18-f1e2-544a9cb1beaf@HIDDEN>
 <831qf3pd1y.fsf@HIDDEN>
 <28a7916e-92d5-77ab-a61e-f85b59ac76b1@HIDDEN>
 <83sf7jnq0m.fsf@HIDDEN>
 <5c493f86-0af5-256f-41a7-7d886ab4c5e4@HIDDEN>
 <83ledanvzw.fsf@HIDDEN>
 <bc4bb3de-d433-7249-cb53-2f312656589d@HIDDEN>
 <83r0n2m7qz.fsf@HIDDEN>
 <26afa109-9ba3-78a3-0e68-7585ae8e3a19@HIDDEN>
 <83il8dna30.fsf@HIDDEN> <e46f996a-0bed-8796-23d7-c9c8b7d428be@HIDDEN>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 13 Sep 2023 23:38:29 +0300
> Cc: luangruo@HIDDEN, sbaugh@HIDDEN, yantar92@HIDDEN,
>  64735 <at> debbugs.gnu.org
> From: Dmitry Gutov <dmitry@HIDDEN>
> 
> > I think these questions are slightly premature.  We should first have
> > the implementation of that filter, and then look for candidates that
> > could benefit from it.
> 
> The implementation in that patch looks almost complete to me, unless you 
> have any further comments.

Fine, then please post a complete patch with all the bells and
whistles, and let's have it reviewed more widely.  (I suggest a new
bug report, as this one is already prohibitively long to follow,
includes unrelated issues, and I fear some people will ignore patches
posted to it).  I think there are a few subtleties we still need to
figure out.

> The main difference would be the change in 
> the dispatch comparison from
> 
>    if (p->filter == Qinternal_default_process_filter)
> 
> to
> 
>    if (p->filter == Qbuffer)

Btw, both of the above are mistakes: you cannot compare Lisp objects
as if they were simple values.  You must use EQ.

>     This function gives PROCESS the filter function FILTER.  If FILTER
>       is ‘nil’, it gives the process the default filter, which inserts
>       the process output into the process buffer.  If FILTER is ‘t’,
>       Emacs stops accepting output from the process, unless it’s a
>       network server process that listens for incoming connections.
> 
> What can we add?
> 
>    If FILTER is ‘buffer’, it works like the default one, only a bit faster.
> 
> ?

  If FILTER is the symbol ‘buffer’, it works like the default filter,
  but makes some shortcuts to be faster: it doesn't adjust markers and
  the process mark (something else?).

Of course, the real text will depend on what the final patch will look
like: I'm not yet sure I understand which parts of
internal-default-process-filter you want to keep in this alternative
filter.  (If you intend to keep all of them, it might be better to
replace internal-default-process-filter completely, perhaps first with
some variable exposed to Lisp which we could use to see if the new one
causes issues.)

> > My tendency is to change only callers which
> > are in many cases expected to get a lot of stuff from a subprocess, so
> > shell buffers are probably out.  But we could discuss that later.
> 
> When I'm thinking of start-file-process-shell-command, I have in mind 
> project--files-in-directory, which currently uses 
> process-file-shell-command. Though I suppose most cases would be more 
> easily converted to use make-process (like xref-matches-in-files uses 
> process-file for launching a shell pipeline already).
> 
> I was also thinking about Flymake backends because those work in the 
> background. The outputs are usually small, but can easily grow in rare 
> cases, without particular limit. Flymake also runs in the background, 
> meaning whatever extra work it has to do (or especially GC pressure), 
> affects the delays when editing.

I think we will have to address these on a case by case basis.  The
issues and aspects are not trivial and sometimes subtle.  We might
even introduce knobs to allow different pipe sizes if there's no
one-fits-all value for a specific function using these primitives.

> >> I'm not sure what negatives to test for, though. Raising the limit 10x
> >> is unlikely to lead to an OOM, but I guess some processes could grow
> >> higher latency?..
> > 
> > With a large buffer and small subprocess output we will ask the OS for
> > a large memory increment for no good reason.  Then the following GC
> > will want to compact the gap, which means it will be slower.
> 
> I wonder what scenario that might become apparent in. Launching many 
> small processes at once? Can't think of a realistic test case.

One process suffices.  The effect might not be significant, but
slowdowns due to new features are generally considered regressions.

Last modified: Thu, 14 Sep 2023 05:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.