GNU bug report logs - #49264
28.0.50; project.el+tramp performance issue

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Ergus <spacibba@HIDDEN>; dated Mon, 28 Jun 2021 22:12:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 30 Jun 2021 13:35:50 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 30 09:35:50 2021
Received: from localhost ([127.0.0.1]:56990 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyaNZ-0006Q6-R0
	for submit <at> debbugs.gnu.org; Wed, 30 Jun 2021 09:35:50 -0400
Received: from eggs.gnu.org ([209.51.188.92]:38196)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lyaNY-0006Pp-5x
 for 49264 <at> debbugs.gnu.org; Wed, 30 Jun 2021 09:35:48 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:38542)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lyaNR-0004lT-ME; Wed, 30 Jun 2021 09:35:41 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3953
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1lyaNP-0001Q4-PX; Wed, 30 Jun 2021 09:35:41 -0400
Date: Wed, 30 Jun 2021 16:35:41 +0300
Message-Id: <83h7hfmq4y.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Phil Sainty <psainty@HIDDEN>
In-Reply-To: <116428124d5c9911116bc992b360f2d7@HIDDEN> (message
 from Phil Sainty on Thu, 01 Jul 2021 01:25:07 +1200)
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
 <83mtr8ooz4.fsf@HIDDEN> <20210629222142.jbyegdo72wkp6vv2@Ergus>
 <83sg0zmsf1.fsf@HIDDEN>
 <116428124d5c9911116bc992b360f2d7@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org, spacibba@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Thu, 01 Jul 2021 01:25:07 +1200
> From: Phil Sainty <psainty@HIDDEN>
> Cc: Ergus <spacibba@HIDDEN>, 49264 <at> debbugs.gnu.org
> 
> > AFAIR, that's not really true, and ISTR project.el aims to support the
> > use cases with several different VC backends.
> 
> It's probably worth considering that while one *can* have multiple VC
> backends active in a single directory, it's *extremely* common not to.

I think Dmitry once told me this isn't so, but maybe I'm dreaming.

> If there was a user option which effectively opted out of the multiple-
> backend support in favour of performance-oriented assumptions

You mean, ask the user to specify the backend for a project?  That
could work, I think.

But again, Jimmy's use case was unbearably slow even with a single VC
backend.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 30 Jun 2021 13:25:15 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 30 09:25:15 2021
Received: from localhost ([127.0.0.1]:56889 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyaDL-0003t0-6P
	for submit <at> debbugs.gnu.org; Wed, 30 Jun 2021 09:25:15 -0400
Received: from smtp-4.orcon.net.nz ([60.234.4.59]:37365)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <psainty@HIDDEN>) id 1lyaDK-0003ss-9D
 for 49264 <at> debbugs.gnu.org; Wed, 30 Jun 2021 09:25:14 -0400
Received: from [10.253.37.70] (port=41403 helo=webmail.orcon.net.nz)
 by smtp-4.orcon.net.nz with esmtpa (Exim 4.90_1)
 (envelope-from <psainty@HIDDEN>)
 id 1lyaDE-0000Df-79; Thu, 01 Jul 2021 01:25:09 +1200
Received: from ip-116-251-162-85.kinect.net.nz ([116.251.162.85])
 via [10.253.37.253] by webmail.orcon.net.nz
 with HTTP (HTTP/1.1 POST); Thu, 01 Jul 2021 01:25:07 +1200
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Thu, 01 Jul 2021 01:25:07 +1200
From: Phil Sainty <psainty@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
In-Reply-To: <83sg0zmsf1.fsf@HIDDEN>
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
 <83mtr8ooz4.fsf@HIDDEN> <20210629222142.jbyegdo72wkp6vv2@Ergus>
 <83sg0zmsf1.fsf@HIDDEN>
Message-ID: <116428124d5c9911116bc992b360f2d7@HIDDEN>
X-Sender: psainty@HIDDEN
User-Agent: Orcon Webmail
X-GeoIP: --
X-Spam_score: -2.9
X-Spam_score_int: -28
X-Spam_bar: --
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org, Ergus <spacibba@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

On 2021-07-01 00:46, Eli Zaretskii wrote:
>> As a note here, when N files are in the same directory the normal 
>> thing
>> is that all of them share the VCS. So calling a check function for all
>> of them is redundant and slow.
> 
> AFAIR, that's not really true, and ISTR project.el aims to support the
> use cases with several different VC backends.

It's probably worth considering that while one *can* have multiple VC
backends active in a single directory, it's *extremely* common not to.

If there was a user option which effectively opted out of the multiple-
backend support in favour of performance-oriented assumptions and
caching, along with some mechanism for flushing the cache on demand
(advertised to the user as part of the user option documentation),
then users could then enable that option as a performance measure
provided that they were confident that the default functionality was
redundant for their use-cases (as I suspect it would be for many
people).


-Phil





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 30 Jun 2021 12:46:33 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 30 08:46:33 2021
Received: from localhost ([127.0.0.1]:56711 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyZbs-0000YM-MN
	for submit <at> debbugs.gnu.org; Wed, 30 Jun 2021 08:46:32 -0400
Received: from eggs.gnu.org ([209.51.188.92]:52442)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lyZbr-0000YB-JO
 for 49264 <at> debbugs.gnu.org; Wed, 30 Jun 2021 08:46:31 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:36904)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lyZbm-0001Ty-B4; Wed, 30 Jun 2021 08:46:26 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4872
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1lyZbl-0003th-V4; Wed, 30 Jun 2021 08:46:26 -0400
Date: Wed, 30 Jun 2021 15:46:26 +0300
Message-Id: <83sg0zmsf1.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ergus <spacibba@HIDDEN>
In-Reply-To: <20210629222142.jbyegdo72wkp6vv2@Ergus> (message from Ergus on
 Wed, 30 Jun 2021 00:21:42 +0200)
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
 <83mtr8ooz4.fsf@HIDDEN> <20210629222142.jbyegdo72wkp6vv2@Ergus>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Wed, 30 Jun 2021 00:21:42 +0200
> From: Ergus <spacibba@HIDDEN>
> Cc: 49264 <at> debbugs.gnu.org
> 
> >AFAICT, most of the time is taken by 'apply', but the profile doesn't
> >show which function is called by 'apply'.  Can you tell which function
> >is that?
> >
> apply is called in vc-responsible-backend and looking at the percentages
> it is obvious that the times are going in
> 
>    23%                      + vc-svn-responsible-p
>    18%                      + vc-bzr-responsible-p
>    15%                      + vc-hg-responsible-p
>     6%                      + vc-git-responsible-p
>     2%                      + vc-cvs-responsible-p
>     2%                      + vc-rcs-responsible-p
>     1%                      + vc-sccs-responsible-p
>     1%                      + vc-src-responsible-p
> ------
>    68%
> 
> These are the backends' functions that are passed to vc-call-backend in
> the mapcar in vc-responsible-backend and the output of
> vc-find-backend-function.

But if you still need to wait for dozens of seconds with just 2
backends, which take only 20% of the time according to the above, then
how do you want to speed this up in your case?  How about not using
ivy (which AFAICT is the root cause of this slowness)?

> >... after removing the "unused" VC back-ends, you say that the code
> >still runs very slowly.  So is the issue with VC back-ends still
> >relevant, and if so, how?
> >
> Yes, it is. It is still slow, lets say around 20-40 seconds to complete
> the command. But that's that's much faster than before (3-5 minutes);
> but still too slow to make the command usable.

It's still unacceptably slow, so that couldn't be a solution,
certainly not a general one, IMO.

> As a note here, when N files are in the same directory the normal thing
> is that all of them share the VCS. So calling a check function for all
> of them is redundant and slow.

AFAIR, that's not really true, and ISTR project.el aims to support the
use cases with several different VC backends.

And again, waiting for 30 seconds when you have just 10 buffers is
unacceptable.  E.g., in the session where I'm writing this, I have 80
buffers that visit files in a single branch of the Emacs Git
repository, almost an order of magnitude more than your 10 buffers.

So we must find some better way of getting reasonable performance in
this use case.  If the round-trip of the VC backend to a remote
filesystem is the bottleneck, let's try to speed that up in some way.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 30 Jun 2021 00:01:58 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 29 20:01:58 2021
Received: from localhost ([127.0.0.1]:55969 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyNfy-0002Tw-Dj
	for submit <at> debbugs.gnu.org; Tue, 29 Jun 2021 20:01:58 -0400
Received: from sonic313-14.consmr.mail.bf2.yahoo.com ([74.6.133.124]:43913)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <spacibba@HIDDEN>) id 1lyNfv-0002TX-Oh
 for 49264 <at> debbugs.gnu.org; Tue, 29 Jun 2021 20:01:56 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aol.com; s=a2048;
 t=1625011309; bh=EqRW6YirdOg81X5igmaWIywu4/rACWUoRm+vlng+3sE=;
 h=Date:From:To:Cc:Subject:References:In-Reply-To:From:Subject:Reply-To;
 b=kA/0IduXXOmJdTvm2DzTUyKsHjOD0CxVA09snzlTOXOfV3C1zeji/8RUhHJ6TVFuFHCsW/kAx8BBbItwZkpPlijDXNMUM4dU+ph+ZmD041b4e8SUcYq3fFmQ1tMlcV0ucIl7xoLad4j8KYwVKbrwsy4RYoRfLIev9WU4NtsgTu2pxZQgxmjPP2/d5manDtwy8mpmld7wQAa7qZlCrGrXDuwen9B7KZQKpGEM+LMu74Pie45k645B4JLiJz6ZaExqKB35YvtGimOxApn+AptFVJALA5L+o9LSvSrl9C0/rxF96FZyId3ngVYxrzN9iVrsAeQNwYrzwyKp4uptKugcmg==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1625011309; bh=8ZqG1CAmQP0mDkAegwHMcFcwIkN+VqelIsffoCszDRg=;
 h=X-Sonic-MF:Date:From:To:Subject:From:Subject;
 b=ahCHQG2FX09dU+EBJLP6+lUqd66edEVnDyOgRv2ZDSkken7sP+TQ9/ly8odU7i1OYQOBBbzA9lgQGOBwdgY8HY6ugiX0GXQb9huawq4omuUVCW/bHJ3QHXzgLXN7gjV362SCWjK4TqaQsZy6EapO+7fLsjrxxDIh2EL9H00rrDPhlWyCf/Qv8WVm7f6t9javjos+4/ISilwAd9eFAPS33eIM72JA/3PS8pM0hITlsVRN/YHfR8rUyHJHuT99F1mrzSnAawBM/GHnfhUQYN1gmEB4nAAMHerb3ZJNU2celiAlDKFU1yobAr+hwAa2rLvlct2cWaO/WAcknkIVbmV5aA==
X-YMail-OSG: HLF4qaYVM1mSPKzIQ9_Oiz00BFK6CmlJad2iUQjJgUlRNyHeangd4xyT0FL8289
 rebB4QwqMbHt4EuI3kVR5EoxqDZ.rrBGiIYGLN3G0.W5xYFOwwGDe.IV0L_T2lHvk6JRcPn4iK_2
 iR8O1xuXVlWFCtVnsABMVsOQZaDrPvW4Z006FZ8W2OpTQuD9LKaLi2luoR32YM2ryKs17LMhhEYF
 5uVOmqFGENkIzbBrPaPo8AgwFdX62G4sADfBu0WDNrdXx5d.0jJqLSUSoEE_ngMuLwlVLtssPM1j
 cn41kH_UjTfURzuyKXHkF5oaW38waFOTD8UCLF8oBTBWK8kJZh0FakjKoAevOYLh6jD5LEjFrSXD
 dE.X5FE1E1WoGjsADgcJl6i_gbHbNNtVUMGEd8rljhI8K71KG_HRWs20jpTQTHK0LYBHSpJhG9UH
 BEJoyXgEHYfMByNOn.o6a1mhF8Y6N4L4YnlZcv9XXmYG_5Oq7bxf.F5nAHW4zy3nU_uSqgpBhkOL
 FENuRVfLB_snLeXDeIWS_1mTJFdTjtGCpTYLSGgZQWUvCk7Wo49lJQeOmvx7ivgP8r4l3Lp91_nl
 5fSZVH67h9nr85NVHszKrYUpSVsIpPMXF0ZhLTZ54f24_1qWWJAtpX0QyovHBwafBeTSpCskPORd
 PqrwfJLXzGWGJgmulPv7Iq.tHvCIpmnEeVa_N_JUGe1m1HDRpaoYoPpa0kQYQROcmVAyfznzMyFE
 TR_KlU0Xl9tvhX.qKuqWR9.Qo4UGD.m07psg436_96fEmacZZGFXu85dFaRC.jue6vu0IsPX.nwg
 2SBUCTxDjIP28l1gQih_LGnHDO9kjOt0tI9NMZF_37C7Dj4L3mVRXrjG3_EQIaYOm2o6TPk6xxMZ
 UUl7A9xwKaDJzvzQvVUMjMAptBxKvSNBH9MO1gflsVcMLOInQAS2jQOobCWPDw6bKhuiKin6_zvn
 Ql4r576Ild10qn0Es_.w4DmmPgWxEoiDeiePnmuGOI8Ad6CCl5xTSaUP5OOHltGn.Pe.TSVNG0SY
 retPrifBz65ff9D06.qIIIL4WgZ3_XvbwfZZWBslx6OtZs0pibEttsBSQ1HIiItlX2xSEHrqvAL2
 9HluKn3X4GjDugdLxyx7VNaBLskf4FbmwllHFvVR6RySMJDwRR7AVorYaSnl9292PCQ0NtkrCyRl
 Ej81Mk.m_9SBMUbuIxnlVO_jU6e9HkfCL3ABCKIXeDGsjIxaV841THVZboYjj2noKjc8KSCWI5R.
 GAhbmD.dwedKtIV_47GIWN1fjzZiLMvKkNhz_cy3kHw5k0kVEQOSf8C51W4Z6Ja3qWRfplqGeI8P
 srQ9em5dIrq01u3jM9mBAKb20rtrhs_jSUNirRbFwZsvsZE_nmefVzPy6PyocV48ErclW84T6zcI
 4QWRcWwpe5kQYXcfmRN8RxGw7rnwiSK14dnca_Kqplzx7ziRa.ihExVtrNO8_yFztmjwwq44K4Fs
 wt_orRUy5SUku824FdsSBu1u.SmDzgLRFNij0CZqm0IL1QFnlVFP59hY52UIW2v1s10APLVHZ39t
 dBvKlZnFOe9FYtCVAL07ZOBEv3OYPMBXBQ2iB0h.yxfISwCPE_aQRrw.yEjlgJutUu8jp17vsDE1
 kUyYLpKixp2XZFEqfsjdVA.G1XrmqPKybikzGWd07HggLvjIt9LGvuyT.MXjoHdi0.onjpinJKU6
 aJu6JU94MJvYNHNot_jLuGXuC8SM_GX_xv2sdyUziGemFaPTJmjL4qTsMBG8m7iJtkwsZxTV05FG
 CjJvXI8Es1GzDqVOzA5PJCAXP5RPJAJAxYSp5xZHm.Vzki.v76nyxHcQJeVYcq8hmYiyPUuJ5ue6
 dtwjVmLmgA_SYxITqdRe_74LLnCiOQ9rLLVJTr.oxAVUxvwmj8v9cFWB2jlMOh6synQ77O3oYBGf
 8_hUZ1zsxdOb0eMbr_GTD_cnyK3FDPMC19DCEBgPZ9uPeNHJvR1O.v038Q8flgIQi5.748krfgfa
 XHs_ue.egobm2j1F_vZ72aAw557XBRsbzmbC23H2sAiyhn36__EtPKvnxGqU6og7TS0veyEaLs5w
 FbVv4yi0jLfZpUuToV5UxC4Oa0VYgdvg09fAdTFUzYZyXoR_Rp2ccNGSGKIYhw7inI6s5OFbO2hP
 ciQF.3wC7EqddDs4AwBSr3xi5G43jo.6JtRr9GQGG0ZjfVqVUsvJYx_PpRZqJKVffDUEPwXuSvNp
 KlerUGpmqE231YTS6Hz3frFJKotQsg0kNedyC6M9qK6Lx8FDeEl2H2fikGSs0HezeR.EcmBbCeN5
 vkEtrgslEgFpbUx0awWVIZEYgRHgyQR8WOU8BrtL7KL2s1D6WB2etZyO3_rKyD8RW4QzU1qKhNuX
 yJyVie_NCkwYb4ErB2Kqbrmv8Moczl9ivMv716KoqolNPZZKN
X-Sonic-MF: <spacibba@HIDDEN>
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic313.consmr.mail.bf2.yahoo.com with HTTP; Wed, 30 Jun 2021 00:01:49 +0000
Received: by kubenode527.mail-prod1.omega.ir2.yahoo.com (VZM Hermes SMTP
 Server) with ESMTPA ID 13e9fed445d0719f034e7e6e061c8855; 
 Wed, 30 Jun 2021 00:01:45 +0000 (UTC)
Date: Wed, 30 Jun 2021 02:01:36 +0200
From: Ergus <spacibba@HIDDEN>
To: Dmitry Gutov <dgutov@HIDDEN>
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
Message-ID: <20210630000136.o2uqaa3ldxxtn5go@Ergus>
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
 <72c1b336-ae02-36c0-db40-f608bafecdf3@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <72c1b336-ae02-36c0-db40-f608bafecdf3@HIDDEN>
X-Mailer: WebService/1.1.18469
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.aol
Content-Length: 2882
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

On Tue, Jun 29, 2021 at 04:00:02PM +0300, Dmitry Gutov wrote:
>Hi!
>
>Thanks for the report, I'll follow up on this discussion some more 
>later, but some initial observations:
>
>On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss 
>army knife of text editors wrote:
>>As a workaround I removed all the uninteresting handlers from
>>vc-handled-backends and I get better times now, but IMHO it is still
>>very inefficient (almost a minute for project-switch-to-buffer is
>>excessive). And make it practically unusable.
>
>Could you evaluate (benchmark 1 '(project-current)) in one of your 
>buffers? That should give us an estimate how long it takes to find the 
>"current project" on your remote system.
>
Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

>If I'm right, project-switch-to-buffer should take 25 x (that time).
>
>If you indeed only have 25 buffers (including hidden ones).
>
>>In any case:
>>
>>Maybe (I think I mentioned this before) `project.el` needs a sort of
>>cache to speedup some functions like `project-current` that are called
>>very frequently inside the project.el code.
>
>The difficulty here is probably with the large latency to the remote 
>system. And our current approach calls the "find current project" 
>logic for each open buffer.
>
>Even if we add the "current project" cache, it will only take effect 
>at second and further invocations. Your first project-switch-to-buffer 
>call will still take 3-5 minutes, which is unacceptable.
>
>Please get back to us with the requested measurements, and perhaps 
>other observations (any research initiative is welcome), but if 
>finding the current vc root for each buffer is unavoidably slow, we'll 
>finally need to switch to a "project-contains-buffer-p generic" 
>approach, previously discussed in e.g. "Speed up project-kill-buffers" 
>thread on emacs-devel.

Another (and maybe even simpler) optimization, may be to consider that
all the buffers with the same default-directory should have the same vcs
and vc root (and probably belong to same project). (I think that all vcs
backends only do the search based on default-directory at the end.)

So if the association in the cache is for `default-directory` instead of
individual file names; then, less files will need to evaluate the search
of vc root, and vc-backend only 1/subdir will search the first time. It
is a very good trade of IMO.

This will reduce the time even the first time we iterate project-current
over all opened buffers if multiple files are in the same directory
(example: sources in src, includes in include and so on). And I think it
will cover 99% of normal use cases.

This won't affect nested projects, git-submodules or similar, because
those will be in different subdirs.





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 29 Jun 2021 22:22:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 29 18:22:03 2021
Received: from localhost ([127.0.0.1]:55872 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyM7H-00045x-D8
	for submit <at> debbugs.gnu.org; Tue, 29 Jun 2021 18:22:03 -0400
Received: from sonic302-3.consmr.mail.bf2.yahoo.com ([74.6.135.42]:45451)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <spacibba@HIDDEN>) id 1lyM7G-00045T-DK
 for 49264 <at> debbugs.gnu.org; Tue, 29 Jun 2021 18:22:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aol.com; s=a2048;
 t=1625005316; bh=yzt3sAygmvlmvZYsmxoNHLcVJ9Fu7PIIfrhAMJeryH0=;
 h=Date:From:To:Cc:Subject:References:In-Reply-To:From:Subject:Reply-To;
 b=UY7ITrCueIzz8qg33W8vpBnNBwRr8N3Z/p8Qvs/y28gtWeQ/ZBNxsqKsQE2KSWS+9zfxmS97aJCfR8sy8lmARlnehpG3NmiiYJQubS4O96W+oUq6Yeih9JknTe3e90WDAf5sKOOVQ5KgAWnwHCDvpZHD8XV0DcvoQRgLbIAVuyRy9qX9xauJmEyl1rvphOAQA5SHyVuYvG2NSQgAorjdmFPFvWdj/H4X/r6Rfwlb3ivRt/lQchbQ49um6mWmGISGUenjL3LJ1ceaSMqO+1PcAgVnmw/mgU41Cx0cXG49Di7i7ul4iHBG2cYplruyyQCMo0yJYqjEQzqjtINpBuRdOQ==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1625005316; bh=XpwTwltnenT/PdFMocFu4PJL8L8diMDe6nuxVqavA9X=;
 h=X-Sonic-MF:Date:From:To:Subject:From:Subject;
 b=ZOCjtfxKgmaBogHzJt63wiU1l3+T8ObHmw0MzOadQRi4e2Is2STQXynYba2BvLy2Ds/5crkeBydEaiq929ruzssCFsOa/A9omBAp5NkmlI5ku90AgEuuo0gQU/4uXDFL/wpnAHehsiCWO4ucrAeUrZxYzoSba56R8J772xnEdTvjlG1BeQNk6dsqW87DZ1uCyLyTY84BTJInFS3aIHw9Y8o5ipNCwAI8aw/nVNP9FCHmQZxZ3D/S66fUfMz/1ASorZ8NtTAOZAcBHF5vAd5z+7urSqsFUePrai/+7eqxb4NHVv6KebjnqiVwkcn+1OPhrsBivb+TWDIx6X43DRSw9g==
X-YMail-OSG: vPZbcjoVM1m2.KE4s5Ig14Vg6CVVy3gVoxo2doiORuOBmke4_42uQLA7gL9fp3e
 Qg.2NJgZjNtTBsLVQsC0Lvr0a94GJ2o3mYMqQVplbbjiQeBKstMwDBYbKdCCt_mPzxSFqJrIpiJN
 LJm5BGyv6PD3Kl2tW8DbcI0I4CmI6n3FCVGnLGL7zBJ0MS19ZxlywvyG4Ph8TQhhpeFpepZhWeAp
 ZdtpN6F0CFwFVj9Gx0MrEWz8fBpummRBp7wK5gQaPcNBJ_nQKReSCbObn9l4LRyvXRAsWYw.uTDi
 hvIjFT8gHF3iHN5IRlT.oQ7Aj.IsDSLv0KeHqB3zLcgF2WBwwcJ96KvxuCBwfjZhptfBhaFTJaW9
 DgKlF0qTv_cPb.aAb64Qw8MxzjNl9zgrR0qwTZ4dgg3AaCJNgv_HFUgpWFxz92lTY3qrREWETJ3J
 eruEWwxT61FGEU_T2lQUJLDTjFxh4EuGoQw3vwyeyx4niymNTmtaMMY7sSXaucZ5OaZV5lOt1Fy6
 RGwsL99lONBc75Mbu4PEt6OAuGEgoEVzplUVDtuhad7En27Y5DrKA7iYTUqKomRLTez5.71nV0lM
 YKv8N6lp13Sa8XSR67NOslbcg2IuVrnkb4iRv.r68JVNy1WC0sFcjgs5WUx7t63gVjbMd.Y26ia6
 6FNpmEtJmPDrT_WDQIMY3Os9T_I45dY6pAYE6MaDOtyh2jfYWOsFIGYS06TWFOkeVO.7.YFB0JTT
 0tTRuCmButK3q1A.PLXzCbZ3jduTEdGoGNoclqe7OaNM067wiv3UL682zJ1E0FSJDrZUUKKX14_u
 Yl_OLffU.6CaYwOlef6CnRTaGCy7pILQ_yEtf8Nxp0w1OwMDpZtenJ3WWTQAdjQ8mwg79hsBmCDl
 3E9xc4SFnDIgI5VdQzDRSKHsExthv8srGfHnNwPJP892rufO1OKoMREfSximpF3SRouOwY.0nTMD
 EaEvw.BII_LHLQq6hRqjyVpkH9hFFYyrExoOGzGxxNGrwrInO08NfSxaZgJcEGGUD6ZwjnrMG4qi
 E8ZymomtJ..kzrwCg7FP2xPTse3rBHs.SuhZgQez3b35DmBTacTf3YEf0Jk1y69i5.UeIE121BDI
 N6sLnk1nye9BEOifw9TYX3myAq_KILiNkggeD5F1cnviAJqhpFl_Ex.GVoKB7zbQFG__Xfs0nal3
 3BQAllqod_fDVFMLGZ7mOC6xrIzK2lX8hypA9VV8caabzzfLPEzzmhTXKon.mLY0pqs073mVjcTa
 mcpcAgXbh9mQk0ydT2rOTBIK2P93RCU5Rrim2_1F.JUn_.v77sbuh2AYDZ7wm3j5jkE5ANTLBPhC
 RBRxeL85kpQkChsGzmNDMTimOM2TUzR_gxqInXrvykGchEpyPrYxgXt5rVhlej6YxYlZNtJQuj_1
 dxNQUQMfFBn5x6RhgQ5qp6yWh7CGEynikBTrfXVU2I2jeOUggS_IXPZt5b9tHaC2ET8B2JtlmJYG
 5Gg4PEwMlfSmpStXZF_61viZQLaay2nwPHnJiAoDQsARGGe97GqobFbCv6Hdujgh7zgarOfdco2C
 jABOhCY9aDaPCFHWBO3y07P3m2nWduKEPIlawYkC.Cr7BXZgl6406ovHo6QupEdziIQE8mjMI7lq
 fQOkftLz17IfpjTyzxaHeHcgohtOzhXu75FARyhNx8Kdhs1uLT8AlTJ_YPBFQzFdjlMJ3g3Kkdfa
 m1jP8X8prHNcpaeL67e1xjP9c0640h07d.5ZXkAayZE53m8AdO_teHZvgGlN4Dcw4EvACT.d10aP
 zrOOkDJm4NvUpE_IXVjlnWM5NpUoZlPdDHHk.YnIVC4R7QvMjoc51zf.5HS.jd.VKVI400rh_fuJ
 XaycWPGr4aQdIqNpEhH5FFVhSjR2KCQfLjrGGrkp6doEdR_4CFUJA9CUSGqwb1yCvmHT3OWCBIoV
 UwS5nIZ.GfY04ZWMQWW4k6LfUmu_bfmFNdJ2bNxYOMUaI.mTuTqEoroik8w9DZa3YB51KwsclX_7
 a6L1miXXYSCSrP2xGH1K46HhtCq_OmQKjGhLy06bQ4YTm0UQ7PfV6mGZYFPZdFfR49zQ87fpNSeh
 tCXg2RnHR3xzxMAUc54EkNVUxDoyzHRUoPixleZLIRzwPv6QHwPmwX6lb9ihTW9Ju.6PYyVzynQj
 64H5SDsQd9JDZpfZOuPvTHjOFlFZ7V1YKrgSu2UYdMQwHdEIlejyaRmqfP7T_PS7pCq6nBk6phBW
 SEEsBZRekTaA3gHFzUo6iXq8XOUahMr7uaNv2ZvrHyZXT98Ny.17OUpzSl0nirro.EhZGcjBOjft
 Oj3meNBAUwnR1KvDqeqIhHjLgXoMrVrAdZBYYZZVs_MyNdg8MwfJAZQAFlcOVauEaou.is28QO2p
 OXbZjrpV694xjIS0FiNHeMe0PsKxFIqH_2p8-
X-Sonic-MF: <spacibba@HIDDEN>
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic302.consmr.mail.bf2.yahoo.com with HTTP; Tue, 29 Jun 2021 22:21:56 +0000
Received: by kubenode525.mail-prod1.omega.ir2.yahoo.com (VZM Hermes SMTP
 Server) with ESMTPA ID 462f400b8bb535964166e44dab381a4a; 
 Tue, 29 Jun 2021 22:21:52 +0000 (UTC)
Date: Wed, 30 Jun 2021 00:21:42 +0200
From: Ergus <spacibba@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
Message-ID: <20210629222142.jbyegdo72wkp6vv2@Ergus>
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
 <83mtr8ooz4.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <83mtr8ooz4.fsf@HIDDEN>
X-Mailer: WebService/1.1.18469
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.aol
Content-Length: 8614
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Eli:

On Tue, Jun 29, 2021 at 03:05:35PM +0300, Eli Zaretskii wrote:
>> Date: Tue, 29 Jun 2021 00:11:00 +0200
>> From:  Ergus via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN>
>>
>> Using tramp I tried to use project.el with a command like
>> project-switch-to-buffer and it took like 10 minutes to complete.
>>
>> I ran a profiler and I found that most of the time was taken by an
>> external function: global-tags-try-project-root
>
>That doesn't follow from the profile you show.  According to the
>profile, global-tags-try-project-root takes just 6% of the CPU time.
>
The profile is `After some optimization in an external package`. As
explained 2 paragraphs next. Before the fix it was taking more or less
the same percent than vc-responsible-backend.

>> project-current is called in a loop for all the opened buffers it calls
>> project--find-in-directory that calls project-find-functions and there
>> is going all the time.
>
>I don't see project-find-functions in the profile.  Where is it and
>how does it come into this picture?

project-find-functions is a hook called in project--find-in-directory

(defun project--find-in-directory (dir)
   (run-hook-with-args-until-success 'project-find-functions dir))

>
>> After some optimization in an external package; now the time is half
>> than before but still very slow to use the command (around 3-5 minutes
>> to complete) and running again the profiler I get this:
>>
>>          5637  89% - command-execute
>>          5549  88%  - byte-code
>>          5549  88%   - project--read-project-buffer
>>          5549  88%    - let*
>>          5336  85%     - read-buffer
>>          5323  84%      - ivy-completing-read
>>          5323  84%       - ivy-read
>>          4941  78%        - ivy--reset-state
>>          4941  78%         - ivy--buffer-list
>>          4941  78%          - internal-complete-buffer
>>          4941  78%           - #<lambda -0x1a357caf01243d61>
>>          4941  78%            - and
>>          4941  78%             - equal
>>          4941  78%              - save-current-buffer
>>          4941  78%               - project-current
>>          4941  78%                - project--find-in-directory
>>          4548  72%                 - project-try-vc
>>          4537  72%                  - vc-responsible-backend
>>          4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>>          4478  71%                    - vc-call-backend
>>          4478  71%                     - apply
>>          1470  23%                      + vc-svn-responsible-p
>>          1142  18%                      + vc-bzr-responsible-p
>>           970  15%                      + vc-hg-responsible-p
>>           390   6%                      + vc-git-responsible-p
>>           156   2%                      + vc-cvs-responsible-p
>>           126   2%                      + vc-rcs-responsible-p
>>           108   1%                      + vc-sccs-responsible-p
>>            98   1%                      + vc-src-responsible-p
>>            57   0%                   + tramp-file-name-handler
>>            11   0%                  + vc-file-getprop
>>           393   6%                 + global-tags-try-project-root
>>           375   5%        + read-from-minibuffer
>>            13   0%      + if
>>           213   3%     + project-current
>>            88   1%  + funcall-interactively
>>           572   9% + ...
>>            51   0% + timer-event-handler
>>             8   0% + redisplay_internal (C function)
>>
>>
>> As you can see most of the time is still taken by project-current and I
>> can't really understand why:
>
>AFAICT, most of the time is taken by 'apply', but the profile doesn't
>show which function is called by 'apply'.  Can you tell which function
>is that?
>
apply is called in vc-responsible-backend and looking at the percentages
it is obvious that the times are going in

   23%                      + vc-svn-responsible-p
   18%                      + vc-bzr-responsible-p
   15%                      + vc-hg-responsible-p
    6%                      + vc-git-responsible-p
    2%                      + vc-cvs-responsible-p
    2%                      + vc-rcs-responsible-p
    1%                      + vc-sccs-responsible-p
    1%                      + vc-src-responsible-p
------
   68%

These are the backends' functions that are passed to vc-call-backend in
the mapcar in vc-responsible-backend and the output of
vc-find-backend-function.

>> 1) Are so many samples 4548 seems a very high number for only 25 opened
>> buffers.
>
>These two numbers are unrelated.  4548 is the number of time the
>profiler found the program counter inside project-try-vc and the
>functions it calls.  This number has no relation to the number of
>buffers you have, it just means that code runs slowly.
>
Ok

>> 2) why project-try-vc still takes so much...? Specially for unfrequent
>> vc systems in our days like svn or bzr that I am not using.
>
>That was explained on emacs-devel.  However, ...
>
>> As a workaround I removed all the uninteresting handlers from
>> vc-handled-backends and I get better times now, but IMHO it is still
>> very inefficient (almost a minute for project-switch-to-buffer is
>> excessive). And make it practically unusable.
>
>... after removing the "unused" VC back-ends, you say that the code
>still runs very slowly.  So is the issue with VC back-ends still
>relevant, and if so, how?
>
Yes, it is. It is still slow, lets say around 20-40 seconds to complete
the command. But that's that's much faster than before (3-5 minutes);
but still too slow to make the command usable.

>More importantly, what is the profile after you remove the extra VC
>calls?
>
          790  83% - command-execute
          631  66%  - byte-code
          631  66%   - project--read-project-buffer
          436  46%    + ivy-completing-read
          188  19%    - project-current
          188  19%     - project--find-in-directory
          188  19%      - project-try-vc
          135  14%       - vc-responsible-backend
          135  14%        - #<compiled -0x11223b7c225fdde9>
          135  14%         - vc-call-backend
          135  14%          - apply
          105  11%           - vc-hg-responsible-p
          105  11%            - vc-find-root
          102  10%             - locate-dominating-file
           76   8%              + tramp-file-name-handler
           26   2%              + abbreviate-file-name
           30   3%           - vc-git-responsible-p
           30   3%            - vc-find-root
           30   3%             - locate-dominating-file
           30   3%              + tramp-file-name-handler
           41   4%       - project--vc-merge-submodules-p
           41   4%        - project--value-in-dir
           41   4%         - hack-dir-local-variables-non-file-buffer
           41   4%          - hack-dir-local-variables
           37   3%           - dir-locals-find-file
           27   2%            - locate-dominating-file
           21   2%             + abbreviate-file-name
            6   0%             + dir-locals--all-files
           10   1%            + dir-locals--all-files
           12   1%       + vc-call-backend
            7   0%    + #<compiled 0x34145703cd88c72>


Only with git and mercurial backends and with 10 remote buffers open (in
the first profile there were like 25). Remember that the times grow
linearly with the number of buffers.

As a note here, when N files are in the same directory the normal thing
is that all of them share the VCS. So calling a check function for all
of them is redundant and slow.

>> VCS changing is not something that happens very often to require a check
>> of all the backends everytime, several times for every buffer in many
>> project.el functions right? Specially when using tramp.
>
>Once again, given what you say above, this doesn't sound important,
>does it?  The slow processing is elsewhere, and without seeing a
>profile with VC calls removed, it's hard to make progress in this
>matter, or give you some advice regarding potential reason(s).
>

Reducing 3-5 minutes to 20-40 seconds when there is only hg and git
sounds relevant for me. Still slow for a command, but important
considering that it is still asking the filesystem for every
file/buffer.

>> vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
>> it's result there to avoid repeating time consuming computations?
>
>Again: is this issue relevant, given that without the VC calls the
>code is still very slow?





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 29 Jun 2021 13:00:13 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 29 09:00:13 2021
Received: from localhost ([127.0.0.1]:53829 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyDLY-0004dS-RX
	for submit <at> debbugs.gnu.org; Tue, 29 Jun 2021 09:00:13 -0400
Received: from mail-wm1-f42.google.com ([209.85.128.42]:46872)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <raaahh@HIDDEN>) id 1lyDLX-0004cC-2f
 for 49264 <at> debbugs.gnu.org; Tue, 29 Jun 2021 09:00:12 -0400
Received: by mail-wm1-f42.google.com with SMTP id
 v20-20020a05600c2154b02901dcefb16af0so2368514wml.5
 for <49264 <at> debbugs.gnu.org>; Tue, 29 Jun 2021 06:00:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=sender:subject:to:references:from:message-id:date:user-agent
 :mime-version:in-reply-to:content-language:content-transfer-encoding;
 bh=0CdbJzPJ/Ieg63bHkigcGJjzD9G5B4zTFFDqC0GaQYY=;
 b=TD7WGZk/f2VzWhHCsUmJdDjTqCPDCmNXL5zif4Ylwj8jfMrBxThh6wG/0tXoir7JgE
 JOrQ7cfEomXXEm3WHA0ALRno+qJhxxrQygiAPDczzQviSLazYaAlt9QBx4fO3cJWxqqf
 MEB6xdK5bauHZ0I961srVlQtwz4bWQKjnZhOjSQucU3FMg7lFt/FezwcyVViL1wV83Te
 Bbr3tkhC3vS0Q2GAiBzyvLVZtssZpTVCYZ7tzsZUWFPPGsxxNh1JZPS5ftFnQ2+eVUu5
 9+9THjUZRHIfrpt9wEGxWRjp8jwpLlJ7/QAzxFLEJLSfSz/9BM0DFpMj8JIjaRj7iIhY
 kyRA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:sender:subject:to:references:from:message-id
 :date:user-agent:mime-version:in-reply-to:content-language
 :content-transfer-encoding;
 bh=0CdbJzPJ/Ieg63bHkigcGJjzD9G5B4zTFFDqC0GaQYY=;
 b=Nix555b79QwJ6OxwepyhnO6YdkyLbQThOvN/m8X3rSPBS4uS6eJHjA0TdvH4VBo4mR
 AdyDBuVOxVlOetWzuBXwynEuybp1E/FUZPklc7rCKwf4vV6y/37R5vMhlQZggcRl4syL
 0knG6gwTtZOpsrv/nO50XWV93UOzIyBtGXTW4gXhLHvOuQ/iHH2AN5hOqsu1L5M77LqG
 GvtsA5k9jHA1+1aOVXpmpPgtQ8iJ/1FdgR3w0VVTyAM2/5/szTEra9wHwvVGsF5fOa7G
 BrHBj2dDC8pxm8C1aBduAToyIYy5lJvYsPCCz19Zzf0pZ0FFl+ZPGO8c/reqrDC1JKdb
 OFdg==
X-Gm-Message-State: AOAM532BR/EDz2IrODuEJae38rPn6ZpIt48NdtdWTTgj1DiA1OsKbZdT
 QinjNeBiaJLqhT9P6Eoyikb7lZOKjck=
X-Google-Smtp-Source: ABdhPJzxwv1AJ++K6loHriqg6/5bK5XCH7ss2AxhLjhGfC+ExJaUB7C9LbispZ71Mkrd7PS7Z6RBIg==
X-Received: by 2002:a1c:7f4b:: with SMTP id a72mr5200048wmd.48.1624971605062; 
 Tue, 29 Jun 2021 06:00:05 -0700 (PDT)
Received: from [192.168.0.6] ([46.251.119.176])
 by smtp.googlemail.com with ESMTPSA id f62sm6426880wmf.22.2021.06.29.06.00.03
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Tue, 29 Jun 2021 06:00:04 -0700 (PDT)
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
To: Ergus <spacibba@HIDDEN>, 49264 <at> debbugs.gnu.org
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
From: Dmitry Gutov <dgutov@HIDDEN>
Message-ID: <72c1b336-ae02-36c0-db40-f608bafecdf3@HIDDEN>
Date: Tue, 29 Jun 2021 16:00:02 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <87fsx13aiz.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.5 (/)
X-Debbugs-Envelope-To: 49264
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.5 (/)

Hi!

Thanks for the report, I'll follow up on this discussion some more 
later, but some initial observations:

On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> As a workaround I removed all the uninteresting handlers from
> vc-handled-backends and I get better times now, but IMHO it is still
> very inefficient (almost a minute for project-switch-to-buffer is
> excessive). And make it practically unusable.

Could you evaluate (benchmark 1 '(project-current)) in one of your 
buffers? That should give us an estimate how long it takes to find the 
"current project" on your remote system.

If I'm right, project-switch-to-buffer should take 25 x (that time).

If you indeed only have 25 buffers (including hidden ones).

> In any case:
> 
> Maybe (I think I mentioned this before) `project.el` needs a sort of
> cache to speedup some functions like `project-current` that are called
> very frequently inside the project.el code.

The difficulty here is probably with the large latency to the remote 
system. And our current approach calls the "find current project" logic 
for each open buffer.

Even if we add the "current project" cache, it will only take effect at 
second and further invocations. Your first project-switch-to-buffer call 
will still take 3-5 minutes, which is unacceptable.

Please get back to us with the requested measurements, and perhaps other 
observations (any research initiative is welcome), but if finding the 
current vc root for each buffer is unavoidably slow, we'll finally need 
to switch to a "project-contains-buffer-p generic" approach, previously 
discussed in e.g. "Speed up project-kill-buffers" thread on emacs-devel.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at 49264 <at> debbugs.gnu.org:


Received: (at 49264) by debbugs.gnu.org; 29 Jun 2021 12:05:43 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Tue Jun 29 08:05:43 2021
Received: from localhost ([127.0.0.1]:53786 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lyCUo-0003Jd-Jf
	for submit <at> debbugs.gnu.org; Tue, 29 Jun 2021 08:05:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:51698)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lyCUn-0003JR-7Q
 for 49264 <at> debbugs.gnu.org; Tue, 29 Jun 2021 08:05:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:51772)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lyCUh-00058m-OQ; Tue, 29 Jun 2021 08:05:35 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1397
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <eliz@HIDDEN>)
 id 1lyCUh-00058G-A7; Tue, 29 Jun 2021 08:05:35 -0400
Date: Tue, 29 Jun 2021 15:05:35 +0300
Message-Id: <83mtr8ooz4.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Ergus <spacibba@HIDDEN>
In-Reply-To: <87fsx13aiz.fsf@HIDDEN> (bug-gnu-emacs@HIDDEN)
Subject: Re: bug#49264: 28.0.50; project.el+tramp performance issue
References: <87fsx13aiz.fsf.ref@HIDDEN> <87fsx13aiz.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 49264
Cc: 49264 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Tue, 29 Jun 2021 00:11:00 +0200
> From:  Ergus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@HIDDEN>
> 
> Using tramp I tried to use project.el with a command like
> project-switch-to-buffer and it took like 10 minutes to complete.
> 
> I ran a profiler and I found that most of the time was taken by an
> external function: global-tags-try-project-root

That doesn't follow from the profile you show.  According to the
profile, global-tags-try-project-root takes just 6% of the CPU time.

> project-current is called in a loop for all the opened buffers it calls
> project--find-in-directory that calls project-find-functions and there
> is going all the time.

I don't see project-find-functions in the profile.  Where is it and
how does it come into this picture?

> After some optimization in an external package; now the time is half
> than before but still very slow to use the command (around 3-5 minutes
> to complete) and running again the profiler I get this:
> 
>          5637  89% - command-execute
>          5549  88%  - byte-code
>          5549  88%   - project--read-project-buffer
>          5549  88%    - let*
>          5336  85%     - read-buffer
>          5323  84%      - ivy-completing-read
>          5323  84%       - ivy-read
>          4941  78%        - ivy--reset-state
>          4941  78%         - ivy--buffer-list
>          4941  78%          - internal-complete-buffer
>          4941  78%           - #<lambda -0x1a357caf01243d61>
>          4941  78%            - and
>          4941  78%             - equal
>          4941  78%              - save-current-buffer
>          4941  78%               - project-current
>          4941  78%                - project--find-in-directory
>          4548  72%                 - project-try-vc
>          4537  72%                  - vc-responsible-backend
>          4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>          4478  71%                    - vc-call-backend
>          4478  71%                     - apply
>          1470  23%                      + vc-svn-responsible-p
>          1142  18%                      + vc-bzr-responsible-p
>           970  15%                      + vc-hg-responsible-p
>           390   6%                      + vc-git-responsible-p
>           156   2%                      + vc-cvs-responsible-p
>           126   2%                      + vc-rcs-responsible-p
>           108   1%                      + vc-sccs-responsible-p
>            98   1%                      + vc-src-responsible-p
>            57   0%                   + tramp-file-name-handler
>            11   0%                  + vc-file-getprop
>           393   6%                 + global-tags-try-project-root
>           375   5%        + read-from-minibuffer
>            13   0%      + if
>           213   3%     + project-current
>            88   1%  + funcall-interactively
>           572   9% + ...
>            51   0% + timer-event-handler
>             8   0% + redisplay_internal (C function)
> 
> 
> As you can see most of the time is still taken by project-current and I
> can't really understand why:

AFAICT, most of the time is taken by 'apply', but the profile doesn't
show which function is called by 'apply'.  Can you tell which function
is that?

> 1) Are so many samples 4548 seems a very high number for only 25 opened
> buffers.

These two numbers are unrelated.  4548 is the number of time the
profiler found the program counter inside project-try-vc and the
functions it calls.  This number has no relation to the number of
buffers you have, it just means that code runs slowly.

> 2) why project-try-vc still takes so much...? Specially for unfrequent
> vc systems in our days like svn or bzr that I am not using.

That was explained on emacs-devel.  However, ...

> As a workaround I removed all the uninteresting handlers from
> vc-handled-backends and I get better times now, but IMHO it is still
> very inefficient (almost a minute for project-switch-to-buffer is
> excessive). And make it practically unusable.

... after removing the "unused" VC back-ends, you say that the code
still runs very slowly.  So is the issue with VC back-ends still
relevant, and if so, how?

More importantly, what is the profile after you remove the extra VC
calls?

> VCS changing is not something that happens very often to require a check
> of all the backends everytime, several times for every buffer in many
> project.el functions right? Specially when using tramp.

Once again, given what you say above, this doesn't sound important,
does it?  The slow processing is elsewhere, and without seeing a
profile with VC calls removed, it's hard to make progress in this
matter, or give you some advice regarding potential reason(s).

> vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
> it's result there to avoid repeating time consuming computations?

Again: is this issue relevant, given that without the VC calls the
code is still very slow?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 28 Jun 2021 22:11:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Jun 28 18:11:31 2021
Received: from localhost ([127.0.0.1]:53227 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lxzTX-00024u-8G
	for submit <at> debbugs.gnu.org; Mon, 28 Jun 2021 18:11:31 -0400
Received: from lists.gnu.org ([209.51.188.17]:40232)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <spacibba@HIDDEN>) id 1lxzTT-00024m-Op
 for submit <at> debbugs.gnu.org; Mon, 28 Jun 2021 18:11:30 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:51574)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <spacibba@HIDDEN>) id 1lxzTS-0007w9-1p
 for bug-gnu-emacs@HIDDEN; Mon, 28 Jun 2021 18:11:27 -0400
Received: from sonic302-3.consmr.mail.bf2.yahoo.com ([74.6.135.42]:41670)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <spacibba@HIDDEN>) id 1lxzTN-000887-HB
 for bug-gnu-emacs@HIDDEN; Mon, 28 Jun 2021 18:11:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aol.com; s=a2048;
 t=1624918279; bh=yXMBLpUkKtciNPAdYKF0xinWWVzc328j8BDTdQPc5Fg=;
 h=From:To:Cc:Subject:Date:References:From:Subject:Reply-To;
 b=fyj53isZtmkLBs/SBIJds6xb/TTHdfQlSJohVPmzIpxO1X+4jdKM29WXLpgRHC7y2t8ywvKXcyZ1O02vgJlkoTQVIhsQlmtsGC+KCg+Vg+3hkN/e1egPgJtGgStdmQuNnReSYq+Zvmh0yRVnrk8gj0HCnvrXMKunzumkGHyBq3n7yckfnbVFP3XuzyswpC0PBywSLSifYMGXbTkH5ILhvdCibwusZItX5bSrWj9NC83p9dh1Ogi8Ht+Z7zh6rAiQPTYf1sDZqaNrD3mqLtkD4HuMbdSawXVA/quG1Neb4c34kHCXXvNBIrcZHe4o4/6iNNTd6o0TFiAZed+eDQBEJA==
X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048;
 t=1624918279; bh=wH2frLZevEKxC6JMKP/jJVQ6D2KA+qNU9BTZIJDvhRw=;
 h=X-Sonic-MF:From:To:Subject:Date:From:Subject;
 b=IT2PycGsTWQvWFs4adtBuIonpY0nwKUkVWh9PFy9IC821DKe+iFb4f34ZsSFmTW6JNYT2xicLq+Xdi4kbZaw+hfAYGEZCexAvTFGfUEQcQma7wLwSP08D9AMhqcH+BEAbzRTEkNBKp6FhPUKMXUey/9aE5TPEeH2I80F9DXyZpBDJmnDDs6j7EtTDHmENXGAmlM5V+dZBzt4lqYR6NO9I+ViCn6/EJw435sUSJFXF0DvyJpgyhsm72Ex4VmtKTuNYbBKeeBB0R1zRsVHyIN7mqHck/43WAlnrW+KEeqaQq71AkkwFoJeIcvdZ9jj0tGCAKE9Mu0x44be3d+5h3VqbA==
X-YMail-OSG: WN7SN1wVM1nzAbeJV7E9LtPCBrHIeQf1a_NwYHycAAr9lT1ZAUR.9E1zO8PmMY3
 Jh.hpjMzdiJfWr7bQjkGK.3EC90wTcaknCiARy.3qwdVFW0GywARC9g3R6MjAw.w98evw_lu8aLU
 Ay9vLk6AdphvGMCx33dMl0c7XSKQ68nqIs3wqczQtLpxWFLSFJpwtM1lPZ_TL3Xn5sqKysdjkTJv
 rHSjG4rpTqlHjGhW5fnr7QfJN5H5NX_kPK9flWp16vucEHTQ2xEKn9xVdTtIZjzGl37UlzKzlJEF
 xelCSN.kfcwHUlADEi.e_6WiXZAOFDVi91fZE4vqSTJL1Iw7xVn3ubjGbXMxLRRzRqGlsZxh40Wa
 3wudgkttiTF..w0ijj0Ymtx.n7XzfQVHeH2SbcoEvqyW9FkM2wiqiZyTtN_4n7O9EMU7IFBbvBtm
 ERpayLBrhESKraTf_Ag_5IW46gv6URVB5IKsfwYNxa2swoDsPa2hIcDX9h_Tpea4lhCL_Lae0TER
 3kc.OU.bgb1mFmgDyAUc34Dp09P1B1i0ZD0l0fKJEfVZXq5S.PextjLfZowffriQNrW_NGWcE.mS
 KIFjzU15QbSSqKVL653T.oXidTtbUlnTB37H1w9jqZAz8tnxw7QNgR_NHvgg0JlvaCNAOsATBgjy
 2PDQlmyLwU_8PB6EGozmihaHz3udwViNMNv9rXGj7SKyYM9X9OfJw0SM96LsvRpI7Y1hOOozX5UO
 EXH6A8xA8ip0oTLbetege0YM9e6YqugsniGoOnQz2fr2DUTeboyr3eyCECOyWqXgEQpRzDE18i84
 PgmPnAbYSf37ZmpM4UgBfXKa.W8Q02Vh.ZSGYrJgw5TINRbc3QE030M2Od_3K4Of9x2n0X9V6AXc
 TzzPkmGangw0oKVwh5ghURJLfVkBu20wJ5VwjuxiXuzdceXgAd8_Dn_O1zTHfehe74r5jMch3Avd
 QY_ZppXU86hAWlQxWrGKAWkH5ynlcgjK2cUmYJH7ACeDQkkaMCxyWH3mJFo.Oe4.bGf2lCpHls0T
 IJk2ltKJscmfixKryDBsoo3dpucDeauuWKTY9wJ7odIy3i6hLvPw.BKm9Di3BHzO4HX6BVWobsnw
 tICzEAA0oAJInpLDB_Vz7xoxmKlnqYYXeyk6J51ea5KPNMQnHKmK5IPzdju8a0tfRDbjfGUIaaZn
 HgEkApx3UDNxEU9jDaA.uj.mrRD7XMc3ECQzXjplW9e6rN2i5H_uGCD4fsOeLJXGWDIS6VDGAwFG
 rRHtkgO.HVzeoQ_PWJxOXUlIFvUvQKstSUL7LSa8WLWazEnRWIxUeXut_ba1euqnuKpB5gwLpxsj
 yNQyhh9xvoA6gXbOclTr7gNXOKFSvwFSZNxHmJn7lWdqvQ1G7213sZf3oZwSX_cby45nkTXbbcz8
 5FtRZZiQ8gRnQ0.j0.hlAVaKm7b_uA4z9IjXPcqUqlLk7qwn5g71vtjD4Okrpc3Zfl1tFMI7kNHO
 qGOQSjikZlj6GQ2Pr0MArHF5Ipxc43YOCRw5rmMOHzZYl0AytNlQzaE9vFiMZEUSsdmwYt0ihfN0
 XLIsreaUwkHTxr41Fa1Lm1z1_Cjl7LLDJZpCWYRmnz2hycoY6UXOvuqtIeCB1Wyf482x1IewZrtR
 4KDMY7kVqcAYouCGKEkYf_XutWCWuh1cgGROEtv6Uq0BrOIj_aA.KqIuhnWu033oPadz8WfgKElU
 CximrCC5u02A7Y5pIK3ahReDLaHBVQF3Mkkw06gxKdJKcksBtVyK7z1MBvhDkwMiDs8XVOuYSvdm
 Q4K9TQtAIRQQx.QxWxJQNq71IQkHf_jkK6b74YYoWCXlK8eKKIxbl3A0AKsaUY.t2PIxXvj7nkeH
 eFYRTyyNPyy3t7i5E6fOjj5qAV7_HJdplmcIPsfdsqTM26KVv8QBpY4M7MKhG.qpLvzk8CxwwnOw
 7kkVlauDAU7rLUe7dqjL6K7HI1IWxHVt5sGq0uxDyCESlvOT6j40Jv8YQEPdM2yU0tKUrmz6hvRf
 gLxQRdV93.8J02IxwCEcy4ng2MjQrXKj.3YuXfi4hwXok3UfMXoL3XqL_d_hZHhcera3_WxMQiTK
 jUc44tKujJ9o2mMwYbOIPSmM9zKjFu8XCXUf.gvcc_IxVBhqgXk._Lu7k0MGJxTDLlqaqdL1TCR2
 ZlgXrimw2hmTZ_FwC36E2mRdb3JW5si41_kdK5rSGH1NesMej3_toCETCSKkjEeD8LhSwUIVPaDQ
 EQXzPUzDmL6GrAmkBrCn4EBq60u4m3omlQ3DCVzRdlFJ4xZ1oHpSBuR8fK8znZwSTB_HU5Acsrzx
 JPH0oLQEb.t4XYi04wVsXBY_IMg--
X-Sonic-MF: <spacibba@HIDDEN>
Received: from sonic.gate.mail.ne1.yahoo.com by
 sonic302.consmr.mail.bf2.yahoo.com with HTTP; Mon, 28 Jun 2021 22:11:19 +0000
Received: by kubenode503.mail-prod1.omega.ir2.yahoo.com (VZM Hermes SMTP
 Server) with ESMTPA ID e2ff5b4c72c268af2e9aa79d29da4757; 
 Mon, 28 Jun 2021 22:11:17 +0000 (UTC)
From: Ergus <spacibba@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Subject: 28.0.50; project.el+tramp performance issue
Date: Tue, 29 Jun 2021 00:11:00 +0200
Message-ID: <87fsx13aiz.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
References: <87fsx13aiz.fsf.ref@HIDDEN>
X-Mailer: WebService/1.1.18469
 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.aol
Content-Length: 12506
Received-SPF: pass client-ip=74.6.135.42; envelope-from=spacibba@HIDDEN;
 helo=sonic302-3.consmr.mail.bf2.yahoo.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)


Hi:

Using tramp I tried to use project.el with a command like
project-switch-to-buffer and it took like 10 minutes to complete.

I ran a profiler and I found that most of the time was taken by an
external function: global-tags-try-project-root

project-current is called in a loop for all the opened buffers it calls
project--find-in-directory that calls project-find-functions and there
is going all the time.

After some optimization in an external package; now the time is half
than before but still very slow to use the command (around 3-5 minutes
to complete) and running again the profiler I get this:

         5637  89% - command-execute
         5549  88%  - byte-code
         5549  88%   - project--read-project-buffer
         5549  88%    - let*
         5336  85%     - read-buffer
         5323  84%      - ivy-completing-read
         5323  84%       - ivy-read
         4941  78%        - ivy--reset-state
         4941  78%         - ivy--buffer-list
         4941  78%          - internal-complete-buffer
         4941  78%           - #<lambda -0x1a357caf01243d61>
         4941  78%            - and
         4941  78%             - equal
         4941  78%              - save-current-buffer
         4941  78%               - project-current
         4941  78%                - project--find-in-directory
         4548  72%                 - project-try-vc
         4537  72%                  - vc-responsible-backend
         4478  71%                   - #<compiled 0xd3f2e32af0966f7>
         4478  71%                    - vc-call-backend
         4478  71%                     - apply
         1470  23%                      + vc-svn-responsible-p
         1142  18%                      + vc-bzr-responsible-p
          970  15%                      + vc-hg-responsible-p
          390   6%                      + vc-git-responsible-p
          156   2%                      + vc-cvs-responsible-p
          126   2%                      + vc-rcs-responsible-p
          108   1%                      + vc-sccs-responsible-p
           98   1%                      + vc-src-responsible-p
           57   0%                   + tramp-file-name-handler
           11   0%                  + vc-file-getprop
          393   6%                 + global-tags-try-project-root
          375   5%        + read-from-minibuffer
           13   0%      + if
          213   3%     + project-current
           88   1%  + funcall-interactively
          572   9% + ...
           51   0% + timer-event-handler
            8   0% + redisplay_internal (C function)


As you can see most of the time is still taken by project-current and I
can't really understand why:

1) Are so many samples 4548 seems a very high number for only 25 opened
buffers.

2) why project-try-vc still takes so much...? Specially for unfrequent
vc systems in our days like svn or bzr that I am not using.

As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient (almost a minute for project-switch-to-buffer is
excessive). And make it practically unusable.

In any case:

Maybe (I think I mentioned this before) `project.el` needs a sort of
cache to speedup some functions like `project-current` that are called
very frequently inside the project.el code.

Related with https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966

VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right? Specially when using tramp.

vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
it's result there to avoid repeating time consuming computations?

Either in the local system the performance penalty seems to be
significant to me.

Best,
Ergus



In GNU Emacs 28.0.50 (build 50, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.17.4)
 of 2021-06-26 built on Ergus
Repository revision: b8f9e58ef72402e69a1f0960816184d52e5d2d29
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-json
 --with-x-toolkit=gtk3 --with-xft --with-wide-int --with-modules
 --with-cairo --with-harfbuzz --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++//law

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  diff-hl-margin-mode: t
  windmove-mode: t
  subword-mode: t
  hide-ifdef-mode: t
  preproc-font-lock-mode: t
  shell-dirtrack-mode: t
  show-paren-mode: t
  global-auto-revert-mode: t
  xclip-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  electric-pair-mode: t
  flyspell-mode: t
  company-mode: t
  flycheck-mode: t
  counsel-mode: t
  ivy-mode: t
  composable-mark-mode: t
  composable-mode: t
  repeat-mode: t
  xterm-mouse-mode: t
  winner-mode: t
  save-place-mode: t
  which-key-mode: t
  override-global-mode: t
  delete-selection-mode: t
  savehist-mode: t
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/cmake-mode hides /home/ergo/.emacs.d/elpa/cmake-mode-20210104.1831/cmake-mode
/usr/share/emacs/site-lisp/notmuch-crypto hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-crypto
/usr/share/emacs/site-lisp/notmuch-compat hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-compat
/usr/share/emacs/site-lisp/notmuch-hello hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-hello
/usr/share/emacs/site-lisp/notmuch hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch
/usr/share/emacs/site-lisp/notmuch-show hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-show
/usr/share/emacs/site-lisp/notmuch-maildir-fcc hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-maildir-fcc
/usr/share/emacs/site-lisp/coolj hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/coolj
/usr/share/emacs/site-lisp/notmuch-draft hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-draft
/usr/share/emacs/site-lisp/notmuch-tree hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-tree
/usr/share/emacs/site-lisp/notmuch-parser hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-parser
/usr/share/emacs/site-lisp/notmuch-lib hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-lib
/usr/share/emacs/site-lisp/notmuch-mua hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-mua
/usr/share/emacs/site-lisp/notmuch-message hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-message
/usr/share/emacs/site-lisp/notmuch-address hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-address
/usr/share/emacs/site-lisp/notmuch-wash hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-wash
/usr/share/emacs/site-lisp/notmuch-tag hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-tag
/usr/share/emacs/site-lisp/notmuch-print hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-print
/usr/share/emacs/site-lisp/notmuch-query hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-query
/usr/share/emacs/site-lisp/notmuch-jump hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-jump
/usr/share/emacs/site-lisp/notmuch-company hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-company
/home/ergo/.emacs.d/elpa/transient-20210619.1100/transient hides /home/ergo/.local/share/emacs/28.0.50/lisp/transient

Features:
(shadow sort notmuch-company notmuch-lib notmuch-version notmuch-compat
mm-view mml-smime smime dig mail-extr emacsbug sendmail magit-extras
hi-lock magit-bookmark magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff
git-commit log-edit message rmc puny rfc822 mml mml-sec epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader add-log magit-core
magit-autorevert magit-margin magit-transient magit-process with-editor
server magit-mode transient magit-git magit-section magit-utils crm
tramp-cmds global-tags ht generator async counsel-gtags pulse
mc-separate-operations mc-edit-lines mc-hide-unmatched-lines-mode
mc-mark-more mc-cycle-cursors multiple-cursors-core rect move-dup
diff-hl-margin eieio-opt speedbar ezimage dframe shortdoc help-fns
radix-tree vc-annotate amx s windmove misearch multi-isearch ffap
url-parse url-vars face-remap vc-hg macrostep-c cmacexp macrostep
cap-words superword subword hideif preproc-font-lock cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
dired-aux diff-hl-dired diff-hl log-view pcvs-util vc-dir ewoc vc
tramp-cache tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete parse-time iso8601 time-date
ls-lisp format-spec auth-source password-cache thingatpt vc-git
diff-mode vc-dispatcher bookmark pp paren autorevert filenotify xclip
yasnippet-snippets yasnippet elec-pair flyspell-correct-ivy
flyspell-correct flyspell ispell company-keywords company-gtags
company-dabbrev-code company-dabbrev company-files company-semantic
company-template company-capf company flycheck json map find-func dash
counsel xdg xref project dired-x dired dired-loaddefs compile
text-property-search comint ansi-color swiper ivy-avy avy ivy flx
ivy-faces ivy-overlay colir pcase term/tmux term/xterm xterm jka-compr
init composable composable-mark powerline comp comp-cstr warnings subr-x
powerline-separators color powerline-themes repeat xt-mouse
simple-16-theme winner ring saveplace diminish edmacro kmacro which-key
advice configmail cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
use-package-core disp-table delsel savehist easy-mmode
display-fill-column-indicator display-line-numbers info ede/auto
eieio-base cl-seq seq eieio byte-opt bytecomp byte-compile cconv
eieio-core cl-macs gv eieio-loaddefs cl-loaddefs cl-lib tex-site rx
slime-autoloads early-init iso-transl tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 811583 75249)
 (symbols 48 30651 0)
 (strings 32 107218 13868)
 (string-bytes 1 4408538)
 (vectors 16 57480)
 (vector-slots 8 1324399 45245)
 (floats 8 342 1198)
 (intervals 56 53473 1428)
 (buffers 992 37))




Acknowledgement sent to Ergus <spacibba@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#49264; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Wed, 30 Jun 2021 13:45:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.