GNU logs - #56342, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Paul Pogonyshev <pogonyshev@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 01 Jul 2022 17:15:02 +0000
Resent-Message-ID: <handler.56342.B.16566956662785 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 56342 <at> debbugs.gnu.org
MIME-Version: 1.0
From: Paul Pogonyshev <pogonyshev@HIDDEN>
Date: Fri, 1 Jul 2022 19:14:08 +0200
Message-ID: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
Content-Type: multipart/alternative; boundary="0000000000004ef69f05e2c184de"
Received-SPF: pass client-ip=2607:f8b0:4864:20::b34;
 envelope-from=pogonyshev@HIDDEN; helo=mail-yb1-xb34.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

--0000000000004ef69f05e2c184de
Content-Type: text/plain; charset="UTF-8"

Emacs version: GNU Emacs 28.1.50 (build 1, x86_64-pc-linux-gnu, GTK+
Version 2.24.33, cairo version 1.16.0) of 2022-05-24

I have an internet connection with a relatively large ping, so TRAMP for me
works quite slowly. I notice this when editing and saving files (also when
Emacs stores backup copies), with Magit and otherwise.

I tried to understand why it is _so_ slow for me. I have found out variable
`tramp-verbose' and used it to figure out which commands TRAMP executes on
the remote machine. For testing, I used a real library's (Logview's) buffer
refreshing command  that boils down to this:

    (with-temp-buffer
      (insert-file-contents the-file nil from-somewhere-in-the-middle nil)
      ; do something about the data
      )

In other words, it inserts into a buffer not the whole file, but a part of
it (the reason is irrelevant here).

Well, this results in _eleven_ commands from TRAMP! Here they are:

1) check if connection is alive (`echo are you awake');
2) test if the file exists;
3) creating a temporary file for the chunk to be inserted; I guess it tries
until it finds an unused filename, e.g. here it seems to be done after
`test -e /tmp/tramp.OD3cCu', which doesn't exist;
4) 'touch' on the temporary file, presumably to create it;
5) 'chmod' on the temporary, presumably so that other users cannot read it;
6) copying the requested chunk from the full file into the temporary (using
`dd');
7) finding the real name of the temporary with `readlink';
8) finding attributes of the temporary with `stat';
9) gzipping the temporary for transmition from the remote to the local
machine;
10) testing if the temporary is a directory (WTF?);
11) removing the temporary.

I guess it should be obvious that this is a bit too much for one
`insert-file-contents' call.

Suggested improvements:

* TRAMP should issue just one `stat' command to find out most of the things
about a file: whether it exists, if it is a directory, its real name when
dereferencing links and whatever stats it is used to find now; from `$ stat
--help' this seems to be possible. In other words, TRAMP shouldn't use
simple commands like `test -e': any ping, even nominal, will negate any
gains from using a tad faster command. Instead, if it needs to find
anything about a file, it should ask the remote about as many things as
possible in one go: it is very likely that the additional information will
be needed soon and even if not, this is basically free compared to ping
anyway.

* TRAMP code should prefer the approach "try do something and handle
resulting errors" where possible. For example, don't check if the file
exists, try to read it right away and handle failures properly. Code like
`(when (file-exists-p ...) do-something)' adds an unnecessary command call
and creates a racing condition anyway. Also, error-free requests should be
more frequent, so they should be the main optimization goal. I'm not sure
if it is applicable to TRAMP itself and doesn't come from a higher level,
though.

--0000000000004ef69f05e2c184de
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Emacs version:=C2=A0GNU Emacs 28.1.50 (build 1, x86_6=
4-pc-linux-gnu, GTK+ Version 2.24.33, cairo version 1.16.0) of 2022-05-24</=
div><div><br></div><div>I have an internet connection with a relatively lar=
ge ping, so TRAMP for me works quite slowly. I notice this when editing and=
 saving files (also when Emacs stores backup copies), with Magit and otherw=
ise.</div><div><br></div><div>I tried to understand why it is _so_ slow for=
 me. I have found out variable `tramp-verbose&#39; and used it to figure ou=
t which commands TRAMP executes on the remote machine. For testing, I used =
a real library&#39;s (Logview&#39;s) buffer refreshing command=C2=A0 that b=
oils down to this:</div><div><br></div><div>=C2=A0 =C2=A0 (with-temp-buffer=
</div><div>=C2=A0 =C2=A0 =C2=A0 (insert-file-contents the-file nil from-som=
ewhere-in-the-middle nil)<br></div><div>=C2=A0 =C2=A0 =C2=A0 ; do something=
 about the data</div><div>=C2=A0 =C2=A0 =C2=A0 )</div><div><br></div><div>I=
n other words, it inserts into a buffer not the whole file, but a part of i=
t (the reason is irrelevant here).</div><div><br></div><div>Well, this resu=
lts in _eleven_ commands from TRAMP! Here they are:</div><div><br></div><di=
v>1) check if connection is alive (`echo are you awake&#39;);</div><div>2) =
test if the file exists;</div><div>3) creating a temporary file for the chu=
nk to be inserted; I guess it tries until it finds an unused filename, e.g.=
 here it seems to be done after `test -e /tmp/tramp.OD3cCu&#39;, which does=
n&#39;t exist;</div><div>4) &#39;touch&#39; on the temporary file, presumab=
ly to create it;</div><div>5) &#39;chmod&#39; on the temporary, presumably =
so that other users cannot read it;</div><div>6) copying the requested chun=
k from the full file into the temporary (using `dd&#39;);</div><div>7) find=
ing the real name of the temporary with `readlink&#39;;</div><div>8) findin=
g attributes of the temporary with `stat&#39;;</div><div>9) gzipping the te=
mporary for transmition from the remote to the local machine;</div><div>10)=
 testing if the temporary is a directory (WTF?);</div><div>11) removing the=
 temporary.</div><div><br></div><div>I guess it should be obvious that this=
 is a bit too much for one `insert-file-contents&#39; call.</div><div><br><=
/div><div>Suggested improvements:</div><div><br></div><div>* TRAMP should i=
ssue just one `stat&#39; command to find out most of the things about a fil=
e: whether it exists, if it is a directory, its real name when dereferencin=
g links and whatever stats it is used to find now; from `$ stat --help&#39;=
 this seems to be possible. In other words, TRAMP shouldn&#39;t use simple =
commands like `test -e&#39;: any ping, even nominal, will negate any gains =
from using a tad faster command. Instead, if it needs to find anything abou=
t a file, it should ask the remote about as many things as possible in one =
go: it is very likely that the additional information will be needed soon a=
nd even if not, this is basically free compared to ping anyway.</div><div><=
br></div><div>* TRAMP code should prefer the approach &quot;try do somethin=
g and handle resulting errors&quot; where possible. For example, don&#39;t =
check if the file exists, try to read it right away and handle failures pro=
perly. Code like `(when (file-exists-p ...) do-something)&#39; adds an unne=
cessary command call and creates a racing condition anyway. Also, error-fre=
e requests should be more frequent, so they should be the main optimization=
 goal. I&#39;m not sure if it is applicable to TRAMP itself and doesn&#39;t=
 come from a higher level, though.</div></div>

--0000000000004ef69f05e2c184de--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Subject: bug#56342: Acknowledgement (TRAMP (sh) issues way too many
 commands, thus being very slow over high-ping networks)
Message-ID: <handler.56342.B.16566956662785.ack <at> debbugs.gnu.org>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
X-Gnu-PR-Message: ack 56342
X-Gnu-PR-Package: emacs
Reply-To: 56342 <at> debbugs.gnu.org
Date: Fri, 01 Jul 2022 17:15:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 56342 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
56342: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D56342
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 02 Jul 2022 15:59:01 +0000
Resent-Message-ID: <handler.56342.B56342.16567775342923 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
Date: Sat, 02 Jul 2022 17:58:25 +0200
In-Reply-To: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 (Paul Pogonyshev's message of "Fri, 1 Jul 2022 19:14:08 +0200")
Message-ID: <8735fjh5ge.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Paul Pogonyshev <pogonyshev@HIDDEN> writes:

Hi Paul,

> 1) check if connection is alive (`echo are you awake');
> 2) test if the file exists;
> 3) creating a temporary file for the chunk to be inserted; I guess it
> tries until it finds an unused filename, e.g. here it seems to be done
> after `test -e /tmp/tramp.OD3cCu', which doesn't exist;
> 4) 'touch' on the temporary file, presumably to create it;
> 5) 'chmod' on the temporary, presumably so that other users cannot
> read it;
> 6) copying the requested chunk from the full file into the temporary
> (using `dd');
> 7) finding the real name of the temporary with `readlink';
> 8) finding attributes of the temporary with `stat';
> 9) gzipping the temporary for transmition from the remote to the local
> machine;
> 10) testing if the temporary is a directory (WTF?);
> 11) removing the temporary.
>
> I guess it should be obvious that this is a bit too much for one
> `insert-file-contents' call.

In general, I agree. However, some of the commands are caused by
primitive file operations, like file-exists-p. Tramp cannot know what
will be the next call, and it doesn't have all the opportunities to
optimize, compared with the overall picture you see in the eleven steps.

> Suggested improvements:
>
> * TRAMP should issue just one `stat' command to find out most of the
> things about a file: whether it exists, if it is a directory, its real
> name when dereferencing links and whatever stats it is used to find
> now; from `$ stat --help' this seems to be possible. In other words,
> TRAMP shouldn't use simple commands like `test -e': any ping, even
> nominal, will negate any gains from using a tad faster command.
> Instead, if it needs to find anything about a file, it should ask the
> remote about as many things as possible in one go: it is very likely
> that the additional information will be needed soon and even if not,
> this is basically free compared to ping anyway.

Not all remote hosts carry a stat command, and not all existing stat's
are GNU compatible. But yes, if possible, Tramp shall gather as much
information in one run, and cache the results for further use.

I will see what could be done. Will come back with a proposal next days
(note that this will be for Emacs 29, ie git master).

> * TRAMP code should prefer the approach "try do something and handle
> resulting errors" where possible. For example, don't check if the file
> exists, try to read it right away and handle failures properly. Code
> like `(when (file-exists-p ...) do-something)' adds an unnecessary
> command call and creates a racing condition anyway. Also, error-free
> requests should be more frequent, so they should be the main
> optimization goal. I'm not sure if it is applicable to TRAMP itself
> and doesn't come from a higher level, though.

Indeed, this is not Tramp's responsibility. Tramp is a stupid
library. If there is a call for file-exists-p, it must return the
answer. It doesn't know what will be the next request. So I'm rather
pesimistic that Tramp can improve here.

Best regards, Michael.

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 2 Jul 2022 15:59:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jul 02 11:59:20 2022
Received: from localhost ([127.0.0.1]:42620 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o7fWi-0000m5-A0
	for submit <at> debbugs.gnu.org; Sat, 02 Jul 2022 11:59:20 -0400
Received: from mout.gmx.net ([212.227.15.18]:56227)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o7fWe-0000ls-Rl
 for control <at> debbugs.gnu.org; Sat, 02 Jul 2022 11:59:18 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656777550;
 bh=H+olClqt9cZzEb1MWHjSdj16jYpS3GluRkoNjw4SMzY=;
 h=X-UI-Sender-Class:Date:To:From:Subject;
 b=h8NwlG0cGQxFGIDM7e2PwouQomP9aJRSf4e6PfNTDgq1ziE5pFS6eZUgKZ3N1wIff
 N6D7Icp0ZcGT5t47Y3C8V2cIFQntpeocFY3u0dAUEUTtvIIhfunY/58QtkH/U7cSVU
 RYX3agJ4uii+rRUz2ToPGBVCsCTbDNOPnxgDQ5dY=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([79.140.124.34]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N7zBR-1nUweb2RRD-014yx0 for
 <control <at> debbugs.gnu.org>; Sat, 02 Jul 2022 17:59:10 +0200
Date: Sat, 02 Jul 2022 17:59:09 +0200
Message-Id: <871qv3h5f6.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: Michael Albinus <michael.albinus@HIDDEN>
Subject: control message for bug #56342
X-Provags-ID: V03:K1:XXO7rm60tOqdcraQFXydRjVBZldW26QIIvf6UN58H+sbfalNkOQ
 u2b3m+hlqJCv2FxoQ1L8dMRTlaldgRjfHRuxWqyWp6id/1trViL3Dm1du+FNSzJe2ZrXEHL
 bi8pEkbDnYOyR0UyQpWmWPgSb6j7+E1TYafdwPI2hnkmMxuFNPRvR9pVc6iEYE87VS3X4vC
 O+vNYGhf/1fdbYrxsW0yw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:r3Q+6ozTWh8=:KB0IOcHZ7wthiGne01jQwG
 cRg5aHsyL6aLE4ZTq4rzgAJ6gUy1P5ww12dQ71l5kCxwvkIbeF47vUyAuJAAVDZ4FB6im9p6b
 woASeaSuHLVSwLr/+aGQGlp7/kLJvaRiGYgLGMZjLe7XA0qkttwRMg45Oc17oM+297Tyqyyq4
 8Z2laYCXDBqlgF2jqhC/HXpEyRaw8nNZVMWMEm/xLdL59qrLm9mJZXX1D/czAd80hynyX4ipq
 xfvYpiq1M2OcQa6jppUjUeNqydVoOcBAvt6bg2MLIceZFdnf2ExQk06VfjtGsyl1cGjkgEJ6v
 CtgP02sXK7QJ7u9BTDH9gBSYnyzbElk17/RnBBVQkSnmmPTdOToPnH4YmWwu7QdpY+xfTmH5Q
 Z3udkKvCFVHqFF3huDgDybsl4FxkiKNQtDIQBoMXHuev3ulb+WhRwPLlB50O52uoqDPkYeGRi
 PdbR3ZW+pSQ9UUqYuWofa2tp821PmvYpRQrT2+GD9sMAqFdHzkc/dukoelMp/WtrcxlTlCUI9
 m6Z+m4pvK1Bx91Y3dYvMThA0NjE75AnMCNHwcgj9DWvfH8NrOYSwn149QAJQVpLYY8XWzW0XH
 vcaCE9yxNJGBJuERUs7ImbfEpGd7PQSh65BC5LLUxhuZATVPMqdE54RbQ9J4SQYHnbPU88lru
 Gt/aTMu7byht3FDH33xDg57mY0IDAvepE55TuURVZ3DPOnNQv89axdrRKsJn0Guuw6zc16t0N
 QIessUc7ZR7wIcuZx18RQ5+/IJ2EwEEN7Fg72KnMvPEV5piSatjzig359lpwzd47lb6z7mQ0k
 mNZQlPVkJkSL1wgBztRJ43mQHyGdmTtKpwvFIH7fPUjN/wgPkAjzlg7yHm9UIOrmtkTZF9aLr
 49seJ74NkB6amnrBLRl/eVSuIFayixK3j97jyLSSbfeKmUXnhWM//d/W60tMRVMQQEpNNLDQa
 4CEVLZA1RK7FWagpM+j/8DNy+9/5scogk1BohecQYLSqFAfhNbOTWqkULMsFj7awlGtO8tdBu
 Ce1C8MEhZ56LSAvslmRhOr2jvNC93walPGrg7ie28Un1DBGl9bnQdF0aO2i1UIbgmnviG2itc
 kG33jurBaDOnrnPsv++ZPGwqV8zOVIja6Dgm3AECbStb4ry/ZCSEs0YuQ==
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

severity 56342 wishlist
quit

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Paul Pogonyshev <pogonyshev@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 02 Jul 2022 18:15:02 +0000
Resent-Message-ID: <handler.56342.B56342.16567856961055 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.16567856961055
          (code B ref 56342); Sat, 02 Jul 2022 18:15:02 +0000
Received: (at 56342) by debbugs.gnu.org; 2 Jul 2022 18:14:56 +0000
Received: from localhost ([127.0.0.1]:42831 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o7hdw-0000Gv-1K
	for submit <at> debbugs.gnu.org; Sat, 02 Jul 2022 14:14:56 -0400
Received: from mail-yw1-f176.google.com ([209.85.128.176]:35345)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pogonyshev@HIDDEN>) id 1o7hds-0000Gg-8f
 for 56342 <at> debbugs.gnu.org; Sat, 02 Jul 2022 14:14:54 -0400
Received: by mail-yw1-f176.google.com with SMTP id
 00721157ae682-31c782f7d96so19207707b3.2
 for <56342 <at> debbugs.gnu.org>; Sat, 02 Jul 2022 11:14:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=AokJDcy7hKPJp8AABywEI8gLvXO/gAn1sDPkj8v0nvw=;
 b=VpyjD+nmKe/0UJF2lt5CYkrt0FbhbREJDpswwsiZaIjcRpjuhJ6nivlsYoa0ZE3A2U
 YVwee4XzOeDrDLnXsXutcw5DzMPxkreh9tgMLQt0OqCkW9WprtU2CEeeH6rZRVqm1XJd
 fi92TuLTUDNayqizT+jVx4gUAOCnjXqSVro6IJeXYjgCdtERh5uWzr9d5aGDMCLeoyEk
 aSRpPi3VtHsndOBfT5GyMTHX4RWbcKcamFGLCAe6KGQo+e1I4iN4EANIHWZmraJG82sg
 sF9+rZ5DtXVVSn9PJ9dPGYhU9MpCgMvMxKnPNDkuH6ntXmx/3BvwnwpmYQYrDQv2gbpg
 5jfw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=AokJDcy7hKPJp8AABywEI8gLvXO/gAn1sDPkj8v0nvw=;
 b=EEOEpQ/NgrKqa0ytHXsFHYbeDM/w75z/goQhBNnPDK9XakMrbmIiasG0j4gQmfdb4X
 kHfqhoSSYHoWYkYpSA/PPGcNDA2IRt1Vlaf4TkYMs9nI8BFQGUN60DGvUZgNkfC3Iv4T
 GbqDCn75eR4/eEgveZPFnHiNJETm4GwwzyIKaEfhBBeLvOLxpjMoi84zlb6ee7ehEnHS
 MYvAb52yk3WlAFXTX+gtunk+R74kDIaf36Ff5YOonyjJwVkQvp1BFCbRbQFvMccIIkl0
 iOetKJ2Vjh5nuIjJyl9GY+J1nkjsxkVI8ZnK97who6IHNWeoqd9AYH3ctGseNnOvBhCO
 ug1Q==
X-Gm-Message-State: AJIora+Ja5bdNlBoZI8+8eFmRVdNR2eSyItMIl9HNivZU6LUSzFDo0Ah
 9600s7U//gFqPVDGufE6rPU8EE4b6eBCgSZW1JHPnXm40+pM
X-Google-Smtp-Source: AGRyM1ttpVTMFnoUhAR8kCduK8sNpw2zwPlijRyhK5gNMiLkUdDZR2UQ7726+cptSLpTWbpmeo9hS7X0gxrnSK0EFZ0=
X-Received: by 2002:a81:6cd2:0:b0:31c:95d:552 with SMTP id
 h201-20020a816cd2000000b0031c095d0552mr23512852ywc.28.1656785686496; 
 Sat, 02 Jul 2022 11:14:46 -0700 (PDT)
MIME-Version: 1.0
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
In-Reply-To: <8735fjh5ge.fsf@HIDDEN>
From: Paul Pogonyshev <pogonyshev@HIDDEN>
Date: Sat, 2 Jul 2022 20:14:35 +0200
Message-ID: <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
Content-Type: multipart/alternative; boundary="00000000000050098e05e2d67ade"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--00000000000050098e05e2d67ade
Content-Type: text/plain; charset="UTF-8"

Some more thoughts. Why does it even need `echo are you awake'? It's a
network connection, it can still fail even if it worked fine 1 ms before
when you checked. So, why not just let the first command fail if the
connection is dead and restart the connection if it fails in such a way as
to suspect that it is dead (i.e. no output)? Maybe limit this to read
commands.

A way to let higher-level code avoid certain `file-exists-p' calls: add a
dynamic variable that tells TRAMP to skip certain commands if the result is
not available from a cache. Something similar to
`process-file-side-effects'. Calling code could then do sth. like this:

    (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown
,file))))
            (file-exists-p file))  ; TRAMP will return t or nil if it knows
or 'unknown if not cached; for local files there is no effect
      ...)

Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED
ARGUMENT...). Any element of the list with unknown function name etc. would
be simply ignored.

Code that doesn't let-bind this variable will behave as before. Code that
cares can be optimized.

Paul

On Sat, 2 Jul 2022 at 17:58, Michael Albinus <michael.albinus@HIDDEN> wrote:

> Paul Pogonyshev <pogonyshev@HIDDEN> writes:
>
> Hi Paul,
>
> > 1) check if connection is alive (`echo are you awake');
> > 2) test if the file exists;
> > 3) creating a temporary file for the chunk to be inserted; I guess it
> > tries until it finds an unused filename, e.g. here it seems to be done
> > after `test -e /tmp/tramp.OD3cCu', which doesn't exist;
> > 4) 'touch' on the temporary file, presumably to create it;
> > 5) 'chmod' on the temporary, presumably so that other users cannot
> > read it;
> > 6) copying the requested chunk from the full file into the temporary
> > (using `dd');
> > 7) finding the real name of the temporary with `readlink';
> > 8) finding attributes of the temporary with `stat';
> > 9) gzipping the temporary for transmition from the remote to the local
> > machine;
> > 10) testing if the temporary is a directory (WTF?);
> > 11) removing the temporary.
> >
> > I guess it should be obvious that this is a bit too much for one
> > `insert-file-contents' call.
>
> In general, I agree. However, some of the commands are caused by
> primitive file operations, like file-exists-p. Tramp cannot know what
> will be the next call, and it doesn't have all the opportunities to
> optimize, compared with the overall picture you see in the eleven steps.
>
> > Suggested improvements:
> >
> > * TRAMP should issue just one `stat' command to find out most of the
> > things about a file: whether it exists, if it is a directory, its real
> > name when dereferencing links and whatever stats it is used to find
> > now; from `$ stat --help' this seems to be possible. In other words,
> > TRAMP shouldn't use simple commands like `test -e': any ping, even
> > nominal, will negate any gains from using a tad faster command.
> > Instead, if it needs to find anything about a file, it should ask the
> > remote about as many things as possible in one go: it is very likely
> > that the additional information will be needed soon and even if not,
> > this is basically free compared to ping anyway.
>
> Not all remote hosts carry a stat command, and not all existing stat's
> are GNU compatible. But yes, if possible, Tramp shall gather as much
> information in one run, and cache the results for further use.
>
> I will see what could be done. Will come back with a proposal next days
> (note that this will be for Emacs 29, ie git master).
>
> > * TRAMP code should prefer the approach "try do something and handle
> > resulting errors" where possible. For example, don't check if the file
> > exists, try to read it right away and handle failures properly. Code
> > like `(when (file-exists-p ...) do-something)' adds an unnecessary
> > command call and creates a racing condition anyway. Also, error-free
> > requests should be more frequent, so they should be the main
> > optimization goal. I'm not sure if it is applicable to TRAMP itself
> > and doesn't come from a higher level, though.
>
> Indeed, this is not Tramp's responsibility. Tramp is a stupid
> library. If there is a call for file-exists-p, it must return the
> answer. It doesn't know what will be the next request. So I'm rather
> pesimistic that Tramp can improve here.
>
> Best regards, Michael.
>

--00000000000050098e05e2d67ade
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Some more thoughts. Why does it even need `echo are you aw=
ake&#39;? It&#39;s a network connection, it can still fail even if it worke=
d fine 1 ms before when you checked. So, why not just let the first command=
 fail if the connection is dead and restart the connection if it fails in s=
uch a way as to suspect that it is dead (i.e. no output)? Maybe limit this =
to read commands.<div><br></div><div>A way to let higher-level code avoid c=
ertain `file-exists-p&#39; calls: add a dynamic variable that tells TRAMP t=
o skip certain commands if the result is not available from a cache. Someth=
ing similar to `process-file-side-effects&#39;. Calling code could then do =
sth. like this:</div><div><br></div><div>=C2=A0 =C2=A0 (when (let ((tramp-m=
ay-skip-if-not-cached `((file-exists-p unknown ,file))))</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (file-exists-p file))=C2=A0 ; TRAMP will=
 return t or nil if it knows or &#39;unknown if not cached; for local files=
 there is no effect</div><div>=C2=A0 =C2=A0 =C2=A0 ...)</div><div><br></div=
><div>Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED A=
RGUMENT...). Any element of the list with unknown function name etc. would =
be simply ignored.</div><div><br></div><div>Code that doesn&#39;t let-bind =
this variable will behave as before. Code that cares can be optimized.</div=
><div><br></div><div>Paul</div></div><br><div class=3D"gmail_quote"><div di=
r=3D"ltr" class=3D"gmail_attr">On Sat, 2 Jul 2022 at 17:58, Michael Albinus=
 &lt;<a href=3D"mailto:michael.albinus@HIDDEN" target=3D"_blank">michael.al=
binus@HIDDEN</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex">Paul Pogonyshev &lt;<a href=3D"mailto:pogonyshev@HIDDEN" tar=
get=3D"_blank">pogonyshev@HIDDEN</a>&gt; writes:<br>
<br>
Hi Paul,<br>
<br>
&gt; 1) check if connection is alive (`echo are you awake&#39;);<br>
&gt; 2) test if the file exists;<br>
&gt; 3) creating a temporary file for the chunk to be inserted; I guess it<=
br>
&gt; tries until it finds an unused filename, e.g. here it seems to be done=
<br>
&gt; after `test -e /tmp/tramp.OD3cCu&#39;, which doesn&#39;t exist;<br>
&gt; 4) &#39;touch&#39; on the temporary file, presumably to create it;<br>
&gt; 5) &#39;chmod&#39; on the temporary, presumably so that other users ca=
nnot<br>
&gt; read it;<br>
&gt; 6) copying the requested chunk from the full file into the temporary<b=
r>
&gt; (using `dd&#39;);<br>
&gt; 7) finding the real name of the temporary with `readlink&#39;;<br>
&gt; 8) finding attributes of the temporary with `stat&#39;;<br>
&gt; 9) gzipping the temporary for transmition from the remote to the local=
<br>
&gt; machine;<br>
&gt; 10) testing if the temporary is a directory (WTF?);<br>
&gt; 11) removing the temporary.<br>
&gt;<br>
&gt; I guess it should be obvious that this is a bit too much for one<br>
&gt; `insert-file-contents&#39; call.<br>
<br>
In general, I agree. However, some of the commands are caused by<br>
primitive file operations, like file-exists-p. Tramp cannot know what<br>
will be the next call, and it doesn&#39;t have all the opportunities to<br>
optimize, compared with the overall picture you see in the eleven steps.<br=
>
<br>
&gt; Suggested improvements:<br>
&gt;<br>
&gt; * TRAMP should issue just one `stat&#39; command to find out most of t=
he<br>
&gt; things about a file: whether it exists, if it is a directory, its real=
<br>
&gt; name when dereferencing links and whatever stats it is used to find<br=
>
&gt; now; from `$ stat --help&#39; this seems to be possible. In other word=
s,<br>
&gt; TRAMP shouldn&#39;t use simple commands like `test -e&#39;: any ping, =
even<br>
&gt; nominal, will negate any gains from using a tad faster command.<br>
&gt; Instead, if it needs to find anything about a file, it should ask the<=
br>
&gt; remote about as many things as possible in one go: it is very likely<b=
r>
&gt; that the additional information will be needed soon and even if not,<b=
r>
&gt; this is basically free compared to ping anyway.<br>
<br>
Not all remote hosts carry a stat command, and not all existing stat&#39;s<=
br>
are GNU compatible. But yes, if possible, Tramp shall gather as much<br>
information in one run, and cache the results for further use.<br>
<br>
I will see what could be done. Will come back with a proposal next days<br>
(note that this will be for Emacs 29, ie git master).<br>
<br>
&gt; * TRAMP code should prefer the approach &quot;try do something and han=
dle<br>
&gt; resulting errors&quot; where possible. For example, don&#39;t check if=
 the file<br>
&gt; exists, try to read it right away and handle failures properly. Code<b=
r>
&gt; like `(when (file-exists-p ...) do-something)&#39; adds an unnecessary=
<br>
&gt; command call and creates a racing condition anyway. Also, error-free<b=
r>
&gt; requests should be more frequent, so they should be the main<br>
&gt; optimization goal. I&#39;m not sure if it is applicable to TRAMP itsel=
f<br>
&gt; and doesn&#39;t come from a higher level, though.<br>
<br>
Indeed, this is not Tramp&#39;s responsibility. Tramp is a stupid<br>
library. If there is a call for file-exists-p, it must return the<br>
answer. It doesn&#39;t know what will be the next request. So I&#39;m rathe=
r<br>
pesimistic that Tramp can improve here.<br>
<br>
Best regards, Michael.<br>
</blockquote></div>

--00000000000050098e05e2d67ade--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 03 Jul 2022 12:17:02 +0000
Resent-Message-ID: <handler.56342.B56342.165685059610771 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.165685059610771
          (code B ref 56342); Sun, 03 Jul 2022 12:17:02 +0000
Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 12:16:36 +0000
Received: from localhost ([127.0.0.1]:43487 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o7yWh-0002nI-To
	for submit <at> debbugs.gnu.org; Sun, 03 Jul 2022 08:16:36 -0400
Received: from mout.gmx.net ([212.227.15.18]:49201)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o7yWe-0002gE-Rw
 for 56342 <at> debbugs.gnu.org; Sun, 03 Jul 2022 08:16:33 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656850586;
 bh=Bq8goL2n1LU3p25fZjzxCenjliNxFIJnopPzgnaFLa8=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To;
 b=IbSOV6fBaXy9MwUhayzqavELHxVoTUFWjS65D14Mt3pmFCgFnO9i3rNxH40kCmAyr
 N19wAc6ceEXYOkG5N+AuQzGQUOL3VgcFtxI8/TkXgLRL8pgAaogVyeYPwxBDXHrnj1
 yYmpR/vZY01h/7ZK0DzJzyyrnbu9aJFvXlctMEtg=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([213.220.148.177]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mk0Ne-1njfGb3mnY-00kPpU; Sun, 03
 Jul 2022 14:16:26 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
Date: Sun, 03 Jul 2022 14:16:24 +0200
In-Reply-To: <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 (Paul Pogonyshev's message of "Sat, 2 Jul 2022 20:14:35 +0200")
Message-ID: <87v8sefl2f.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:fA846orykYZJmc49cCPmkSQuNSGii/X/NhHmuSWc1Ww8X/qRXPT
 LL1o1avDHZ0vLoT7y5istjnWj91Cy6C8yDqXX1qaParzH7pT+iu1EX3JOp0hvHsWLbf7nK8
 Dwyj8eYLiCSnvQ51KkCHSB2dYUz5D9ky9JZR4Q2d10CoHt5QT7L4SInBC6juLF/981uAuJs
 VpH8GUm9jvqHPzljwoozg==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:ovNbRz7LHho=:ALNJHY0u8kzTLOoEr9GL3y
 qepwazmOimzJMrTuOo7UhpK8DHMkoUCOBabDOUjCkLQx9uFDZBQi5zKhAYumKZS6y3K+FPN06
 bCj+5Ulyf7+qns16KEpExki4MP+iE4KfcP/RJ+8fYpu0QZjVNHQ+AGWX9UxBxZuqdKG487APg
 KvQZ4kNu4U5TZxNG74I99iAy2R3gUARVLFheohj9uTpmayRvDq4T1Pa0DdIIcZQQvRkKIGzpg
 7mR0wbNDDF2+OHL4qjAuWLdA2Ltvgy6xxaNPd2VXGAXTDDqyTl4vtI3imoiFrnQHck8XPENPW
 OCBHGDSXNqMmTYql+9xl2pyEnjFi2LNJrvM7UMMxQSrN7Y3I0YUs6fOC6jGmfUj3QkPfFO5ji
 viW2t2EgpPrCvQdc4c8c/TdcImxxR9kJxB1Twd0dS7tY/17BhZoTybce6zH88sE5x38bWH+Na
 vkVgD6ZB+eM9Oa3JxmBkwQk+FGrD+kYrbdEVkf5IfJ+tf0P/5vqtc+AitYlB6puALye+O9Kvi
 8WfqnEPxoEmR8FeRSckDKlvtnoP6qv2D+V2aiP/NcFncWcroh8b5CvlURx8s7SPVwz5tSXgq6
 AkBOUdtA7Wv6Ui8DoKVfPfgf686m90rHdWDs5QFQJksZtA1d2TY9thCMB1QjS/w5QDCpEY3rM
 hob6xqadZiANZEJ+YB3gFhegkja+FoXUZufhiMNSP51r+KF9YWziUYhsZiG8b+kSEQoTP20Yd
 dHVR0+9csLH5BOnZDNpyEBMVkEbbHQLm3tP79fyXvL9Ewtx2CQ/plG5FVmSfE6RQ8oJpeSlkD
 UYZFY0JOcxKpsqKe1iUOdNn4xggm5qykzKaXQwBottoACLrriQ1P6MYrIelXRbLEeFW+sHZk8
 OjJXwzgKQt1iz3Qx0W371YXTd34Zjm0ObY3tpW7TcCJtvwfnLAzsCNaPTZoYl0YwfoH/KzgsH
 3RC5KVKyI2YSJatcc4BtcTstLKavmZ+Vyhfh5RE2N0iX27p18DFNtLEcDiyLG7E/LokHLaQoH
 S5evzV+tXDKZdRi0KnRWb4zBZ8NPs2vqmhfq4oOn3Y1w5XNxfi3xdFbQwFanNi4JDXPI+ENgP
 m/Y8PDicwGN0nUSIAf8kVQjuA8onKuTZQSijuFyvs/J5focZ87HRQBKow==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Paul Pogonyshev <pogonyshev@HIDDEN> writes:

Hi Paul,

> Some more thoughts. Why does it even need `echo are you awake'? It's a
> network connection, it can still fail even if it worked fine 1 ms
> before when you checked. So, why not just let the first command fail
> if the connection is dead and restart the connection if it fails in
> such a way as to suspect that it is dead (i.e. no output)? Maybe limit
> this to read commands.

This is a sanity check. It avoids to hang in a blocked connection,
because this special command is surrounded by a timeout of 10 sec. Other
commands w/o this protection could hang forever. See also the comment in
tramp-maybe-open-connection.

> A way to let higher-level code avoid certain `file-exists-p' calls:
> add a dynamic variable that tells TRAMP to skip certain commands if
> the result is not available from a cache. Something similar to
> `process-file-side-effects'. Calling code could then do sth. like
> this:
>
>     (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown
> ,file))))
>             (file-exists-p file))  ; TRAMP will return t or nil if it
> knows or 'unknown if not cached; for local files there is no effect
>       ...)
>
> Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED
> ARGUMENT...). Any element of the list with unknown function name etc.
> would be simply ignored.
>
> Code that doesn't let-bind this variable will behave as before. Code
> that cares can be optimized.

If a caller can live w/o a valid result of file-exists-p, it shouldn't
call it. Everything else is too sophisticated and good for trouble, I believe.

In general, packages shall not care what's the implementation of a given
function like file-exists-p. If they care, they could still use
file-remote-p in order to distinguish.

> Paul

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Paul Pogonyshev <pogonyshev@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 03 Jul 2022 14:02:02 +0000
Resent-Message-ID: <handler.56342.B56342.165685686410022 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.165685686410022
          (code B ref 56342); Sun, 03 Jul 2022 14:02:02 +0000
Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 14:01:04 +0000
Received: from localhost ([127.0.0.1]:45325 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o809o-0002bZ-9Y
	for submit <at> debbugs.gnu.org; Sun, 03 Jul 2022 10:01:04 -0400
Received: from mail-yb1-f171.google.com ([209.85.219.171]:46781)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pogonyshev@HIDDEN>) id 1o809i-0002at-Gl
 for 56342 <at> debbugs.gnu.org; Sun, 03 Jul 2022 10:01:02 -0400
Received: by mail-yb1-f171.google.com with SMTP id l11so12357139ybu.13
 for <56342 <at> debbugs.gnu.org>; Sun, 03 Jul 2022 07:00:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=8MQccKLGauc625dU0r+abtvxNPTltMfSoLHCk2kna38=;
 b=UgPa7HV/u+FHbRp0etIy+Q3z/61JybvBH9xCHhNDbJYCk/+cVCPQtV6NUfyag79WrQ
 s1JBM+48Fh57OAuSXPEEifRxpT9r4n57FLVJwbWWwlaWoVNX7WRLjACn18SQy/IJimx4
 WeDWSnxsVykTrbVQ1K8nDFTorx69/dvC2K3X6EUkTy8vK4LKpvGJcpu9knw2bSS8xs92
 v+crZLmaOQReP27BPsdtj7zDBu5zJPm3UlByWT/jCl0LPQ4Pk4U7Xq3gcyMfscPZ/Oup
 fHXSSDEdEkUgzlkKFPp1K4BncbCDKA/s7AVv4lwVuOYlTNEAStDJ9p10ITc7g1IviBjF
 SuVA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=8MQccKLGauc625dU0r+abtvxNPTltMfSoLHCk2kna38=;
 b=kI+G+g6u7uNYWt7dj3S6FNttHMyFIotduW8X78otrdN6R/GEs0VAy7HfBNWqjEiatc
 5f1LBnLhi4nFV30w25KPBWuPbPRYf63zgEqrRp7DyFLbGh1W6pYrT+Z0Pmk/Ne0aU1bh
 pcG78YNDmFxPIGsqB43YETBSusRpjpql6DSWQc1KzXeAWoMp6m/BZ33WvKsnxTS+oS3O
 aXzUlwlS4F868HkPse+5u3UWCqBoNdxKhmQXZo5lqz7SiNzTzh4p4kFEzVLoxddRsdeu
 4aqVyReykkaQxxuM5BEl6VeIk0oey7B/xZwumMTUkiVWXdXti7VvUlGfI7YjL/BetJ1e
 u81w==
X-Gm-Message-State: AJIora8WKMNraJM/w0KJoq9JtjI1vktQcNw7hX47HcIV2MwDWQf3md48
 EBHiJMp6qBQNDNBhHJFsANAePYCgexSOI6fiPqqHNHmbSdpK
X-Google-Smtp-Source: AGRyM1uw/Ncn67MUAS3Z6Wo48buIOUELe8K+tSZZTiHBZnhc8l9Cjhe0gW5RErIIKuwQnMrtcRKGSFv3lS2GlOgahlo=
X-Received: by 2002:a25:870e:0:b0:66c:b173:f047 with SMTP id
 a14-20020a25870e000000b0066cb173f047mr25644243ybl.635.1656856852723; Sun, 03
 Jul 2022 07:00:52 -0700 (PDT)
MIME-Version: 1.0
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
In-Reply-To: <87v8sefl2f.fsf@HIDDEN>
From: Paul Pogonyshev <pogonyshev@HIDDEN>
Date: Sun, 3 Jul 2022 16:00:41 +0200
Message-ID: <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000267a0705e2e70c21"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000267a0705e2e70c21
Content-Type: text/plain; charset="UTF-8"

> This is a sanity check. It avoids to hang in a blocked connection,
> because this special command is surrounded by a timeout of 10 sec. Other
> commands w/o this protection could hang forever. See also the comment in
> tramp-maybe-open-connection.

Maybe you could then single out commands that are supposed to return result
nearly-instantly anyway, e.g. `stat' or `test' (though the latter I have
suggested to replace with `stat' before). If they fail to produce the
result in 10 seconds, the connection can be assumed dead, just as with
`echo are you awake'. Commands that involve transmitting (potentially)
large amount of data this or that way can work as now. So, you would
usually avoid one extra command (which with a high ping means sth. like 0.2
s, already pretty noticeable in UI) and achieve the same result.

Sanity checks are good. but not if they visibly slow down normal operation.

> If a caller can live w/o a valid result of file-exists-p, it shouldn't
> call it. Everything else is too sophisticated and good for trouble, I
believe.

It might be easier to convince the rest of Emacs developers to use an extra
variable to possibly skip `file-exists-p' on only remote files than to
remove the call altogether. Though in either case it is required that
`file-missing' error is handled properly which is, incidentally, easier to
test if the call to `file-exists-p' is dropped. A target here would be
`insert-file-contents' and whatever it calls, and this is a part of Emacs,
not some external library.

Yet another idea: removing temporary should be done asynchronously (I
haven't checked, maybe it's already the case, but likely not). The caller
doesn't really care about call result and even if it has succeeded. Again,
not sure if this is easy to achieve interface-wise, maybe it's from a
higher level. But I suspect it's something like `with-temp-file' and can be
optimized: one-time optimization are worth it even if that costs
readability.

Paul


On Sun, 3 Jul 2022 at 14:16, Michael Albinus <michael.albinus@HIDDEN> wrote:

> Paul Pogonyshev <pogonyshev@HIDDEN> writes:
>
> Hi Paul,
>
> > Some more thoughts. Why does it even need `echo are you awake'? It's a
> > network connection, it can still fail even if it worked fine 1 ms
> > before when you checked. So, why not just let the first command fail
> > if the connection is dead and restart the connection if it fails in
> > such a way as to suspect that it is dead (i.e. no output)? Maybe limit
> > this to read commands.
>
> This is a sanity check. It avoids to hang in a blocked connection,
> because this special command is surrounded by a timeout of 10 sec. Other
> commands w/o this protection could hang forever. See also the comment in
> tramp-maybe-open-connection.
>
> > A way to let higher-level code avoid certain `file-exists-p' calls:
> > add a dynamic variable that tells TRAMP to skip certain commands if
> > the result is not available from a cache. Something similar to
> > `process-file-side-effects'. Calling code could then do sth. like
> > this:
> >
> >     (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown
> > ,file))))
> >             (file-exists-p file))  ; TRAMP will return t or nil if it
> > knows or 'unknown if not cached; for local files there is no effect
> >       ...)
> >
> > Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED
> > ARGUMENT...). Any element of the list with unknown function name etc.
> > would be simply ignored.
> >
> > Code that doesn't let-bind this variable will behave as before. Code
> > that cares can be optimized.
>
> If a caller can live w/o a valid result of file-exists-p, it shouldn't
> call it. Everything else is too sophisticated and good for trouble, I
> believe.
>
> In general, packages shall not care what's the implementation of a given
> function like file-exists-p. If they care, they could still use
> file-remote-p in order to distinguish.
>
> > Paul
>
> Best regards, Michael.
>

--000000000000267a0705e2e70c21
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&gt; This is a sanity check. It avoids to hang in a blocke=
d connection,<br>&gt;=C2=A0because this special command is surrounded by a =
timeout of 10 sec. Other<br>&gt;=C2=A0commands w/o this protection could ha=
ng forever. See also the comment in<br>&gt;=C2=A0tramp-maybe-open-connectio=
n.<span class=3D"gmail-im" style=3D"color:rgb(80,0,80)"><br></span><div><br=
></div><div>Maybe you could then single out commands that are supposed to r=
eturn result nearly-instantly anyway, e.g. `stat&#39; or `test&#39; (though=
 the latter I have suggested to replace with `stat&#39; before). If they fa=
il to produce the result in 10 seconds, the connection can be assumed dead,=
 just as with `echo are you awake&#39;. Commands that involve transmitting =
(potentially) large amount of data this or that way can work as now. So, yo=
u would usually avoid one extra command (which with a high ping means sth. =
like 0.2 s, already pretty noticeable in UI) and achieve the same result.</=
div><div><br></div><div>Sanity checks are good. but not if they visibly slo=
w down normal operation.</div><div><br></div><div>&gt; If a caller can live=
 w/o a valid result of file-exists-p, it shouldn&#39;t</div>&gt; call it. E=
verything else is too sophisticated and good for trouble, I believe.<br><di=
v><br></div><div>It might be easier to convince the rest of Emacs developer=
s to use an extra variable to possibly skip `file-exists-p&#39; on only rem=
ote files than to remove the call altogether. Though in either case it is r=
equired that `file-missing&#39; error is handled properly which is, inciden=
tally, easier to test if the call to `file-exists-p&#39; is dropped. A targ=
et here would be `insert-file-contents&#39; and whatever it calls, and this=
 is a part of Emacs, not some external library.</div><div><br></div><div>Ye=
t another idea: removing temporary should be done asynchronously (I haven&#=
39;t checked, maybe it&#39;s already the case, but likely not). The caller =
doesn&#39;t really care about call result and even if it has succeeded. Aga=
in, not sure if this is easy to achieve interface-wise, maybe it&#39;s from=
 a higher level. But I suspect it&#39;s something like `with-temp-file&#39;=
 and can be optimized: one-time optimization are worth it even if that cost=
s readability.</div><div><br></div><div>Paul</div><div><br></div></div><br>=
<div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sun, 3 =
Jul 2022 at 14:16, Michael Albinus &lt;<a href=3D"mailto:michael.albinus@gm=
x.de">michael.albinus@HIDDEN</a>&gt; wrote:<br></div><blockquote class=3D"g=
mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204=
,204,204);padding-left:1ex">Paul Pogonyshev &lt;<a href=3D"mailto:pogonyshe=
v@HIDDEN" target=3D"_blank">pogonyshev@HIDDEN</a>&gt; writes:<br>
<br>
Hi Paul,<br>
<br>
&gt; Some more thoughts. Why does it even need `echo are you awake&#39;? It=
&#39;s a<br>
&gt; network connection, it can still fail even if it worked fine 1 ms<br>
&gt; before when you checked. So, why not just let the first command fail<b=
r>
&gt; if the connection is dead and restart the connection if it fails in<br=
>
&gt; such a way as to suspect that it is dead (i.e. no output)? Maybe limit=
<br>
&gt; this to read commands.<br>
<br>
This is a sanity check. It avoids to hang in a blocked connection,<br>
because this special command is surrounded by a timeout of 10 sec. Other<br=
>
commands w/o this protection could hang forever. See also the comment in<br=
>
tramp-maybe-open-connection.<br>
<br>
&gt; A way to let higher-level code avoid certain `file-exists-p&#39; calls=
:<br>
&gt; add a dynamic variable that tells TRAMP to skip certain commands if<br=
>
&gt; the result is not available from a cache. Something similar to<br>
&gt; `process-file-side-effects&#39;. Calling code could then do sth. like<=
br>
&gt; this:<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0(when (let ((tramp-may-skip-if-not-cached `((file-e=
xists-p unknown<br>
&gt; ,file))))<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(file-exists-p file))=
=C2=A0 ; TRAMP will return t or nil if it<br>
&gt; knows or &#39;unknown if not cached; for local files there is no effec=
t<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0...)<br>
&gt;<br>
&gt; Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED<br=
>
&gt; ARGUMENT...). Any element of the list with unknown function name etc.<=
br>
&gt; would be simply ignored.<br>
&gt;<br>
&gt; Code that doesn&#39;t let-bind this variable will behave as before. Co=
de<br>
&gt; that cares can be optimized.<br>
<br>
If a caller can live w/o a valid result of file-exists-p, it shouldn&#39;t<=
br>
call it. Everything else is too sophisticated and good for trouble, I belie=
ve.<br>
<br>
In general, packages shall not care what&#39;s the implementation of a give=
n<br>
function like file-exists-p. If they care, they could still use<br>
file-remote-p in order to distinguish.<br>
<br>
&gt; Paul<br>
<br>
Best regards, Michael.<br>
</blockquote></div>

--000000000000267a0705e2e70c21--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 03 Jul 2022 18:48:01 +0000
Resent-Message-ID: <handler.56342.B56342.16568740426982 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.16568740426982
          (code B ref 56342); Sun, 03 Jul 2022 18:48:01 +0000
Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 18:47:22 +0000
Received: from localhost ([127.0.0.1]:45510 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o84cs-0001oX-0n
	for submit <at> debbugs.gnu.org; Sun, 03 Jul 2022 14:47:22 -0400
Received: from mout.gmx.net ([212.227.15.18]:36119)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o84cp-0001oJ-Gk
 for 56342 <at> debbugs.gnu.org; Sun, 03 Jul 2022 14:47:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656874032;
 bh=avHaiaUxfjUxgDetkRUCbin0jIBb51PTeasHARgeyTQ=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To;
 b=S+JDwa4ssNjRTuhCM/1jgkTeTXOx87xGU8wN1tltfayAb/GaWnBiHAgwjqeef0+4E
 ljgaZcLqUQfr2JxXuY3PHjuNPbenVf7EYbXpC6Ct9id5PgkZ1ywPSOyIIT7IlcxRyw
 pJAQIvoRBTz1Rg0dNZ+2ro8JLFiAxU3+wjrJP+yM=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([213.220.148.177]) by mail.gmx.net (mrgmx005
 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N6KUd-1nWBi50iU5-016dK6; Sun, 03
 Jul 2022 20:47:12 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
 <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
Date: Sun, 03 Jul 2022 20:47:10 +0200
In-Reply-To: <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
 (Paul Pogonyshev's message of "Sun, 3 Jul 2022 16:00:41 +0200")
Message-ID: <874jzyc9u9.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:AXFntJXEX1BgdvCrzNfIfkOe393XcCFTQneHiG9OsRjupM0OQvb
 TSr/L3wKL4esZTb4vV8swdG/tnoGpe1Zmr3/cxXFUXXHrpTxAPKaieDIhXiFw/GqTK0Wm16
 yO8v6QS4b0szZbhd+pquri23pCr9QQBTFUNQFSXvG4JuEvKdJF75hGXkYWIWSHcXJ4c7RHd
 USvQmw4JBkPT8sjfMahWg==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:hN7A+WAmyrs=:Gf29JUJT7XqHPRFolhI0yW
 PP5SKT26g5NUGLy3un6yPfsNL350DIxGSB3E5Fcfr+sv/3JgJ1x/oyUJZgt7FGxjnzGiY5TgX
 ORVfXPn+kWr2odiCTsknZwTEJ63A3G2DJrQjPSVsDKw4MS6SkD7LzMCcgRNf9AAn3bpJqqCOQ
 6EKlXiPS4BfGyGL/YIsVRnWyG5OabPPQT34B9hl+BRCaRFLyKncZepu1kREgV8AAsAX5Tzz4R
 VWMcc4iGFe0QkA3QJm3krz9bT0DUDc5qINbUqzjP0IS5+tEtZ1njLAzwPxo2b6Q+sJzUQ11om
 mMda0lfQE2/oxFsbiGIox6CDHv/WSw4UfirsmN/+eXfRDcKJ7VNGMFqZNQ4aisGM5nyWheXsM
 AGzL/Io0tSJECSvcFND2sACKi4bsKZVO6SptJGom3emuk2s2jzGajhnaOq0bg11lgjJlwPDyQ
 eL8out5lpHK1Z1mMhdhTzOhkrP/tYpCakhqxP+CpVL7FOAmXwAA6i29odxQ3zQdkvqZG4Rb2k
 1lYf9OZCKHmzCepUgPVpj4y1/SrvMXbHgCoscQfSeLbinsyRQAvh4ZXelBdxjMwtShZJl3Izh
 aADwEcpiDPpw4fzxNUdynG/IVHAF6LpJ5+WTiW0rkQE0JT0DOQig2DxlUohIOnmSVeqSn5oTD
 zAH7HQG21FDRq23MboOOnjX9iJbRR143T/3+OomGRLElYipVoYpkdrEW+Bce2iuBPdTOeRp1U
 z68Ca/RcC0X6zR1j9H7FHEDV9F/wgAoo9XvuNNdVn7RQoBqCkbTRThnaR+BxADa7kI+I7aQPj
 Vkwryq4E5AMKHJLnOBtSg6uSDT+JNM2T2JQEcfi7EvQoB47NrE8UfgaSONnzxDAQFszAxAFr0
 cY3jVMTm/2emEyYaL3WJ0qHvOHhBAcUiEI44yFCIjqSoybtZBW96RZFZT9gvmD46ETOQkk7j9
 Q2Nn3DwLMlFaNNG9DmcuON6jHsV6vpahU1cqILcJCX7vEEaGV6diY/0rG3adGXvVmW0mMD62G
 C/l7lXFs6pE45hzpmkvQs9IgxCxW8JZFPBGoea6SBquV9SPvOw++rSP4exA4vGY0mikYxguel
 qhXVYIBJyDWHmf4FK/5FCCMcyMGH133iWdQlwwur72YgLt0xjL8VjdbEw==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Paul Pogonyshev <pogonyshev@HIDDEN> writes:

Hi Paul,

> Maybe you could then single out commands that are supposed to return
> result nearly-instantly anyway, e.g. `stat' or `test' (though the
> latter I have suggested to replace with `stat' before). If they fail
> to produce the result in 10 seconds, the connection can be assumed
> dead, just as with `echo are you awake'. Commands that involve
> transmitting (potentially) large amount of data this or that way can
> work as now. So, you would usually avoid one extra command (which with
> a high ping means sth. like 0.2 s, already pretty noticeable in UI)
> and achieve the same result.
>
> Sanity checks are good. but not if they visibly slow down normal
> operation.

I will see whether I can do something along these lines.

> It might be easier to convince the rest of Emacs developers to use an
> extra variable to possibly skip `file-exists-p' on only remote files
> than to remove the call altogether. Though in either case it is
> required that `file-missing' error is handled properly which is,
> incidentally, easier to test if the call to `file-exists-p' is
> dropped. A target here would be `insert-file-contents' and whatever it
> calls, and this is a part of Emacs, not some external library.

Tramp has an own implementation of insert-file-contents, called
tramp-handle-insert-file-contents. And indeed, file-exists-p is called
in order to raise the file-missing error if needed. I have no idea how
we could generate this otherwise. Perhaps changing the order: First try
to insert the file contents, and if this errs out, a call with
file-exists-p in order to raise the error. I'll play with this.

OTOH, with longer cache expiration time (see below), file-exists-p
doesn't cost a roundtrip.

> Yet another idea: removing temporary should be done asynchronously (I
> haven't checked, maybe it's already the case, but likely not). The
> caller doesn't really care about call result and even if it has
> succeeded. Again, not sure if this is easy to achieve interface-wise,
> maybe it's from a higher level. But I suspect it's something like
> `with-temp-file' and can be optimized: one-time optimization are worth
> it even if that costs readability.

Doing it asynchronously would require a second connection to the remote
host. Performance would rather degrade.

One thing you could do w/o code change is to change the value of
remote-file-name-inhibit-cache. Its default value is 10, meaning caches
expire already after 10 seconds. With your slow connection, a higher
value should help already.

> Paul

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Paul Pogonyshev <pogonyshev@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 03 Jul 2022 19:54:01 +0000
Resent-Message-ID: <handler.56342.B56342.165687799513208 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.165687799513208
          (code B ref 56342); Sun, 03 Jul 2022 19:54:01 +0000
Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 19:53:15 +0000
Received: from localhost ([127.0.0.1]:45521 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o85ec-0003Qy-BE
	for submit <at> debbugs.gnu.org; Sun, 03 Jul 2022 15:53:14 -0400
Received: from mail-yb1-f175.google.com ([209.85.219.175]:40624)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pogonyshev@HIDDEN>) id 1o85ea-0003Ql-S9
 for 56342 <at> debbugs.gnu.org; Sun, 03 Jul 2022 15:53:13 -0400
Received: by mail-yb1-f175.google.com with SMTP id o2so8160367yba.7
 for <56342 <at> debbugs.gnu.org>; Sun, 03 Jul 2022 12:53:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=Y6nTzgBZF6uyYLCi3FOGFGUC5Pr8bjYuApmIWbAbww8=;
 b=OIoVpAfy4ZXc+2nmPaj7dKMAvXsYJzgeGuUmyAC6uCuE1QO4F+hm4VOsGNEyT0f3Pt
 WntbOExra7rBz+qP0fzNKeIQZIxT9ceIv3Ig18zOzBa2NRlm/TvFZsi8rcisJ2C5KF69
 YJG+82MBKdwCkR1oxAZAKz4tm2hY8n7fndRzZb6cyuM/+i0ZUvQTdCJX6rTphPgoSOL0
 etzQ+hwQUIiODi5DD9W5B79EaN7zMAuuShcbuKesowC3TCZIuYAW2eXqp24jT7rNCaM3
 KEMkSgB4LAuxhVhd9YFr8afuhEKWugMiBacGW6AzgiRIRiJNmOo4R7g0uuvkbA+434o9
 CKvA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=Y6nTzgBZF6uyYLCi3FOGFGUC5Pr8bjYuApmIWbAbww8=;
 b=lgNsrIL2slheyDyWmt9HxfWcyCax/ZC0eqrcvDYjnLKzciypfDLSXkwqqdHMKn+RNz
 HvChPcyVIvRNH8KI2zrHhbKd1Ux+G0lIs6qrNYp3S1Z79YIcCih4LvwdVDaygVgbBRQ5
 Li51sMDiGDzJuDYXEM+q60wcgU3jr7gaTqgWIhtWFZ+gmHWO9RaZw/bNrK2d768gdaHI
 75FKSwKwFsPDB0J4SHw8Ksx3+Lj/bIPc3rsz3ou3nGNlmUHDK/Ocwu7BMDJjTW2Z9cW5
 9RF4xmlZgsyUrZc/opbiz6bp4yrCwGTmTlA+grP2iCRa7OMNXzqxgYDBlneOUa7f+C15
 ZOAQ==
X-Gm-Message-State: AJIora+ZCdaEK4HZVM+1fg5+emjuSbM/VjU5+/mHOxqKTEHAOE2adYEA
 /AzHOwXgBghOg8k+wz5f40YL8BT7JEjg2NluIw==
X-Google-Smtp-Source: AGRyM1sZlDcIci6yR9h2PtXpySS3lMgI+4YA2JrY0h4X8C667yrvwSp/Hsz0DZdhIHojNdAfuSWe8a1lbtopvVx5QfU=
X-Received: by 2002:a25:6ed5:0:b0:669:8b84:bb57 with SMTP id
 j204-20020a256ed5000000b006698b84bb57mr26798090ybc.227.1656877986971; Sun, 03
 Jul 2022 12:53:06 -0700 (PDT)
MIME-Version: 1.0
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
 <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
 <874jzyc9u9.fsf@HIDDEN>
In-Reply-To: <874jzyc9u9.fsf@HIDDEN>
From: Paul Pogonyshev <pogonyshev@HIDDEN>
Date: Sun, 3 Jul 2022 21:52:55 +0200
Message-ID: <CAG7BparKdinfykywLCf4av+foAifZXFBdPJA-044Xv0ZBTNV3w@HIDDEN>
Content-Type: multipart/alternative; boundary="000000000000d9855f05e2ebf70a"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--000000000000d9855f05e2ebf70a
Content-Type: text/plain; charset="UTF-8"

> I will see whether I can do something along these lines.

Thank you. I'm just spewing out ideas here, you know the code better and
what will and will not work.

> Perhaps changing the order: First try
> to insert the file contents, and if this errs out, a call with
> file-exists-p in order to raise the error. I'll play with this.

Yes, this would be a better way from my point of view. I don't care how
slowly (unless it is like 50 times slower) it fails, that is an unlikely
case. The most likely one, when the file exists, should be the benchmark.

> Doing it asynchronously would require a second connection to the remote
> host. Performance would rather degrade.

Maybe not really asynchronously, just let it return early, not waiting for
the result? I'm not sure how TRAMP receives responses, but can it just keep
executing commands sequentially, as now, but give control back to the
caller in case of commands where the result doesn't really matter (cleanup,
e.g. deleting a temporary file). Of course, this means that if an
"important" command is issued right away, it has to wait for the response
to the previous inessential command. But when such an inessential command
is the last in a batch, this waiting would be effectively merged with
Emacs' idling in the normal UI command loop, making things more responsive
for the user.

> One thing you could do w/o code change is to change the value of
> remote-file-name-inhibit-cache.

Thank you for the hint, will try.

Paul


On Sun, 3 Jul 2022 at 20:47, Michael Albinus <michael.albinus@HIDDEN> wrote:

> Paul Pogonyshev <pogonyshev@HIDDEN> writes:
>
> Hi Paul,
>
> > Maybe you could then single out commands that are supposed to return
> > result nearly-instantly anyway, e.g. `stat' or `test' (though the
> > latter I have suggested to replace with `stat' before). If they fail
> > to produce the result in 10 seconds, the connection can be assumed
> > dead, just as with `echo are you awake'. Commands that involve
> > transmitting (potentially) large amount of data this or that way can
> > work as now. So, you would usually avoid one extra command (which with
> > a high ping means sth. like 0.2 s, already pretty noticeable in UI)
> > and achieve the same result.
> >
> > Sanity checks are good. but not if they visibly slow down normal
> > operation.
>
> I will see whether I can do something along these lines.
>
> > It might be easier to convince the rest of Emacs developers to use an
> > extra variable to possibly skip `file-exists-p' on only remote files
> > than to remove the call altogether. Though in either case it is
> > required that `file-missing' error is handled properly which is,
> > incidentally, easier to test if the call to `file-exists-p' is
> > dropped. A target here would be `insert-file-contents' and whatever it
> > calls, and this is a part of Emacs, not some external library.
>
> Tramp has an own implementation of insert-file-contents, called
> tramp-handle-insert-file-contents. And indeed, file-exists-p is called
> in order to raise the file-missing error if needed. I have no idea how
> we could generate this otherwise. Perhaps changing the order: First try
> to insert the file contents, and if this errs out, a call with
> file-exists-p in order to raise the error. I'll play with this.
>
> OTOH, with longer cache expiration time (see below), file-exists-p
> doesn't cost a roundtrip.
>
> > Yet another idea: removing temporary should be done asynchronously (I
> > haven't checked, maybe it's already the case, but likely not). The
> > caller doesn't really care about call result and even if it has
> > succeeded. Again, not sure if this is easy to achieve interface-wise,
> > maybe it's from a higher level. But I suspect it's something like
> > `with-temp-file' and can be optimized: one-time optimization are worth
> > it even if that costs readability.
>
> Doing it asynchronously would require a second connection to the remote
> host. Performance would rather degrade.
>
> One thing you could do w/o code change is to change the value of
> remote-file-name-inhibit-cache. Its default value is 10, meaning caches
> expire already after 10 seconds. With your slow connection, a higher
> value should help already.
>
> > Paul
>
> Best regards, Michael.
>

--000000000000d9855f05e2ebf70a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&gt; I will see whether I can do something along these lin=
es.<br><div><br></div><div>Thank you. I&#39;m just spewing out ideas here, =
you know the code better and what will and will not work.</div><div><br></d=
iv><div>&gt; Perhaps changing the order: First try</div>&gt; to insert the =
file contents, and if this errs out, a call with<br>&gt; file-exists-p in o=
rder to raise the error. I&#39;ll play with this.<div><br></div><div>Yes, t=
his would be a better way from my point of view. I don&#39;t care how slowl=
y (unless it is like 50 times slower) it fails, that is an unlikely case. T=
he most likely one, when the file exists, should be the benchmark.</div><di=
v><br></div><div>&gt; Doing it asynchronously would require a second connec=
tion to the remote<br>&gt; host. Performance would rather degrade.</div><di=
v><br></div><div>Maybe not really asynchronously, just let it return early,=
 not waiting for the result? I&#39;m not sure how TRAMP receives responses,=
 but can it just keep executing commands sequentially, as now, but give con=
trol back to the caller in case of commands where the result doesn&#39;t re=
ally matter (cleanup, e.g. deleting a temporary file). Of course, this mean=
s that if an &quot;important&quot; command is issued right away, it has to =
wait for the response to the previous inessential command. But when such an=
 inessential command is the last in a batch, this waiting would be effectiv=
ely merged with Emacs&#39; idling in the normal UI command loop, making thi=
ngs more responsive for the user.<br><br>&gt; One thing you could do w/o co=
de change is to change the value of<br>&gt; remote-file-name-inhibit-cache.=
</div><div><br></div><div>Thank you for the hint, will try.</div><div><br><=
/div><div>Paul<br><div><br></div></div></div><br><div class=3D"gmail_quote"=
><div dir=3D"ltr" class=3D"gmail_attr">On Sun, 3 Jul 2022 at 20:47, Michael=
 Albinus &lt;<a href=3D"mailto:michael.albinus@HIDDEN">michael.albinus@gmx.=
de</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex=
">Paul Pogonyshev &lt;<a href=3D"mailto:pogonyshev@HIDDEN" target=3D"_bl=
ank">pogonyshev@HIDDEN</a>&gt; writes:<br>
<br>
Hi Paul,<br>
<br>
&gt; Maybe you could then single out commands that are supposed to return<b=
r>
&gt; result nearly-instantly anyway, e.g. `stat&#39; or `test&#39; (though =
the<br>
&gt; latter I have suggested to replace with `stat&#39; before). If they fa=
il<br>
&gt; to produce the result in 10 seconds, the connection can be assumed<br>
&gt; dead, just as with `echo are you awake&#39;. Commands that involve<br>
&gt; transmitting (potentially) large amount of data this or that way can<b=
r>
&gt; work as now. So, you would usually avoid one extra command (which with=
<br>
&gt; a high ping means sth. like 0.2 s, already pretty noticeable in UI)<br=
>
&gt; and achieve the same result.<br>
&gt;<br>
&gt; Sanity checks are good. but not if they visibly slow down normal<br>
&gt; operation.<br>
<br>
I will see whether I can do something along these lines.<br>
<br>
&gt; It might be easier to convince the rest of Emacs developers to use an<=
br>
&gt; extra variable to possibly skip `file-exists-p&#39; on only remote fil=
es<br>
&gt; than to remove the call altogether. Though in either case it is<br>
&gt; required that `file-missing&#39; error is handled properly which is,<b=
r>
&gt; incidentally, easier to test if the call to `file-exists-p&#39; is<br>
&gt; dropped. A target here would be `insert-file-contents&#39; and whateve=
r it<br>
&gt; calls, and this is a part of Emacs, not some external library.<br>
<br>
Tramp has an own implementation of insert-file-contents, called<br>
tramp-handle-insert-file-contents. And indeed, file-exists-p is called<br>
in order to raise the file-missing error if needed. I have no idea how<br>
we could generate this otherwise. Perhaps changing the order: First try<br>
to insert the file contents, and if this errs out, a call with<br>
file-exists-p in order to raise the error. I&#39;ll play with this.<br>
<br>
OTOH, with longer cache expiration time (see below), file-exists-p<br>
doesn&#39;t cost a roundtrip.<br>
<br>
&gt; Yet another idea: removing temporary should be done asynchronously (I<=
br>
&gt; haven&#39;t checked, maybe it&#39;s already the case, but likely not).=
 The<br>
&gt; caller doesn&#39;t really care about call result and even if it has<br=
>
&gt; succeeded. Again, not sure if this is easy to achieve interface-wise,<=
br>
&gt; maybe it&#39;s from a higher level. But I suspect it&#39;s something l=
ike<br>
&gt; `with-temp-file&#39; and can be optimized: one-time optimization are w=
orth<br>
&gt; it even if that costs readability.<br>
<br>
Doing it asynchronously would require a second connection to the remote<br>
host. Performance would rather degrade.<br>
<br>
One thing you could do w/o code change is to change the value of<br>
remote-file-name-inhibit-cache. Its default value is 10, meaning caches<br>
expire already after 10 seconds. With your slow connection, a higher<br>
value should help already.<br>
<br>
&gt; Paul<br>
<br>
Best regards, Michael.<br>
</blockquote></div>

--000000000000d9855f05e2ebf70a--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 04 Jul 2022 10:34:02 +0000
Resent-Message-ID: <handler.56342.B56342.16569308362722 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.16569308362722
          (code B ref 56342); Mon, 04 Jul 2022 10:34:02 +0000
Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 10:33:56 +0000
Received: from localhost ([127.0.0.1]:46179 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o8JOu-0000hp-2l
	for submit <at> debbugs.gnu.org; Mon, 04 Jul 2022 06:33:56 -0400
Received: from mout.gmx.net ([212.227.17.21]:44073)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o8JOn-0000hV-No
 for 56342 <at> debbugs.gnu.org; Mon, 04 Jul 2022 06:33:55 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656930823;
 bh=oHvJohFq3CH1KTNEH1WX4jkHgUcsxvAqLcrXPi6Amkc=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To;
 b=M8Ch6KF4agNgTvscoHx4C0miGP/xHA2pYmPmBRF+9RKJxu5R++AVYoxktgVGqHiJm
 COuIMBudUYfmlG2xclaR3cnAwY2YMiZs+S62eC0Jc7nUbkrDoo/oKzA6jlumSOB74u
 /BE4pnGMQJfO2t2SNT2Gvq3wsy0Tv0vZPfj64pHk=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx104
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1Mn2aN-1nidhB2C2B-00k8qV; Mon, 04
 Jul 2022 12:33:43 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
Date: Mon, 04 Jul 2022 12:33:39 +0200
In-Reply-To: <8735fjh5ge.fsf@HIDDEN> (Michael Albinus's message of "Sat, 02
 Jul 2022 17:58:25 +0200")
Message-ID: <87wnctb20s.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:6kO8ar5uvbON1/o84b/sOfWDqYIFvce4kN/1TBOaygJSzJSlphm
 YbpLeZUoNoyyLvba23pn7iiQq3MZe57OZZdl8nl2GbwmoWiBLNNbX6aTDDwdD/GAhgCXMFH
 HQC4g389TK9v2+kVaK6sITNOMZUWiPMP5yMWd50X8XewYNf/+IeJUyC9QQbNV5teID4RetU
 8PAx7N7xKQk/SiL6Mh6cw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:HzkWuYWpokc=:3vx0tzUnXWz2y5jSdA/8sH
 xBoE2IfIQvqPfHvHi5tXqeGN/z1Kh1Z7uub1a46izbFNxNxMzCyVRTc/WAIZdbARfmGZwTCvm
 I0GLTfvY0dezTgEMyAOCpkoOxTp/T58u3O/QhG9ABV2c5sCblSyCpXutg75620yUIdSF/F0PN
 jXzGJaMpHyvDkYmZz4k1JR4X5O5g3d5RBKcEUG6G+PlXYAPnByh3nX9uVSFLkpiAyai4Cbst/
 BoOvfBrpj2Vq/2jG7m7LsDEy3zXV5RHwiUO2e9NAlQhGbHRDNxQTetvu24jmizdziD/Z884WW
 UmcMoJHMOdbIwOqymgHokMvRvzf+Mml0+Nk5cjwZxJ2ySkLPFuP4lrRdJLPVglqmxKgIPLmFr
 8VKAlVyYyTUwi9fRZeiqvsDAASOb1rT0bocBizb6i6chnE86noB9Sgc4tx9CmPC+toHhmIDin
 LZYL4mZcT1qOHN4Z5rZKFVQ7sKvprI5d/fmDnzOOVrTiOJQw9gGGduTEt/+pzA8hS4bqCWQZM
 SOnxJvewF24YpIjo2pCuaogyAZirqrYBgC48vGAt8WNxC7pAF2NR0R1iBKkruyclyDGsN90dy
 fk+BRs2NAJvUtkFQ2eSQYgJROGxIujjJwNzL9EKa2U+S4hj+YoTqgds+RB+m1xpCXJ2MFWDBY
 umq7PCLpcftDVQB9aAoQlPe25B9uDvDkIn/q9+Fkds/rw1QqDV23C+3dnp+/r7Y1PYtxD4OdV
 bFEmH3DiGUUmXWgPYevxPbR37bIetJYEYUXdqoRar46oL/YZ7wD5bO0TrOC9RUN/toOf4KkwC
 ih7Um6BrH1Nj14Gbuvo7yy/uGSW0uX+3I5Z9puBNpAbj+7q/LuLXM8cKkJGPwHyFJzGuTlYsP
 w94QC3DTN5ZRyF/3tkDU/oML2mZKSQ0kr4M3z4iSjGGQAlDSWEE40Qa9r7PooslepvqOgJKkr
 l6Nb200yrSjafYebzIVQV7guv+MeSxdlLT65vH0rS+DfDQXFaxvyal8lMWGb/NT3u9xJcnPIH
 1vXvmf+M4XolAT2fKuQe+OGp1DmdE3XwYRQm6mE46Xn/pj/Vi+zngpHOtBNlHQQNgeY2Bu3dZ
 qvOIaQMS4LgThYLp8x2PT+bl6zrvldlRKsRq+HHloWT/o6Id65O2mlCxQ==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Michael Albinus <michael.albinus@HIDDEN> writes:

Hi Paul,

>> Suggested improvements:
>>
>> * TRAMP should issue just one `stat' command to find out most of the
>> things about a file: whether it exists, if it is a directory, its real
>> name when dereferencing links and whatever stats it is used to find
>> now; from `$ stat --help' this seems to be possible. In other words,
>> TRAMP shouldn't use simple commands like `test -e': any ping, even
>> nominal, will negate any gains from using a tad faster command.
>> Instead, if it needs to find anything about a file, it should ask the
>> remote about as many things as possible in one go: it is very likely
>> that the additional information will be needed soon and even if not,
>> this is basically free compared to ping anyway.
>
> Not all remote hosts carry a stat command, and not all existing stat's
> are GNU compatible. But yes, if possible, Tramp shall gather as much
> information in one run, and cache the results for further use.

FTR, stat is not sufficient to detect the real file name. Example:

# touch /tmp/a
# ln -s /tmp/a /tmp/b
# ln -s /tmp/b /tmp/c
# stat -c %N /tmp/c

It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must
still use "readlink --canonicalize".

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 04 Jul 2022 11:20:02 +0000
Resent-Message-ID: <handler.56342.B56342.165693359424722 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.165693359424722
          (code B ref 56342); Mon, 04 Jul 2022 11:20:02 +0000
Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 11:19:54 +0000
Received: from localhost ([127.0.0.1]:46308 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o8K7O-0006Qg-BP
	for submit <at> debbugs.gnu.org; Mon, 04 Jul 2022 07:19:54 -0400
Received: from mout.gmx.net ([212.227.17.21]:54269)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o8K7N-0006QS-14
 for 56342 <at> debbugs.gnu.org; Mon, 04 Jul 2022 07:19:53 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656933586;
 bh=h8vZtzxJxknb48sioZgkMRt+/8FaBbm7di2v8kCQUYQ=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To;
 b=CRSaYIjC/RAMxWLztBvzs1bLh23U2SSBW0ap5a+6q3uMvPq0ci54UqSD9zHnOVheJ
 WwMQVLBuamaap+/O1OHUeT2Ej4co0qIcMa3LADZywzIvENBEb8svokkdnxwnlHRZIF
 uxxHz+4/VSojPyaf7Iwgui/5szWL37irlrzPD0a8=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M26r3-1oAn5n0e1h-002VEG; Mon, 04
 Jul 2022 13:19:46 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
 <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
 <874jzyc9u9.fsf@HIDDEN>
 <CAG7BparKdinfykywLCf4av+foAifZXFBdPJA-044Xv0ZBTNV3w@HIDDEN>
Date: Mon, 04 Jul 2022 13:19:45 +0200
In-Reply-To: <CAG7BparKdinfykywLCf4av+foAifZXFBdPJA-044Xv0ZBTNV3w@HIDDEN>
 (Paul Pogonyshev's message of "Sun, 3 Jul 2022 21:52:55 +0200")
Message-ID: <87sfnhazvy.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:a048jtjG92FRil+jbLcLKcGHynb52eVKmf+Plm7GSvhccMZwJL8
 JGid2wv1fc2wCkzVxWwQSccDB3UW+dmTrbIYrQKexSRiBgrL+03r5UUEFG+n0VyBJHHtZYt
 kQGm0yf2TrZlG1cl5HSDVsLtpi9+84at3g2ydHSmj9PZSLRwaZGHrluEXLEkPXoHyDibPzx
 1AKVK4Oy53XQx6UiL1dJw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:h1Vx4CkIH2A=:PrGW5p5mfT8yVfqGZW6UCr
 17pVHaGG93PWmI2+KX89wAwlcXzJYYgsRA4ZWhrAW1WJWbXjdesfLZNPWd61lWl3/GRtXbUEa
 FCI1+RnKcG4J3zTnBih4R+gMiwCXTbG/pAHlhtq8XmOhaj5+/iv7TpUTPuNffq5naON3dCbEs
 g2Jfyt6mufgUulLDQMUKQHYxggd22wbcpiQFTfcUCnYZxPCXd9H6VT6njRh/xAXmB1y3erOZj
 cKtvuwlxW4p3ZiSsRmXdQ4jMUxiumaviAsspSEifeiSO3WesR1kYFSLdAF87EtFT7wZxAVwTG
 39FIIZzgOl4zpd5NQvc4vEx/PavtamqlFJT330rDukoE3wZLvPIsLvm2HV+FWUAoD9h4bmpHt
 IVA9Iu3ZryFUsBg9+Hqo62imgzpEQhpOJRn+Qi+tJZ8x3Eoye3XVUdZ0v6iAe6KRIBePDEDLd
 Pvd5TjKeduQRhUyKoaDhVzzsuJrh13I+u/peXDJcKAHRmV3r9KkAZw6tJhziXSnlXNd/PB9e6
 KaHAtDMszsl9ClZeeBxXBGvqmMvynwzSPeWVku2mINHkBzw2FHLWjMrhnTXd2qTDjSLMbhjmm
 EP/nHWAdAYgnbi7QGn+ynG4/QMVKdgYPo+MV0r4KFDpXyZbEUm+jYAr1cfMfx4Z4griimFr7Z
 El43jCOb3f9WfuXUK5qMK9trDC6alw28DrYN5F8crrCImHVXy50OHaXmM2ux7OLxPaJcOB2Nu
 /XIOar/TmxbqbWbfIovuwnonMcKNCQTPGt2Skz9yQXzXffqK2JBX4sgE0e3NMQyJxGs6G2gHf
 f++mzr5oOHecIh5azwrHKu1iUBR9+1eGqrQJcYnqA02sqfIj01+D9HEnUf19TfNXxw5JhPdFU
 n5CmE62EV1rTy6/VJTYAFVqgHXg0QSZhtjrwbOyr3D3sxilermj84E/BzjvQDllKdUO0Tzuga
 aqURjB32uAopA8VCN5DeopYKP9VQvuXwrpeLe6SXDXf/Pg2ND2KwOhFRnVNQyOxdFQ8Kk7tJo
 U+r+4TPGpTl5PRf3KsOeCJaxgrfD6iB78pUFhHF7DSGKKtr7ZDANdKCceX4u053pJHVjwt4gg
 vponUMXAYXgdT6Ds8jIOZ3DIYW8faVfVz3dGLdLg3DhL1Yu0nuJcWg+Bg==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Paul Pogonyshev <pogonyshev@HIDDEN> writes:

Hi Paul,

>> Doing it asynchronously would require a second connection to the remote
>> host. Performance would rather degrade.
>
> Maybe not really asynchronously, just let it return early, not waiting
> for the result? I'm not sure how TRAMP receives responses, but can it
> just keep executing commands sequentially, as now, but give control
> back to the caller in case of commands where the result doesn't really
> matter (cleanup, e.g. deleting a temporary file). Of course, this
> means that if an "important" command is issued right away, it has to
> wait for the response to the previous inessential command. But when
> such an inessential command is the last in a batch, this waiting would
> be effectively merged with Emacs' idling in the normal UI command
> loop, making things more responsive for the user.

Tramps communication with the remote host is like a REPL engine. It
sends shell commands to the remote hosts, reads the result, and waits
for the shell prompt. If it doesn't wait for the final shell prompt, it
is likely that the result or the shell prompt will be seen when reading
the output of the next command. This confuses. So no, I don't see a
chance to implement this kind of "asynchronity".

> Paul

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Paul Pogonyshev <pogonyshev@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 04 Jul 2022 14:43:01 +0000
Resent-Message-ID: <handler.56342.B56342.165694576723101 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.165694576723101
          (code B ref 56342); Mon, 04 Jul 2022 14:43:01 +0000
Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 14:42:47 +0000
Received: from localhost ([127.0.0.1]:48379 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o8NHj-00060U-2V
	for submit <at> debbugs.gnu.org; Mon, 04 Jul 2022 10:42:47 -0400
Received: from mail-yw1-f177.google.com ([209.85.128.177]:33289)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pogonyshev@HIDDEN>) id 1o8NHg-000605-1A
 for 56342 <at> debbugs.gnu.org; Mon, 04 Jul 2022 10:42:45 -0400
Received: by mail-yw1-f177.google.com with SMTP id
 00721157ae682-31c89111f23so33827777b3.0
 for <56342 <at> debbugs.gnu.org>; Mon, 04 Jul 2022 07:42:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=;
 b=aVLdYbFOrw/Mvt6pMTOFwKu2t/KdMW9zzbpfuhz50F8S/GRnAz46IkZEpKDQs+NMSt
 tW7UzegHDhwOJowEXPLlw1jvVQ9YgWpQY8V7vLtkpal7BqoBP4AHq+MMVn2p4h8bWMGf
 coTferDfS4ZcBk80jza+PQjGGu0Ialwl5WX1fdDJyRzd99oBRy8xNoUZB59ZQdCbEK2U
 617kv/iIh68LNH7GBZlmlt0jCcOton0Xcau8cbIq9/a0kU3qUZS3dwYf0u/Y/HLddwlK
 rDI87HggTP3XFYwRTz2FRhL3YQiZlSQ0vB1fW5Y2MjmsQcGCNiGx2BWS8NWxytdLv1Uq
 wdqQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=;
 b=UEK8ov/YQH35vhgO9AHdHfTCSEw/WdwqmWZbGeZ5bDcfQ7DTwFw3NLmHEfg+petPwc
 +lM5mAC2u7yI3FffNlcs+zz0y5bAnoqUMxSaQJUwiS/hHo3PX+jwyZ+KugJIJ5gB7ELG
 bj0ofpMWlc60Ouk1K5k9AoHHRjwjQPrRjRNY+BiJrILwLfdIQI/Ohts/PimBCCWSGOCE
 QECb/woHIacUaxtIhup20V+38uSv0Xop0rfiJUvGYWgHuMacCICDxNMSTcGa4NMfyIwN
 +qVF6fX66hkcvFMTpT38R5Mxwl6SmjtL9QMENmvH09nmJodQztxZ6wHVsBnfQkHS6qse
 LetQ==
X-Gm-Message-State: AJIora+QLFWRly+uz55ZmlpClm+++iXwvwJ6H6JyG0ieLe3qGoU/HMM6
 kLqJcqQk7tnFYSg472iuXRjWE2wdKbeptHCEKC97jeSxmA==
X-Google-Smtp-Source: AGRyM1sLDVjumx9IIZh7xBV0DKu+JO5nEBX5w6SveNf1SekWHkYzHsnwA4+yMFFZ7rg+bwdgLtCD4YD+ikYyK0Z7580=
X-Received: by 2002:a81:106:0:b0:2d0:e682:8a7a with SMTP id
 6-20020a810106000000b002d0e6828a7amr34241796ywb.257.1656945758306; Mon, 04
 Jul 2022 07:42:38 -0700 (PDT)
MIME-Version: 1.0
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
 <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
 <874jzyc9u9.fsf@HIDDEN>
 <CAG7BparKdinfykywLCf4av+foAifZXFBdPJA-044Xv0ZBTNV3w@HIDDEN>
 <87sfnhazvy.fsf@HIDDEN>
In-Reply-To: <87sfnhazvy.fsf@HIDDEN>
From: Paul Pogonyshev <pogonyshev@HIDDEN>
Date: Mon, 4 Jul 2022 16:42:26 +0200
Message-ID: <CAG7BpapO9Rnppfxgnng8S-kG__TB=XdAXTpkzrEuEVu8fC-A1w@HIDDEN>
Content-Type: multipart/alternative; boundary="00000000000056054c05e2fbbf5c"
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--00000000000056054c05e2fbbf5c
Content-Type: text/plain; charset="UTF-8"

> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must
> still use "readlink --canonicalize".

According to a quick search, it is possible to merge output of several
shell commands together. This seems to work even with dumb `sh', it's not a
Bash extension:

    $ sh -c '{ stat xxx && readlink xxx; }'

I guess TRAMP could just sth. similar, as I understand it doesn't have to
be strictly one executable call, just one command given to the remote shell.

> Tramps communication with the remote host is like a REPL engine. It
> sends shell commands to the remote hosts, reads the result, and waits
> for the shell prompt. If it doesn't wait for the final shell prompt, it
> is likely that the result or the shell prompt will be seen when reading
> the output of the next command. This confuses. So no, I don't see a
> chance to implement this kind of "asynchronity".

I see parameter `nooutput' to `tramp-send-command'. Couldn't that be used?

Even if not, I could imagine sth. like this:

    (defvar pending-commands nil)
    (defvar reading-output nil)

    (defun send-command (x output-inessential)
      (if output-inessential
          (setf pending-commands (nconc pending-commands (list x)))
        (while reading-output  ; make sure the connection is free for the
next essential command
          (read-next-output-chunk)
          (when (and (not reading-output) pending-commands)
            (do-send-command (pop pending-commands))))
        (do-send-command x)
        (read-output-now)))

    (defun do-send-command (x)
      (really-do-send-it x)
      (setf reading-output t))

    (defun read-output-now ()
      (while reading-output
        (read-next-output-chunk))
      (extract-received-output-from-process-buffer))

    (defun emacs-idling ()  ; hooked up using `run-with-idle-timer' or
something like that
      (cond (reading-output
             (read-next-output-chunk))
            (pending-commands
             (do-send-command (pop pending-commands)))))

    (defun read-next-output-chunk ()
      (when reading-output
        (do-read-output-chunk)  ; this should be non-blocking
        (when (end-of-command-output)
          (setf reading-output nil))))

Paul

On Mon, 4 Jul 2022 at 13:19, Michael Albinus <michael.albinus@HIDDEN> wrote:

> Paul Pogonyshev <pogonyshev@HIDDEN> writes:
>
> Hi Paul,
>
> >> Doing it asynchronously would require a second connection to the remote
> >> host. Performance would rather degrade.
> >
> > Maybe not really asynchronously, just let it return early, not waiting
> > for the result? I'm not sure how TRAMP receives responses, but can it
> > just keep executing commands sequentially, as now, but give control
> > back to the caller in case of commands where the result doesn't really
> > matter (cleanup, e.g. deleting a temporary file). Of course, this
> > means that if an "important" command is issued right away, it has to
> > wait for the response to the previous inessential command. But when
> > such an inessential command is the last in a batch, this waiting would
> > be effectively merged with Emacs' idling in the normal UI command
> > loop, making things more responsive for the user.
>
> Tramps communication with the remote host is like a REPL engine. It
> sends shell commands to the remote hosts, reads the result, and waits
> for the shell prompt. If it doesn't wait for the final shell prompt, it
> is likely that the result or the shell prompt will be seen when reading
> the output of the next command. This confuses. So no, I don't see a
> chance to implement this kind of "asynchronity".
>
> > Paul
>
> Best regards, Michael.
>

--00000000000056054c05e2fbbf5c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&gt; It returns &quot;&#39;/tmp/c&#39; -&gt; &#39;/tmp/b&#=
39;&quot;. However, we need &quot;/tmp/a&quot;. So we must<br>&gt; still us=
e &quot;readlink --canonicalize&quot;.<br><div><br></div><div>According to =
a quick search, it is possible to merge output of several shell commands to=
gether. This seems to work even with dumb `sh&#39;, it&#39;s not a Bash ext=
ension:</div><div><br></div><div>=C2=A0 =C2=A0 $ sh -c &#39;{ stat xxx &amp=
;&amp; readlink xxx; }&#39;<br></div><div><br></div><div>I guess TRAMP coul=
d just sth. similar, as I understand it doesn&#39;t have to be strictly one=
 executable call, just one command given to the remote shell.</div><div><br=
></div><div>&gt; Tramps communication with the remote host is like a REPL e=
ngine. It<br>&gt;=C2=A0sends shell commands to the remote hosts, reads the =
result, and waits<br>&gt;=C2=A0for the shell prompt. If it doesn&#39;t wait=
 for the final shell prompt, it<br>&gt;=C2=A0is likely that the result or t=
he shell prompt will be seen when reading<br>&gt;=C2=A0the output of the ne=
xt command. This confuses. So no, I don&#39;t see a<br>&gt;=C2=A0chance to =
implement this kind of &quot;asynchronity&quot;.<br></div><div><br></div><d=
iv>I see parameter `nooutput&#39; to `tramp-send-command&#39;. Couldn&#39;t=
 that be used?</div><div><br></div><div>Even if not, I could imagine sth. l=
ike this:</div><div><br></div><div>=C2=A0 =C2=A0 (defvar pending-commands n=
il)<br>=C2=A0 =C2=A0 (defvar reading-output nil)<br><br>=C2=A0 =C2=A0 (defu=
n send-command (x output-inessential)<br>=C2=A0 =C2=A0 =C2=A0 (if output-in=
essential<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf pending-commands (nco=
nc pending-commands (list x)))<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (while readin=
g-output =C2=A0; make sure the connection is free for the next essential co=
mmand<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-next-output-chunk)<br>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (and (not reading-output) pending-com=
mands)<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command (pop p=
ending-commands))))<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command x)<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-output-now)))<br><br>=C2=A0 =C2=A0 (defun=
 do-send-command (x)<br>=C2=A0 =C2=A0 =C2=A0 (really-do-send-it x)<br>=C2=
=A0 =C2=A0 =C2=A0 (setf reading-output t))<br><br>=C2=A0 =C2=A0 (defun read=
-output-now ()<br>=C2=A0 =C2=A0 =C2=A0 (while reading-output<br>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 (read-next-output-chunk))<br>=C2=A0 =C2=A0 =C2=A0 (extrac=
t-received-output-from-process-buffer))<br><br>=C2=A0 =C2=A0 (defun emacs-i=
dling () =C2=A0; hooked up using `run-with-idle-timer&#39; or something lik=
e that<br>=C2=A0 =C2=A0 =C2=A0 (cond (reading-output<br>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk))<br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (pending-commands<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0(do-send-command (pop pending-commands)))))<br><br>=
=C2=A0 =C2=A0 (defun read-next-output-chunk ()<br>=C2=A0 =C2=A0 =C2=A0 (whe=
n reading-output<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-read-output-chunk) =C2=
=A0; this should be non-blocking<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (end-=
of-command-output)<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf reading-outp=
ut nil))))</div><div><br></div><div>Paul</div></div><br><div class=3D"gmail=
_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, 4 Jul 2022 at 13:19, =
Michael Albinus &lt;<a href=3D"mailto:michael.albinus@HIDDEN" target=3D"_bl=
ank">michael.albinus@HIDDEN</a>&gt; wrote:<br></div><blockquote class=3D"gm=
ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,=
204,204);padding-left:1ex">Paul Pogonyshev &lt;<a href=3D"mailto:pogonyshev=
@gmail.com" target=3D"_blank">pogonyshev@HIDDEN</a>&gt; writes:<br>
<br>
Hi Paul,<br>
<br>
&gt;&gt; Doing it asynchronously would require a second connection to the r=
emote<br>
&gt;&gt; host. Performance would rather degrade.<br>
&gt;<br>
&gt; Maybe not really asynchronously, just let it return early, not waiting=
<br>
&gt; for the result? I&#39;m not sure how TRAMP receives responses, but can=
 it<br>
&gt; just keep executing commands sequentially, as now, but give control<br=
>
&gt; back to the caller in case of commands where the result doesn&#39;t re=
ally<br>
&gt; matter (cleanup, e.g. deleting a temporary file). Of course, this<br>
&gt; means that if an &quot;important&quot; command is issued right away, i=
t has to<br>
&gt; wait for the response to the previous inessential command. But when<br=
>
&gt; such an inessential command is the last in a batch, this waiting would=
<br>
&gt; be effectively merged with Emacs&#39; idling in the normal UI command<=
br>
&gt; loop, making things more responsive for the user.<br>
<br>
Tramps communication with the remote host is like a REPL engine. It<br>
sends shell commands to the remote hosts, reads the result, and waits<br>
for the shell prompt. If it doesn&#39;t wait for the final shell prompt, it=
<br>
is likely that the result or the shell prompt will be seen when reading<br>
the output of the next command. This confuses. So no, I don&#39;t see a<br>
chance to implement this kind of &quot;asynchronity&quot;.<br>
<br>
&gt; Paul<br>
<br>
Best regards, Michael.<br>
</blockquote></div>

--00000000000056054c05e2fbbf5c--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 04 Jul 2022 16:31:01 +0000
Resent-Message-ID: <handler.56342.B56342.16569522263376 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 56342
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Paul Pogonyshev <pogonyshev@HIDDEN>
Cc: 56342 <at> debbugs.gnu.org
Received: via spool by 56342-submit <at> debbugs.gnu.org id=B56342.16569522263376
          (code B ref 56342); Mon, 04 Jul 2022 16:31:01 +0000
Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 16:30:26 +0000
Received: from localhost ([127.0.0.1]:48553 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1o8Oxt-0000rm-Iu
	for submit <at> debbugs.gnu.org; Mon, 04 Jul 2022 12:30:26 -0400
Received: from mout.gmx.net ([212.227.17.22]:48375)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1o8Oxp-0000gC-35
 for 56342 <at> debbugs.gnu.org; Mon, 04 Jul 2022 12:30:23 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
 s=badeba3b8450; t=1656952214;
 bh=8saIDYDSHn71BE2qAjBSOZH16WAO72ZcpW6fT26qWPE=;
 h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To;
 b=G2e2fBNBOr0lGb0xoY6JT0jE6Z9MRX3PR1zXGZb2IT5KcKG6sTqeENOUfVKotEkok
 vPwEx7nKf2X1OHOucjLGpXMXXhiAeXZ/V2QmOswH0b2h+QPgiqEwPq3gSUyzQlj17+
 ehTkv6MytyRQeBknfCQY2pP0loi2f6W/cld9U3iA=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx105
 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MiJZO-1neGrw1bOw-00fOCq; Mon, 04
 Jul 2022 18:30:14 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <CAG7BpapEuJqV6HiBQJt=ABKc1uDYH2LpNmwkpSxpdNqcTks7gA@HIDDEN>
 <8735fjh5ge.fsf@HIDDEN>
 <CAG7BpapA6=kvxNX8siHSEyVHytRdYVA_S2W=gwHJvN2LjNkt2A@HIDDEN>
 <87v8sefl2f.fsf@HIDDEN>
 <CAG7Bpap0NmMvc+tfJ5v4K-XJrXv0s6MaRo_WgdBDi8hzytsoyg@HIDDEN>
 <874jzyc9u9.fsf@HIDDEN>
 <CAG7BparKdinfykywLCf4av+foAifZXFBdPJA-044Xv0ZBTNV3w@HIDDEN>
 <87sfnhazvy.fsf@HIDDEN>
 <CAG7BpapO9Rnppfxgnng8S-kG__TB=XdAXTpkzrEuEVu8fC-A1w@HIDDEN>
Date: Mon, 04 Jul 2022 18:30:12 +0200
In-Reply-To: <CAG7BpapO9Rnppfxgnng8S-kG__TB=XdAXTpkzrEuEVu8fC-A1w@HIDDEN>
 (Paul Pogonyshev's message of "Mon, 4 Jul 2022 16:42:26 +0200")
Message-ID: <87k08sc02z.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:dSQ9HtSdG2WGRcqeFpEQTABfATtY5LGBM3X23ZKTyrkUL8fyCQd
 N0hARi61W+PRuDIqy+/W5PxNTaoiNvDTJIjhhPWbhdp1QFrTrOLPcSz1qiDKUnivhF8fGfQ
 oORRhuemBVly3jHRR4EPP6JWbRhq2GpNalAx3S3Iz2/LgGGU8FyT0za6+XDmbG5V8IdbljY
 QQKSTayQrrkGaE5NGy3KQ==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:6cvVaUrf+jI=:omIe9eaw0N6lh8fXQZqZg+
 /M1x2LBCt64QygNxzEMQ2kxj8QhZ8maE7vLrphok2cixRyfGSULt8Dw6HwQFta3sKry5vJg84
 VObZTJ53z0SwoLDiogI+doAySbqRpJtJ2ZDaoyRjMvKY6PnkakaAJAG7ZVuAxMu2orRRRvMjh
 igmek8LSIXKFJ44mC6c4ydzYfpY73zFz7hVdBmhTtPLlkCtgcrxbpZ5r99FHU3LZjvJG9ZB+C
 9+dfZ7T57onHilV1QNLa/dp/L9FqhbIAEFU+/WZAF90SRbifcVqV9tY/TZ+yl3d+SlqSXKlyn
 DmYLtE4Q2flJXpIq7n5uJI9LwYlpjWdCY+wLEyS4YXeaUi6HSj+AC2NMUm/jXzJ9PpNJcdS0R
 vDLTMibHdR3MugzAhUSGUjSjomrTg3HI9C5zVr/Pt11bAfmM4SgDys4rCWhYoIQQc++B55LDC
 bTmvpPd9iaJ93e3B2qY3aPUX14UTeSYa+CGFsl0Ea4OfIJh7/wFFNeinHZ3Zn4eNlBDNMAtnK
 oKxd5CICv2RLfhQ6SM9jk503rau06xDQ/G9UBzuFTTsyY2SN5NKA0WBrUm2Rz1EJuQZ7K1idL
 4JY53hMxPWVpwmP/grQx92lupKv6wUG801E9j3zz1t7y/xsPTo6brmY+8ov20EIRS9M0sdexY
 Z8J0Gqf16WM7TCiIWtY/DvzzfrB0xdrFyZHHRIfUkR1R/fmRyqnNgq0bzcMVUk8FGbcMIz9y2
 8zGsreWA5XaHl0U0X1gA90CzBrLFwO8KcxWYH+wo+NdvUfM+u5HB0I7tSxS8xNl9BSiHvx7Om
 729w+Yaa0DNZeOfllzhPaqDjDxu5MlOH6KRtz5UlfL2Z8ePGqEA6IKGnP7CEUa1hRLgOwvRLv
 vqbZmE+y2JXD3Bab413LtlVDjyifva28IrwCad00i36yyEmcuteLNByPxwQQXZ5arXYS6eRft
 Ck3O7vM71M8pu43s5PWcYPXpKylTySujsXJq0jKr1lcylURDi098XJNiQ5GjetIVg38gsu77Z
 u4qs5dpXIW1vt0dVBf1ifPRdcMia6UOW3MXQzawfcPdgj1QQ6NVIdtU7DpMJrr0NQDHJub+w1
 1lxKA3kG7CCc4fJmz8iM5CnAmvqBfUxzCjcfHovNEGeUi7th9niEGDlmA==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Paul Pogonyshev <pogonyshev@HIDDEN> writes:

Hi Paul,

>> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must
>> still use "readlink --canonicalize".
>
> According to a quick search, it is possible to merge output of several
> shell commands together. This seems to work even with dumb `sh', it's
> not a Bash extension:
>
>     $ sh -c '{ stat xxx && readlink xxx; }'
>
> I guess TRAMP could just sth. similar, as I understand it doesn't have
> to be strictly one executable call, just one command given to the
> remote shell.

I'll check, perhaps it could be used in this special case.

But Tramp uses already combination of shell commands, and it uses even
private shell functions when applicable. Don't expect too much in
general.

>> Tramps communication with the remote host is like a REPL engine. It
>> sends shell commands to the remote hosts, reads the result, and waits
>> for the shell prompt. If it doesn't wait for the final shell prompt, it
>> is likely that the result or the shell prompt will be seen when reading
>> the output of the next command. This confuses. So no, I don't see a
>> chance to implement this kind of "asynchronity".
>
> I see parameter `nooutput' to `tramp-send-command'. Couldn't that be
> used?

No. This argument tells tramp-send-command not to call the final
accept-process-output. But this is only because on the caller side,
accept-process-output will be called with other arguments but the
default ones.

> Even if not, I could imagine sth. like this:
>
>     (defvar pending-commands nil)
>     (defvar reading-output nil)
>
>     (defun send-command (x output-inessential)
>       (if output-inessential
>           (setf pending-commands (nconc pending-commands (list x)))
>         (while reading-output  ; make sure the connection is free for
> the next essential command
>           (read-next-output-chunk)
>           (when (and (not reading-output) pending-commands)
>             (do-send-command (pop pending-commands))))
>         (do-send-command x)
>         (read-output-now)))
>
>     (defun do-send-command (x)
>       (really-do-send-it x)
>       (setf reading-output t))
>
>     (defun read-output-now ()
>       (while reading-output
>         (read-next-output-chunk))
>       (extract-received-output-from-process-buffer))
>
>     (defun emacs-idling ()  ; hooked up using `run-with-idle-timer' or
> something like that
>       (cond (reading-output
>              (read-next-output-chunk))
>             (pending-commands
>              (do-send-command (pop pending-commands)))))
>
>     (defun read-next-output-chunk ()
>       (when reading-output
>         (do-read-output-chunk)  ; this should be non-blocking
>         (when (end-of-command-output)
>           (setf reading-output nil))))

Something like this could be done, yes. Perhaps not with an (idle)
timer, because they are known to interrupt Tramp's REPL engine at any
point, with (sometimes) desastrous results. In general, one shall try to
avoid Tramp actions in timers, process sentinels and process filters.

So we could use this for deleting temporary files and alike, but there
won't be too many performance boosters I fear.

I will play with this idea, perhaps it helps. But for sure it will be an
opt-in feature only.

> Paul

Best regards, Michael.

Last modified: Mon, 4 Jul 2022 16:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.