GNU bug report logs - #31862
26.1; [feature request] Improve copying over a network connection in dired

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 16 Jun 2018 18:23:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 16 14:23:41 2018
Received: from localhost ([127.0.0.1]:52001 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1fUFrR-0002Tm-3a
	for submit <at> debbugs.gnu.org; Sat, 16 Jun 2018 14:23:41 -0400
Received: from eggs.gnu.org ([208.118.235.92]:43252)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michelbach94@HIDDEN>) id 1fUFrP-0002TZ-6H
 for submit <at> debbugs.gnu.org; Sat, 16 Jun 2018 14:23:39 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <michelbach94@HIDDEN>) id 1fUFrJ-00016C-6z
 for submit <at> debbugs.gnu.org; Sat, 16 Jun 2018 14:23:34 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: *
X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_50,
 FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,T_DKIM_INVALID autolearn=disabled
 version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:59085)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <michelbach94@HIDDEN>)
 id 1fUFrJ-000164-2S
 for submit <at> debbugs.gnu.org; Sat, 16 Jun 2018 14:23:33 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:34223)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <michelbach94@HIDDEN>) id 1fUFrG-0007Oy-Ck
 for bug-gnu-emacs@HIDDEN; Sat, 16 Jun 2018 14:23:32 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <michelbach94@HIDDEN>) id 1fUFrF-00014s-H4
 for bug-gnu-emacs@HIDDEN; Sat, 16 Jun 2018 14:23:30 -0400
Received: from mail-wm0-x22c.google.com ([2a00:1450:400c:c09::22c]:38310)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
 (Exim 4.71) (envelope-from <michelbach94@HIDDEN>)
 id 1fUFrF-00014X-9f
 for bug-gnu-emacs@HIDDEN; Sat, 16 Jun 2018 14:23:29 -0400
Received: by mail-wm0-x22c.google.com with SMTP id 69-v6so8847999wmf.3
 for <bug-gnu-emacs@HIDDEN>; Sat, 16 Jun 2018 11:23:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=message-id:subject:from:to:date:mime-version
 :content-transfer-encoding;
 bh=BnsHjInBr+tdxqxuuz9uGycDw2UsA4xc+r53HvMeAT0=;
 b=KswPqBHBtWrS1L6bTgODYe7cXMEQJj0V0Ti3vt/SpDLLZieodQTlo7gtiifnMO3MtX
 CYxGF5HIejelHmgvZG0JS2WiNZFD1Mv0+dZ7vcPfQ4pQr3UmX6sofxMFe47TR5WJwGkr
 G9oVqmupHLWBwOOf4jx5HVHNeb/TKUFlsD4jqtzP78hIRlMWB6mGhNnsR33HM6d0lhCb
 QUEH8qMmh6L1lrPfcp5QzDXfBnG5gRlLZoiYD3Sxe9BV0I71OWPBrZ4svlceXHbg5A5E
 3EIH/uwEsf0MnMgrQntSu8VPSi7eLCvZovA9LvqlIdMovhwe8/7J3lXTpB1JxwFJzmLO
 euMQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:message-id:subject:from:to:date:mime-version
 :content-transfer-encoding;
 bh=BnsHjInBr+tdxqxuuz9uGycDw2UsA4xc+r53HvMeAT0=;
 b=GFv32Qbd7zAbTZOJVIIMkka9eQ42WDuLCTF+SozefUEp76cQjIhtkn9klGCj7OYnEG
 7a2pSYZVdREJcKm/PuxwoPoof+oZeXNj/CL+lPyzPiKvpNAu/cRfawK96VJdAIaV+VnJ
 Mvvoq5kiaiGQwNR3gi74aO13RqzJMcLzbZsDTH2bLg+DNIomeir9xmlURmY++nmKsDjs
 HhGFyYM5EEdkubeOk5voyThmw4rEcjmNewaJPYXtcBWDGv7NZ2mMiEk4vFIib/NN8CYg
 DZq3cnMZIuTGS8Iqhz9mz9/CRNN44Th17Tk2K1MlP+N4suBvVdTyuvcaGqnlcz781XUb
 qDRA==
X-Gm-Message-State: APt69E2+6v3k++car9KLmm0fVy8REnjeJDDFTqys4tOcYkPQLTONc5Qh
 3aQRqiRH53M83ClG6VxFI/492A==
X-Google-Smtp-Source: ADUXVKKmvjxFbFXXVLXGMVKLyYUI1ad0PTYZoqLTNGhbkSIN7bvEgsyE0QjQDKLcTm1+c4ibbR1VTw==
X-Received: by 2002:a1c:f45:: with SMTP id 66-v6mr4392953wmp.124.1529173407877; 
 Sat, 16 Jun 2018 11:23:27 -0700 (PDT)
Received: from ?IPv6:2a02:8071:21b0:3d00:197e:8105:a299:fab6?
 ([2a02:8071:21b0:3d00:197e:8105:a299:fab6])
 by smtp.googlemail.com with ESMTPSA id
 p38-v6sm24163004wrc.11.2018.06.16.11.23.26
 for <bug-gnu-emacs@HIDDEN>
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Sat, 16 Jun 2018 11:23:27 -0700 (PDT)
Message-ID: <1529173406.941.8.camel@HIDDEN>
Subject: 26.1; [feature request] Improve copying over a network connection
 in dired
From: Christoph Michelbach <michelbach94@HIDDEN>
To: bug-gnu-emacs@HIDDEN
Date: Sat, 16 Jun 2018 20:23:26 +0200
Content-Type: text/plain; charset="UTF-8"
X-Mailer: Evolution 3.18.5.2-0ubuntu3.2 
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -3.8 (---)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -4.8 (----)

I noticed that copying many files via dired takes a lot of time when it's done
over a network connection. This seems to be because every single file is
compressed and then transmitted in sequence.

To illustrate the problem: Create files on a server using these commands:

mkdir /tmp/dir
cd /tmp/dir
dd if=/dev/urandom bs=1K count=10000 of=bigFile
mkdir smallFiles
cd smallFiles
for i in {0..9999}; do dd if=/dev/urandom bs=1K count=1 of="smallFile""$i"; done

Then copy the files over to your local machine using dired.

Copying the big file took 5 seconds (including user interaction) via my network
connection. Copying the small files with the same amount of data took almost 20
minutes.

Can't you simply compress and transfer several small files at once (up to some
limit so the server doesn't run out of RAM)?

Of course copying small files takes longer than copying a big one of the same
size as the small ones combined but this disparity is just way out of
proportion.