GNU logs - #18626, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: stephen_leake@HIDDEN
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 04 Oct 2014 06:48:02 +0000
Resent-Message-ID: <handler.18626.B.14124052709363 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 18626 <at> debbugs.gnu.org
From: stephen_leake@HIDDEN
Date: Sat, 04 Oct 2014 01:47:16 -0500
Message-ID: <85wq8ge3gr.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

This may be related to http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18420

I've run into another problem with Emacs communication with external
processes; there seems to be a delay on every task switch. This shows up
both when sending data to the subprocess, and when reading from it.

When sending the contents of an Emacs buffer (source code to be parsed)
to the subprocess, if the read buffer in the subprocess is
large enough to hold the entire contents, no delay is apparent. However,
if the buffer is smaller, so that several reads are needed, then a delay
appears.

Similarly when sending data from the subprocess to Emacs.

To measure the delay:

(defun pipe-torture (exe &rest args)
  ""
  (let ((proc (apply 'start-process "pipe-reader" "pipe-read" exe args))
	start-time)

    (unless (process-live-p proc)
      (error "process died"))

    (setq start-time (time-to-seconds (current-time)))
    (process-send-region proc (point-min) (point-max))
    (message "send time %f" (- (time-to-seconds (current-time)) start-time))
    ))

(find-file "emacs-24.3.94/src/xdisp.c")
(pipe-torture "cat.exe")
    => 11.255173

The time varies slightly with each run. "cat.exe" here is from Cygwin.

For comparison, in a Cygwin bash shell on the same system:

$ cd /tmp
$ time cat < /Projects/emacs/emacs-24.3.94/src/xdisp.c > xdisp.c
real	0m0.051s
user	0m0.000s
sys	0m0.030s

Doing (setq process-adaptive-read-buffering nil) did not change the
timing.

To show the effect of changing the number of subprocess reads, here is a
trivial Ada program that reads from stdin into a buffer whose size is
set from the command line, and does no writes:

with Ada.Command_Line;
with GNAT.OS_Lib;
procedure Debug
is
   use GNAT.OS_Lib;

   Chunk : constant Integer := Integer'Value (Ada.Command_Line.Argument (1));

   File       : constant File_Descriptor := GNAT.OS_Lib.Standin;
   Read_Bytes : Integer;
   Text       : aliased String (1 .. Chunk);
begin
   loop
      Read_Bytes := Read (File, Text'Address, Chunk);
      exit when Read_Bytes < Chunk;
   end loop;
end Debug;


in xdisp.c:

(pipe-torture "debug.exe" "4096")
4.797574

(pipe-torture "debug.exe" "40000")
0.523758

The test with "debug.exe" can also be run in Emacs 24.3.1, since it does
not mix reads and writes (see bug 18420); it gives a time comparable to
the command-line "cat": 0.053000


To show the delay on reading subprocess output:

(defun pipe-torture-read (file)
  ""
  (let ((proc (apply 'start-process "pipe-reader" "pipe-read" (list "bash" "-c" (concat "cat < " file))))
	start-time)

    (unless (process-live-p proc)
      (error "process died"))

    (setq start-time (time-to-seconds (current-time)))
    (while (process-live-p proc)
      (accept-process-output proc))

    (message "read time %f" (- (time-to-seconds (current-time)) start-time))
    ))

In emacs 24.3.94:
(pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
"read time 11.965184"

In Windows native emacs 24.3.1:
(pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
"read time 11.992000"



In GNU Emacs 24.3.94.1 (x86_64-pc-mingw32)
 of 2014-10-04 on TAKVER
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --prefix=/mingw32'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: cp1252

Major mode: Text

Minor modes in effect:
  diff-auto-refine-mode: t
  display-time-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-s a d a <C-f12> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> M-x b u g - r e p o r t <return>
<M-S-left> <M-S-left> r e p o r <tab> <return>

Recent messages:
Loading c:/Projects/emacs_stephe.work/emacs_stephe/stephe_settings.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Loading c:/Projects/org.emacs.dvc.stephe/dvc-load.el (source)...done
Loading c:/Projects/emacs_stephe.work/emacs_stephe/dvc-patches.el (source)...
Loading c:/Projects/emacs_stephe.work/emacs_stephe/vc-mode-keys.el (source)...done
Loading c:/Projects/emacs_stephe.work/emacs_stephe/dvc-patches.el (source)...done
Package iswitchb is obsolete!
Mark saved where search started
Function xmtn-match--generate-branch is already compiled
Function xmtn-match--contains-match-variable-p is already compiled

Load-path shadows:
/Projects/org.emacs.ada-mode.stephe-1/ada-mode hides c:/Projects/emacs/emacs-24.3.94/lisp/progmodes/ada-mode

Features:
(shadow sort mail-extr emacsbug sendmail xmtn-dvc dvc-status dvc-diff
dvc-fileinfo diff-mode xmtn-conflicts xmtn-ids xmtn-match xmtn-automate
xmtn-compat parse-time xmtn-run xmtn-basic-io xmtn-base misearch
multi-isearch xgit-core bzr-core xmtn-minimal dvc-log log-edit message
format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils
mailheader pcvs-util vc vc-dispatcher ediff-merg ediff-wind ediff-diff
ediff-mult ediff-help ediff-init ediff-util ediff dvc-unified dvc-tips
ffap thingatpt url-parse auth-source eieio byte-opt bytecomp
byte-compile cconv eieio-core gnus-util mm-util help-fns mail-prsvr
password-cache url-vars add-log dvc-autoloads dvc-core dvc-lisp
dvc-buffers dvc-ui dvc-register dvc-utils dvc-emacs ewoc dvc-defs
dvc-site time delsel cus-start cus-load color-theme cl-macs edmacro
kmacro wid-edit cl gv cl-loaddefs cl-lib noutline outline easy-mmode
iswitchb whitespace dired-x dired-aux dired compile comint ansi-color
ring info easymenu bbdb-autoloads package epg-config time-date tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp w32-common-fns disp-table w32-win w32-vars tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 8 260868 6068)
 (symbols 32 28706 0)
 (miscs 32 46 139)
 (strings 16 49862 6082)
 (string-bytes 1 1215255)
 (vectors 8 19963)
 (vector-slots 4 495165 5152)
 (floats 8 177 242)
 (intervals 28 289 12)
 (buffers 512 13))

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.503 (Entity 5.503)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: stephen_leake@HIDDEN
Subject: bug#18626: Acknowledgement (24.3.94; communication with
 subprocess is slow)
Message-ID: <handler.18626.B.14124052709363.ack <at> debbugs.gnu.org>
References: <85wq8ge3gr.fsf@HIDDEN>
X-Gnu-PR-Message: ack 18626
X-Gnu-PR-Package: emacs
Reply-To: 18626 <at> debbugs.gnu.org
Date: Sat, 04 Oct 2014 06:48:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 18626 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
18626: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D18626
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 04 Oct 2014 08:08:02 +0000
Resent-Message-ID: <handler.18626.B18626.141241003717330 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: stephen_leake@HIDDEN
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Date: Sat, 04 Oct 2014 11:07:14 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85wq8ge3gr.fsf@HIDDEN>
Message-id: <83oatsgswd.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: stephen_leake@HIDDEN
> Date: Sat, 04 Oct 2014 01:47:16 -0500
> 
> (find-file "emacs-24.3.94/src/xdisp.c")
> (pipe-torture "cat.exe")
>     => 11.255173
> 
> The time varies slightly with each run. "cat.exe" here is from Cygwin.
> 
> For comparison, in a Cygwin bash shell on the same system:
> 
> $ cd /tmp
> $ time cat < /Projects/emacs/emacs-24.3.94/src/xdisp.c > xdisp.c
> real	0m0.051s
> user	0m0.000s
> sys	0m0.030s

Can you present similar timings from some Posix platform for
comparison?

> Doing (setq process-adaptive-read-buffering nil) did not change the
> timing.

AFAIK, this variable only matters for reading the subprocess output.
Since you didn't present the results for that, there's no evidence yet
which would support this conclusion.

> (pipe-torture "debug.exe" "4096")
> 4.797574
> 
> (pipe-torture "debug.exe" "40000")
> 0.523758
> 
> The test with "debug.exe" can also be run in Emacs 24.3.1, since it does
> not mix reads and writes (see bug 18420); it gives a time comparable to
> the command-line "cat": 0.053000

That's actually evidence that playing with the 2 parameters mentioned
on emacs-devel might make the time lower.

Still, the question is what happens on GNU/Linux or some other popular
Posix platform, in comparison with Windows.  I get ~1.2 sec on one
such machine, which is still a significant slowdown wrt file I/O
outside of Emacs, by a factor of about 50.

> In emacs 24.3.94:
> (pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
> "read time 11.965184"
> 
> In Windows native emacs 24.3.1:
> (pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
> "read time 11.992000"

That's identical, and expected: the changeset that solved the deadlock
case only changes the way Emacs treats the write end of the pipe.  The
read end behaves in Emacs 24.4 identically to what it did in 24.3.

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#18626: 24.3.94; communication with subprocess is slow
References: <85wq8ge3gr.fsf@HIDDEN>
In-Reply-To: <85wq8ge3gr.fsf@HIDDEN>
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 04 Oct 2014 08:12:02 +0000
Resent-Message-ID: <handler.18626.B18626.141241030017759 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 18626 <at> debbugs.gnu.org
From: Stephen Leake <stephen_leake@HIDDEN>
Date: Sat, 04 Oct 2014 03:11:26 -0500
Message-ID: <85d2a8dzkh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

As Eli suggested, I tried changing the pipe buffer size in w32.c pipe2,
and the delay in process.c send_process, call to
wait_reading_process_output.

The pipe buffer size should affect reading from the subprocess, but it
doesn't change the timing significantly:

(pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")

size                time
------------------------
0 (uses OS default) 11.965184
9192                12.012684
80000               12.018934

The send_process time delay is the cause of the slowdown in write:

(find-file "xdisp.c")
(pipe-torture "debug.exe" "4096")

delay               time
------------------------
20 * 1000 * 1000    4.450068
10 * 1000 * 1000    2.443788
 0 * 1000 * 1000    hangs until C-g

xdisp.c has 974233 bytes; with a subprocess read buffer of 4096 and a
delay of 0.020 seconds, that gives a send time of (* 0.020 (/ 974233
4096)) = 4.74, which is close to what we see above.

One way around the write delay is to use a large enough read buffer in
the subprocess. In my current real application, this is easy and also
gives other benefits.

I'm not clear why Emacs hangs with a delay of 0. The subprocess should
be running on a separate CPU core, so even if the Emacs process is in a
busy wait loop, the subprocess should read input.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 04 Oct 2014 08:33:02 +0000
Resent-Message-ID: <handler.18626.B18626.141241152319755 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141241152319755
          (code B ref 18626); Sat, 04 Oct 2014 08:33:02 +0000
Received: (at 18626) by debbugs.gnu.org; 4 Oct 2014 08:32:03 +0000
Received: from localhost ([127.0.0.1]:60737 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XaKl1-00058Z-5C
	for submit <at> debbugs.gnu.org; Sat, 04 Oct 2014 04:32:03 -0400
Received: from mtaout27.012.net.il ([80.179.55.183]:51044)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1XaKkx-000588-To
 for 18626 <at> debbugs.gnu.org; Sat, 04 Oct 2014 04:32:01 -0400
Received: from conversion-daemon.mtaout27.012.net.il by mtaout27.012.net.il
 (HyperSendmail v2007.08) id <0NCW00900VDTY200@HIDDEN> for
 18626 <at> debbugs.gnu.org; Sat, 04 Oct 2014 11:26:37 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by mtaout27.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0NCW00OFSVGCX680@HIDDEN>; Sat, 04 Oct 2014 11:26:37 +0300 (IDT)
Date: Sat, 04 Oct 2014 11:32:00 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85d2a8dzkh.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <83lhowgrr3.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN> <85d2a8dzkh.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Date: Sat, 04 Oct 2014 03:11:26 -0500
> 
> The send_process time delay is the cause of the slowdown in write:
> 
> (find-file "xdisp.c")
> (pipe-torture "debug.exe" "4096")
> 
> delay               time
> ------------------------
> 20 * 1000 * 1000    4.450068
> 10 * 1000 * 1000    2.443788
>  0 * 1000 * 1000    hangs until C-g
> 
> xdisp.c has 974233 bytes; with a subprocess read buffer of 4096 and a
> delay of 0.020 seconds, that gives a send time of (* 0.020 (/ 974233
> 4096)) = 4.74, which is close to what we see above.
> 
> One way around the write delay is to use a large enough read buffer in
> the subprocess. In my current real application, this is easy and also
> gives other benefits.
> 
> I'm not clear why Emacs hangs with a delay of 0. The subprocess should
> be running on a separate CPU core, so even if the Emacs process is in a
> busy wait loop, the subprocess should read input.

Did you try with 5 msec instead of zero?

Anyway, for 24.4 all I can suggest is a variable to replace the fixed
20 msec value.  Doing anything more complicated than that would be too
dangerous this late into the pretest.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stefan Monnier <monnier@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 04 Oct 2014 22:30:02 +0000
Resent-Message-ID: <handler.18626.B18626.141246178721629 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org, Stephen Leake <stephen_leake@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141246178721629
          (code B ref 18626); Sat, 04 Oct 2014 22:30:02 +0000
Received: (at 18626) by debbugs.gnu.org; 4 Oct 2014 22:29:47 +0000
Received: from localhost ([127.0.0.1]:33293 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XaXpj-0005cm-2H
	for submit <at> debbugs.gnu.org; Sat, 04 Oct 2014 18:29:47 -0400
Received: from ironport2-out.teksavvy.com ([206.248.154.181]:58661)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <monnier@HIDDEN>) id 1XaXpg-0005cb-CU
 for 18626 <at> debbugs.gnu.org; Sat, 04 Oct 2014 18:29:44 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ArYGAIDvNVNFxKjo/2dsb2JhbABZgwaDSr0vgw6BFxd0giUBAQEBAgFWIwULCzQSFBgNJIgECNIZF456B4Q4BKkZgWqDTCE
X-IPAS-Result: ArYGAIDvNVNFxKjo/2dsb2JhbABZgwaDSr0vgw6BFxd0giUBAQEBAgFWIwULCzQSFBgNJIgECNIZF456B4Q4BKkZgWqDTCE
X-IronPort-AV: E=Sophos;i="4.97,753,1389762000"; d="scan'208";a="91685632"
Received: from 69-196-168-232.dsl.teksavvy.com (HELO ceviche.home)
 ([69.196.168.232])
 by ironport2-out.teksavvy.com with ESMTP/TLS/DHE-RSA-AES256-SHA;
 04 Oct 2014 18:29:42 -0400
Received: by ceviche.home (Postfix, from userid 20848)
 id 17C7466094; Sat,  4 Oct 2014 18:29:42 -0400 (EDT)
From: Stefan Monnier <monnier@HIDDEN>
Message-ID: <jwvppe7wjt3.fsf-monnier+emacsbugs@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
Date: Sat, 04 Oct 2014 18:29:42 -0400
In-Reply-To: <83lhowgrr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 04 Oct
 2014 11:32:00 +0300")
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.3 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)

> Anyway, for 24.4 all I can suggest is a variable to replace the fixed
> 20 msec value.

But why is this needed in 24.4?  What has changed w.r.t 24.3?


        Stefan

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 02:47:02 +0000
Resent-Message-ID: <handler.18626.B18626.141247717414378 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stefan Monnier <monnier@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org, stephen_leake@HIDDEN
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141247717414378
          (code B ref 18626); Sun, 05 Oct 2014 02:47:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 02:46:14 +0000
Received: from localhost ([127.0.0.1]:33384 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xabpu-0003jq-0c
	for submit <at> debbugs.gnu.org; Sat, 04 Oct 2014 22:46:14 -0400
Received: from mtaout29.012.net.il ([80.179.55.185]:38515)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1Xabpq-0003jf-D4
 for 18626 <at> debbugs.gnu.org; Sat, 04 Oct 2014 22:46:12 -0400
Received: from conversion-daemon.mtaout29.012.net.il by mtaout29.012.net.il
 (HyperSendmail v2007.08) id <0NCY0070088ZY400@HIDDEN> for
 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 05:45:23 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by mtaout29.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0NCY00BQVABN5J20@HIDDEN>; Sun, 05 Oct 2014 05:45:23 +0300 (IDT)
Date: Sun, 05 Oct 2014 05:46:12 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <jwvppe7wjt3.fsf-monnier+emacsbugs@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <837g0fnsi3.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <jwvppe7wjt3.fsf-monnier+emacsbugs@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stefan Monnier <monnier@HIDDEN>
> Cc: Stephen Leake <stephen_leake@HIDDEN>,  18626 <at> debbugs.gnu.org
> Date: Sat, 04 Oct 2014 18:29:42 -0400
> 
> > Anyway, for 24.4 all I can suggest is a variable to replace the fixed
> > 20 msec value.
> 
> But why is this needed in 24.4?  What has changed w.r.t 24.3?

Unlike in 24.3, we now set the write end of the pipe to be in no-wait
mode, and use the EWOULDBLOCK machinery in send_process to send large
amounts of material down the pipe.  This evidently makes the writing
slower.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 13:21:02 +0000
Resent-Message-ID: <handler.18626.B18626.141251525614693 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141251525614693
          (code B ref 18626); Sun, 05 Oct 2014 13:21:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 13:20:56 +0000
Received: from localhost ([127.0.0.1]:33563 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xalk7-0003ou-Jj
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:20:56 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.230]:15564
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1Xalk1-0003oh-SO
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:20:53 -0400
Received: from [70.94.38.149] ([70.94.38.149:49533] helo=TAKVER)
 by dnvrco-oedge02 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 57/AF-08316-FA541345; Sun, 05 Oct 2014 13:20:48 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN> <83oatsgswd.fsf@HIDDEN>
Date: Sun, 05 Oct 2014 08:20:47 -0500
In-Reply-To: <83oatsgswd.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 04 Oct
 2014 11:07:14 +0300")
Message-ID: <858ukuejps.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.130:25
X-Authority-Analysis: v=2.1 cv=ReIeCjdv c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=wi5yxs08qe8tc_r_UQEA:9 a=ii61gXl28gQA:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

> Can you present similar timings from some Posix platform for
> comparison?

On Debian wheezy, with emacs 24.3.94:

$ cd /tmp
$ time cat < /home/Projects/emacs/emacs-24.3.94/src/xdisp.c > xdisp.c
real	0m0.007s
user	0m0.000s
sys	0m0.004s


(find-file "emacs-24.3.94/src/xdisp.c")
(pipe-torture "cat.exe")
1.556597

much faster than Windows, but still slow relative to plain file IO.

(pipe-torture "debug.exe" "4096")

This dies, because "debug.exe" exits early. It turns out that Emacs is
sending one line of the buffer at a time. I have not yet tried to figure
out why that doesn't happen on Windows.

(find-file "emacs-24.3.94/src/xdisp.c")
(pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
0.077341

much faster than Windows, 10 times slower than file IO.

>> Doing (setq process-adaptive-read-buffering nil) did not change the
>> timing.
>
> AFAIK, this variable only matters for reading the subprocess output.
> Since you didn't present the results for that, there's no evidence yet
> which would support this conclusion.

Right; process-adaptive-read-buffering should only affect
pipe-torture-read.

On Windows:

    (find-file "emacs-24.3.94/src/xdisp.c")
    (pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")

    process-adaptive-read-buffering t:
    11.997684

    process-adaptive-read-buffering nil:
    12.012685

process-adaptive-read-buffering has no effect.

Simililary, on Debian, process-adaptive-read-buffering has no effect.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 13:35:02 +0000
Resent-Message-ID: <handler.18626.B18626.141251607715911 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141251607715911
          (code B ref 18626); Sun, 05 Oct 2014 13:35:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 13:34:37 +0000
Received: from localhost ([127.0.0.1]:33572 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XalxM-00048Y-4h
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:34:36 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.231]:45990
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XalxH-00048O-JR
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:34:32 -0400
Received: from [70.94.38.149] ([70.94.38.149:49541] helo=TAKVER)
 by dnvrco-oedge02 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 42/C4-08316-6E841345; Sun, 05 Oct 2014 13:34:30 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
Date: Sun, 05 Oct 2014 08:34:30 -0500
In-Reply-To: <83lhowgrr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 04 Oct
 2014 11:32:00 +0300")
Message-ID: <854mviej2x.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.130:25
X-Authority-Analysis: v=2.1 cv=ReIeCjdv c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=fMqkbcfAKPJPCgNnFaoA:9 a=ii61gXl28gQA:10 a=FMD1XRJ_Ct4A:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Stephen Leake <stephen_leake@HIDDEN>
>> 
>> (find-file "xdisp.c")
>> (pipe-torture "debug.exe" "4096")
>> 
>> delay               time
>> ------------------------
>> 20 * 1000 * 1000    4.450068
>> 10 * 1000 * 1000    2.443788
>>  0 * 1000 * 1000    hangs until C-g
>> 
>
> Did you try with 5 msec instead of zero?

No, that seemed obvious :).

> Anyway, for 24.4 all I can suggest is a variable to replace the fixed
> 20 msec value.  Doing anything more complicated than that would be too
> dangerous this late into the pretest.

Right, and I have a good work-around for it (use a large enough buffer
in the subprocess).

But I'd like to understand the delay on reading from subprocesses.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 13:42:01 +0000
Resent-Message-ID: <handler.18626.B18626.141251647416557 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141251647416557
          (code B ref 18626); Sun, 05 Oct 2014 13:42:01 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 13:41:14 +0000
Received: from localhost ([127.0.0.1]:33576 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xam3l-0004Iy-AD
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:41:13 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.225]:14688
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1Xam3i-0004Ip-Bz
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 09:41:11 -0400
Received: from [70.94.38.149] ([70.94.38.149:49542] helo=TAKVER)
 by dnvrco-oedge02 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 9E/E7-08316-57A41345; Sun, 05 Oct 2014 13:41:09 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN> <83oatsgswd.fsf@HIDDEN>
 <858ukuejps.fsf@HIDDEN>
Date: Sun, 05 Oct 2014 08:41:08 -0500
In-Reply-To: <858ukuejps.fsf@HIDDEN> (Stephen Leake's message of
 "Sun, 05 Oct 2014 08:20:47 -0500")
Message-ID: <85zjdad47f.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.130:25
X-Authority-Analysis: v=2.1 cv=ReIeCjdv c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=uK4GQZ33S1GECPavXOwA:9 a=FMD1XRJ_Ct4A:10 a=ii61gXl28gQA:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Stephen Leake <stephen_leake@HIDDEN> writes:

> Eli Zaretskii <eliz@HIDDEN> writes:
>
>> Can you present similar timings from some Posix platform for
>> comparison?
>
> On Debian wheezy, with emacs 24.3.94:
>
> $ cd /tmp
> $ time cat < /home/Projects/emacs/emacs-24.3.94/src/xdisp.c > xdisp.c
> real	0m0.007s
> user	0m0.000s
> sys	0m0.004s
>
>
> (find-file "emacs-24.3.94/src/xdisp.c")
> (pipe-torture "cat.exe")
> 1.556597
>
> much faster than Windows, but still slow relative to plain file IO.
>
> (pipe-torture "debug.exe" "4096")
>
> This dies, because "debug.exe" exits early. It turns out that Emacs is
> sending one line of the buffer at a time. I have not yet tried to figure
> out why that doesn't happen on Windows.

I rewrote debug.adb to handle this case by exiting when the total bytes
read equals the file size:

with Ada.Command_Line;
with Ada.Text_IO; use Ada.Text_IO;
with GNAT.OS_Lib;
procedure Debug_Counted
is
   use GNAT.OS_Lib;

   Chunk     : constant Integer := Integer'Value (Ada.Command_Line.Argument (1));
   File_Size : constant Integer := Integer'Value (Ada.Command_Line.Argument (2));

   File             : constant File_Descriptor := GNAT.OS_Lib.Standin;
   Read_Bytes       : Integer;
   Total_Read_Bytes : Integer                  := 0;
   Text             : aliased String (1 .. Chunk);
   Short_Count      : Integer                  := 0;
   Count            : Integer                  := 0;
begin
   loop
      Read_Bytes := Read (File, Text'Address, Chunk);
      Total_Read_Bytes := Total_Read_Bytes + Read_Bytes;
      if Read_Bytes < Chunk then
         Short_Count := Short_Count + 1;
      else
         Count := Count + 1;
      end if;
      exit when Total_Read_Bytes >= File_Size;
   end loop;
   Put_Line ("Short_Count => " & Integer'Image (Short_Count) & " Count => " & Integer'Image (Count));
end Debug_Counted;

(find-file "xdisp.c")
(pipe-torture
"/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
"4096" "974230")

0.344027

This is comparable to the time on Windows with a large buffer in the
subprocess. So apparently it is not using the same delay in
send_process. Or the IO system is providing a large buffer.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 14:41:02 +0000
Resent-Message-ID: <handler.18626.B18626.141252001623193 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141252001623193
          (code B ref 18626); Sun, 05 Oct 2014 14:41:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 14:40:16 +0000
Received: from localhost ([127.0.0.1]:34821 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xamyt-00061z-Hv
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 10:40:15 -0400
Received: from mtaout21.012.net.il ([80.179.55.169]:34115)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1Xamym-00061n-U7
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 10:40:10 -0400
Received: from conversion-daemon.a-mtaout21.012.net.il by
 a-mtaout21.012.net.il (HyperSendmail v2007.08) id
 <0NCZ001007AF7U00@HIDDEN> for 18626 <at> debbugs.gnu.org;
 Sun, 05 Oct 2014 17:40:07 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by a-mtaout21.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0NCZ001GL7EU2WA0@HIDDEN>;
 Sun, 05 Oct 2014 17:40:07 +0300 (IDT)
Date: Sun, 05 Oct 2014 17:40:12 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <854mviej2x.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <8338b2oa0j.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Cc: 18626 <at> debbugs.gnu.org
> Date: Sun, 05 Oct 2014 08:34:30 -0500
> 
> But I'd like to understand the delay on reading from subprocesses.

Did you now?

in any case, reading from subprocesses was not changed in any way,
only writing to subprocesses was.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 14:44:02 +0000
Resent-Message-ID: <handler.18626.B18626.141252020723488 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141252020723488
          (code B ref 18626); Sun, 05 Oct 2014 14:44:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 14:43:27 +0000
Received: from localhost ([127.0.0.1]:34825 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xan1y-00066l-I9
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 10:43:27 -0400
Received: from mtaout20.012.net.il ([80.179.55.166]:33104)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1Xan1v-00066Y-C1
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 10:43:24 -0400
Received: from conversion-daemon.a-mtaout20.012.net.il by
 a-mtaout20.012.net.il (HyperSendmail v2007.08) id
 <0NCZ007007EC8300@HIDDEN> for 18626 <at> debbugs.gnu.org;
 Sun, 05 Oct 2014 17:43:21 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by a-mtaout20.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0NCZ007UK7K94B20@HIDDEN>;
 Sun, 05 Oct 2014 17:43:21 +0300 (IDT)
Date: Sun, 05 Oct 2014 17:43:27 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85zjdad47f.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <831tqmo9v4.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN> <83oatsgswd.fsf@HIDDEN>
 <858ukuejps.fsf@HIDDEN> <85zjdad47f.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Cc: 18626 <at> debbugs.gnu.org
> Date: Sun, 05 Oct 2014 08:41:08 -0500
> 
> (find-file "xdisp.c")
> (pipe-torture
> "/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
> "4096" "974230")
> 
> 0.344027
> 
> This is comparable to the time on Windows with a large buffer in the
> subprocess. So apparently it is not using the same delay in
> send_process. Or the IO system is providing a large buffer.

Evidently, subprocess I/O is more efficient on GNU/Linux than it is on
Windows.  Perhaps this is related to the fact that by default
GNU/Linux uses PTYs for that, not pipes.  Did you try your experiments
with process-connection-type bound to nil?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 18:20:01 +0000
Resent-Message-ID: <handler.18626.B18626.141253316515746 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141253316515746
          (code B ref 18626); Sun, 05 Oct 2014 18:20:01 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 18:19:25 +0000
Received: from localhost ([127.0.0.1]:34894 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XaqOy-00045u-Na
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 14:19:25 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.232]:31357
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XaqOu-00045i-V3
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 14:19:22 -0400
Received: from [70.94.38.149] ([70.94.38.149:49600] helo=TAKVER)
 by dnvrco-oedge03 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 11/DF-27958-6AB81345; Sun, 05 Oct 2014 18:19:19 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN> <83oatsgswd.fsf@HIDDEN>
 <858ukuejps.fsf@HIDDEN> <85zjdad47f.fsf@HIDDEN>
 <831tqmo9v4.fsf@HIDDEN>
Date: Sun, 05 Oct 2014 13:19:18 -0500
In-Reply-To: <831tqmo9v4.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 05 Oct
 2014 17:43:27 +0300")
Message-ID: <85vbnycrbt.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.142:25
X-Authority-Analysis: v=2.1 cv=L8aTQoj8 c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=GwgV5YMEs9_6KFwOCs8A:9 a=ii61gXl28gQA:10 a=FMD1XRJ_Ct4A:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Stephen Leake <stephen_leake@HIDDEN>
>> Cc: 18626 <at> debbugs.gnu.org
>> Date: Sun, 05 Oct 2014 08:41:08 -0500
>> 
>> (find-file "xdisp.c")
>> (pipe-torture
>> "/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
>> "4096" "974230")
>> 
>> 0.344027
>> 
>> This is comparable to the time on Windows with a large buffer in the
>> subprocess. So apparently it is not using the same delay in
>> send_process. Or the IO system is providing a large buffer.
>
> Evidently, subprocess I/O is more efficient on GNU/Linux than it is on
> Windows.  Perhaps this is related to the fact that by default
> GNU/Linux uses PTYs for that, not pipes.  Did you try your experiments
> with process-connection-type bound to nil?

On Debian:

(setq process-connection-type nil)

(pipe-torture-read "/home/Projects/emacs/emacs-24.3.94/src/xdisp.c")
0.074980

no signficant change

(find-file "xdisp.c")
(pipe-torture
"/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
"4096" "974230")

0.294485, 0.304046,  0.343767

(pipe-torture
"/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
"40000" "974230")
0.341241, 0.342165

Slower than pty, but much faster than Windows, and still independent of
subprocess read buffer size.


(setq process-connection-type nil) also eliminates the line-by-line
read; for a buffer size of 4096, Read returns 4096 until the end of the
text. For a buffer size of 40000, Read returns 40000 or 25536 until the
end of the text. (+ 40000 25536) = 65536, which makes sense for the
default pipe buffer size.

Given a send_process delay of 0.020 on EWOULD_BLOCK, and a buffer size
of 65536, the expected write time is (* 0.020 (/ 974230 65536)) = 0.28,
comparable to what we see.

Apparently pty's do the line by line send. Which makes sense for a
"psuedo-terminal". And they also apparently use a larger internal
buffer. 

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 18:26:02 +0000
Resent-Message-ID: <handler.18626.B18626.141253355716311 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141253355716311
          (code B ref 18626); Sun, 05 Oct 2014 18:26:02 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 18:25:57 +0000
Received: from localhost ([127.0.0.1]:34898 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XaqVI-0004F0-WB
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 14:25:57 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.231]:37155
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XaqVG-0004Er-Fg
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 14:25:55 -0400
Received: from [70.94.38.149] ([70.94.38.149:49601] helo=TAKVER)
 by dnvrco-oedge03 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 29/C2-27958-13D81345; Sun, 05 Oct 2014 18:25:53 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
Date: Sun, 05 Oct 2014 13:25:53 -0500
In-Reply-To: <8338b2oa0j.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 05 Oct
 2014 17:40:12 +0300")
Message-ID: <85r3ymcr0u.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.142:25
X-Authority-Analysis: v=2.1 cv=L8aTQoj8 c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=-tpc-rJyMVjctZf2x1kA:9 a=ii61gXl28gQA:10 a=FMD1XRJ_Ct4A:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Stephen Leake <stephen_leake@HIDDEN>
>> Cc: 18626 <at> debbugs.gnu.org
>> Date: Sun, 05 Oct 2014 08:34:30 -0500
>> 
>> But I'd like to understand the delay on reading from subprocesses.
>
> Did you now?

No.

> in any case, reading from subprocesses was not changed in any way,
> only writing to subprocesses was.

Yes. It is still a problem on Windows; 12 seconds to read a megabyte is
unacceptably slow for my real application.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 05 Oct 2014 19:05:01 +0000
Resent-Message-ID: <handler.18626.B18626.141253585319925 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141253585319925
          (code B ref 18626); Sun, 05 Oct 2014 19:05:01 +0000
Received: (at 18626) by debbugs.gnu.org; 5 Oct 2014 19:04:13 +0000
Received: from localhost ([127.0.0.1]:34906 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xar6H-0005BF-9z
	for submit <at> debbugs.gnu.org; Sun, 05 Oct 2014 15:04:13 -0400
Received: from mtaout24.012.net.il ([80.179.55.180]:45828)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1Xar6B-0005B4-C4
 for 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 15:04:07 -0400
Received: from conversion-daemon.mtaout24.012.net.il by mtaout24.012.net.il
 (HyperSendmail v2007.08) id <0NCZ00I00IPXQY00@HIDDEN> for
 18626 <at> debbugs.gnu.org; Sun, 05 Oct 2014 21:58:00 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by mtaout24.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0NCZ00H6NJCNXI30@HIDDEN>; Sun, 05 Oct 2014 21:58:00 +0300 (IDT)
Date: Sun, 05 Oct 2014 22:04:07 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85r3ymcr0u.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <83y4sumj88.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Cc: 18626 <at> debbugs.gnu.org
> Date: Sun, 05 Oct 2014 13:25:53 -0500
> 
> > in any case, reading from subprocesses was not changed in any way,
> > only writing to subprocesses was.
> 
> Yes. It is still a problem on Windows; 12 seconds to read a megabyte is
> unacceptably slow for my real application.

I can help you time the various parts of the machinery involved in
reading from the subprocesses on Windows, if you'd like to pursue this
further.

What about writing to subprocesses -- is that sufficiently fast if the
delay in send_process is decreased to 10 ms?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 08 Oct 2014 13:31:02 +0000
Resent-Message-ID: <handler.18626.B18626.141277502828481 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141277502828481
          (code B ref 18626); Wed, 08 Oct 2014 13:31:02 +0000
Received: (at 18626) by debbugs.gnu.org; 8 Oct 2014 13:30:28 +0000
Received: from localhost ([127.0.0.1]:37554 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XbrJl-0007Oy-LZ
	for submit <at> debbugs.gnu.org; Wed, 08 Oct 2014 09:30:27 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.225]:51312
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XbrJb-0007OP-NK
 for 18626 <at> debbugs.gnu.org; Wed, 08 Oct 2014 09:30:06 -0400
Received: from [70.94.38.149] ([70.94.38.149:50132] helo=TAKVER)
 by dnvrco-oedge02 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id CB/96-18752-95C35345; Wed, 08 Oct 2014 13:30:01 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN>
Date: Wed, 08 Oct 2014 08:29:59 -0500
In-Reply-To: <85r3ymcr0u.fsf@HIDDEN> (Stephen Leake's message of
 "Sun, 05 Oct 2014 13:25:53 -0500")
Message-ID: <85ppe2vgdk.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.130:25
X-Authority-Analysis: v=2.1 cv=G9mSErU5 c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=Sg0YA1n-PdpcZRv8HlEA:9
 a=FMD1XRJ_Ct4A:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Stephen Leake <stephen_leake@HIDDEN> writes:

> It is still a problem on Windows; 12 seconds to read a megabyte is
> unacceptably slow for my real application.

The cause for this is the Sleep in _sys_read_ahead in w32.c. This is
called from reader_thread after the wait for a single char read
succeeds. The default wait is 50 ms, set by the elisp variable
w32-pipe-read-delay. Setting that to 0 gives:

(pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
0.180000, 0.095000

only a factor of 2 slower than command-line cat.

w32-pipe-read-delay is a global variable. The comment in _sys_read_ahead
says there are problems with subprocess interaction when setting this to
0 on some systems. If that is still true, we may need to change this to
a per-subprocess setting.

In my real application, let-binding w32-pipe-read-delay for a short time
is possible, and a reasonable workaround.

--
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 08 Oct 2014 13:54:01 +0000
Resent-Message-ID: <handler.18626.B18626.141277638430528 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141277638430528
          (code B ref 18626); Wed, 08 Oct 2014 13:54:01 +0000
Received: (at 18626) by debbugs.gnu.org; 8 Oct 2014 13:53:04 +0000
Received: from localhost ([127.0.0.1]:37572 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1Xbrfr-0007wK-QE
	for submit <at> debbugs.gnu.org; Wed, 08 Oct 2014 09:53:04 -0400
Received: from mtaout20.012.net.il ([80.179.55.166]:41618)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1Xbrfp-0007vs-Mp
 for 18626 <at> debbugs.gnu.org; Wed, 08 Oct 2014 09:53:03 -0400
Received: from conversion-daemon.a-mtaout20.012.net.il by
 a-mtaout20.012.net.il (HyperSendmail v2007.08) id
 <0ND400L00OYUYV00@HIDDEN> for 18626 <at> debbugs.gnu.org;
 Wed, 08 Oct 2014 16:52:59 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by a-mtaout20.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0ND400MHNP8A2100@HIDDEN>;
 Wed, 08 Oct 2014 16:52:59 +0300 (IDT)
Date: Wed, 08 Oct 2014 16:53:12 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85ppe2vgdk.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <837g0ak6rb.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN> <85ppe2vgdk.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Cc: 18626 <at> debbugs.gnu.org
> Date: Wed, 08 Oct 2014 08:29:59 -0500
> 
> The cause for this is the Sleep in _sys_read_ahead in w32.c. This is
> called from reader_thread after the wait for a single char read
> succeeds. The default wait is 50 ms, set by the elisp variable
> w32-pipe-read-delay. Setting that to 0 gives:
> 
> (pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
> 0.180000, 0.095000
> 
> only a factor of 2 slower than command-line cat.
> 
> w32-pipe-read-delay is a global variable. The comment in _sys_read_ahead
> says there are problems with subprocess interaction when setting this to
> 0 on some systems. If that is still true, we may need to change this to
> a per-subprocess setting.
> 
> In my real application, let-binding w32-pipe-read-delay for a short time
> is possible, and a reasonable workaround.

I think I will set that to -1, on the trunk, except on Windows 9X
systems.  That should cause the reader thread yield its time slice
before returning.  Can you see how this affects your use case?

As for the effect of this on non-Windows 9X systems, I don't see the
problem described in the comment with the "dir" command on Windows XP
when I set this variable's value to zero.  And running DOS programs
from a w32 Emacs is much less probable use case these days, what with
the proliferation of 64-bit Windows systems that don't have a DOS
emulator.  So I think we can safely make this value zero or -1 on
modern systems.

Thanks for tracking this down.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 10 Oct 2014 08:08:02 +0000
Resent-Message-ID: <handler.18626.B18626.141292842930758 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.141292842930758
          (code B ref 18626); Fri, 10 Oct 2014 08:08:02 +0000
Received: (at 18626) by debbugs.gnu.org; 10 Oct 2014 08:07:09 +0000
Received: from localhost ([127.0.0.1]:39711 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XcVEC-0007zz-H3
	for submit <at> debbugs.gnu.org; Fri, 10 Oct 2014 04:07:08 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.227]:57138
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XcVE9-0007zp-76
 for 18626 <at> debbugs.gnu.org; Fri, 10 Oct 2014 04:07:07 -0400
Received: from [70.94.38.149] ([70.94.38.149:51190] helo=TAKVER)
 by dnvrco-oedge01 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id C6/2F-23176-7A397345; Fri, 10 Oct 2014 08:07:03 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN> <85ppe2vgdk.fsf@HIDDEN>
 <837g0ak6rb.fsf@HIDDEN>
Date: Fri, 10 Oct 2014 03:07:02 -0500
In-Reply-To: <837g0ak6rb.fsf@HIDDEN> (Eli Zaretskii's message of "Wed, 08 Oct
 2014 16:53:12 +0300")
Message-ID: <85lhoouz4p.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.118:25
X-Authority-Analysis: v=2.1 cv=d8V7yHTE c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=mDV3o1hIAAAA:8
 a=eTH9Wmw1qlWkzL1QRvYA:9 a=ii61gXl28gQA:10 a=FMD1XRJ_Ct4A:10
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: Stephen Leake <stephen_leake@HIDDEN>
>> Cc: 18626 <at> debbugs.gnu.org
>> Date: Wed, 08 Oct 2014 08:29:59 -0500
>> 
>> The cause for this is the Sleep in _sys_read_ahead in w32.c. This is
>> called from reader_thread after the wait for a single char read
>> succeeds. The default wait is 50 ms, set by the elisp variable
>> w32-pipe-read-delay. Setting that to 0 gives:
>> 
>> (pipe-torture-read "/Projects/emacs/emacs-24.3.94/src/xdisp.c")
>> 0.180000, 0.095000
>> 
>> only a factor of 2 slower than command-line cat.
>> 
>> w32-pipe-read-delay is a global variable. The comment in _sys_read_ahead
>> says there are problems with subprocess interaction when setting this to
>> 0 on some systems. If that is still true, we may need to change this to
>> a per-subprocess setting.
>> 
>> In my real application, let-binding w32-pipe-read-delay for a short time
>> is possible, and a reasonable workaround.
>
> I think I will set that to -1, on the trunk, except on Windows 9X
> systems.  That should cause the reader thread yield its time slice
> before returning.  Can you see how this affects your use case?

That has essentially no effect when running on a multi-core machine; the
subprocess is scheduled separately.

> As for the effect of this on non-Windows 9X systems, I don't see the
> problem described in the comment with the "dir" command on Windows XP
> when I set this variable's value to zero.  And running DOS programs
> from a w32 Emacs is much less probable use case these days, what with
> the proliferation of 64-bit Windows systems that don't have a DOS
> emulator.  So I think we can safely make this value zero or -1 on
> modern systems.

Ok.

> Thanks for tracking this down.

You're welcome. Thanks for your work on Emacs for Windows!

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Stephen Leake <stephen_leake@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 10 Oct 2014 08:48:02 +0000
Resent-Message-ID: <handler.18626.B18626.14129308402410 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.14129308402410
          (code B ref 18626); Fri, 10 Oct 2014 08:48:02 +0000
Received: (at 18626) by debbugs.gnu.org; 10 Oct 2014 08:47:20 +0000
Received: from localhost ([127.0.0.1]:39716 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XcVr5-0000co-4n
	for submit <at> debbugs.gnu.org; Fri, 10 Oct 2014 04:47:19 -0400
Received: from dnvrco-outbound-snat.email.rr.com ([107.14.73.226]:60243
 helo=dnvrco-oedge-vip.email.rr.com)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <stephen_leake@HIDDEN>) id 1XcVr0-0000cd-M2
 for 18626 <at> debbugs.gnu.org; Fri, 10 Oct 2014 04:47:16 -0400
Received: from [70.94.38.149] ([70.94.38.149:51224] helo=TAKVER)
 by dnvrco-oedge03 (envelope-from <stephen_leake@HIDDEN>)
 (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP
 id 06/0E-27958-11D97345; Fri, 10 Oct 2014 08:47:14 +0000
From: Stephen Leake <stephen_leake@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN>
Date: Fri, 10 Oct 2014 03:47:12 -0500
In-Reply-To: <85r3ymcr0u.fsf@HIDDEN> (Stephen Leake's message of
 "Sun, 05 Oct 2014 13:25:53 -0500")
Message-ID: <85h9zcux9r.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (windows-nt)
MIME-Version: 1.0
Content-Type: text/plain
X-RR-Connecting-IP: 107.14.64.142:25
X-Authority-Analysis: v=2.1 cv=L8aTQoj8 c=1 sm=1 tr=0
 a=AppmJ/7ZOOFWL/q6u6u93g==:117 a=AppmJ/7ZOOFWL/q6u6u93g==:17 a=ayC55rCoAAAA:8
 a=9XSUBuVRJI8A:10 a=o_R75loqY_IA:10 a=9i_RQKNPAAAA:8 a=fnrp1_RlzECi3FzGUUoA:9
X-Cloudmark-Score: 0
X-Spam-Score: 0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.7 (/)

There remains a problem on Linux; with either a pty or a pipe, there is
a default internal buffer of 2^16 bytes, which means we encounter the
send_process EWOULDBLOCK delay of 20 ms for every 2^16 bytes; that takes
0.3 seconds for 1 megabyte:

(find-file "xdisp.c")
(pipe-torture
"/home/Projects/org.emacs.ada-mode.stephe-1/build/wisi/debug_counted"
"974233" "974233")
send time 0.291322


The pipe is created by emacs_pipe in sysdep.c; there is no parameter for
the internal buffer size.

The pty is created by allocate_pty in process.c; I don't see a size
parameter there, but there are lots of macros.

However, changing the value for the nsecs parameter in the call to
wait_reading_process_output in send_process at process.c:4282 has no
effect, so the 20 ms delay is coming from something else.

-- 
-- Stephe

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#18626: 24.3.94; communication with subprocess is slow
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 10 Oct 2014 09:14:02 +0000
Resent-Message-ID: <handler.18626.B18626.14129324175009 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 18626
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Stephen Leake <stephen_leake@HIDDEN>
Cc: 18626 <at> debbugs.gnu.org
Reply-To: Eli Zaretskii <eliz@HIDDEN>
Received: via spool by 18626-submit <at> debbugs.gnu.org id=B18626.14129324175009
          (code B ref 18626); Fri, 10 Oct 2014 09:14:02 +0000
Received: (at 18626) by debbugs.gnu.org; 10 Oct 2014 09:13:37 +0000
Received: from localhost ([127.0.0.1]:39732 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.80)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1XcWGX-0001Ii-2X
	for submit <at> debbugs.gnu.org; Fri, 10 Oct 2014 05:13:37 -0400
Received: from mtaout22.012.net.il ([80.179.55.172]:60959)
 by debbugs.gnu.org with esmtp (Exim 4.80)
 (envelope-from <eliz@HIDDEN>) id 1XcWGR-0001IR-JW
 for 18626 <at> debbugs.gnu.org; Fri, 10 Oct 2014 05:13:35 -0400
Received: from conversion-daemon.a-mtaout22.012.net.il by
 a-mtaout22.012.net.il (HyperSendmail v2007.08) id
 <0ND800F001F6HU00@HIDDEN> for 18626 <at> debbugs.gnu.org;
 Fri, 10 Oct 2014 12:13:30 +0300 (IDT)
Received: from HOME-C4E4A596F7 ([87.69.4.28]) by a-mtaout22.012.net.il
 (HyperSendmail v2007.08) with ESMTPA id
 <0ND800F2B1MH0H70@HIDDEN>;
 Fri, 10 Oct 2014 12:13:30 +0300 (IDT)
Date: Fri, 10 Oct 2014 12:13:27 +0300
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <85lhoouz4p.fsf@HIDDEN>
X-012-Sender: halo1@HIDDEN
Message-id: <831tqgb83s.fsf@HIDDEN>
References: <85wq8ge3gr.fsf@HIDDEN>
 <85d2a8dzkh.fsf@HIDDEN> <83lhowgrr3.fsf@HIDDEN>
 <854mviej2x.fsf@HIDDEN> <8338b2oa0j.fsf@HIDDEN>
 <85r3ymcr0u.fsf@HIDDEN> <85ppe2vgdk.fsf@HIDDEN>
 <837g0ak6rb.fsf@HIDDEN> <85lhoouz4p.fsf@HIDDEN>
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <http://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <http://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <http://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 1.0 (+)

> From: Stephen Leake <stephen_leake@HIDDEN>
> Cc: 18626 <at> debbugs.gnu.org
> Date: Fri, 10 Oct 2014 03:07:02 -0500
> 
> > I think I will set that to -1, on the trunk, except on Windows 9X
> > systems.  That should cause the reader thread yield its time slice
> > before returning.  Can you see how this affects your use case?
> 
> That has essentially no effect when running on a multi-core machine; the
> subprocess is scheduled separately.

Well, yes, but the other cores could be busy, e.g., by "make -jN", or
even by other Emacs subprocesses running in parallel.  Also, don't
forget that Emacs itself has at least 3 active threads when even a
single subprocess is running: the main (a.k.a. "Lisp") thread, the
input thread which reads Windows messages, and the reader thread.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.