GNU bug report logs - #32728
Emacs slow when reading process output

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>; merged with #32729; dated Thu, 13 Sep 2018 15:14:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 25 Oct 2019 07:00:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 25 03:00:48 2019
Received: from localhost ([127.0.0.1]:37195 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iNtaa-0003jy-HQ
	for submit <at> debbugs.gnu.org; Fri, 25 Oct 2019 03:00:48 -0400
Received: from eggs.gnu.org ([209.51.188.92]:57832)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iNtaY-0003je-HO; Fri, 25 Oct 2019 03:00:47 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:33049)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iNtaS-00011r-Vb; Fri, 25 Oct 2019 03:00:41 -0400
Received: from [176.228.60.248] (port=2992 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iNtaS-0003p6-33; Fri, 25 Oct 2019 03:00:40 -0400
Date: Fri, 25 Oct 2019 10:00:25 +0300
Message-Id: <83woctwbqu.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: "Benninghofen\, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
In-reply-to: <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN>
 (benjamin.benninghofen@HIDDEN)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>	<87y2xplufp.fsf@HIDDEN>
 <83r23hkqr3.fsf@HIDDEN>	<87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN>
 <87r23fiu66.fsf@HIDDEN>
 <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, larsi@HIDDEN, 32729 <at> debbugs.gnu.org,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
> CC: "layer@HIDDEN" <layer@HIDDEN>, "32729 <at> debbugs.gnu.org"
> 	<32729 <at> debbugs.gnu.org>, "32728 <at> debbugs.gnu.org" <32728 <at> debbugs.gnu.org>
> Date: Fri, 25 Oct 2019 06:38:16 +0000
> 
> For me your discussion about the performance problem appears very theoretical.

I don't think it was theoretical.  It was an attempt to identify which
factor(s) affect(s) this issue, so that further investigation could be
more focused.

> Why do you not just compare the relevant GNU Emacs Source Code with the Xemacs Source Code?

Did you try doing that?  Every time I did I found that the code is so
different that any comparison is very hard at best, and usually simply
meaningless.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 25 Oct 2019 06:38:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 25 02:38:28 2019
Received: from localhost ([127.0.0.1]:37185 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iNtEy-0003BJ-5j
	for submit <at> debbugs.gnu.org; Fri, 25 Oct 2019 02:38:28 -0400
Received: from mo1.myeers.net ([87.190.7.232]:37247)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1iNtEw-0003Az-Kl; Fri, 25 Oct 2019 02:38:27 -0400
X-IronPort-AV: E=Sophos;i="5.68,227,1569276000"; d="scan'208";a="108953666"
Received: from ec2-44-225-67-160.us-west-2.compute.amazonaws.com (HELO
 DE0-03HUB-P03.central.mail.corp) ([44.225.67.160])
 by de0-44iro-p02-out.myeers.net with ESMTP/TLS/AES256-SHA;
 25 Oct 2019 08:38:19 +0200
Received: from esa2e.demail.de.airbusds.corp (10.67.144.34) by
 DE0-03HUB-P03.central.mail.corp (44.225.67.164) with Microsoft SMTP Server id
 15.0.1473.3; Fri, 25 Oct 2019 08:38:17 +0200
Received: from unknown (HELO CD1-4DDAG02-P01.cdmail.common.airbusds.corp)
 ([10.67.164.142])
 by esa2i.demail.de.airbusds.corp with ESMTP; 25 Oct 2019 08:38:16 +0200
Received: from CD1-4BDAG02-P03.cdmail.common.airbusds.corp (10.67.164.139) by
 CD1-4DDAG02-P01.cdmail.common.airbusds.corp (10.67.164.142) with
 Microsoft
 SMTP Server (TLS) id 15.0.1473.3; Fri, 25 Oct 2019 08:38:16 +0200
Received: from CD1-4BDAG02-P03.cdmail.common.airbusds.corp ([10.67.164.139])
 by CD1-4BDAG02-P03.cdmail.common.airbusds.corp ([10.67.164.139]) with mapi id
 15.00.1473.003; Fri, 25 Oct 2019 08:38:16 +0200
From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Subject: RE: bug#32729: Xemacs 23 times as fast as GNU Emacs
Thread-Topic: bug#32729: Xemacs 23 times as fast as GNU Emacs
Thread-Index: AQHVgLFJdXJGPCQprkOKuUrCkgOMJKdWnzbCgACsFgKAAO+gGYAAoE9/gAAQkl+AAOzrSYARIZUQ
Date: Fri, 25 Oct 2019 06:38:16 +0000
Message-ID: <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN>
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>	<87y2xplufp.fsf@HIDDEN>
 <83r23hkqr3.fsf@HIDDEN>	<87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN>
 <87r23fiu66.fsf@HIDDEN>
In-Reply-To: <87r23fiu66.fsf@HIDDEN>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-ms-exchange-transport-fromentityheader: Hosted
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
X-GM-Security: forwarded
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: "layer@HIDDEN" <layer@HIDDEN>,
 "32729 <at> debbugs.gnu.org" <32729 <at> debbugs.gnu.org>,
 "32728 <at> debbugs.gnu.org" <32728 <at> debbugs.gnu.org>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

For me your discussion about the performance problem appears very theoretic=
al.
Why do you not just compare the relevant GNU Emacs Source Code with the Xem=
acs Source Code?

Benjamin Benninghofen

-----Original Message-----
From: Lars Ingebrigtsen [mailto:larsi@HIDDEN] =

Sent: Monday, October 14, 2019 10:54 AM
To: Eli Zaretskii
Cc: layer@HIDDEN; 32729 <at> debbugs.gnu.org; Benninghofen, Benjamin Dr.; 327=
28 <at> debbugs.gnu.org
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs

Actually, my benchmarking is somewhat wrong.

start-process with a filter, but discard output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
      (set-process-filter proc (lambda (proc string)))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=3D> (18.828236636 59 13.315468088000017)

filter, but insert the output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer (get-buffer " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=3D> (21.120281346 59 13.250166416000013)

With the default filter:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=3D> (34.046986424 116 26.025843717999976)

(!)

So the default filter is really slow?

Anyway, compare with call-process:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") n=
il "if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
=3D> (1.694743653 0 0.0)

So what makes start-process 10x slower than call-process?  If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.

-- =

(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

The information in this e-mail is confidential. The contents may not be dis=
closed or used by anyone other than the addressee. Access to this e-mail by=
 anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and=
 delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of=
 this e-mail as it has been sent over public networks. If you have any conc=
erns over the content of this message or its Accuracy or Integrity, please =
contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus =
scanning software but you should take whatever measures you deem to be appr=
opriate to ensure that this message and any attachments are virus free.





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 14 Oct 2019 10:18:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 06:18:49 2019
Received: from localhost ([127.0.0.1]:38186 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJxRA-0000YV-QF
	for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 06:18:49 -0400
Received: from eggs.gnu.org ([209.51.188.92]:59244)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJxR6-0000YA-2X; Mon, 14 Oct 2019 06:18:44 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:50206)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJxR0-0004u8-IA; Mon, 14 Oct 2019 06:18:38 -0400
Received: from [176.228.60.248] (port=3772 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJxQz-0007S4-Tt; Mon, 14 Oct 2019 06:18:38 -0400
Date: Mon, 14 Oct 2019 13:18:30 +0300
Message-Id: <83y2xniqa1.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87r23fiu66.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 14 Oct 2019 10:54:25 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
 <87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN> <87r23fiu66.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, benjamin.benninghofen@HIDDEN, 32729 <at> debbugs.gnu.org,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: layer@HIDDEN,  32729 <at> debbugs.gnu.org,
>   benjamin.benninghofen@HIDDEN,  32728 <at> debbugs.gnu.org
> Date: Mon, 14 Oct 2019 10:54:25 +0200
> 
> So what makes start-process 10x slower than call-process?

The fact that it goes through pselect, and only accepts the process
output when Emacs is idle?

> this obviously requires a more in-depth dive into process.c.

Agreed.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 14 Oct 2019 09:15:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 05:15:25 2019
Received: from localhost ([127.0.0.1]:38141 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJwRo-0007Tc-MJ
	for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 05:15:24 -0400
Received: from eggs.gnu.org ([209.51.188.92]:50760)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJwRm-0007TI-SG; Mon, 14 Oct 2019 05:15:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:49425)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJwRh-0001hX-28; Mon, 14 Oct 2019 05:15:17 -0400
Received: from [176.228.60.248] (port=2273 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJwRg-0006dy-CZ; Mon, 14 Oct 2019 05:15:16 -0400
Date: Mon, 14 Oct 2019 12:15:09 +0300
Message-Id: <8336fvk7s2.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87v9sriv0k.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 
 14 Oct 2019 10:36:11 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
 <87sgnwbl8w.fsf@HIDDEN> <83blujkaf8.fsf@HIDDEN>
 <87v9sriv0k.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: benjamin.benninghofen@HIDDEN,  layer@HIDDEN,
>   32729 <at> debbugs.gnu.org,  32728 <at> debbugs.gnu.org
> Date: Mon, 14 Oct 2019 10:36:11 +0200
> 
> Eli Zaretskii <eliz@HIDDEN> writes:
> 
> > I don't understand what would trigger these callbacks, and how do you
> > specify the region in advance, without knowing what will be inserted.
> 
> accept_process_output inserts the data into the buffer and then calls
> the callback with the region in question.  Well,
> read_and_dispose_of_process_output, I guess...

Filter functions are called even if the Lisp program never calls
accept-process-output, so your proposal doesn't seem to be equivalent
to what we have now, right?

OTOH, if one has to call accept-process-output, then why do we need
callbacks?  Just extend accept-process-output to call a function with
the received output.  No?

> > Without understanding this, I don't think I see the utility, and most
> > important: why this would be faster.
> 
> It would avoid creating (and garbaging) the strings.

I'm not sure I see how.

The way it works now is that we get the process output as a C string;
we then decode it and make a Lisp string from the result of decoding;
and then we invoke the filter with that Lisp string.  (If the filter
is nil, we invoke internal-default-process-filter instead, but it
still gets the text as a string.)

Which part(s) of this will be avoided under your proposal?

> > Btw, unlike what I originally implied, the default filter also
> > receives a Lisp string, so the question why by default reading dd
> > output is so much faster than when you define a non-default filter
> > function still stands.
> 
> Oh!  That is curious indeed.  Are the Lisp_Object strings somehow
> ... special here when they never leave C land?

No, I don't think so.

> The speed differential is completely repeatable...  hm...  Is the
> only difference that gc isn't given a chance to run in the
> non-filter case?

You could test that hypothesis by setting gc-cons-threshold to a very
high value.

Bottom line: I think we must understand better what takes the time in
your last test case, before we discuss solutions.  I'd start by
profiling that with "M-x profiler-start".




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 14 Oct 2019 08:54:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:54:31 2019
Received: from localhost ([127.0.0.1]:38116 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJw7b-0006wY-NK
	for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:54:31 -0400
Received: from [80.91.231.51] (port=54068 helo=quimby.gnus.org)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJw7a-0006wM-H4; Mon, 14 Oct 2019 04:54:30 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJw7V-00043Q-JR; Mon, 14 Oct 2019 10:54:28 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
 <87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN>
Date: Mon, 14 Oct 2019 10:54:25 +0200
In-Reply-To: <834l0clbzt.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct
 2019 21:46:30 +0300")
Message-ID: <87r23fiu66.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  Actually, my benchmarking is somewhat wrong. start-process
 with a filter, but discard output: (let ((coding-system-for-read 'binary))
 (kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc
 (start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero"
 "bs=4096" "co [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 1.3 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  Actually, my benchmarking is somewhat wrong. start-process
 with a filter, but discard output: (let ((coding-system-for-read 'binary))
 (kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc
 (start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero"
 "bs=4096" "co [...] 
 Content analysis details:   (1.3 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 SPF_NONE               SPF: sender does not publish an SPF Record
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 0.0 URIBL_BLOCKED          ADMINISTRATOR NOTICE: The query to URIBL was
 blocked.  See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
 for more information. [URIs: ingebrigtsen.no]
 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, benjamin.benninghofen@HIDDEN, 32729 <at> debbugs.gnu.org,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)

Actually, my benchmarking is somewhat wrong.

start-process with a filter, but discard output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (18.828236636 59 13.315468088000017)

filter, but insert the output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer (get-buffer " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (21.120281346 59 13.250166416000013)

With the default filter:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (34.046986424 116 26.025843717999976)

(!)

So the default filter is really slow?

Anyway, compare with call-process:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=4096" "count=250000")))
=> (1.694743653 0 0.0)

So what makes start-process 10x slower than call-process?  If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 14 Oct 2019 08:36:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:36:19 2019
Received: from localhost ([127.0.0.1]:38100 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJvpy-0006WN-Kx
	for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:36:18 -0400
Received: from [80.91.231.51] (port=53630 helo=quimby.gnus.org)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJvpx-0006WB-AF; Mon, 14 Oct 2019 04:36:17 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJvps-0003rW-FV; Mon, 14 Oct 2019 10:36:15 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
 <87sgnwbl8w.fsf@HIDDEN> <83blujkaf8.fsf@HIDDEN>
Date: Mon, 14 Oct 2019 10:36:11 +0200
In-Reply-To: <83blujkaf8.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 14 Oct
 2019 11:18:03 +0300")
Message-ID: <87v9sriv0k.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  Eli Zaretskii <eliz@HIDDEN> writes: > I don't understand
 what would trigger these callbacks, and how do you > specify the region in
 advance, without knowing what will be inserted. accept_process_output inserts
 the data into the buffer and then calls the callback with the region in
 question. Well, read_and_dispose_of_process_output, I guess... 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 1.3 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  Eli Zaretskii <eliz@HIDDEN> writes: > I don't understand
 what would trigger these callbacks, and how do you > specify the region in
 advance, without knowing what will be inserted. accept_process_output inserts
 the data into the buffer and then calls the callback with the region in
 question. Well, read_and_dispose_of_process_output, I guess... 
 Content analysis details:   (1.3 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 0.0 URIBL_BLOCKED          ADMINISTRATOR NOTICE: The query to URIBL was
 blocked.  See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
 for more information. [URIs: gnu.org]
 0.0 SPF_NONE               SPF: sender does not publish an SPF Record
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)

Eli Zaretskii <eliz@HIDDEN> writes:

> I don't understand what would trigger these callbacks, and how do you
> specify the region in advance, without knowing what will be inserted.

accept_process_output inserts the data into the buffer and then calls
the callback with the region in question.  Well,
read_and_dispose_of_process_output, I guess...

> Without understanding this, I don't think I see the utility, and most
> important: why this would be faster.

It would avoid creating (and garbaging) the strings.

> Btw, unlike what I originally implied, the default filter also
> receives a Lisp string, so the question why by default reading dd
> output is so much faster than when you define a non-default filter
> function still stands.

Oh!  That is curious indeed.  Are the Lisp_Object strings somehow
... special here when they never leave C land?  The speed differential
is completely repeatable...  hm...  Is the only difference that gc isn't
given a chance to run in the non-filter case?  Even if you subtract the
gc time, that doesn't explain the difference fully.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 14 Oct 2019 08:18:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:18:19 2019
Received: from localhost ([127.0.0.1]:38057 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJvYZ-0003yE-4M
	for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:18:19 -0400
Received: from eggs.gnu.org ([209.51.188.92]:43228)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJvYW-0003xw-Rc; Mon, 14 Oct 2019 04:18:17 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48753)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJvYR-0002uA-Bd; Mon, 14 Oct 2019 04:18:11 -0400
Received: from [176.228.60.248] (port=2756 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJvYQ-0001me-8H; Mon, 14 Oct 2019 04:18:10 -0400
Date: Mon, 14 Oct 2019 11:18:03 +0300
Message-Id: <83blujkaf8.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87sgnwbl8w.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 
 13 Oct 2019 19:36:47 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87sgnwbl8w.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: benjamin.benninghofen@HIDDEN,  layer@HIDDEN,
>   32729 <at> debbugs.gnu.org,  32728 <at> debbugs.gnu.org
> Date: Sun, 13 Oct 2019 19:36:47 +0200
> 
> So this is the design I want to do:
> 
> process-add-callback PROCESS FUNCTION
> process-remove-callback PROCESS FUNCTION
> 
> FUNCTION takes three parameters: The PROCESS and the start/end of the
> region inserted.  Perhaps it would make sense to do something with the
> return values -- if the function returns non-nil, then further callbacks
> are inhibited?

I don't understand what would trigger these callbacks, and how do you
specify the region in advance, without knowing what will be inserted.

Without understanding this, I don't think I see the utility, and most
important: why this would be faster.

> > However, I would begin by measuring the effect of this resizing on the
> > time it takes to receive large amounts of data.  Maybe other factors
> > make this part negligible.
> 
> Sure.  My simple dd test (without a filter) surprised me by being as
> fast as it was, so Emacs was able to grow that buffer quicker than I
> expected.  But it's also a pretty simple test case -- I can try to see
> what happens if I call enlarge_buffer_text to 1GB first and see what the
> effects are.

Btw, unlike what I originally implied, the default filter also
receives a Lisp string, so the question why by default reading dd
output is so much faster than when you define a non-default filter
function still stands.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 18:46:52 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 14:46:51 2019
Received: from localhost ([127.0.0.1]:36966 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJitF-0006jU-V8
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 14:46:51 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47624)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJitD-0006dx-P7; Sun, 13 Oct 2019 14:46:48 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34745)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJit8-0006mD-Ep; Sun, 13 Oct 2019 14:46:42 -0400
Received: from [176.228.60.248] (port=1042 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJit2-0002y3-Ct; Sun, 13 Oct 2019 14:46:41 -0400
Date: Sun, 13 Oct 2019 21:46:30 +0300
Message-Id: <834l0clbzt.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87k198bkrf.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 
 13 Oct 2019 19:47:16 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87k198bkrf.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: benjamin.benninghofen@HIDDEN,  layer@HIDDEN,
>   32729 <at> debbugs.gnu.org,  32728 <at> debbugs.gnu.org
> Date: Sun, 13 Oct 2019 19:47:16 +0200
> 
> >> Nope.  Does Emacs need to do a lot of recomputing when going to
> >> multibyte buffers?
> >
> > Of course: we try to display the multibyte text as characters, search
> > for fonts, invoke bidi reordering, etc.
> 
> I tried:
> 
> (benchmark-run
>     1
>   (call-process "dd" nil (with-current-buffer
> 			     (get-buffer-create " *zeroes*")
> 			   (set-buffer-multibyte nil)
> 			   (current-buffer))
> 		nil "if=/dev/zero" "bs=4096" "count=250000"))
> 
> and then jumped to the buffer, and Emacs hung anyway.  (That is, I
> killed Emacs after a minute, so I don't know whether it would have
> recovered after a while...)

Do that, _and_ move point to the beginning of the buffer.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 18:45:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 14:45:04 2019
Received: from localhost ([127.0.0.1]:36959 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJirX-00058l-P1
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 14:45:04 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47440)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJirW-00056n-Nq; Sun, 13 Oct 2019 14:45:03 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:34717)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJirR-0004vx-1r; Sun, 13 Oct 2019 14:44:57 -0400
Received: from [176.228.60.248] (port=4907 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJirP-0001qM-AL; Sun, 13 Oct 2019 14:44:55 -0400
Date: Sun, 13 Oct 2019 21:44:48 +0300
Message-Id: <835zkslc2n.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87wod8blsl.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 
 13 Oct 2019 19:24:58 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN>
 <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> <87wod8blsl.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: psainty@HIDDEN, layer@HIDDEN, 32729 <at> debbugs.gnu.org,
 benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Date: Sun, 13 Oct 2019 19:24:58 +0200
> Cc: Kevin Layer <layer@HIDDEN>, "Benninghofen,
>  Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32729 <at> debbugs.gnu.org,
>  32728 <at> debbugs.gnu.org
> 
> It is unfortunate that stuff like this makes Emacs hang, though.  It'd
> be nice if Emacs had a "OK, we just give up" mode if the buffer is too
> intractable, where "too intractable" may be, for instance, if it finds a
> line that's longer than a few megabytes, perhaps?
> 
> I don't know what "just give up" would entail, though.  Just put point
> at the start of the buffer and refuse to scroll or do anything?  Just
> about anything would be better than the current situation where you have
> to kill Emacs if you're playing with (some) binary files and try to
> display the buffer.

The problem is exactly to decide what to do when you "give up" in a
way that will still get you a functional editor that can display
something reasonable.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 17:47:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:47:22 2019
Received: from localhost ([127.0.0.1]:36850 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJhxh-0007rV-Ox
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:47:21 -0400
Received: from quimby.gnus.org ([80.91.231.51]:34924)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJhxg-0007rL-R8; Sun, 13 Oct 2019 13:47:21 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJhxc-0003Vk-T2; Sun, 13 Oct 2019 19:47:19 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
Date: Sun, 13 Oct 2019 19:47:16 +0200
In-Reply-To: <83r23hkqr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct
 2019 11:13:04 +0300")
Message-ID: <87k198bkrf.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> >> (Note: Don't visit
 the " *zeroes*" buffer after this, because that will >> >> hang Emacs totally.
 I guess the long-line display problem hasn't been >> >> fixed after all?)
 >> > >> > It's unrelat [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will
>> >> hang Emacs totally.  I guess the long-line display problem hasn't been
>> >> fixed after all?)
>> >
>> > It's unrelated.  Did you try to insert that into a unibyte buffer
>> > instead?
>> 
>> Nope.  Does Emacs need to do a lot of recomputing when going to
>> multibyte buffers?
>
> Of course: we try to display the multibyte text as characters, search
> for fonts, invoke bidi reordering, etc.

I tried:

(benchmark-run
    1
  (call-process "dd" nil (with-current-buffer
			     (get-buffer-create " *zeroes*")
			   (set-buffer-multibyte nil)
			   (current-buffer))
		nil "if=/dev/zero" "bs=4096" "count=250000"))

and then jumped to the buffer, and Emacs hung anyway.  (That is, I
killed Emacs after a minute, so I don't know whether it would have
recovered after a while...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 17:36:55 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:36:55 2019
Received: from localhost ([127.0.0.1]:36839 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJhna-0007aU-UX
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:36:55 -0400
Received: from quimby.gnus.org ([80.91.231.51]:34758)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJhnY-0007aF-Nf; Sun, 13 Oct 2019 13:36:53 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJhnU-0003Ol-B9; Sun, 13 Oct 2019 19:36:50 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
 <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
Date: Sun, 13 Oct 2019 19:36:47 +0200
In-Reply-To: <83r23hkqr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct
 2019 11:13:04 +0300")
Message-ID: <87sgnwbl8w.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> No sentinel is called,
 because the process status doesn't change, >> typically. All the relevant
 network protocols do not close after having >> "done something" (IMAP, HTTP,
 etc), but instead use i [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> No sentinel is called, because the process status doesn't change,
>> typically.  All the relevant network protocols do not close after having
>> "done something" (IMAP, HTTP, etc), but instead use in-protocol markers
>> to say that an operation is done.  So a filter has to be used.
>
> OK, so not in sentinel, but in a timer or somesuch.

A timer wastes resources by triggering work when there's nothing to do,
and conversely, makes network connections sluggish by not having the
code happen immediately when there's data ready.

After thinking about this a bit more, I'm reminded of the problems I had
in this area when doing the NSM, because I really wanted to put a filter
on (some of) the processes, but I couldn't, because we only allow one
filter per process.

So this is the design I want to do:

process-add-callback PROCESS FUNCTION
process-remove-callback PROCESS FUNCTION

FUNCTION takes three parameters: The PROCESS and the start/end of the
region inserted.  Perhaps it would make sense to do something with the
return values -- if the function returns non-nil, then further callbacks
are inhibited?

Anyway, with this, the current filters can trivially be implemented as a
callback instead -- the filters would just be wrapped in a function that
does (funcall filter-function (buffer-substring start end)) so that
we don't need duplicated code for these mechanisms on the C level.  So
set-process-filter and related moves to the Lisp level...

> We could provide a Lisp interface for such cases.  The C
> implementation is in enlarge_buffer_text (but we need to remember the
> decoding of human-readable text, when applicable).
>
> However, I would begin by measuring the effect of this resizing on the
> time it takes to receive large amounts of data.  Maybe other factors
> make this part negligible.

Sure.  My simple dd test (without a filter) surprised me by being as
fast as it was, so Emacs was able to grow that buffer quicker than I
expected.  But it's also a pretty simple test case -- I can try to see
what happens if I call enlarge_buffer_text to 1GB first and see what the
effects are.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 17:25:07 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:25:07 2019
Received: from localhost ([127.0.0.1]:36824 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJhcB-0005Df-0b
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:25:07 -0400
Received: from quimby.gnus.org ([80.91.231.51]:34510)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJhc7-0005DO-0t; Sun, 13 Oct 2019 13:25:05 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJhc2-0003JB-Mq; Sun, 13 Oct 2019 19:25:01 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Phil Sainty <psainty@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN>
 <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN>
Date: Sun, 13 Oct 2019 19:24:58 +0200
In-Reply-To: <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> (Phil
 Sainty's message of "Sun, 13 Oct 2019 23:49:41 +1300")
Message-ID: <87wod8blsl.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Phil Sainty <psainty@HIDDEN> writes: > On 12/10/19 4:57
 PM, Lars Ingebrigtsen wrote: >> (Note: Don't visit the " *zeroes*" buffer
 after this, because that >> will hang Emacs totally. I guess the long-line
 display problem >> hasn't been f [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, "Benninghofen,
 Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Phil Sainty <psainty@HIDDEN> writes:

> On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote:
>> (Note: Don't visit the " *zeroes*" buffer after this, because that
>> will hang Emacs totally.  I guess the long-line display problem
>> hasn't been fixed after all?)
>
> I imagine you're thinking of so-long.el here, in which case there may
> be a misconception.

Yes, I hadn't followed the development closely, but just registered that
... things ... had gotten better.  :-)

> The biggest problem in this case (by far) is that point has been left
> at the end of the line following the insertion, and in order for the
> redisplay to *display* the end of the line, it's having to process a
> gigabyte of data.  My largest test case for so-long.el was a couple of
> orders of magnitude smaller than this, and Emacs wasn't remotely happy
> showing the end of that line, let alone this one.

Ah, I see.  Thanks for the explanation.

It is unfortunate that stuff like this makes Emacs hang, though.  It'd
be nice if Emacs had a "OK, we just give up" mode if the buffer is too
intractable, where "too intractable" may be, for instance, if it finds a
line that's longer than a few megabytes, perhaps?

I don't know what "just give up" would entail, though.  Just put point
at the start of the buffer and refuse to scroll or do anything?  Just
about anything would be better than the current situation where you have
to kill Emacs if you're playing with (some) binary files and try to
display the buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 10:49:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 06:49:48 2019
Received: from localhost ([127.0.0.1]:35247 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJbRc-0007ss-9S
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 06:49:48 -0400
Received: from smtp-2.orcon.net.nz ([60.234.4.43]:33701)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <psainty@HIDDEN>)
 id 1iJbRZ-0007sg-Ag; Sun, 13 Oct 2019 06:49:46 -0400
Received: from [116.251.203.173] (port=2746 helo=[192.168.20.103])
 by smtp-2.orcon.net.nz with esmtpa (Exim 4.90_1)
 (envelope-from <psainty@HIDDEN>)
 id 1iJbRV-0003Bn-5Z; Sun, 13 Oct 2019 23:49:41 +1300
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
To: Lars Ingebrigtsen <larsi@HIDDEN>
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN>
From: Phil Sainty <psainty@HIDDEN>
Message-ID: <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN>
Date: Sun, 13 Oct 2019 23:49:41 +1300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <871rviobu2.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-GB
Content-Transfer-Encoding: 7bit
X-GeoIP: NZ
X-Spam_score: -2.9
X-Spam_score_int: -28
X-Spam_bar: --
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 32728
Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, "Benninghofen,
 Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote:
> (Note: Don't visit the " *zeroes*" buffer after this, because that
> will hang Emacs totally.  I guess the long-line display problem
> hasn't been fixed after all?)

I imagine you're thinking of so-long.el here, in which case there may
be a misconception.  so-long helps tremendously in certain situations,
but it doesn't (and cannot) eliminate all sources of potential
slowness in the face of extremely long lines -- ultimately the C code
still needs time to process the lines being displayed, and so-long.el
can't make that code work any faster.

The biggest problem in this case (by far) is that point has been left
at the end of the line following the insertion, and in order for the
redisplay to *display* the end of the line, it's having to process a
gigabyte of data.  My largest test case for so-long.el was a couple of
orders of magnitude smaller than this, and Emacs wasn't remotely happy
showing the end of that line, let alone this one.

(What so-long is really good at is preventing situations whereby
merely visiting a file -- with point still at the beginning of the
buffer -- caused Emacs to hang badly; which can easily happen when
various modes and options are active.  It can work wonders in those
cases.  If you try to display the end of a really gargantuan line of
data like this, though, you're simply at the mercy of other factors.)

If you added a (goto-char (point-min)) before switching to the
*zeroes* buffer then, given that there's not much else happening in
that buffer which would create issues, Emacs shouldn't hang when you
switch to it.

The further you move into that line, though, the worse the performance
is going to get (so the key take-away here would be to avoid doing that
when dealing with this kind of data, if at all possible).

Also note that `so-long' doesn't actually trigger automatically if you
simply create a buffer and throw a ton of data into it.  The automatic
behaviour is tied to `set-auto-mode' and intended to deal with users
visiting files.  Auto-detecting the sudden insertion of large lines
into an existing buffer is a different kettle of fish, and the current
library does not attempt to cater for it.  So if you wanted to trigger
it in this example you would need to call it explicitly.


-Phil





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 13 Oct 2019 08:13:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 04:13:21 2019
Received: from localhost ([127.0.0.1]:34884 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJZ0D-0003uE-3Q
	for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 04:13:21 -0400
Received: from eggs.gnu.org ([209.51.188.92]:40287)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJZ0B-0003tv-JR; Sun, 13 Oct 2019 04:13:20 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:55395)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJZ05-00040m-RG; Sun, 13 Oct 2019 04:13:13 -0400
Received: from [176.228.60.248] (port=1844 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJZ04-0004RJ-Vm; Sun, 13 Oct 2019 04:13:13 -0400
Date: Sun, 13 Oct 2019 11:13:04 +0300
Message-Id: <83r23hkqr3.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <87y2xplufp.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sat, 
 12 Oct 2019 19:55:54 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Cc: benjamin.benninghofen@HIDDEN,  layer@HIDDEN,
>   32729 <at> debbugs.gnu.org,  32728 <at> debbugs.gnu.org
> Date: Sat, 12 Oct 2019 19:55:54 +0200
> 
> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will
> >> hang Emacs totally.  I guess the long-line display problem hasn't been
> >> fixed after all?)
> >
> > It's unrelated.  Did you try to insert that into a unibyte buffer
> > instead?
> 
> Nope.  Does Emacs need to do a lot of recomputing when going to
> multibyte buffers?

Of course: we try to display the multibyte text as characters, search
for fonts, invoke bidi reordering, etc.

> >> And it's a real world problem: When reading data from any network
> >> source, you have to use filters because the protocol is usually based on
> >> parsing the output to find out when it's over, so you can't use
> >> sentinels.
> >
> > Why can't you use the default filter, and in the sentinel work on the
> > buffer with the complete response?
> 
> No sentinel is called, because the process status doesn't change,
> typically.  All the relevant network protocols do not close after having
> "done something" (IMAP, HTTP, etc), but instead use in-protocol markers
> to say that an operation is done.  So a filter has to be used.

OK, so not in sentinel, but in a timer or somesuch.

> > Please also note that, GC aside, inserting stuff of size that is
> > unknown in advance is not free, either: we need to enlarge the buffer
> > text each time more stuff arrives.
> 
> Yes, I was wondering about that -- how slow is resizing buffers, really?
> Does Emacs rely on OS-level realloc that doesn't necessitate actually
> copying all that data all the time?

Not necessarily realloc, sometimes mmap or similar.  And yes, this
requires copying the data when the OS cannot grow the previous
allocation, but instead gives us a memory block at another address.
In most systems this copying is under the hood, but it does happen.

> Also -- some of the networking operations know in advance how much data
> is going to be received.  For instance, when using non-chunked HTTP
> (quite common when fetching "data", i.e., images, PDFs etc), we get the
> content size in a header.  Would it make sense to have a way to tell
> Emacs about this in advance so that it could pre-size the buffer?

We could provide a Lisp interface for such cases.  The C
implementation is in enlarge_buffer_text (but we need to remember the
decoding of human-readable text, when applicable).

However, I would begin by measuring the effect of this resizing on the
time it takes to receive large amounts of data.  Maybe other factors
make this part negligible.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 12 Oct 2019 17:56:03 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 12 13:56:03 2019
Received: from localhost ([127.0.0.1]:34097 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJLcZ-0001t1-0k
	for submit <at> debbugs.gnu.org; Sat, 12 Oct 2019 13:56:03 -0400
Received: from quimby.gnus.org ([80.91.231.51]:36670)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJLcV-0001sV-BU; Sat, 12 Oct 2019 13:56:01 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJLcQ-0006TB-IJ; Sat, 12 Oct 2019 19:55:57 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
Date: Sat, 12 Oct 2019 19:55:54 +0200
In-Reply-To: <83imouo1jp.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 12 Oct
 2019 10:39:22 +0300")
Message-ID: <87y2xplufp.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> (benchmark-run 1
 (call-process
 "dd" nil (get-buffer-create " >> *zeroes*") nil "if=/dev/zero" "bs=1000"
 "count=1000000")) >> => (4.703641145 0 0.0) >> >> (Note: Don't visit the "
 *zeroes*" buffer a [...] 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> (benchmark-run 1 (call-process "dd" nil (get-buffer-create "
>> *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
>> => (4.703641145 0 0.0)
>> 
>> (Note: Don't visit the " *zeroes*" buffer after this, because that will
>> hang Emacs totally.  I guess the long-line display problem hasn't been
>> fixed after all?)
>
> It's unrelated.  Did you try to insert that into a unibyte buffer
> instead?

Nope.  Does Emacs need to do a lot of recomputing when going to
multibyte buffers?

>> And it's a real world problem: When reading data from any network
>> source, you have to use filters because the protocol is usually based on
>> parsing the output to find out when it's over, so you can't use
>> sentinels.
>
> Why can't you use the default filter, and in the sentinel work on the
> buffer with the complete response?

No sentinel is called, because the process status doesn't change,
typically.  All the relevant network protocols do not close after having
"done something" (IMAP, HTTP, etc), but instead use in-protocol markers
to say that an operation is done.  So a filter has to be used.

> Please also note that, GC aside, inserting stuff of size that is
> unknown in advance is not free, either: we need to enlarge the buffer
> text each time more stuff arrives.

Yes, I was wondering about that -- how slow is resizing buffers, really?
Does Emacs rely on OS-level realloc that doesn't necessitate actually
copying all that data all the time?

Also -- some of the networking operations know in advance how much data
is going to be received.  For instance, when using non-chunked HTTP
(quite common when fetching "data", i.e., images, PDFs etc), we get the
content size in a header.  Would it make sense to have a way to tell
Emacs about this in advance so that it could pre-size the buffer?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 12 Oct 2019 07:39:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 12 03:39:42 2019
Received: from localhost ([127.0.0.1]:60675 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJC05-0001Wl-Vj
	for submit <at> debbugs.gnu.org; Sat, 12 Oct 2019 03:39:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:41731)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>)
 id 1iJC03-0001WS-A7; Sat, 12 Oct 2019 03:39:40 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:33890)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1iJBzx-0007ld-Ko; Sat, 12 Oct 2019 03:39:33 -0400
Received: from [176.228.60.248] (port=2912 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1iJBzx-0001eq-2w; Sat, 12 Oct 2019 03:39:33 -0400
Date: Sat, 12 Oct 2019 10:39:22 +0300
Message-Id: <83imouo1jp.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>
In-reply-to: <871rviobu2.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sat, 
 12 Oct 2019 05:57:09 +0200)
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
 <871rviobu2.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 32728
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Lars Ingebrigtsen <larsi@HIDDEN>
> Date: Sat, 12 Oct 2019 05:57:09 +0200
> Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org
> 
> (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
> => (4.703641145 0 0.0)
> 
> (Note: Don't visit the " *zeroes*" buffer after this, because that will
> hang Emacs totally.  I guess the long-line display problem hasn't been
> fixed after all?)

It's unrelated.  Did you try to insert that into a unibyte buffer
instead?

> So it would seem that the Emacs filter method is just unnecessarily
> slow, which I've long suspected.  Creating the strings before calling
> the filter is probably what's taking quite a bit of this time, but the
> rest is taken up by garbage collecting as it spends 13 of these 20
> seconds doing that.
> 
> And it's a real world problem: When reading data from any network
> source, you have to use filters because the protocol is usually based on
> parsing the output to find out when it's over, so you can't use
> sentinels.

Why can't you use the default filter, and in the sentinel work on the
buffer with the complete response?

Please also note that, GC aside, inserting stuff of size that is
unknown in advance is not free, either: we need to enlarge the buffer
text each time more stuff arrives.

> So for Emacs 28 I want to explore adding a new type of filter to
> processes: One that doesn't take a string argument, but which just
> inserts the data into the buffer, and then calls the filter with the
> region positions of what was inserted, which is just as powerful, but
> should allow streams to be 10x more efficient.

Doesn't the default filter already do that?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.

Message received at 32728 <at> debbugs.gnu.org:


Received: (at 32728) by debbugs.gnu.org; 12 Oct 2019 03:57:19 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 11 23:57:19 2019
Received: from localhost ([127.0.0.1]:60551 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1iJ8Wt-0006lR-8W
	for submit <at> debbugs.gnu.org; Fri, 11 Oct 2019 23:57:19 -0400
Received: from quimby.gnus.org ([80.91.231.51]:50562)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>)
 id 1iJ8Wo-0006lB-1F; Fri, 11 Oct 2019 23:57:14 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
 by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.89) (envelope-from <larsi@HIDDEN>)
 id 1iJ8Wj-0006U3-Bq; Sat, 12 Oct 2019 05:57:11 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: "Benninghofen\, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
Date: Sat, 12 Oct 2019 05:57:09 +0200
Message-ID: <871rviobu2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  The example is a bit convoluted,
 but it's an interesting question
 -- is Emacs' handling of process output fast or not? As a baseline, let's
 read a GB of zeroes and output to /dev/null: larsi@marnie:~/src/emacs/trunk$
 time dd if=/dev/zero bs=1000 count=1000000 > /dev/null 1000000+0 records
 in 1000000+0 records out 1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672
 s, 489 MB/s 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32728
Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

The example is a bit convoluted, but it's an interesting question -- is
Emacs' handling of process output fast or not?

As a baseline, let's read a GB of zeroes and output to /dev/null:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 > /dev/null
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672 s, 489 MB/s

real	0m2.064s
user	0m0.173s
sys	0m1.684s

(benchmark-run 1 (call-process "dd" nil nil nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (0.665899839 0 0.0)

So that's better in Emacs than in the shell, but we're just setting
stdout to /dev/zero here, so we're not actually seeing the data at all.

But let's insert the data somewhere:

(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (4.703641145 0 0.0)

(Note: Don't visit the " *zeroes*" buffer after this, because that will
hang Emacs totally.  I guess the long-line display problem hasn't been
fixed after all?)

4.7s isn't horrible, but it's not good, either.  But most of that time
is taken up in coding system conversion, so:

(let ((coding-system-for-read 'binary))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")))
=> (1.750549617 0 0.0)

Which is faster than writing to a file:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 of=/tmp/foo
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.21987 s, 450 MB/s

real	0m2.325s
user	0m0.168s
sys	0m1.957s

So that's OK.  But what happens when we add a process filter?

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
				 ))
      (while (and (process-live-p proc)
		  (accept-process-output proc 0.01))))))
=> (16.878995199 993 12.469541476)

That's slow, and we're just discarding the data.  If we output the data,
it's even slower, but not a surprising amount:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer
				     (get-buffer-create " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 0.01))))))
=> (19.801399562 1000 12.700370797000001)

Byte-compiling the function makes no difference.

So it would seem that the Emacs filter method is just unnecessarily
slow, which I've long suspected.  Creating the strings before calling
the filter is probably what's taking quite a bit of this time, but the
rest is taken up by garbage collecting as it spends 13 of these 20
seconds doing that.

And it's a real world problem: When reading data from any network
source, you have to use filters because the protocol is usually based on
parsing the output to find out when it's over, so you can't use
sentinels.

So for Emacs 28 I want to explore adding a new type of filter to
processes: One that doesn't take a string argument, but which just
inserts the data into the buffer, and then calls the filter with the
region positions of what was inserted, which is just as powerful, but
should allow streams to be 10x more efficient.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





Information forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.
Changed bug title to 'Emacs slow when reading process output' from 'Xemacs 23 times as fast as GNU Emacs' Request was from Lars Ingebrigtsen <larsi@HIDDEN> to control <at> debbugs.gnu.org. Full text available.
Merged 32728 32729. Request was from Glenn Morris <rgm@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 13 Sep 2018 15:13:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Sep 13 11:13:42 2018
Received: from localhost ([127.0.0.1]:39367 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1g0TJO-0006HD-58
	for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 11:13:42 -0400
Received: from eggs.gnu.org ([208.118.235.92]:49839)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0NDB-0002R8-PM
 for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 04:42:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0ND5-000819-IV
 for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 04:42:48 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,HTML_MESSAGE
 autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:38881)
 by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0ND5-00080w-Bo
 for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 04:42:47 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42580)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0ND4-0003gz-71
 for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 04:42:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0ND0-0007yl-41
 for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 04:42:46 -0400
Received: from mo3.myeers.net ([87.190.7.238]:58927)
 by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
 (Exim 4.71) (envelope-from <benjamin.benninghofen@HIDDEN>)
 id 1g0NCz-0007x1-Ew
 for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 04:42:42 -0400
X-IronPort-AV: E=Sophos;i="5.53,368,1531778400"; 
 d="txt'?scan'208,217";a="39114592"
Received: from unknown (HELO DE0-03HUB-P01.central.mail.corp) ([44.225.67.170])
 by de0-03iro-p04-out.myeers.net with ESMTP/TLS/AES256-SHA;
 13 Sep 2018 10:42:37 +0200
Received: from esa2e.demail.de.airbusds.corp (10.67.144.34) by
 DE0-03HUB-P01.central.mail.corp (44.225.67.172) with Microsoft SMTP Server id
 15.0.1365.1; Thu, 13 Sep 2018 10:42:32 +0200
Received: from unknown (HELO CD1-4DDAG02-P01.cdmail.common.airbusds.corp)
 ([10.67.164.142])
 by esa2i.demail.de.airbusds.corp with ESMTP; 13 Sep 2018 10:42:32 +0200
Received: from CD1-4DDAG02-P01.cdmail.common.airbusds.corp (10.67.164.142) by
 CD1-4DDAG02-P01.cdmail.common.airbusds.corp (10.67.164.142) with
 Microsoft
 SMTP Server (TLS) id 15.0.1365.1; Thu, 13 Sep 2018 10:42:32 +0200
Received: from CD1-4DDAG02-P01.cdmail.common.airbusds.corp ([10.67.164.142])
 by CD1-4DDAG02-P01.cdmail.common.airbusds.corp ([10.67.164.142]) with mapi id
 15.00.1365.000; Thu, 13 Sep 2018 10:42:32 +0200
From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
To: "bug-gnu-emacs@HIDDEN" <bug-gnu-emacs@HIDDEN>
Subject: Xemacs 23 times as fast as GNU Emacs
Thread-Topic: Xemacs 23 times as fast as GNU Emacs
Thread-Index: AdRLPalnSe9KPpEWS6qma41+du5D7Q==
Date: Thu, 13 Sep 2018 08:42:32 +0000
Message-ID: <08c31622831a463da10ca3750f3d54fa@HIDDEN>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
x-ms-exchange-transport-fromentityheader: Hosted
Content-Type: multipart/mixed;
 boundary="_004_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_"
MIME-Version: 1.0
X-GM-Security: forwarded
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
 recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Thu, 13 Sep 2018 11:13:41 -0400
Cc: Kevin Layer <layer@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.1 (-----)

--_004_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_
Content-Type: multipart/alternative;
	boundary="_000_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_"

--_000_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable



The versions of Xemacs and GNU Emacs are those that come with RHEL 7.5.

The attachment contains 2 files:

demo.el   : to be evaluated in Emacs Lisp
prog.sh  :  to be placed in the home directory

Furthermore a large text file "input.txt"  is needed in the home directory.=
 The file should have 1000000 lines, each line longer than 80 characters. T=
he file is not included in the attachment because this would be too big.


The file I used was created with the following ANSI-COMMON-LISP function:

(defun print-nums (&key (first 1) (last 1000000))
  (check-type first fixnum)
  (check-type last fixnum)
  (loop for k of-type fixnum from first to last do (format t "~%~B ^3 =3D ~=
B" k (expt k 3)))
  t)

Alternatively the "input.txt" file can be created as follows:

#! /bin/bash
  last=3D1000000
  function base2 {
      echo "obase=3D2;$1" | bc
  }
  for k in $(seq 1 $last); do
      x=3D$(( k * k * k ))
      echo $(base2 $k) '^3 =3D' $(base2 $x)
  done

generate the input like this:

  $ ./gen.sh > input.txt



The benchmark is executed by
(M-x) demo

At the end the time is printed and I received the following results:

Xemacs    : 51 seconds
GNU Emacs : 1205 seconds

So the Xemacs is more than 23 times as fast as the GNU Emacs.

---
Benjamin Benninghofen


The information in this e-mail is confidential. The contents may not be dis=
closed or used by anyone other than the addressee. Access to this e-mail by=
 anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and=
 delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of=
 this e-mail as it has been sent over public networks. If you have any conc=
erns over the content of this message or its Accuracy or Integrity, please =
contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus =
scanning software but you should take whatever measures you deem to be appr=
opriate to ensure that this message and any attachments are virus free.

--_000_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_
Content-Type: text/html; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii">
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left:=
 #800000 2px solid; } --></style>
</head>
<body>
<font face=3D"Calibri" size=3D"2"><span style=3D"font-size:11pt;">
<div> </div>
<div>&nbsp;</div>
<div>The versions of Xemacs and GNU Emacs are those that come with RHEL 7.5=
.</div>
<div>&nbsp;</div>
<div>The attachment contains 2 files:</div>
<div>&nbsp;</div>
<div>demo.el&nbsp;&nbsp; : to be evaluated in Emacs Lisp</div>
<div>prog.sh&nbsp; :&nbsp; to be placed in the home directory</div>
<div>&nbsp;</div>
<div>Furthermore a large text file &quot;input.txt&quot;&nbsp; is needed in=
 the home directory. The file should have 1000000 lines, each line longer t=
han 80 characters. The file is not included in the attachment because this =
would be too big.</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>The file I used was created with the following ANSI-COMMON-LISP functi=
on:</div>
<div>&nbsp;</div>
<div>(defun print-nums (&amp;key (first 1) (last 1000000))</div>
<div>&nbsp; (check-type first fixnum)</div>
<div>&nbsp; (check-type last fixnum)</div>
<div>&nbsp; (loop for k of-type fixnum from first to last do (format t &quo=
t;~%~B ^3 =3D ~B&quot; k (expt k 3)))</div>
<div>&nbsp; t)</div>
<div>&nbsp;</div>
<div>Alternatively the &quot;input.txt&quot; file can be created as follows=
:</div>
<div>&nbsp;</div>
<div>#! /bin/bash</div>
<div>&nbsp; last=3D1000000</div>
<div>&nbsp; function base2 {</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; echo &quot;obase=3D2;$1&quot; | bc</div>
<div>&nbsp; }</div>
<div>&nbsp; for k in $(seq 1 $last); do</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; x=3D$(( k * k * k ))</div>
<div>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; echo $(base2 $k) '^3 =3D' $(base2 $x)</=
div>
<div>&nbsp; done</div>
<div>&nbsp;</div>
<div>generate the input like this:</div>
<div>&nbsp;</div>
<div>&nbsp; $ ./gen.sh &gt; input.txt</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>The benchmark is executed by </div>
<div>(M-x) demo</div>
<div>&nbsp;</div>
<div>At the end the time is printed and I received the following results:</=
div>
<div>&nbsp;</div>
<div>Xemacs&nbsp;&nbsp;&nbsp; : 51 seconds</div>
<div>GNU Emacs : 1205 seconds</div>
<div>&nbsp;</div>
<div>So the Xemacs is more than 23 times as fast as the GNU Emacs.</div>
<div>&nbsp;</div>
<div>---</div>
<div>Benjamin Benninghofen</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
</span></font>
<font style=3D"font-size: 9px;">The information in this e-mail is confident=
ial. The contents may not be disclosed or used by anyone other than the add=
ressee. Access to this e-mail by anyone else is unauthorised.<br>If you are=
 not the intended recipient, please notify Airbus immediately and delete th=
is e-mail.<br>Airbus cannot accept any responsibility for the accuracy or c=
ompleteness of this e-mail as it has been sent over public networks. If you=
 have any concerns over the content of this message or its Accuracy or Inte=
grity, please contact Airbus immediately.<br>All outgoing e-mails from Airb=
us are checked using regularly updated virus scanning software but you shou=
ld take whatever measures you deem to be appropriate to ensure that this me=
ssage and any attachments are virus free.</font></body>
</html>

--_000_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_--

--_004_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_
Content-Type: text/plain;
	name="A_POLICY_VIOLATED_FILE_WAS_DETECTED_AND_REMOVED.TXT"
Content-Description: A_POLICY_VIOLATED_FILE_WAS_DETECTED_AND_REMOVED.TXT
Content-Disposition: attachment;
	filename="A_POLICY_VIOLATED_FILE_WAS_DETECTED_AND_REMOVED.TXT"; size=189;
	creation-date="Thu, 13 Sep 2018 08:27:00 GMT";
	modification-date="Thu, 13 Sep 2018 08:18:47 GMT"
Content-Transfer-Encoding: base64

77u/U2Nhbk1haWwgZGV0ZWN0ZWQgYW5kIHJlbW92ZWQgYSBmaWxlIG5hbWVkICJmaWxlczIudGd6
IiB0aGF0IHZpb2xhdGVkIGF0dGFjaG1lbnQgYmxvY2tpbmcgcG9saWN5IGZyb20gdGhlIG9yaWdp
bmFsIG1haWwgZW50aXR5LiBZb3UgY2FuIHNhZmVseSBzYXZlIG9yIGRlbGV0ZSB0aGlzIHJlcGxh
Y2VtZW50IGF0dGFjaG1lbnQu

--_004_08c31622831a463da10ca3750f3d54faCD14DDAG02P01cdmailcomm_--





Acknowledgement sent to "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#32728; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.