GNU bug report logs - #48118
27.1; 28; Only first process receives output with multiple running processes

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: emacs; Reported by: Daniel Mendler <mail@HIDDEN>; dated Fri, 30 Apr 2021 13:45:02 UTC; Maintainer for emacs is bug-gnu-emacs@HIDDEN.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 2 May 2021 07:23:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun May 02 03:23:44 2021
Received: from localhost ([127.0.0.1]:41112 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1ld6S8-0007Ol-9E
	for submit <at> debbugs.gnu.org; Sun, 02 May 2021 03:23:44 -0400
Received: from quimby.gnus.org ([95.216.78.240]:55036)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <larsi@HIDDEN>) id 1ld6S6-0007Ob-3p
 for 48118 <at> debbugs.gnu.org; Sun, 02 May 2021 03:23:42 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org;
 s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:
 References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=jX6yH6WHPoLM9d/mzt5U6qHOSr13utPdg7IYl2gBY3U=; b=CXzGApIRnaSK03KFdjcgHmpmzX
 Ht1Baj7kB69j+nImj3WCOJQMzvv9XI85C7HI05n5bF3YTUgJn5NHmj/uoqsvlUjD5yqN9pMQKhAQx
 /olEYC0ITs9ENuTKjpwruIiGLaO7Sl26034c1+mfcCk9smfk1/gg9MeImMZ/AhCQ6yXU=;
Received: from cm-84.212.220.105.getinternet.no ([84.212.220.105] helo=xo)
 by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.92) (envelope-from <larsi@HIDDEN>)
 id 1ld6Rx-0003mH-Vd; Sun, 02 May 2021 09:23:36 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
 <83o8dvbyyz.fsf@HIDDEN>
 <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
 <83bl9vbw8h.fsf@HIDDEN>
 <70ea83e2-fc9e-6feb-240c-ed41abac5254@HIDDEN>
X-Now-Playing: Zola Jesus's _New Amsterdam_: "Odessa"
Date: Sun, 02 May 2021 09:23:33 +0200
In-Reply-To: <70ea83e2-fc9e-6feb-240c-ed41abac5254@HIDDEN> (Daniel
 Mendler's message of "Fri, 30 Apr 2021 18:17:49 +0200")
Message-ID: <87v981k39m.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 @@CONTACT_ADDRESS@@ for details.
 Content preview:  Daniel Mendler <mail@HIDDEN> writes: > This is
 what we assumed. Emacs could check the second process the next > time. This
 way one may get a slightly more fair behavior. Yes, more fairness in how
 Emacs handles process output would be nice, indeed. 
 Content analysis details:   (-2.9 points, 5.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.0 ALL_TRUSTED            Passed through trusted hosts only via SMTP
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
 [score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 48118
Cc: Eli Zaretskii <eliz@HIDDEN>, 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Daniel Mendler <mail@HIDDEN> writes:

> This is what we assumed. Emacs could check the second process the next
> time. This way one may get a slightly more fair behavior.

Yes, more fairness in how Emacs handles process output would be nice,
indeed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 18:06:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 14:06:41 2021
Received: from localhost ([127.0.0.1]:59813 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcXXF-0006TO-E3
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 14:06:41 -0400
Received: from eggs.gnu.org ([209.51.188.92]:42620)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcXXE-0006TD-F6
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 14:06:40 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:53643)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcXX8-0003dr-BC; Fri, 30 Apr 2021 14:06:34 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2678
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcXX4-000491-Rx; Fri, 30 Apr 2021 14:06:32 -0400
Date: Fri, 30 Apr 2021 21:06:22 +0300
Message-Id: <83y2czabq9.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <70ea83e2-fc9e-6feb-240c-ed41abac5254@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 18:17:49 +0200)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
 <83o8dvbyyz.fsf@HIDDEN>
 <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
 <83bl9vbw8h.fsf@HIDDEN>
 <70ea83e2-fc9e-6feb-240c-ed41abac5254@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48118 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 18:17:49 +0200
> 
> > Bytes read has a problem when processes produce output a very
> > different rates.  Time spent to handle may (and usually does) mean the
> > filter function does something expensive, it doesn't necessarily tell
> > anything about the output from the subprocess.
> 
> Of course it is not possible to find a perfect scheduling algorithm. But
> how does the OS handle it if you have multiple processes which produce
> output with vastly different rates? I am not claiming this problem has
> been solved, but there are certainly some heuristics. Emacs is also
> dependent on the OS scheduling, depending on how Emacs schedules its
> reads/writes from the processes, the OS scheduler adjusts accordingly.
> This furthermore complicates the picture.

I'm sure patches to tune the scheduling to specific use cases will be
welcome.  My gut feeling is that we will need some variables to allow
Lisp programs to tell Emacs how to handle the various kinds of
processes and combinations thereof, but if you can come up with
patches that automatically adapt to the process's behavior, that would
be even better.

You could also try playing with the value of read-process-output-max,
perhaps enlarging it will make the problem in your case less severe.

> > If you read the code, you will see this isn't what happens.  What
> > happens is that Emacs reads a chunk of output from the first process
> > it sees ready, then it goes back and re-checks which processes are
> > ready -- and in your scenario I think it again sees that the first
> > process is ready.
> 
> This is what we assumed. Emacs could check the second process the next
> time. This way one may get a slightly more fair behavior. It would
> certainly not be perfect and you could throw scenarios at it which would
> make it behave unexpectedly. It may behave a bit more expectedly in the
> common case?

We could try that (conditioned on some new variable) and see if this
has downsides.

> > I suggest to read the code of wait_reading_process_output, it has some
> > non-trivial logic in this department.
> 
> I will do that. Has this problem discussed before?

I don't think so, but my memory is not to be trusted.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 17:52:42 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 13:52:42 2021
Received: from localhost ([127.0.0.1]:59779 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcXJi-00064t-2u
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 13:52:42 -0400
Received: from eggs.gnu.org ([209.51.188.92]:39744)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcXJg-00064h-MX
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 13:52:41 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:53143)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcXJb-0005n5-2A; Fri, 30 Apr 2021 13:52:35 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1823
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcXJa-0004TJ-9J; Fri, 30 Apr 2021 13:52:34 -0400
Date: Fri, 30 Apr 2021 20:52:29 +0300
Message-Id: <837dkjbqxu.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: jakanakaevangeli@HIDDEN
In-Reply-To: <87pmyb6963.fsf@miha-pc> (jakanakaevangeli@HIDDEN)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <87pmyb6963.fsf@miha-pc>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: mail@HIDDEN, 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: jakanakaevangeli@HIDDEN
> Cc: eliz@HIDDEN, 48118 <at> debbugs.gnu.org
> Date: Fri, 30 Apr 2021 18:15:16 +0200
> 
> Excuse me if I'm mistaken, but doesn't pselect return all file
> descriptors that are ready for reading? Couldn't we then simply process
> all of them (in a for loop) after pselect returns?

That's what we do, but with a twist that I described.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 16:18:00 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 12:18:00 2021
Received: from localhost ([127.0.0.1]:59699 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcVq4-0001NI-5S
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 12:18:00 -0400
Received: from server.qxqx.de ([178.63.65.180]:60019 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcVq2-0001N3-4t
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 12:17:58 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=BsGnEcKcdaN2xAZepPtfvABAJL0pvmjEU9hq49zvY5o=; b=XIBWkH0NADrVJA+ZnnK0LRn+jj
 e/Mb4/v1TBPx2YY5CM2uJhPbE2gVs3ipBh/3sgx086uMsD3qKPk01AjVcaaqfuO4lgWQXww8Jk7aS
 5Aqk3AyjdiFd0RY33FfPB1irmAzu30n3PxvT0uDaSc+Y1EePB/3ND1xMnydVg4Xy/Xl0=;
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
To: Eli Zaretskii <eliz@HIDDEN>
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
 <83o8dvbyyz.fsf@HIDDEN>
 <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
 <83bl9vbw8h.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <70ea83e2-fc9e-6feb-240c-ed41abac5254@HIDDEN>
Date: Fri, 30 Apr 2021 18:17:49 +0200
MIME-Version: 1.0
In-Reply-To: <83bl9vbw8h.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 4/30/21 5:58 PM, Eli Zaretskii wrote:
> A call to accept-process-output prioritizes a process only if it
> explicitly requests output from that single process.  Which is not
> necessarily true in all cases.

Yes, I have seen that in the documentation.

>>> What does this mean, exactly?  Which quantity should be doled in a
>>> round-robin fashion? bytes read from the processes? something else?
>>>
>>> If the bytes read, then how do you suggest to handle two processes
>>> which produce output at very different rates?
>>
>> For example bytes read or time spent to handle a process (time spent in
>> the filter function?).
> 
> Bytes read has a problem when processes produce output a very
> different rates.  Time spent to handle may (and usually does) mean the
> filter function does something expensive, it doesn't necessarily tell
> anything about the output from the subprocess.

Of course it is not possible to find a perfect scheduling algorithm. But
how does the OS handle it if you have multiple processes which produce
output with vastly different rates? I am not claiming this problem has
been solved, but there are certainly some heuristics. Emacs is also
dependent on the OS scheduling, depending on how Emacs schedules its
reads/writes from the processes, the OS scheduler adjusts accordingly.
This furthermore complicates the picture.

>> When I stumbled over this issue, it astonished me that Emacs
>> does not seem to do any scheduling at all and handles only a single
>> process.
> 
> If you read the code, you will see this isn't what happens.  What
> happens is that Emacs reads a chunk of output from the first process
> it sees ready, then it goes back and re-checks which processes are
> ready -- and in your scenario I think it again sees that the first
> process is ready.

This is what we assumed. Emacs could check the second process the next
time. This way one may get a slightly more fair behavior. It would
certainly not be perfect and you could throw scenarios at it which would
make it behave unexpectedly. It may behave a bit more expectedly in the
common case?

> I suggest to read the code of wait_reading_process_output, it has some
> non-trivial logic in this department.

I will do that. Has this problem discussed before?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 16:10:18 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 12:10:18 2021
Received: from localhost ([127.0.0.1]:59677 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcVic-0001A8-Bk
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 12:10:18 -0400
Received: from chiru.no ([142.4.209.132]:51768)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <jakanakaevangeli@HIDDEN>) id 1lcVia-00019x-GN
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 12:10:16 -0400
Received: from localhost (BSN-77-156-43.static.siol.net [193.77.156.43])
 by chiru.no (Postfix) with ESMTPSA id EF8EE128001B;
 Fri, 30 Apr 2021 16:10:14 +0000 (UTC)
From: jakanakaevangeli@HIDDEN
To: Daniel Mendler <mail@HIDDEN>
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
In-Reply-To: <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
 (message from Daniel Mendler on Fri, 30 Apr 2021 17:39:35 +0200)
Date: Fri, 30 Apr 2021 18:15:16 +0200
Message-ID: <87pmyb6963.fsf@miha-pc>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 48118
Cc: eliz@HIDDEN, 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Excuse me if I'm mistaken, but doesn't pselect return all file
descriptors that are ready for reading? Couldn't we then simply process
all of them (in a for loop) after pselect returns?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 15:58:24 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 11:58:23 2021
Received: from localhost ([127.0.0.1]:59645 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcVX5-00078M-J8
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:58:23 -0400
Received: from eggs.gnu.org ([209.51.188.92]:43130)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcVX4-000786-F3
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:58:22 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:50072)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcVWy-0003r3-HC; Fri, 30 Apr 2021 11:58:16 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2799
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcVWw-0003Lm-1u; Fri, 30 Apr 2021 11:58:14 -0400
Date: Fri, 30 Apr 2021 18:58:06 +0300
Message-Id: <83bl9vbw8h.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 17:39:35 +0200)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
 <83o8dvbyyz.fsf@HIDDEN>
 <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48118 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 17:39:35 +0200
> 
> >> Which scenarios break?
> > 
> > For example, if the filter function call accept-process-output.  Or
> > does anything else that changes output from which processes is or
> > isn't available.
> 
> Does this necessarily prevent scheduling? I interpret
> `accept-process-output` as a function which prioritizes a process, but I
> am unsure if this makes it impossible to implement additional scheduling.

A call to accept-process-output prioritizes a process only if it
explicitly requests output from that single process.  Which is not
necessarily true in all cases.

> > What does this mean, exactly?  Which quantity should be doled in a
> > round-robin fashion? bytes read from the processes? something else?
> > 
> > If the bytes read, then how do you suggest to handle two processes
> > which produce output at very different rates?
> 
> For example bytes read or time spent to handle a process (time spent in
> the filter function?).

Bytes read has a problem when processes produce output a very
different rates.  Time spent to handle may (and usually does) mean the
filter function does something expensive, it doesn't necessarily tell
anything about the output from the subprocess.

> When I stumbled over this issue, it astonished me that Emacs
> does not seem to do any scheduling at all and handles only a single
> process.

If you read the code, you will see this isn't what happens.  What
happens is that Emacs reads a chunk of output from the first process
it sees ready, then it goes back and re-checks which processes are
ready -- and in your scenario I think it again sees that the first
process is ready.

> What is the reason for the current behavior? Is it predictability? If I
> understand correctly, Emacs always reads from the first process. If data
> arrives, Emacs does not read from the second processes at all. Only if
> no data is available from the first process, the second process is
> handled. Is it like this?

In your scenario, yes.  It depends on how large is the output produced
by a process in one go.

I suggest to read the code of wait_reading_process_output, it has some
non-trivial logic in this department.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 15:39:45 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 11:39:45 2021
Received: from localhost ([127.0.0.1]:59593 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcVF3-0004VE-C1
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:39:45 -0400
Received: from server.qxqx.de ([178.63.65.180]:58055 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcVF1-0004V2-No
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 11:39:44 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=HbuKfXyGekFq6RPyNyjMxFCIqaNyGBsrZq5fgVrolhI=; b=NKOjDS4LLMXRMmOmICwoRBO/e8
 jiDPl5UwK7Cu62C4DqRc06a3sIxw/tBbUEiEz43ulpiP4bS9hvM8CQARndUtTRhR1gY0SRiV3gtrF
 VMII/tOHxRPbHA2F0GzyN64qJ0lqdX5kiuSqwXp206bCiEUu/kvtNv8VfQKv7NXSqh/4=;
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
To: Eli Zaretskii <eliz@HIDDEN>
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
 <83o8dvbyyz.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <a190b936-eb10-c9ab-89b1-cbb8bd969650@HIDDEN>
Date: Fri, 30 Apr 2021 17:39:35 +0200
MIME-Version: 1.0
In-Reply-To: <83o8dvbyyz.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 4/30/21 4:59 PM, Eli Zaretskii wrote:
> That's not what I meant.  I meant that if your Lisp program launches a
> subprocess that is known to spill huge amounts of output at high rate,
> and you don't want to starve other subprocesses, your filter function
> can stop the process from time to time to give others an opportunity
> to have their outputs read.

This is true. However I expect Emacs to do that for me. I see the
situation like this - I have multiple packages creating processes and
competing for runtime. Then I expect Emacs to take some kind of
preemptive role and ensure that none of the processes misbehaves.
However one can also take the standpoint that each process should try to
behave well by itself and should be scheduled explicitly; Emacs should
stay out of this business.

>>>> since the underlying Emacs handling of asynchronous processes is
>>>> unable to read from two processes at once?
>>>
>>> No.  The problem is not the _ability_ to read from more than one
>>> subprocess -- the ability does exist.  The problem is that doing so
>>> would run afoul of other scenarios.
>>
>> Which scenarios break?
> 
> For example, if the filter function call accept-process-output.  Or
> does anything else that changes output from which processes is or
> isn't available.

Does this necessarily prevent scheduling? I interpret
`accept-process-output` as a function which prioritizes a process, but I
am unsure if this makes it impossible to implement additional scheduling.

> What does this mean, exactly?  Which quantity should be doled in a
> round-robin fashion? bytes read from the processes? something else?
> 
> If the bytes read, then how do you suggest to handle two processes
> which produce output at very different rates?

For example bytes read or time spent to handle a process (time spent in
the filter function?). If a process has eaten up its time it has to wait
until it gets scheduled next. If you have two process with very
different rates, the slow process may not use up its allotted time slot
and the faster process is still allowed to run.

>> I am not happy with the argument that Emacs cannot do any better than
>> stopping the second process and only handle the first process.
> 
> I'm not saying that Emacs cannot do that, I'm trying to understand
> what that would mean in practice.

Actually I would also like to understand what the best process handling
looks like. When I stumbled over this issue, it astonished me that Emacs
does not seem to do any scheduling at all and handles only a single
process. As far as I know other language runtimes handle this problem
differently, attempting some kind of scheduling.

What is the reason for the current behavior? Is it predictability? If I
understand correctly, Emacs always reads from the first process. If data
arrives, Emacs does not read from the second processes at all. Only if
no data is available from the first process, the second process is
handled. Is it like this?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:59:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:59:17 2021
Received: from localhost ([127.0.0.1]:59526 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcUbt-0003UI-9P
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:59:17 -0400
Received: from eggs.gnu.org ([209.51.188.92]:57414)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcUbr-0003U6-Ne
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:59:16 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48706)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcUbl-0006HR-MD; Fri, 30 Apr 2021 10:59:09 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:3162
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcUbk-00058W-LT; Fri, 30 Apr 2021 10:59:09 -0400
Date: Fri, 30 Apr 2021 17:59:00 +0300
Message-Id: <83o8dvbyyz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 16:45:25 +0200)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
 <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48118 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 16:45:25 +0200
> 
> On 4/30/21 4:34 PM, Eli Zaretskii wrote:
> >> So you say I should repeatedly stop the current process in the filter
> >> function in order to allow the other process to take precedence
> > 
> > Yes.
> 
> This is not a good solution. What if I have multiple packages which read
> from asynchronous processes? Maybe I cannot control all of the processes
> and their scheduling.

That's not what I meant.  I meant that if your Lisp program launches a
subprocess that is known to spill huge amounts of output at high rate,
and you don't want to starve other subprocesses, your filter function
can stop the process from time to time to give others an opportunity
to have their outputs read.

> >> since the underlying Emacs handling of asynchronous processes is
> >> unable to read from two processes at once?
> > 
> > No.  The problem is not the _ability_ to read from more than one
> > subprocess -- the ability does exist.  The problem is that doing so
> > would run afoul of other scenarios.
> 
> Which scenarios break?

For example, if the filter function call accept-process-output.  Or
does anything else that changes output from which processes is or
isn't available.

> > But the general answer to your question is that Emacs knows nothing
> > about the processes, their importance, their output rates, and the
> > respective filter functions.
> 
> Okay good. How can I configure it such that two processes both populate
> their buffers in a round-robin fashion?

What does this mean, exactly?  Which quantity should be doled in a
round-robin fashion? bytes read from the processes? something else?

If the bytes read, then how do you suggest to handle two processes
which produce output at very different rates?

> I am not happy with the argument that Emacs cannot do any better than
> stopping the second process and only handle the first process.

I'm not saying that Emacs cannot do that, I'm trying to understand
what that would mean in practice.

> If you don't want to hardcode the scheduling behavior there could be
> some pluggable scheduler. This would be better than having to write my
> own scheduling by hand for each `make-process` call.

Please hold your horses, you are getting too far ahead of the
discussion.  I asked those questions for a reason: I think we cannot
make any meaningful progress without answering them first.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:45:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:45:41 2021
Received: from localhost ([127.0.0.1]:59510 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcUOd-0003AD-5Z
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:45:41 -0400
Received: from server.qxqx.de ([178.63.65.180]:56285 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcUOb-0003A0-5l
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:45:33 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=cT+XHcynz2B8BbxLZsA6qcxV6cPuLkkwHKdjDp3KpzY=; b=JmxbkwzxHNg1FCSeCUDJaruoW5
 hPz+UCPgojxgA5YE6LWPTMUWCuQ+ClOevu4CzGGUGM5qVxZ3FLLNlMRBn1INFOyQCoDH4XybsYEoT
 ulAm1CHyUnpgoqTH3dmpqxC3RSwtmTEu2sdBZ+YDjWBPjkHzTz6vUmaAn8NaNUodN6E8=;
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
To: Eli Zaretskii <eliz@HIDDEN>
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
 <83pmybc03l.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <c2e637e2-e855-cfb8-b2eb-ff5105428f63@HIDDEN>
Date: Fri, 30 Apr 2021 16:45:25 +0200
MIME-Version: 1.0
In-Reply-To: <83pmybc03l.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

On 4/30/21 4:34 PM, Eli Zaretskii wrote:
>> So you say I should repeatedly stop the current process in the filter
>> function in order to allow the other process to take precedence
> 
> Yes.

This is not a good solution. What if I have multiple packages which read
from asynchronous processes? Maybe I cannot control all of the processes
and their scheduling.

>> since the underlying Emacs handling of asynchronous processes is
>> unable to read from two processes at once?
> 
> No.  The problem is not the _ability_ to read from more than one
> subprocess -- the ability does exist.  The problem is that doing so
> would run afoul of other scenarios.

Which scenarios break?

>> me. What is preventing Emacs from treating multiple processes
>> fairly?
> 
> I asked elsewhere what you mean by "fairly" in this context.
> 
> But the general answer to your question is that Emacs knows nothing
> about the processes, their importance, their output rates, and the
> respective filter functions.

Okay good. How can I configure it such that two processes both populate
their buffers in a round-robin fashion?

> First, what does "fairness" mean in this context?  Given that there
> are multiple simultaneous asynchronous subprocesses that produce
> output at different rates and consume CPU at different levels, what
> would it mean for Emacs to be "fair"?
>
> Second, suppose we have multiple ripgrep subprocesses running, and
> Emacs will somehow read from each one of them in a round-robin
> fashion: what and how do you expect the user to do to handle the
> results of all those subprocesses simultaneously and "fairly"?

I agree with you that fairness is a difficult problem. But the problem
is omnipresent at the os level. There you have scheduling problems in
the io layer, in the process scheduling layer, in the memory management
layer and so on. There is certainly some heuristic that one can apply.
For example by comparing the amount of data produced by multiple
processes one could decide which process is read next. Or one can use a
deadline criterion.

I am not happy with the argument that Emacs cannot do any better than
stopping the second process and only handle the first process.

If you don't want to hardcode the scheduling behavior there could be
some pluggable scheduler. This would be better than having to write my
own scheduling by hand for each `make-process` call.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:34:56 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:34:56 2021
Received: from localhost ([127.0.0.1]:59427 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcUEK-0002pe-Li
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:34:56 -0400
Received: from eggs.gnu.org ([209.51.188.92]:50884)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcUEJ-0002pS-Go
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:34:56 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:48002)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcUEE-0008V9-9n; Fri, 30 Apr 2021 10:34:50 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1660
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcUE7-0005IV-OE; Fri, 30 Apr 2021 10:34:47 -0400
Date: Fri, 30 Apr 2021 17:34:38 +0300
Message-Id: <83pmybc03l.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 16:30:54 +0200)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
 <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48118 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 16:30:54 +0200
> 
> So you say I should repeatedly stop the current process in the filter
> function in order to allow the other process to take precedence

Yes.

> since the underlying Emacs handling of asynchronous processes is
> unable to read from two processes at once?

No.  The problem is not the _ability_ to read from more than one
subprocess -- the ability does exist.  The problem is that doing so
would run afoul of other scenarios.

> me. What is preventing Emacs from treating multiple processes
> fairly?

I asked elsewhere what you mean by "fairly" in this context.

But the general answer to your question is that Emacs knows nothing
about the processes, their importance, their output rates, and the
respective filter functions.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:31:25 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:31:25 2021
Received: from localhost ([127.0.0.1]:59420 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcUAv-0002kc-4k
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:31:25 -0400
Received: from eggs.gnu.org ([209.51.188.92]:50088)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcUAt-0002kP-Fp
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:31:23 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47919)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcUAn-00074q-WF; Fri, 30 Apr 2021 10:31:18 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1448
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcUAm-0007k2-9y; Fri, 30 Apr 2021 10:31:17 -0400
Date: Fri, 30 Apr 2021 17:31:09 +0300
Message-Id: <83sg37c09e.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <09261dcc-92e6-0c4e-e692-7e48979b6b91@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 16:23:34 +0200)
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN>
 <09261dcc-92e6-0c4e-e692-7e48979b6b91@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Cc: 48118 <at> debbugs.gnu.org
> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 16:23:34 +0200
> 
> > Why can't you call stop-process from your sentinel function(s) to
> > avoid the problem?
> 
> What do you mean? I don't want to stop the processes. I want to have
> them running asynchronously and concurrently and Emacs should receive
> the incoming data of both processes. When I have to stop the processes
> the benefit of running the processes asynchronously is lost.

So let's talk about this in more detail, okay?

First, what does "fairness" mean in this context?  Given that there
are multiple simultaneous asynchronous subprocesses that produce
output at different rates and consume CPU at different levels, what
would it mean for Emacs to be "fair"?

Second, suppose we have multiple ripgrep subprocesses running, and
Emacs will somehow read from each one of them in a round-robin
fashion: what and how do you expect the user to do to handle the
results of all those subprocesses simultaneously and "fairly"?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:31:05 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:31:05 2021
Received: from localhost ([127.0.0.1]:59417 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcUAa-0002k4-Tw
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:31:05 -0400
Received: from server.qxqx.de ([178.63.65.180]:55967 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcUAY-0002jZ-Pj
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:31:03 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=gBXXTrvNlb4x4EiA/ryaHJ59sFoBVDSuHXbIK43HhwY=; b=N5jMQ6Y+D8O/Z8BEarGlWp0abl
 PZrUeqm29bzjXzjNrG6qUHm3uNGCQ00VgtJ4G0s1VJ70jxeoVeHg5MdPRNetMYllEUETfPRdqw00N
 Lg3G87nIlMgMvO71/faVWTOc5yH7ktqCWGMxg+NFSy9UnTiQvQih+yGDXkoSfGDIqu3U=;
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
To: Eli Zaretskii <eliz@HIDDEN>
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN> <83tunnc0hz.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <c9b089f1-d844-9121-6054-3d492c6d2055@HIDDEN>
Date: Fri, 30 Apr 2021 16:30:54 +0200
MIME-Version: 1.0
In-Reply-To: <83tunnc0hz.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 4/30/21 4:26 PM, Eli Zaretskii wrote:
>> Date: Fri, 30 Apr 2021 17:17:58 +0300
>> From: Eli Zaretskii <eliz@HIDDEN>
>> Cc: 48118 <at> debbugs.gnu.org
>>
>> Why can't you call stop-process from your sentinel function(s) to
>> avoid the problem?
> 
> Sorry, I meant filter functions, of course.

So you say I should repeatedly stop the current process in the filter
function in order to allow the other process to take precedence, since
the underlying Emacs handling of asynchronous processes is unable to
read from two processes at once? This does not sound like a good
solution to me. What is preventing Emacs from treating multiple
processes fairly?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:26:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:26:17 2021
Received: from localhost ([127.0.0.1]:59405 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcU5w-0002bl-RT
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:26:16 -0400
Received: from eggs.gnu.org ([209.51.188.92]:49192)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcU5w-0002ba-0w
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:26:16 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47848)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcU5q-0003yG-KX; Fri, 30 Apr 2021 10:26:10 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1132
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcU5n-0002if-36; Fri, 30 Apr 2021 10:26:08 -0400
Date: Fri, 30 Apr 2021 17:26:00 +0300
Message-Id: <83tunnc0hz.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: mail@HIDDEN
In-Reply-To: <83wnsjc0vd.fsf@HIDDEN> (message from Eli Zaretskii on Fri, 30
 Apr 2021 17:17:58 +0300)
Subject: Re: bug#48118: 27.1; 28;
 Only first process receives output with multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> Date: Fri, 30 Apr 2021 17:17:58 +0300
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: 48118 <at> debbugs.gnu.org
> 
> Why can't you call stop-process from your sentinel function(s) to
> avoid the problem?

Sorry, I meant filter functions, of course.




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:23:47 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:23:47 2021
Received: from localhost ([127.0.0.1]:59401 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcU3X-0002Xy-A3
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:23:47 -0400
Received: from server.qxqx.de ([178.63.65.180]:53853 helo=mail.qxqx.de)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcU3S-0002Xh-JG
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:23:45 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:
 MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:
 Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:
 Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:
 List-Subscribe:List-Post:List-Owner:List-Archive;
 bh=b8u7W6cdQ4dh5joQ7VGuK2gocpqHKotcit3LRfn+u6o=; b=puXH0wSb6tkbW+xPuUEQJtv/wy
 tiWl9dGCiPGkVMJe6VrfvgCHOfevtZPY9THarl3PY9MDbzLbUdyXX72UBzFUTmJqp0wuu3e8Bjcs5
 z40xMMAbmPJLfgN3NUm+vMCcB31waUydK3Dcfv8eUcdIvLuWvoW5oc4VhPLD+YqXfyWU=;
Subject: Re: bug#48118: 27.1; 28; Only first process receives output with
 multiple running processes
To: Eli Zaretskii <eliz@HIDDEN>
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
 <83wnsjc0vd.fsf@HIDDEN>
From: Daniel Mendler <mail@HIDDEN>
Message-ID: <09261dcc-92e6-0c4e-e692-7e48979b6b91@HIDDEN>
Date: Fri, 30 Apr 2021 16:23:34 +0200
MIME-Version: 1.0
In-Reply-To: <83wnsjc0vd.fsf@HIDDEN>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

On 4/30/21 4:17 PM, Eli Zaretskii wrote:
>> When running multiple asynchronous processes only the output of the
>> first process is handled. This happens when the first process
>> continously produces a huge amount of output, for example when running
>> `ripgrep` as done by my `consult-ripgrep` command (part of my Consult
>> package). Then Emacs is stuck handling the output of the first process.
>> The output of the second process is not read until the first process is
>> terminated. I expect Emacs to treat the running processes fairly.
> 
> Why can't you call stop-process from your sentinel function(s) to
> avoid the problem?

What do you mean? I don't want to stop the processes. I want to have
them running asynchronously and concurrently and Emacs should receive
the incoming data of both processes. When I have to stop the processes
the benefit of running the processes asynchronously is lost.

------
I forgot to append the system information to the report.

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5,
cairo version 1.16.0)
 of 2021-02-09, modified by Debian built on 3df710f593d9
Repository revision: b0229d4bbaea7fcddffced393512c650212830db
Repository branch: deb/emacs/d/sid/master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2 -fdebug-prefix-map=/emacs/emacs=.
 -fstack-protector-strong -Wformat -Werror=format-security -Wall'
 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at 48118 <at> debbugs.gnu.org:


Received: (at 48118) by debbugs.gnu.org; 30 Apr 2021 14:18:21 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 10:18:21 2021
Received: from localhost ([127.0.0.1]:59385 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcTyE-0002P9-LW
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:18:21 -0400
Received: from eggs.gnu.org ([209.51.188.92]:47310)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1lcTy9-0002Ou-Ng
 for 48118 <at> debbugs.gnu.org; Fri, 30 Apr 2021 10:18:17 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:47704)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <eliz@HIDDEN>)
 id 1lcTy3-0007zZ-PE; Fri, 30 Apr 2021 10:18:07 -0400
Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4618
 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1lcTy3-00020H-5h; Fri, 30 Apr 2021 10:18:07 -0400
Date: Fri, 30 Apr 2021 17:17:58 +0300
Message-Id: <83wnsjc0vd.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
To: Daniel Mendler <mail@HIDDEN>
In-Reply-To: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN> (message
 from Daniel Mendler on Fri, 30 Apr 2021 15:44:17 +0200)
Subject: Re: bug#48118: 27.1; 28;
 Only first process receives output with multiple running processes
References: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 48118
Cc: 48118 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

> From: Daniel Mendler <mail@HIDDEN>
> Date: Fri, 30 Apr 2021 15:44:17 +0200
> 
> When running multiple asynchronous processes only the output of the
> first process is handled. This happens when the first process
> continously produces a huge amount of output, for example when running
> `ripgrep` as done by my `consult-ripgrep` command (part of my Consult
> package). Then Emacs is stuck handling the output of the first process.
> The output of the second process is not read until the first process is
> terminated. I expect Emacs to treat the running processes fairly.

Why can't you call stop-process from your sentinel function(s) to
avoid the problem?




Information forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 30 Apr 2021 13:44:29 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 30 09:44:29 2021
Received: from localhost ([127.0.0.1]:57946 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1lcTRV-00017d-3D
	for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 09:44:29 -0400
Received: from lists.gnu.org ([209.51.188.17]:54342)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <mail@HIDDEN>) id 1lcTRT-00017W-MV
 for submit <at> debbugs.gnu.org; Fri, 30 Apr 2021 09:44:27 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:46090)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>)
 id 1lcTRS-0007zY-Cu
 for bug-gnu-emacs@HIDDEN; Fri, 30 Apr 2021 09:44:27 -0400
Received: from server.qxqx.de ([2a01:4f8:121:346::180]:39089 helo=mail.qxqx.de)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <mail@HIDDEN>)
 id 1lcTRO-0004gw-Dj
 for bug-gnu-emacs@HIDDEN; Fri, 30 Apr 2021 09:44:25 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=qxqx.de;
 s=mail1392553390; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date:
 Message-ID:Subject:From:To:Sender:Reply-To:Cc:Content-ID:Content-Description:
 Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:
 In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:
 List-Post:List-Owner:List-Archive;
 bh=R7d1fHyr7lOtHjV0iDEA1AceXEz+str+c4Ia5aNhYtM=; b=ITmDZfGvaEQUyMEnSFMMFHAKZ0
 dfNZ7mP/Y/0cZjjlbjuAbISbDKOZK6CDH4bsR0q7/HI8zCare4RshPqYJVNe+Tfz7YBw76Jr/DZM4
 V6EbhekCfJPeyTwXdJMmnkNQlKIu5jsPphCsiADh4AGD3xXOxiwHR1X0i7YVNIr+vmr4=;
To: bug-gnu-emacs@HIDDEN
From: Daniel Mendler <mail@HIDDEN>
Subject: 27.1; 28; Only first process receives output with multiple running
 processes
Message-ID: <64c194f9-b984-adaa-d5fd-86aa3ed3833a@HIDDEN>
Date: Fri, 30 Apr 2021 15:44:17 +0200
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=2a01:4f8:121:346::180;
 envelope-from=mail@HIDDEN; helo=mail.qxqx.de
X-Spam_score_int: -41
X-Spam_score: -4.2
X-Spam_bar: ----
X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)

When running multiple asynchronous processes only the output of the
first process is handled. This happens when the first process
continously produces a huge amount of output, for example when running
`ripgrep` as done by my `consult-ripgrep` command (part of my Consult
package). Then Emacs is stuck handling the output of the first process.
The output of the second process is not read until the first process is
terminated. I expect Emacs to treat the running processes fairly. The
issue also occurs if a :filter function is specified. Both Emacs 27 and
28 are affected.

Minimal reproducible example by @jakanakaevangeli:

(progn
  (setq pa (make-process
             :name "yes-a"
             :command '("yes")
             :connection-type 'pipe
             :buffer (setq a (generate-new-buffer " *a*"))))
  (setq pb (make-process
             :name "yes-b"
             :command '("yes")
             :connection-type 'pipe
             :buffer (setq b (generate-new-buffer " *b*"))))
  (run-at-time
   1 1
   (lambda ()
     (message "size a: %s\nsize b: %s"
              (buffer-size a) (buffer-size b)))))

Original bug discussions:
https://github.com/minad/consult/issues/272
https://github.com/minad/consult/pull/297




Acknowledgement sent to Daniel Mendler <mail@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs@HIDDEN. Full text available.
Report forwarded to bug-gnu-emacs@HIDDEN:
bug#48118; Package emacs. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Sun, 2 May 2021 07:30:01 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.