GNU logs - #33018, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Thu, 11 Oct 2018 14:59:01 +0000
Resent-Message-ID: <handler.33018.B.15392698969565 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 33018 <at> debbugs.gnu.org
From: "Basil L. Contovounesios" <contovob@HIDDEN>
Message-ID: <87r2gwbmma.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
Date: Thu, 11 Oct 2018 15:57:50 +0100
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

--=-=-=
Content-Type: application/emacs-lisp
Content-Disposition: attachment; filename=test.el
Content-Transfer-Encoding: quoted-printable

;;; test.el --- test -*- lexical-binding: t -*-

;;; Code:

(defun test-debug (&rest args)
  "Print `current-thread' and ARGS in timestamped `message'."
  (push (current-thread) args)
  (message "[%s] %s"
           (format-time-string "%T:%3N")
           (mapconcat #'prin1-to-string args " ")))

(defun test-sentinel (proc msg)
  "Kill buffer of PROC when it exits."
  (test-debug proc 'sentinel (process-status proc) (substring msg 0 -1))
  (and (not (eq (process-status proc) 'run))
       (buffer-live-p (process-buffer proc))
       (kill-buffer   (process-buffer proc))))

(defun test-slave ()
  "Create an asynchronous process and wait for it to exit."
  (let ((proc (make-process
               :name "test"
               :buffer (generate-new-buffer " *test*")
               :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emac=
s")
               :connection-type 'pipe
               :sentinel #'test-sentinel)))
    (while (eq (process-status proc) 'run)
      (test-debug proc 'accept-output (accept-process-output proc 5)))
    (test-debug proc 'exit (process-status proc) (process-exit-status proc)=
)))

(defun test-master (async)
  "Successively run `test-slave'.
With non-nil ASYNC, use a new thread for each invocation."
  (dotimes (i 5)
    (if async
        (thread-join (make-thread #'test-slave (format "test-%d" i)))
      (test-slave))))

(defun test-job (async)
  "Successively run `test-slave'.
Like `test-master', but avoid blocking interactive sessions by
running `test-master' in its own thread."
  (if noninteractive
      (test-master async)
    (make-thread (apply-partially #'test-master async) "test-master")))

(defun test-threads ()
  "Run `test-master' with threads."
  (test-job t))

(defun test-no-threads ()
  "Run `test-master' without threads."
  (test-job nil))

;;; test.el ends here

--=-=-=
Content-Type: text/plain


I attach a sample program test.el whose central function, test-slave,
invokes wget asynchronously before waiting for the process to exit.

The issue I'm facing is that running test-slave twice in succession,
each time in a new thread, causes accept-process-output to hang with no
output (unless a timeout argument is given, in which case the function
returns nil) the second time around.

When this happens, the process sentinel is never called, which is why
I'm assuming accept-process-output is indeed "hanging" in some sense and
it's not just that the process has already exited and so has no further
output.

I could very well be doing or assuming something incorrectly, but what
baffles me is that the "hang" does not occur either when Emacs is run
non-interactively, or when "https://en.wikipedia.org/wiki/Emacs" is
replaced with "https://www.gnu.org/software/emacs/", or when test-slave
is run in the current thread (and not in make-thread).

Since I can reliably reproduce this on both an optimised build of master
and a non-optimised build of emacs-26, I hope to be able to provide
further insights using gdb as time allows.  Please let me know if there
are any specific details/output you would like me to provide.  As a
relatively inexperienced gdb user I welcome any tips and tricks for
debugging threads and processes.

Here are some ways test.el can be run to illustrate the issue:

# All five processes exit successfully.
emacs -batch -l test.el -f test-no-threads

# All five processes (and threads) exit successfully.
emacs -batch -l test.el -f test-threads

# All five processes exit successfully.
emacs -Q -l test.el -f test-no-threads

# First process and thread exit successfully,
# but accept-process-output starts timing out in second thread.
# Warning: may leave empty wget-log files lying around.
emacs -Q -l test.el -f test-threads

Details of the two Emacs versions I'm using follow:

In GNU Emacs 26.1.50 (build 2, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2018-10-11 built on thunk
Repository revision: a7ebc6bf633bd3849ccab032dad6b1fd31b1ef43
Windowing system distributor 'The X.Org Foundation', version 11.0.12001000
System Description:	Debian GNU/Linux testing (buster)

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4 -pipe'
 --config-cache --prefix=/home/blc/.local --program-suffix=26
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 --with-mailutils --with-x-toolkit=lucid --with-modules
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
LCMS2

In GNU Emacs 27.0.50 (build 21, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2018-10-11 built on thunk
Repository revision: 5bd8cfc14d4b0c78c07e65a583f42a10c4cbc06d
Windowing system distributor 'The X.Org Foundation', version 11.0.12001000
System Description: Debian GNU/Linux buster/sid

Configured using:
 'configure --config-cache --prefix=/home/blc/.local --with-mailutils
 --with-x-toolkit=lucid --with-modules --with-file-notification=yes
 --with-x 'CC=ccache gcc' 'CFLAGS=-O2 -march=native -pipe''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON LCMS2 GMP

Thanks,

-- 
Basil

--=-=-=--

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Subject: bug#33018: Acknowledgement (26.1.50; thread starvation with async
 processes and accept-process-output)
Message-ID: <handler.33018.B.15392698969565.ack <at> debbugs.gnu.org>
References: <87r2gwbmma.fsf@HIDDEN>
X-Gnu-PR-Message: ack 33018
X-Gnu-PR-Package: emacs
Reply-To: 33018 <at> debbugs.gnu.org
Date: Thu, 11 Oct 2018 14:59:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 33018 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
33018: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33018
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 12 Oct 2018 08:08:02 +0000
Resent-Message-ID: <handler.33018.B33018.153933166622327 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Date: Fri, 12 Oct 2018 11:07:21 +0300
Message-Id: <83o9bzwpty.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <87r2gwbmma.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN>
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Date: Thu, 11 Oct 2018 15:57:50 +0100
> 
> I attach a sample program test.el whose central function, test-slave,
> invokes wget asynchronously before waiting for the process to exit.
> 
> The issue I'm facing is that running test-slave twice in succession,
> each time in a new thread, causes accept-process-output to hang with no
> output (unless a timeout argument is given, in which case the function
> returns nil) the second time around.

When the hang happens, is there any wget process still alive, or did
they all exit?  Please use OS tools to find that out, don't rely on
what Emacs thinks.

> I could very well be doing or assuming something incorrectly, but what
> baffles me is that the "hang" does not occur either when Emacs is run
> non-interactively, or when "https://en.wikipedia.org/wiki/Emacs" is
> replaced with "https://www.gnu.org/software/emacs/"

Could be different properties of the servers related to async
connections, like TLS handshake or even async getaddrinfo.

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 12 Oct 2018 12:03:01 +0000
Resent-Message-ID: <handler.33018.B33018.153934577627010 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
From: Michael Albinus <michael.albinus@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN>
Date: Fri, 12 Oct 2018 14:02:46 +0200
In-Reply-To: <87r2gwbmma.fsf@HIDDEN> (Basil L. Contovounesios's message of
 "Thu, 11 Oct 2018 15:57:50 +0100")
Message-ID: <875zy7cqzd.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

"Basil L. Contovounesios" <contovob@HIDDEN> writes:

Hi Basil,

> I attach a sample program test.el whose central function, test-slave,
> invokes wget asynchronously before waiting for the process to exit.
>
> The issue I'm facing is that running test-slave twice in succession,
> each time in a new thread, causes accept-process-output to hang with no
> output (unless a timeout argument is given, in which case the function
> returns nil) the second time around.

If you want a process to communicate in a given thread, you must call
`set-process-thread'. See the elisp manual.

> Thanks,

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 12 Oct 2018 12:44:01 +0000
Resent-Message-ID: <handler.33018.B33018.153934823930720 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: contovob@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153934823930720
          (code B ref 33018); Fri, 12 Oct 2018 12:44:01 +0000
Received: (at 33018) by debbugs.gnu.org; 12 Oct 2018 12:43:59 +0000
Received: from localhost ([127.0.0.1]:45994 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gAwnP-0007zQ-7D
	for submit <at> debbugs.gnu.org; Fri, 12 Oct 2018 08:43:59 -0400
Received: from eggs.gnu.org ([208.118.235.92]:41585)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gAwnN-0007zB-Bd
 for 33018 <at> debbugs.gnu.org; Fri, 12 Oct 2018 08:43:57 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gAwnG-0005xL-7r
 for 33018 <at> debbugs.gnu.org; Fri, 12 Oct 2018 08:43:52 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:34923)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gAwnE-0005vR-VC; Fri, 12 Oct 2018 08:43:49 -0400
Received: from [176.228.60.248] (port=1531 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gAwnE-0000kG-9p; Fri, 12 Oct 2018 08:43:48 -0400
Date: Fri, 12 Oct 2018 15:43:51 +0300
Message-Id: <83d0sfwd14.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <875zy7cqzd.fsf@HIDDEN> (message from Michael Albinus on Fri, 12
 Oct 2018 14:02:46 +0200)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: Michael Albinus <michael.albinus@HIDDEN>
> Date: Fri, 12 Oct 2018 14:02:46 +0200
> Cc: 33018 <at> debbugs.gnu.org
> 
> If you want a process to communicate in a given thread, you must call
> `set-process-thread'. See the elisp manual.

But the default is that the process is locked to the thread that
created it, so it sounds like this should have just worked (if that is
the problem).

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 12 Oct 2018 12:50:01 +0000
Resent-Message-ID: <handler.33018.B33018.153934857731263 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: contovob@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153934857731263
          (code B ref 33018); Fri, 12 Oct 2018 12:50:01 +0000
Received: (at 33018) by debbugs.gnu.org; 12 Oct 2018 12:49:37 +0000
Received: from localhost ([127.0.0.1]:46002 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gAwsr-00088B-4x
	for submit <at> debbugs.gnu.org; Fri, 12 Oct 2018 08:49:37 -0400
Received: from mout.gmx.net ([212.227.17.22]:54245)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1gAwsp-00087y-0v
 for 33018 <at> debbugs.gnu.org; Fri, 12 Oct 2018 08:49:35 -0400
Received: from detlef.gmx.de ([212.91.243.207]) by mail.gmx.com (mrgmx103
 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MHXd2-1gEFCd1E3Y-003Lkm; Fri, 12
 Oct 2018 14:49:27 +0200
Received: from detlef.gmx.de ([212.91.243.207]) by mail.gmx.com (mrgmx103
 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MHXd2-1gEFCd1E3Y-003Lkm; Fri, 12
 Oct 2018 14:49:27 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <83d0sfwd14.fsf@HIDDEN>
Date: Fri, 12 Oct 2018 14:49:25 +0200
In-Reply-To: <83d0sfwd14.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 12 Oct
 2018 15:43:51 +0300")
Message-ID: <871s8vcotm.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Provags-ID: V03:K1:6qIHB5ce0VxEcW9PcdryqKVLX7GhWrplb5w9py7gf0kfNPIiv5G
 Cxg/Pekdzdj4/BaZ2tWurb8dYZ8ilTJ9jUUOeVjY8zOPKcgzlt8oUfrFzd4Hb3Uddm0pU0X
 3XpffhSZr40TTAz2MlUnwNeaL7uMECyNIiSPik5pAWEVJII3dbdmQ23J+fUg59zw72c5JHh
 Y5iAmTY2smerFtzKuoK3w==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V01:K0:Hssog6mhvpM=:c2o9KJbHyl9FOo5dyo99XI
 GzuF3YjK+jZa7VtFgW25/MRl8F8LhsdJaiubJQhFBY8FsejaXtItCETazs6dxe2ff6iXbxii+
 lIYBRNnQVHrp061e3D+xzsgJxIzmbyGSqpsMAsHA6FEzGGLZKy/8a8Zh3QgjQYbSMfwfi93CX
 JcerVbLyAzb3bMR5215t4OdVJjgN4gnGGejsLQqRD0XA8sA+qBhTZbfRv4Vq8wzwdOIjRQQNi
 ApVwr2Qo/4HVAhbQHKPe9TkN6fIOWERIOk000SSUCqiWPFdITwZHqcwWqBJg9A5101zJMPHII
 SdrW4Z76MYJTMnlumPKFGQSBvDN9Tkc5H8Y2N0Wj9aYH+CGMoD6QJn0MPkIaf0LGQF8/cztFl
 k1UlM1c0ejfX0Ry44DMyhVS+E+UKPFN+9Ol7ImPQ+26eD3j4oJ26GuwEc2luWe8eM6NWvYzoG
 xKXMVwu+k4/uqZJI8/86wdUk1dGtG1LzYh94LBO4l/wLZtXgqr4hXUgPGPW62L7L6BK1ekEMt
 D33mXy2XOiNqpSRmnxm82WbE7bHTH5Xl/KuqidthuZO4EU56jkKCZbptkgaJcwApFSMiJHjE3
 NpABQ6QyLwFhTqRy/aPRBXUXNgrrwn0i2XoQhvrQslSeEsxD73yrLeyH+gmMpaPTPJDD5Eubc
 Qx4Ke6qBvFlYYqMr4yCE6A6EXFayBXyo4QY5om5kcGugtM1AE4N+7wuv6A2yXjvRFUzcuCp6b
 Lb8SwWtUAaVANIbeVfZJSnUsHVUicStETyfae5+K3PfX+75UEzUKJcr3SuXt5sjRmPZJ7YvoK
 h4ICgi+it+tV83VUcNXj57zuIEiZJguqi3SrcutzwW92BjLH7o=
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> If you want a process to communicate in a given thread, you must call
>> `set-process-thread'. See the elisp manual.
>
> But the default is that the process is locked to the thread that
> created it, so it sounds like this should have just worked (if that is
> the problem).

I'm not sure. In the branch feature/tramp-thread-safe there were also
mysterious blockings in accept-process-output, until I've applied
set-process-thread explicitly.

At least it is worth a try.

Best regards, Michael.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 14 Oct 2018 15:02:01 +0000
Resent-Message-ID: <handler.33018.B33018.1539529270505 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.1539529270505
          (code B ref 33018); Sun, 14 Oct 2018 15:02:01 +0000
Received: (at 33018) by debbugs.gnu.org; 14 Oct 2018 15:01:10 +0000
Received: from localhost ([127.0.0.1]:49436 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gBhtG-000085-Id
	for submit <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:01:10 -0400
Received: from mail-ed1-f41.google.com ([209.85.208.41]:44756)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gBhtE-00007q-Ih
 for 33018 <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:01:09 -0400
Received: by mail-ed1-f41.google.com with SMTP id z21-v6so15489168edb.11
 for <33018 <at> debbugs.gnu.org>; Sun, 14 Oct 2018 08:01:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=e11quNtDda6k3lXSN32F/Ho4Ec37JjL0miEVGKBJXLU=;
 b=JUUrtzBpqsz//VOUrlIJ09jM5NaGEJeLCUdmT6xeB7lN/YSuKvvOtyPuYKrR1A2wzU
 fKnfRA9YMuhWwdetPexRu+zspkTI9b7pAq/0riUZmx01AtbU7SuD/VEp5uinLD1oNOL2
 Es31V7ZnI6vsKCoEHhLIyf0pFkbHOqvM+aV2RdH/qotpCQIYTq4zniIfwFXkuZxa+mof
 cy0Bfgk3nuS1bIoQ+6OZZ4UauKg0kVp6hPf5h5FfAoXHs19RnPWwyU58+Uhv9sZDevmY
 K1g4lTHv5Q86K7gaivMvKIRb5njA3I+b8wRKrD71hNcsk2CGtNaH/+USP1+IzrshzLd4
 zIcA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=e11quNtDda6k3lXSN32F/Ho4Ec37JjL0miEVGKBJXLU=;
 b=MH5hKzUDD/l7Y02xf3FZhupUpImJyA8l9YUQJ45rm6TdTuhnUP8m6pNp2ATcQM23IN
 cIQJ78QV8CDMIIMgK59RgaXz+I8eU/Q4sdxhP7MTWJne/+w5Ug5GfTG1Ci1yPwh3ehnT
 srCtuioFSbFvZvXwLH1EOQrrgPm+IshyLMi32SXA5SfNyQqeCXyFEE3gPhM0RQqGB4Bv
 CIkzk8Vrs1VAT1k5oltP9/nBzfyuzGmHwuffF6IAx2M72tOtBLKmdCyJeotFAcfBqNQb
 NKCDjBNEX1V2627g08oyC/ZhErBatkozmxPdTTtihKKB3wLAVL/BJ6ZkyMBmc0kddjZB
 7GBg==
X-Gm-Message-State: ABuFfoggXfwCJ+ZpP7Y5mcwwxR+aH2M9btrYtR+EU89BDCrTFlvWTuNt
 IiAPurQcNL24Bo/qtnZSFdeOmlqjMNc=
X-Google-Smtp-Source: ACcGV60f2JbHWhaR+rZ7xXvRUCiJnbXWI+AqW6hRC+7CEgJZtltwpwEoKBGut/yIYc+8CKJ/AXABbA==
X-Received: by 2002:a17:906:2ce:: with SMTP id
 14-v6mr15334996ejk.66.1539529262516; 
 Sun, 14 Oct 2018 08:01:02 -0700 (PDT)
Received: from localhost ([213.233.155.142])
 by smtp.gmail.com with ESMTPSA id 7-v6sm1595949ejy.11.2018.10.14.08.01.01
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sun, 14 Oct 2018 08:01:01 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <83o9bzwpty.fsf@HIDDEN>
Date: Sun, 14 Oct 2018 16:00:56 +0100
In-Reply-To: <83o9bzwpty.fsf@HIDDEN> (Eli Zaretskii's message of "Fri, 12 Oct
 2018 11:07:21 +0300")
Message-ID: <87h8hoh8t3.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: "Basil L. Contovounesios" <contovob@HIDDEN>
>> Date: Thu, 11 Oct 2018 15:57:50 +0100
>> 
>> I attach a sample program test.el whose central function, test-slave,
>> invokes wget asynchronously before waiting for the process to exit.
>> 
>> The issue I'm facing is that running test-slave twice in succession,
>> each time in a new thread, causes accept-process-output to hang with no
>> output (unless a timeout argument is given, in which case the function
>> returns nil) the second time around.
>
> When the hang happens, is there any wget process still alive, or did
> they all exit?  Please use OS tools to find that out, don't rely on
> what Emacs thinks.

When the hang happens, the wget process launched by the waiting thread
is still alive but asleep (idle), as reported by ps and top.

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 14 Oct 2018 15:18:02 +0000
Resent-Message-ID: <handler.33018.B33018.15395302401879 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15395302401879
          (code B ref 33018); Sun, 14 Oct 2018 15:18:02 +0000
Received: (at 33018) by debbugs.gnu.org; 14 Oct 2018 15:17:20 +0000
Received: from localhost ([127.0.0.1]:49444 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gBi8u-0000UF-13
	for submit <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:17:20 -0400
Received: from eggs.gnu.org ([208.118.235.92]:51265)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gBi8s-0000Tu-M6
 for 33018 <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:17:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gBi8h-0003eT-Ie
 for 33018 <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:17:11 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:49503)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gBi8f-0003d7-UC; Sun, 14 Oct 2018 11:17:06 -0400
Received: from [176.228.60.248] (port=4262 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gBi8f-0004mu-Cm; Sun, 14 Oct 2018 11:17:05 -0400
Date: Sun, 14 Oct 2018 18:17:13 +0300
Message-Id: <83murgtv5y.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <87h8hoh8t3.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN> <83o9bzwpty.fsf@HIDDEN>
 <87h8hoh8t3.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Cc: <33018 <at> debbugs.gnu.org>
> Date: Sun, 14 Oct 2018 16:00:56 +0100
> 
> > When the hang happens, is there any wget process still alive, or did
> > they all exit?  Please use OS tools to find that out, don't rely on
> > what Emacs thinks.
> 
> When the hang happens, the wget process launched by the waiting thread
> is still alive but asleep (idle), as reported by ps and top.

Any idea why is that?

And if this is the situation, doesn't it explain why
accept-process-output times out?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 14 Oct 2018 15:18:02 +0000
Resent-Message-ID: <handler.33018.B33018.15395302431891 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15395302431891
          (code B ref 33018); Sun, 14 Oct 2018 15:18:02 +0000
Received: (at 33018) by debbugs.gnu.org; 14 Oct 2018 15:17:23 +0000
Received: from localhost ([127.0.0.1]:49446 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gBi8x-0000UR-AT
	for submit <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:17:23 -0400
Received: from mail-ed1-f46.google.com ([209.85.208.46]:45162)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gBi8t-0000Tw-JT
 for 33018 <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:17:19 -0400
Received: by mail-ed1-f46.google.com with SMTP id v18-v6so15520847edq.12
 for <33018 <at> debbugs.gnu.org>; Sun, 14 Oct 2018 08:17:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=XppSNzBezTqqCrMm+3mt6E9JaOGxsiIOA2+sRXzxMa0=;
 b=dd/9zqc34TgwCJ6zyl62zK/bIZA6hRWZc2YbJ9/Mq+ZnlA1Zn1wOyV/jfIiyD9Oikw
 qFh50TIWKWdSh6qN0BmuBqfGxQemF3cIcv6DFou3UImYRcIYFONn6J1JWowRnDKI4QAs
 21cpFwhVyoV1Tp4RBZhfBTbJqL+cG5rdcJk8ZTKGKwBeikOProidQ83aGMVT8DfzVzJN
 VXdLn3n2Ftr/SPlJvZ2ZYl3pfQlqm5yKShZaM3JG3fdLLdvJknHkC9dI4LLaWFl7f5+4
 /A2EZQy5PorEOeRBkF6xjkzXQ7EqyJ4WWiXKgFQG35ENMpHWA/TVRdgjf5OL0qLFi2Pl
 Jm4w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=XppSNzBezTqqCrMm+3mt6E9JaOGxsiIOA2+sRXzxMa0=;
 b=jxejEkiBSuD/h+oQa/dRPDPJPnwJ+HprGdXdZ0bIuuLPAz+ZTtjxblqgPHav6EDIQQ
 KpN9LrboiNsW45qOfzbhu9BJYpmgEiXbt7NL1HBHrs43tFOLlOjGDvP7pvn020g7TDAK
 kIuIzJ4Ox3X+IMIZ9Eht/mFBHThMr9IybcaYkXQe/6GEtORLO2IcBxTsnAgWKRhsTRnI
 uhV2TzMhlacSSlZY0/JRBJYL60M8v8mHRMDLQHPhf8N6kHS7QK0aWC5a+5T245NlN5y1
 nvHHl4hRMFVzV4NiVd3LCOQ1Sj5JPoOaR0OFfNWIFLHFa4IZiKG+aNfRSLJKGcuVqbxR
 n1Mw==
X-Gm-Message-State: ABuFfojMXHm98xbYaeGFsXQIImoDV0OaJdo6t7Hujln7N/O7Ho9c/65R
 cwYVaFPUEq5LkLBe1QjFG7+d1Q==
X-Google-Smtp-Source: ACcGV61Cip2pDn0PgGil7FDoUelUrOTr6NWoZNTAdfV5C8O/aWGvQdM3GpQTGUdUceoRfrYLqYTrzg==
X-Received: by 2002:a50:fc97:: with SMTP id
 f23-v6mr19752846edq.276.1539530233933; 
 Sun, 14 Oct 2018 08:17:13 -0700 (PDT)
Received: from localhost ([213.233.155.133])
 by smtp.gmail.com with ESMTPSA id y47-v6sm3729289edc.27.2018.10.14.08.17.12
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sun, 14 Oct 2018 08:17:13 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
Date: Sun, 14 Oct 2018 16:17:08 +0100
In-Reply-To: <875zy7cqzd.fsf@HIDDEN> (Michael Albinus's message of "Fri, 12
 Oct 2018 14:02:46 +0200")
Message-ID: <878t30h823.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain

Michael Albinus <michael.albinus@HIDDEN> writes:

> "Basil L. Contovounesios" <contovob@HIDDEN> writes:
>
>> I attach a sample program test.el whose central function, test-slave,
>> invokes wget asynchronously before waiting for the process to exit.
>>
>> The issue I'm facing is that running test-slave twice in succession,
>> each time in a new thread, causes accept-process-output to hang with no
>> output (unless a timeout argument is given, in which case the function
>> returns nil) the second time around.
>
> If you want a process to communicate in a given thread, you must call
> `set-process-thread'. See the elisp manual.

Thanks, this is the first thing I tried when earlier experiments started
to hang.  I tried both the following redundant but explicit call:


--=-=-=
Content-Type: text/x-diff
Content-Disposition: inline; filename=test.diff

diff -u --label /tmp/test.el --label \#\<buffer\ /tmp/test.el\> /tmp/test.el /tmp/buffer-content-hY0EDf
--- /tmp/test.el
+++ #<buffer /tmp/test.el>
@@ -24,6 +24,7 @@
                :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emacs")
                :connection-type 'pipe
                :sentinel #'test-sentinel)))
+    (set-process-thread proc (current-thread))
     (while (eq (process-status proc) 'run)
       (test-debug proc 'accept-output (accept-process-output proc 5)))
     (test-debug proc 'exit (process-status proc) (process-exit-status proc))))

Diff finished.  Sun Oct 14 16:04:48 2018

--=-=-=
Content-Type: text/plain


as well as replacing (current-thread) with nil, to unlock the process.

Adding test-debug calls before and after accept-process-output revealed
nothing out of the ordinary, and explicitly un/locking the process
didn't fix the hang.  Should I be calling set-process-thread elsewhere?

-- 
Basil

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sun, 14 Oct 2018 15:37:02 +0000
Resent-Message-ID: <handler.33018.B33018.15395313893565 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15395313893565
          (code B ref 33018); Sun, 14 Oct 2018 15:37:02 +0000
Received: (at 33018) by debbugs.gnu.org; 14 Oct 2018 15:36:29 +0000
Received: from localhost ([127.0.0.1]:49457 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gBiRR-0000vR-DU
	for submit <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:36:29 -0400
Received: from mail-ed1-f53.google.com ([209.85.208.53]:43236)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gBiRP-0000vE-SE
 for 33018 <at> debbugs.gnu.org; Sun, 14 Oct 2018 11:36:28 -0400
Received: by mail-ed1-f53.google.com with SMTP id y20-v6so15552500eds.10
 for <33018 <at> debbugs.gnu.org>; Sun, 14 Oct 2018 08:36:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=k524uxsgCINAvw2xVFPDeEgrsdBmFBA1jtsOc1M8GPA=;
 b=YN0a++J78QoAE6BI/UwBFEsCGTUqwKXBCQTjcdfKGI5byAFFjq1IffC9O9xVi+mdai
 wUAkRKhrG7GCYf1NL6OnuruIRtuOlC/qpFGsZ04rz39hB7QOIYflTSrWPS1vJTBvNlso
 1cWLtrqujCzFdm/GGscUsxOI83EwIG6tTzgeJqkjhfSg4aupk45tCUibOfV768+ZeNJ8
 T4mYdN7mvpDS3x5QdhEAOw0yzpswmBD1b00tJqX/sTWEoj5dEdS0rZa9sMcQoKOZEKAN
 wRcQbMDQBuOXfC5n8j5j7i7o6bFxMIVWQREvIZwGvvRoYlfbJScOhRBYYJCsEEAm2BiB
 JHmg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=k524uxsgCINAvw2xVFPDeEgrsdBmFBA1jtsOc1M8GPA=;
 b=etcG6AmRtXFVDub8+xj1itWcnfsxU83yigz9I17eagtIfNJgeBDkS8i9Y4wEuKeoQ7
 yAd+hwDkCWUEMRsIxelCFePcyC1HwE5nbWkRc4xhPrq2jnmlxtpGQ88lnPIKsEV4h6Om
 uKH7qeViZziCs/7epcCguBWiE66Mv5WXZYPCTImg3SwOfDEY9MiHoKa+iPa6xN+IoI3D
 Lrp7uYYm49cf8fVrvz7umcOObSw9rB2BG83zAteZG84DNCzpkDv/PQIkg+hPvsS4un9c
 UKqDeRnPCVhzp4FQLd+sz/8La3Mc/M2YA2ru+xZo6GUIXYMfkUWtgKai+3PBcUZo0IMA
 BK7g==
X-Gm-Message-State: ABuFfojuN8Q8ZkJH58YCxwNpeD7HujF3IHwCJ/9nJ06ejifl0u7Mha/G
 z9EOQXY10Iiy69ZKnjLF4gZrCQ==
X-Google-Smtp-Source: ACcGV626s0nJdEbtltq/ur0iRZE4jvr2b8N0G7ejMlP37UeRfogYO/1aazRN73uk32E1wwQqHcvyVg==
X-Received: by 2002:a50:8f64:: with SMTP id
 91-v6mr19959285edy.264.1539531382071; 
 Sun, 14 Oct 2018 08:36:22 -0700 (PDT)
Received: from localhost ([213.233.155.142])
 by smtp.gmail.com with ESMTPSA id l42-v6sm2996352edb.24.2018.10.14.08.36.21
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Sun, 14 Oct 2018 08:36:21 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <83o9bzwpty.fsf@HIDDEN>
 <87h8hoh8t3.fsf@HIDDEN> <83murgtv5y.fsf@HIDDEN>
Date: Sun, 14 Oct 2018 16:36:20 +0100
In-Reply-To: <83murgtv5y.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 14 Oct
 2018 18:17:13 +0300")
Message-ID: <87woqkfsln.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: "Basil L. Contovounesios" <contovob@HIDDEN>
>> Cc: <33018 <at> debbugs.gnu.org>
>> Date: Sun, 14 Oct 2018 16:00:56 +0100
>> 
>> > When the hang happens, is there any wget process still alive, or did
>> > they all exit?  Please use OS tools to find that out, don't rely on
>> > what Emacs thinks.
>> 
>> When the hang happens, the wget process launched by the waiting thread
>> is still alive but asleep (idle), as reported by ps and top.
>
> Any idea why is that?

No idea, sorry.  I haven't yet found the time to look into this deeper.

> And if this is the situation, doesn't it explain why
> accept-process-output times out?

Possibly, but the question is why do we enter either of these situations
(process sleeping and accept-process-output timing out) exclusively when
using Emacs threads, no?

By the way, the problem isn't specific to wget; the same thing happens
with curl.  I also find it strange that the first thread doesn't suffer
from the same problem as the second thread, especially given each thread
is created only after the last thread exited.

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Mon, 15 Oct 2018 08:04:02 +0000
Resent-Message-ID: <handler.33018.B33018.153959058316463 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153959058316463
          (code B ref 33018); Mon, 15 Oct 2018 08:04:02 +0000
Received: (at 33018) by debbugs.gnu.org; 15 Oct 2018 08:03:03 +0000
Received: from localhost ([127.0.0.1]:49754 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gBxqB-0004HT-HO
	for submit <at> debbugs.gnu.org; Mon, 15 Oct 2018 04:03:03 -0400
Received: from mout.gmx.net ([212.227.15.19]:50787)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1gBxq9-0004Gz-WE
 for 33018 <at> debbugs.gnu.org; Mon, 15 Oct 2018 04:03:02 -0400
Received: from detlef.gmx.de ([212.86.50.126]) by mail.gmx.com (mrgmx001
 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M2t0Q-1fKgzg3Ilq-00sih9; Mon, 15
 Oct 2018 10:02:54 +0200
Received: from detlef.gmx.de ([212.86.50.126]) by mail.gmx.com (mrgmx001
 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M2t0Q-1fKgzg3Ilq-00sih9; Mon, 15
 Oct 2018 10:02:54 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN>
Date: Mon, 15 Oct 2018 10:02:53 +0200
In-Reply-To: <878t30h823.fsf@HIDDEN> (Basil L. Contovounesios's message of
 "Sun, 14 Oct 2018 16:17:08 +0100")
Message-ID: <87sh177i36.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Provags-ID: V03:K1:Qpnh7mnRu8gLYavULjdmsgd4ECgrnOZHm9C8C5b+M6d/JBuBEhk
 XCUxQ0MCnsE22ccZT5x4Op3+tH+4boJbEvmTu+S7btIC1/K5xBXffAdIeoNLs6ZnuP4wg62
 cx/aagAArvU+L3yGpxveIN62cjzKuAf1B/Ykzb4mwD+m7TY2ZaBUivI51+OheL0biqRYDbL
 70B4j9905VtdUW0wRTDHg==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V01:K0:Y8pTnWUnAlc=:R+jz4/EE2fW6NMVXGPRDzP
 7qk1hU2VQfKRZNB0vGWfKNW2Gf/cE00NkeSJBP71k4d9t7nECwj4itHB2Aj16aayEaJzkAo//
 8yBJ8saY2bP7bc5eT/XO++LYqHo97MkBmUt+h8Ixu1QGIXj6jWdl/ZALedk7+M0+CNbEuz2+Z
 NxLK8TANIEda8CQKCsbpv7dsasIkn/W5PuyAxIbP3nmAwg9NAFUxA8YHaanuwp0ip4W3zcN5I
 6o6qbv4wlhOWPX7HCjH7dcariD2iMy1d29s7afC9ws2/zvKK3TFld8rBRrZLCiN4aKEC8JAKd
 RUa14sUrrZtQCeS7RROKuMDKDKk/LXJ/TMhoVDXucKLHpjMD1+rS/4R7N42/qb0onnvK7Jmnn
 pW8XTyB95xGG+GoMKBC1FGksEb6pEH/rDcj1xeLQ2o5d8bzxzrlcR8aX8x799DTjF5onu3QOy
 5yqaPHdI2B4/2Q2nmn+g92K91cloSgR2tCYH4338kFEbquDJnOWAEEzw52xHFS5+RiNLy0Jzz
 i+iXPAtKsH5P35jNYCuvGG06urkM3dBCx5171vU0kMNRFDriPiXGoknOK0fcKwmSJJ/5Sc1F4
 poKvcwRrWa8hiVrO8SkfEn5TXWPEcbPooSrZIOMrGkRkkD9D9OwxfKlVcs0S8J5Q28pB7g3ci
 OD3iSkELm0dEtXrIOGwRcWmzrNqxqMsPkQJ5JpEp6v9m0bEXOC1Z7Ou1ubmURp1tyj9PQVOgu
 PgmUchmpl/3yqNvZBHuTWVqK+HkxMkoWv1K6zrJryj15ejbV4imcJcmvvDMjhkzlHZ84TksoB
 ngeFFJzd5bjDxBaJbebpqJQywJcKw==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain

"Basil L. Contovounesios" <contovob@HIDDEN> writes:

Hi Basil,

>> If you want a process to communicate in a given thread, you must call
>> `set-process-thread'. See the elisp manual.
>
> Thanks, this is the first thing I tried when earlier experiments started
> to hang.  I tried both the following redundant but explicit call:

Well, I've played with your example. As Eli said, `set-process-thread'
is not needed here.

With your original example, I could reproduce the problem, However, if I
call

emacs -l test.el -f test-threads

the problem does NOT happen. My .emacs is quite long, so I didn't bisect
in order to find out what makes the difference.

I have changed your example a little bit wrt `thread-join', see
appended. This version runs w/o any problem even if emacs is called with
-Q. Maybe this helps you to debug further.

Best regards, Michael.


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment

;;; test.el --- test -*- lexical-binding: t -*-

;;; Code:

(defun test-debug (proc &rest args)
  "Print `current-thread' and ARGS in timestamped `message'."
  (message
   "[%s] %s"
   (format-time-string "%T:%3N")
   (mapconcat
    #'prin1-to-string `(,(current-thread) ,(process-thread proc) ,@args) " ")))

(defun test-sentinel (proc msg)
  "Kill buffer of PROC when it exits."
  (test-debug proc 'sentinel (process-status proc) (substring msg 0 -1))
  (and (not (eq (process-status proc) 'run))
       (buffer-live-p (process-buffer proc))
       (kill-buffer   (process-buffer proc))))

(defun test-slave ()
  "Create an asynchronous process and wait for it to exit."
  (let ((proc (make-process
               :name "test"
               :buffer (generate-new-buffer " *test*")
               :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emacs")
               :connection-type 'pipe
               :sentinel #'test-sentinel)))
    (while (eq (process-status proc) 'run)
      (test-debug proc 'accept-output (accept-process-output proc 5)))
    (test-debug proc 'exit (process-status proc) (process-exit-status proc))))

(defun test-master (async)
  "Successively run `test-slave'.
With non-nil ASYNC, use a new thread for each invocation."
    (if async
	(progn
	  (dotimes (i 5) (make-thread #'test-slave (format "test-%d" i)))
	  (while (cdr (all-threads) (thread-join (cadr (all-threads))))))
      (dotimes (i 5)
	(test-slave))))

(defun test-job (async)
  "Successively run `test-slave'.
Like `test-master', but avoid blocking interactive sessions by
running `test-master' in its own thread."
  (if noninteractive
      (test-master async)
    (make-thread (apply-partially #'test-master async) "test-master")))

(defun test-threads ()
  "Run `test-master' with threads."
  (test-job t))

(defun test-no-threads ()
  "Run `test-master' without threads."
  (test-job nil))

;;; test.el ends here

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 16 Oct 2018 01:16:01 +0000
Resent-Message-ID: <handler.33018.B33018.153965254423776 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153965254423776
          (code B ref 33018); Tue, 16 Oct 2018 01:16:01 +0000
Received: (at 33018) by debbugs.gnu.org; 16 Oct 2018 01:15:44 +0000
Received: from localhost ([127.0.0.1]:51643 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCDxX-0006BQ-US
	for submit <at> debbugs.gnu.org; Mon, 15 Oct 2018 21:15:44 -0400
Received: from mail-ed1-f47.google.com ([209.85.208.47]:37602)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCDxV-0006BB-4A
 for 33018 <at> debbugs.gnu.org; Mon, 15 Oct 2018 21:15:42 -0400
Received: by mail-ed1-f47.google.com with SMTP id c22-v6so19653508edc.4
 for <33018 <at> debbugs.gnu.org>; Mon, 15 Oct 2018 18:15:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=Qu5NrC7ZxHOFneqBJimL1EW028BtpdtZs1JiIJxZQDU=;
 b=OUPKfWOi09riNKml4/YhCEoTgmILvU9HjgtU2Q8ZPZ80sqmFre40i4zhCHmgs+uMwJ
 6opHdInz7cNw5mLcXy0KiNVETQ/nXKR981JOQ6LuyLTXmp+di3IfOCN+0i/yu2T5Yk/z
 oJFjPd5veeVHAqkI6oiScl4n7MppLln0VoFXOxY1pg+sBGfVAUuljojO4maZhb1y+Z4+
 8rOWzAiBQp+stZznN40JcmJGw1lltIkW0vX7uFoRCIYSGir1/yBTj2Qts8RZkyWv7L62
 4DY1o3PsiW4o2UYQeMCjnyN10woE1PzRcEIe/qJjLm9yIFLDmpdaEQ5oEU/zFRDEZMI/
 8n/A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=Qu5NrC7ZxHOFneqBJimL1EW028BtpdtZs1JiIJxZQDU=;
 b=sW2+xfIYsKFyWN16o9Y8NLFwWiXD0Tm91scxNVDis4kToRbd2mro7J8h4FCPtKwwKR
 Nek/UPvCTqg5F/5RtsfVLLAD10zeb2kVJyOXbw8x1PCdYKLqKy9oxvpz/FYuge36LPZf
 RljDnTUp/If9YXG7imEW8lBvaTa23AQpHmaaOumslJ5pFeTabTywT1iHOLfpnGOEDrp7
 D3Il3ZJtz9RN4Z6KlR77q5vdEu3BsgzzSAhWDtFfcKyV9Vk+xhfEl4OzIBehnO7RK4em
 X95ORABZOXPuUiGPc0aH0V8Hcb3UkYW1eG53lSfKPVk9hWpXnSCQKwEi/h03QVnbIX6a
 Kqbw==
X-Gm-Message-State: ABuFfoi6Q5n1J5GwpcIkL4VUkokH3pssvoGDYFkSNCzb6p0LZvgXVhrN
 8mZS+mf5DlPZpnOrE0iwl1Qx5g==
X-Google-Smtp-Source: ACcGV604jjEz3ELxeKcuiQ6Nsru2JXZfoHNcep5yFh/1uQBUQbZzTsViflFdutkeU407SslcHJ11Pg==
X-Received: by 2002:a50:d78b:: with SMTP id
 w11-v6mr28672589edi.19.1539652535019; 
 Mon, 15 Oct 2018 18:15:35 -0700 (PDT)
Received: from localhost (51-171-243-2-dynamic.agg2.clk.blp-srl.eircom.net.
 [51.171.243.2])
 by smtp.gmail.com with ESMTPSA id y47-v6sm5801266edc.27.2018.10.15.18.15.33
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Mon, 15 Oct 2018 18:15:34 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
Date: Tue, 16 Oct 2018 02:15:27 +0100
In-Reply-To: <87sh177i36.fsf@HIDDEN> (Michael Albinus's message of "Mon, 15
 Oct 2018 10:02:53 +0200")
Message-ID: <877eiipu8g.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: application/emacs-lisp
Content-Disposition: attachment; filename=test.el
Content-Transfer-Encoding: quoted-printable

;;; test.el --- test -*- lexical-binding: t -*-

;;; Code:

(defun test-debug (&rest args)
  "Print `current-thread' and ARGS in timestamped `message'."
  (push (current-thread) args)
  (message "[%s] %s"
           (format-time-string "%T:%3N")
           (mapconcat #'prin1-to-string args " ")))

(defun test-sentinel (proc msg)
  "Kill buffer of PROC when it exits."
  (test-debug proc 'sentinel (process-status proc) (substring msg 0 -1))
  (and (not (eq (process-status proc) 'run))
       (buffer-live-p (process-buffer proc))
       (kill-buffer   (process-buffer proc))))

(defun test-slave ()
  "Create an asynchronous process and wait for it to exit."
  (let ((proc (make-process
               :name "test"
               :buffer (generate-new-buffer " *test*")
               :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emac=
s")
               :connection-type 'pipe
               :sentinel #'test-sentinel)))
    (while (eq (process-status proc) 'run)
      (test-debug proc 'accept-output (accept-process-output proc 5)))
    (test-debug proc 'exit (process-status proc) (process-exit-status proc)=
)))

(defun test-master (async)
  "Successively run `test-slave'.
With non-nil ASYNC, use a new thread for each invocation."
  (if async
      (dotimes (_ 2)
        (let (threads)
          (dotimes (i 2)
            (push (make-thread #'test-slave (format "test-%d" i)) threads))
          (mapc #'thread-join threads)))
    (dotimes (_ 2)
      (test-slave))))

(defun test-job (async)
  "Successively run `test-slave'.
Like `test-master', but avoid blocking interactive sessions by
running `test-master' in its own thread."
  (if noninteractive
      (test-master async)
    (make-thread (apply-partially #'test-master async) "test-master")))

(defun test-threads ()
  "Run `test-master' with threads."
  (test-job t))

(defun test-no-threads ()
  "Run `test-master' without threads."
  (test-job nil))

;;; test.el ends here

--=-=-=
Content-Type: text/plain


Michael Albinus <michael.albinus@HIDDEN> writes:

> Well, I've played with your example. As Eli said, `set-process-thread'
> is not needed here.
>
> With your original example, I could reproduce the problem, However, if I
> call
>
> emacs -l test.el -f test-threads
>
> the problem does NOT happen. My .emacs is quite long, so I didn't bisect
> in order to find out what makes the difference.
>
> I have changed your example a little bit wrt `thread-join', see
> appended. This version runs w/o any problem even if emacs is called with
> -Q. Maybe this helps you to debug further.

Thanks, creating all threads before waiting for any of them to exit
indeed does not suffer from the same hang.  Doing this twice (see
attached update), however, still hangs.

There's something about going through a complete create-join cycle more
than once within a non-main-thread which is triggering this behaviour.

-- 
Basil

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Michael Albinus <michael.albinus@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 16 Oct 2018 13:55:02 +0000
Resent-Message-ID: <handler.33018.B33018.153969805220405 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153969805220405
          (code B ref 33018); Tue, 16 Oct 2018 13:55:02 +0000
Received: (at 33018) by debbugs.gnu.org; 16 Oct 2018 13:54:12 +0000
Received: from localhost ([127.0.0.1]:51948 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCPnY-0005J2-Iy
	for submit <at> debbugs.gnu.org; Tue, 16 Oct 2018 09:54:12 -0400
Received: from mout.gmx.net ([212.227.15.15]:51147)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <michael.albinus@HIDDEN>) id 1gCPnW-0005Io-RY
 for 33018 <at> debbugs.gnu.org; Tue, 16 Oct 2018 09:54:11 -0400
Received: from detlef.gmx.de ([212.86.50.126]) by mail.gmx.com (mrgmx003
 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M243n-1fNsbg33mt-00u59Q; Tue, 16
 Oct 2018 15:54:03 +0200
Received: from detlef.gmx.de ([212.86.50.126]) by mail.gmx.com (mrgmx003
 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M243n-1fNsbg33mt-00u59Q; Tue, 16
 Oct 2018 15:54:03 +0200
From: Michael Albinus <michael.albinus@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN>
Date: Tue, 16 Oct 2018 15:54:02 +0200
In-Reply-To: <877eiipu8g.fsf@HIDDEN> (Basil L. Contovounesios's message of
 "Tue, 16 Oct 2018 02:15:27 +0100")
Message-ID: <87sh16ro91.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Provags-ID: V03:K1:KtQC+Mroc7+sdnkSV91A3f8zOsjmlDN2x+Fvp0HvPrvbIsJ/vhi
 xa0Q+8Fx7mjPg9G9F1/K/yvv+5xD2UZECxN8NHDOxu0FeKycZnExUVjuKycLvyLTn5nLZEU
 q2e0vpY7pfeN0DZ1t8wRYsyP+4dIgBqhoqsTpQvMaCwdyXVWmDYEbMnC4Vti03GmJPTz1Vg
 qO7eoDpmIHZ56KXUm3qWw==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V01:K0:nUfzXSnlqr0=:a1S58+3vSGPDsgoSyByalH
 7K3BGD2CpRoIbrFH7iXoNDBhdZb4NVnHmQyaMSUtDqR5Db6VLtt471Ll724I/ecmEiCL8urAM
 qYYKVzqP44Ajbtsvo257R53TWAbb/k9uKLdqSz2S+73uOl+ZLqNga4KconFL2lynG2/iBA2qE
 DJKDrHal94vkWt+ChFuDiwNMFuwoecbogH2eXGfKSh4A8Xak5NN8jiPUf6uCqJjiP3t1nNcIA
 KaFAgTgaMxjNvSVga4zoyeuW3GZMKaCrObr9gjuRHOoOkvOBATJHeGrl72uURdh3F2yuvwX4z
 pdITzneoc7vFwwnsD9IkwzQ/3p0my6d3JPmLaw541N5spTpeEXLGy8JJVJ1nRCVy7YXRyJXPc
 IiFPrPC8XNEOjefzlJcFA0nTrFQF/Qq4YX35P1mIKNv4vABRZ/wAjoM7Yg6lUJhYaBRylrgvA
 0H7xuzgdiIst8y3qnilpwRRXMFyPQaDQQzT8vTMnGnjQGeboTGpszZgAIzkUSK899RvREW725
 tfehBm95A9unx4fZnEN6bBebP5jnhSGt6vZNZG26xt7llHGuLXJV6onNkpBsPljr5/8qdZxp+
 7R79nkjZ3h/CHZaYv0ZpMgrQ+pqhrYxvopZwWHgDcWyEVwyHCrNhAWIeu01jjzhZgsyDfaQEe
 WJSzr920MPwIrtSZhy4utBOV5bQGJbgDP7iq1Rlf6B4e/F9BuSrbJas1Zs0/qBqI7eGO85b9J
 UTyfuvz5jjGGTrLwu7uBZljXuA88P8A/nxOBgXa4NZntYuabOlUHEo+jiqhm+7964/Hhnvxc2
 oQfiGHckvwXwCmZiwvHB+KqgQJxrQ==
X-Spam-Score: -0.7 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

--=-=-=
Content-Type: text/plain

"Basil L. Contovounesios" <contovob@HIDDEN> writes:

> There's something about going through a complete create-join cycle more
> than once within a non-main-thread which is triggering this behaviour.

I'm not sure that it is related to threads. It looks, like some of your
processes do not exit properly, and then thread-join is blocked.

I've modified your example, again. It runs perfectly. And during its
work, you could call "M-x list-threads" and see how the threads are
created and die.

Best regards, Michael.


--=-=-=
Content-Type: text/plain
Content-Disposition: attachment

diff -u /home/albinus/tmp/test.el.\~1\~ /home/albinus/tmp/test.el
--- /home/albinus/tmp/test.el.~1~	2018-10-16 14:46:47.000000000 +0200
+++ /home/albinus/tmp/test.el	2018-10-16 15:47:05.581276541 +0200
@@ -19,9 +19,9 @@
 (defun test-slave ()
   "Create an asynchronous process and wait for it to exit."
   (let ((proc (make-process
-               :name "test"
+               :name (thread-name (current-thread))
                :buffer (generate-new-buffer " *test*")
-               :command '("wget" "-qO-" "https://en.wikipedia.org/wiki/Emacs")
+               :command `("echo" ,(thread-name (current-thread)))
                :connection-type 'pipe
                :sentinel #'test-sentinel)))
     (while (eq (process-status proc) 'run)
@@ -32,10 +32,10 @@
   "Successively run `test-slave'.
 With non-nil ASYNC, use a new thread for each invocation."
   (if async
-      (dotimes (_ 2)
+      (dotimes (i 100)
         (let (threads)
-          (dotimes (i 2)
-            (push (make-thread #'test-slave (format "test-%d" i)) threads))
+          (dotimes (j 100)
+            (push (make-thread #'test-slave (format "test-%d-%d" i j)) threads))
           (mapc #'thread-join threads)))
     (dotimes (_ 2)
       (test-slave))))

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 16 Oct 2018 14:56:01 +0000
Resent-Message-ID: <handler.33018.B33018.153970174227800 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Michael Albinus <michael.albinus@HIDDEN>
Cc: 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153970174227800
          (code B ref 33018); Tue, 16 Oct 2018 14:56:01 +0000
Received: (at 33018) by debbugs.gnu.org; 16 Oct 2018 14:55:42 +0000
Received: from localhost ([127.0.0.1]:53808 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCQl4-0007EK-Ft
	for submit <at> debbugs.gnu.org; Tue, 16 Oct 2018 10:55:42 -0400
Received: from mail-ed1-f45.google.com ([209.85.208.45]:33108)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCQl3-0007E8-07
 for 33018 <at> debbugs.gnu.org; Tue, 16 Oct 2018 10:55:41 -0400
Received: by mail-ed1-f45.google.com with SMTP id l14-v6so12653954edq.0
 for <33018 <at> debbugs.gnu.org>; Tue, 16 Oct 2018 07:55:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=di/Z/+3iqHDOVjBzasbgFTcF610xoUOm9/YVhFlCk8o=;
 b=rWRKD5+gTQ8e4VNUb8YMYwZ1MIsqpV2HrfDvc2kfTFvxXzTPCEEXvaiBo2YndF4jrq
 zA6Op7KCOQ4xnnP94zQ4UmhZloWq4J89nQXqbMUqOiJzLnNs0pX4lNi7BOuNVvg/bkVU
 pDpUZN/I/FthjTRV4dLxRYxYEBuBMB+HgwqdYzU4TREUgck9xP4tqUfPoibA0l6YXTcO
 3FCdZBvduwvHaafD74qWxLCHGzcoqtK9gXOlIm4QZHK1R+dAb0rxw0dVE8ICiZn4vgoM
 2H01Hoz264qDzeup5rQ+0D4/aTcT0W8txWuzTlQZfLiCCvOeJXsm4ur7XhfDN1W1SEWM
 2+LA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=di/Z/+3iqHDOVjBzasbgFTcF610xoUOm9/YVhFlCk8o=;
 b=rjGuHemkkvJRlIDvxB0tpFomDM7VGVJLGpg85ounb8x+MErxSTGiUP67uhXnyGSwB0
 bXmZ2m0558WSfrDCyqu24Dt6klEV1W1Aa5oItttm6hvBsPdNmXRkLG4R40NFDbO0zXAg
 iKp0NubD0L+Fc26XYUAtfC5VB2YgE1wn1a92FHlRrz1Ipr/hn63FvZ+uEEfAEIIU72IZ
 W/23JQCWyZqkgH4/1jLVBm5XWL6W4pBa20R2IMRhTSKcdf4nv8nl8VfMwWFax6eHNZnz
 9RJxk1jBBVbetK82SiyDYXVtBbib9EXArrwyyeloSQQf9OIXpSusMrWLgczmZ4wW3TpJ
 eK3w==
X-Gm-Message-State: ABuFfojdadCewkyZg40QbF5t14dtOmiuVq5a0DFBoJqp0wwIsDYN7AGJ
 t16ID1cFNl3+eBrxTMZf6NkQSQ==
X-Google-Smtp-Source: ACcGV632D2UgZzienNHy7N8oTWChLPw4ZZIAB6RNlBvwB39k9CK8N74XomJSLb0pojg7L1rdHdWzVQ==
X-Received: by 2002:a17:906:4e03:: with SMTP id
 z3-v6mr23263480eju.187.1539701734893; 
 Tue, 16 Oct 2018 07:55:34 -0700 (PDT)
Received: from localhost ([213.233.148.5])
 by smtp.gmail.com with ESMTPSA id x15-v6sm6474371edm.26.2018.10.16.07.55.34
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Tue, 16 Oct 2018 07:55:34 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <87sh16ro91.fsf@HIDDEN>
Date: Tue, 16 Oct 2018 15:55:28 +0100
In-Reply-To: <87sh16ro91.fsf@HIDDEN> (Michael Albinus's message of "Tue, 16
 Oct 2018 15:54:02 +0200")
Message-ID: <87r2gqvt3z.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Michael Albinus <michael.albinus@HIDDEN> writes:

> "Basil L. Contovounesios" <contovob@HIDDEN> writes:
>
>> There's something about going through a complete create-join cycle more
>> than once within a non-main-thread which is triggering this behaviour.
>
> I'm not sure that it is related to threads.

It has to be, because there is never an issue when I run the same
asynchronous wget processes without threads, and with threads the hang
reliably occurs 100% of the time.

> It looks, like some of your processes do not exit properly, and then
> thread-join is blocked.

Indeed, but there is something about the interaction of Emacs threads
and subprocesses which is causing unsuccessful process termination.

Note that I am not ruling out pilot error; I simply haven't debugged
this issue any deeper yet.  The fact that no-one has yet pointed out any
obvious blunders on my part gives me more confidence that there is
indeed some ghost in the wire.

> I've modified your example, again. It runs perfectly.

Indeed, there are many subprocess-within-a-thread examples which don't
suffer from a hang, e.g. by using a different URL.  I would like to get
to the bottom of why network programs like wget/curl in particular
eventually hang, though.

Thanks,

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Tue, 16 Oct 2018 14:59:02 +0000
Resent-Message-ID: <handler.33018.B33018.153970190928041 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153970190928041
          (code B ref 33018); Tue, 16 Oct 2018 14:59:02 +0000
Received: (at 33018) by debbugs.gnu.org; 16 Oct 2018 14:58:29 +0000
Received: from localhost ([127.0.0.1]:53812 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCQnk-0007IC-VQ
	for submit <at> debbugs.gnu.org; Tue, 16 Oct 2018 10:58:29 -0400
Received: from eggs.gnu.org ([208.118.235.92]:52931)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gCQnj-0007I0-GR
 for 33018 <at> debbugs.gnu.org; Tue, 16 Oct 2018 10:58:28 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gCQnZ-0000Ms-Ky
 for 33018 <at> debbugs.gnu.org; Tue, 16 Oct 2018 10:58:20 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41716)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gCQnZ-0000MA-FI; Tue, 16 Oct 2018 10:58:17 -0400
Received: from [176.228.60.248] (port=2614 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gCQnY-0002ti-Al; Tue, 16 Oct 2018 10:58:17 -0400
Date: Tue, 16 Oct 2018 17:58:28 +0300
Message-Id: <83k1mirl9n.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <877eiipu8g.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN> <877eiipu8g.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Date: Tue, 16 Oct 2018 02:15:27 +0100
> Cc: 33018 <at> debbugs.gnu.org
> 
> Thanks, creating all threads before waiting for any of them to exit
> indeed does not suffer from the same hang.  Doing this twice (see
> attached update), however, still hangs.
> 
> There's something about going through a complete create-join cycle more
> than once within a non-main-thread which is triggering this behaviour.

Can you attach a debugger to the wget process that's stuck, and see
where it is stuck?  You will probably need to rebuild wget with debug
info, or install one from your package repository (if they offer
such).  This could give us hints for where to look further.

Thanks.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 17:38:01 +0000
Resent-Message-ID: <handler.33018.B33018.153979783632011 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153979783632011
          (code B ref 33018); Wed, 17 Oct 2018 17:38:01 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 17:37:16 +0000
Received: from localhost ([127.0.0.1]:55567 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCpkx-0008KF-SW
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 13:37:16 -0400
Received: from mail-ed1-f67.google.com ([209.85.208.67]:37725)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCpkv-0008Jz-Lx
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 13:37:14 -0400
Received: by mail-ed1-f67.google.com with SMTP id c22-v6so25681938edc.4
 for <33018 <at> debbugs.gnu.org>; Wed, 17 Oct 2018 10:37:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=UN44GWjQG0etD1hhH28egGDclowuoEeWETCHPiFMuas=;
 b=Q+W/p5NKRiyM07XoUaWlbHJSlc6xI2gsZGIbNV28ivXY+k0mZncPC9a+4hVmKhbgWy
 AzhrNsGLZU3Vel8s4V6POBqWK4tc9I0Lb8dtqq44PNhkzVC+N0tDLClxuJL8JSxhvv3c
 b9aLXbXVybUBp1JbTNJJzQAJQlO3u42nlyXtcLNFmJLSuTQeAqUpgQxODG6hiaAqI46Q
 Cahb+h/I4eCyi+sro0qcKQR5+9H3Ao8u6teJgph/DyJ1+uaOeUCreaAv3ODemzipV6j6
 xkZRbtuxMFG7qAwdEm0aO+lu+NjOBrSKH/K6zuS6AOuEMM1ns7e+Z1Ft578ub/kErom5
 NaAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=UN44GWjQG0etD1hhH28egGDclowuoEeWETCHPiFMuas=;
 b=SY/fDy6HcRq5I6Spk78DSyff8XbqIUWGZPChyZ0Ptf2FXJAJmyXUxENCcozsgnV7lP
 H4WKzCDqMYAADcLm/MJjuAuOkglHIaNAc/1qojC3gpuuMgbOn7dP3A/ViR6FoxRvX/cx
 GUeRvT8D2OgVAh7bNkA4FxUYzgPNQWxP7aby3e50EnMsJSI+VlZcersigQLVVYX7STGi
 Imp4wWbpo9ZA+sm00wr6A3nr4k5aI0v3/S0H/F2aybdN0n+N1052CRUoWKzTM0gNxoYS
 4vpGbvwLrCEXwBxb3xGbkZfL9aJEewjFV0JyJ0QbfqMrp5L02klX7vVjiJKaJ+Vov23o
 EimQ==
X-Gm-Message-State: ABuFfojxxqAjBzVqArEaE+Awa3zKnoUiJmllCBZ+D7v3bcVpL6dfcd/X
 97Qss+hoSuCPzEWJOHC+5wdGTA==
X-Google-Smtp-Source: ACcGV62bMpHUc6gpXDjKQ/PSXW0K+9aGxAhvuCF9p9SM9LYnt+7kyI++LjdROimYDYwfEfesIAN3HA==
X-Received: by 2002:a17:906:3181:: with SMTP id
 1-v6mr27152839ejy.80.1539797827591; 
 Wed, 17 Oct 2018 10:37:07 -0700 (PDT)
Received: from localhost ([213.233.148.7])
 by smtp.gmail.com with ESMTPSA id g48-v6sm7146967edc.93.2018.10.17.10.37.05
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Wed, 17 Oct 2018 10:37:06 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
Date: Wed, 17 Oct 2018 18:37:00 +0100
In-Reply-To: <83k1mirl9n.fsf@HIDDEN> (Eli Zaretskii's message of "Tue, 16 Oct
 2018 17:58:28 +0300")
Message-ID: <874ldkjwzn.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

--=-=-=
Content-Type: text/plain
Content-Disposition: attachment; filename=gdb.txt

#0  0x00007ffff766a2a4 in __GI___libc_write (fd=1, buf=0x5555559c1b30, nbytes=4096)
    at ../sysdeps/unix/sysv/linux/write.c:27
#1  0x00007ffff75fb56d in _IO_new_file_write (f=0x7ffff7739760 <_IO_2_1_stdout_>, 
    data=0x5555559c1b30, n=4096) at fileops.c:1203
#2  0x00007ffff75fa88f in new_do_write (fp=0x7ffff7739760 <_IO_2_1_stdout_>, 
    data=0x5555559c1b30 "pan></span></h2>\n<p>Emacs is primarily a <a href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is designed for manipulating pieces of text, although it is capable of formatting and print"..., to_do=to_do@entry=4096) at fileops.c:457
#3  0x00007ffff75fc6f9 in _IO_new_do_write (fp=<optimized out>, data=<optimized out>, 
    to_do=4096) at fileops.c:433
#4  0x00007ffff75fa6d8 in _IO_new_file_sync (fp=0x7ffff7739760 <_IO_2_1_stdout_>)
    at fileops.c:813
#5  0x00007ffff75ef6ed in __GI__IO_fflush (fp=0x7ffff7739760 <_IO_2_1_stdout_>) at iofflush.c:40
#6  0x00005555555918bb in write_data (out=0x7ffff7739760 <_IO_2_1_stdout_>, out2=0x0, 
    buf=0x5555559a9590 "pan></span></h2>\n<p>Emacs is primarily a <a href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is designed for manipulating pieces of text, although it is capable of formatting and print"..., bufsize=4096, skip=0x7fffffffd0e8, 
    written=0x7fffffffd0e0) at retr.c:207
#7  0x000055555559204f in fd_read_body (downloaded_filename=0x5555555e18a0 "-", fd=4, 
    out=0x7ffff7739760 <_IO_2_1_stdout_>, toread=198224, startpos=0, qtyread=0x7fffffffda20, 
    qtywritten=0x7fffffffd9d0, elapsed=0x7fffffffda28, flags=1, out2=0x0) at retr.c:498
#8  0x000055555557d09d in read_response_body (hs=0x7fffffffd9d0, sock=4, 
    fp=0x7ffff7739760 <_IO_2_1_stdout_>, contlen=198224, contrange=0, 
    chunked_transfer_encoding=false, url=0x5555555ff920 "https://en.wikipedia.org/wiki/Emacs", 
    warc_timestamp_str=0x7fffffffd470 "\f\335\377\377\377\177", 
    warc_request_uuid=0x7fffffffd440 ",", warc_ip=0x0, type=0x5555559c8fc0 "text/html", 
    statcode=200, 
    head=0x5555559b35d0 "HTTP/1.1 200 OK\r\nDate: Wed, 17 Oct 2018 17:29:35 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 198224\r\nConnection: keep-alive\r\nServer: mw1325.eqiad.wmnet\r\nVary: Accept-Encoding,Cookie,A"...) at http.c:1717
#9  0x0000555555581b06 in gethttp (u=0x5555555ff8b0, original_url=0x5555555ff8b0, 
    hs=0x7fffffffd9d0, dt=0x7fffffffdd0c, proxy=0x0, iri=0x5555555ff880, count=1) at http.c:4129
#10 0x0000555555582239 in http_loop (u=0x5555555ff8b0, original_url=0x5555555ff8b0, 
    newloc=0x7fffffffdb68, local_file=0x7fffffffdb58, referer=0x0, dt=0x7fffffffdd0c, 
    proxy=0x0, iri=0x5555555ff880) at http.c:4363
#11 0x0000555555592ca4 in retrieve_url (orig_parsed=0x5555555ff8b0, 
    origurl=0x5555555ff490 "https://en.wikipedia.org/wiki/Emacs", file=0x7fffffffdd18, 
    newloc=0x7fffffffdd10, refurl=0x0, dt=0x7fffffffdd0c, recursive=false, iri=0x5555555ff880, 
    register_status=true) at retr.c:957
#12 0x000055555558b33e in main (argc=3, argv=0x7fffffffdef8) at main.c:2158

--=-=-=
Content-Type: text/plain


Eli Zaretskii <eliz@HIDDEN> writes:

> Can you attach a debugger to the wget process that's stuck, and see
> where it is stuck?  You will probably need to rebuild wget with debug
> info, or install one from your package repository (if they offer
> such).  This could give us hints for where to look further.

I did the following:

0. Build wget 1.19.5 from Debian's repositories:
   apt-get build-dep wget
   apt-get source wget
   configure CC='ccache gcc' CFLAGS='-O0 -g3 -ggdb -gdwarf-4 -pipe'
             --config-cache --enable-assert --with-gnu-ld
   make
1. Substitute resulting wget file name in original test.el program
2. emacs26 -Q -l test.el -f test-threads
3. gdb -p <pid of stuck wget>
4. set logging on
5. bt

I attach the resulting gdb.txt log file.

Thanks,

-- 
Basil

--=-=-=--

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 17:57:02 +0000
Resent-Message-ID: <handler.33018.B33018.15397990131536 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15397990131536
          (code B ref 33018); Wed, 17 Oct 2018 17:57:02 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 17:56:53 +0000
Received: from localhost ([127.0.0.1]:55595 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCq3x-0000Oh-8n
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 13:56:53 -0400
Received: from eggs.gnu.org ([208.118.235.92]:57664)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gCq3v-0000OT-JM
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 13:56:51 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gCq3m-0003hz-28
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 13:56:46 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42408)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gCq3k-0003hB-56; Wed, 17 Oct 2018 13:56:40 -0400
Received: from [176.228.60.248] (port=3903 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gCq3j-0001oO-Lc; Wed, 17 Oct 2018 13:56:40 -0400
Date: Wed, 17 Oct 2018 20:56:31 +0300
Message-Id: <83zhvcqwxc.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <874ldkjwzn.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN> <874ldkjwzn.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
> Date: Wed, 17 Oct 2018 18:37:00 +0100
> 
> #0  0x00007ffff766a2a4 in __GI___libc_write (fd=1, buf=0x5555559c1b30, nbytes=4096)
>     at ../sysdeps/unix/sysv/linux/write.c:27
> #1  0x00007ffff75fb56d in _IO_new_file_write (f=0x7ffff7739760 <_IO_2_1_stdout_>, 
>     data=0x5555559c1b30, n=4096) at fileops.c:1203
> #2  0x00007ffff75fa88f in new_do_write (fp=0x7ffff7739760 <_IO_2_1_stdout_>, 
>     data=0x5555559c1b30 "pan></span></h2>\n<p>Emacs is primarily a <a href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is designed for manipulating pieces of text, although it is capable of formatting and print"..., to_do=to_do@entry=4096) at fileops.c:457
> #3  0x00007ffff75fc6f9 in _IO_new_do_write (fp=<optimized out>, data=<optimized out>, 
>     to_do=4096) at fileops.c:433
> #4  0x00007ffff75fa6d8 in _IO_new_file_sync (fp=0x7ffff7739760 <_IO_2_1_stdout_>)
>     at fileops.c:813
> #5  0x00007ffff75ef6ed in __GI__IO_fflush (fp=0x7ffff7739760 <_IO_2_1_stdout_>) at iofflush.c:40
> #6  0x00005555555918bb in write_data (out=0x7ffff7739760 <_IO_2_1_stdout_>, out2=0x0, 
>     buf=0x5555559a9590 "pan></span></h2>\n<p>Emacs is primarily a <a href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is designed for manipulating pieces of text, although it is capable of formatting and print"..., bufsize=4096, skip=0x7fffffffd0e8, 
>     written=0x7fffffffd0e0) at retr.c:207
> #7  0x000055555559204f in fd_read_body (downloaded_filename=0x5555555e18a0 "-", fd=4, 
>     out=0x7ffff7739760 <_IO_2_1_stdout_>, toread=198224, startpos=0, qtyread=0x7fffffffda20, 
>     qtywritten=0x7fffffffd9d0, elapsed=0x7fffffffda28, flags=1, out2=0x0) at retr.c:498

Looks like the buffer of the pipe through which Emacs reads the stuff
is full, and wget waits for some space there?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 18:07:02 +0000
Resent-Message-ID: <handler.33018.B33018.15397995682527 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15397995682527
          (code B ref 33018); Wed, 17 Oct 2018 18:07:02 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 18:06:08 +0000
Received: from localhost ([127.0.0.1]:55599 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCqCu-0000eh-7n
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:06:08 -0400
Received: from mail-ed1-f45.google.com ([209.85.208.45]:40057)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCqCt-0000eI-5B
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:06:07 -0400
Received: by mail-ed1-f45.google.com with SMTP id r1-v6so25767875edd.7
 for <33018 <at> debbugs.gnu.org>; Wed, 17 Oct 2018 11:06:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=cdjYCVYWXprgtT1Yt2FyTzUtqTPAFHOriGREOju9mfI=;
 b=H/H/pPwYxZ+uOVUnb6K6C7UsyAQCIP0QmEMfWtg3AV4sUC/uLqPnGSlREXxg3xKXHn
 x1HS740wbjYz1Cya5XVUHlPYk91KyOaclk2Ab0ZIVFLnVYDZYzZmmeBm7TpW0T9hDSz5
 j/90yg4g7HhlCuw7IRTarz09ik8sugAWdFwNJfIUPbzbO/SP6fXEOEdJX+qH9CAyZJCn
 hFTED0Vpqw21STBw2Cdpn3Z5TqiRwTs4esXbcg9KB8sgdrZVygk+VMR1+MOaCJr7++lG
 kRHOpUVQuaG3kzQ7aILskBznL28FdouKoVq3U2qhG722ZLeZbe2kCrdhtspWNdE++Bl+
 MXYg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=cdjYCVYWXprgtT1Yt2FyTzUtqTPAFHOriGREOju9mfI=;
 b=BUYi0oR9aa6fgN6Yz6TFFZ6p27PrhAQWfdUOAmto21bhpD9EiftteuloIjDRDmZ9Ub
 aTWSe6393VuFBChU+ptS6ms8XXiziLW9IdVvFCH3px307+vIKY7ii3z77s5ewp/X6cAG
 aZcRAfUFEvtOxAerl5Ow06u34i1YTGLZtoB7mOEk10U4l0BSaDb3h6zWZMznWbXXbnqD
 4YJzB/UWGUf98p1HdZWuyU7YnZoafNgpsZZD3BZWXldITaEEW0POQc6J4CNNnCIPyztn
 0jkhKhkbmiOx/xU6YSDxtLHPMv9L116KT4sGTLLhy34dE3C39EK2E61iQItawqrelVCD
 jrkw==
X-Gm-Message-State: ABuFfohQwHKQe9S/OWbnhuMZhboWL16TyNCVMozeVNQZNZTBvPXVCsDn
 uXIa0mShMTXA8hSmbJ+Tlm+l2w==
X-Google-Smtp-Source: ACcGV601p46nbghJc+SQtDHbtE8isKuuIm34H0TgfSS0GWrAOKEsr+ggNsvaLJjZ+yhrBk+nvVGaTw==
X-Received: by 2002:a50:b8a5:: with SMTP id
 l34-v6mr1284036ede.273.1539799561495; 
 Wed, 17 Oct 2018 11:06:01 -0700 (PDT)
Received: from localhost ([213.233.155.132])
 by smtp.gmail.com with ESMTPSA id e21-v6sm7969509edb.22.2018.10.17.11.06.00
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Wed, 17 Oct 2018 11:06:00 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN>
Date: Wed, 17 Oct 2018 19:05:59 +0100
In-Reply-To: <83zhvcqwxc.fsf@HIDDEN> (Eli Zaretskii's message of "Wed, 17 Oct
 2018 20:56:31 +0300")
Message-ID: <87o9bswirc.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: "Basil L. Contovounesios" <contovob@HIDDEN>
>> Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
>> Date: Wed, 17 Oct 2018 18:37:00 +0100
>> 
>> #0  0x00007ffff766a2a4 in __GI___libc_write (fd=1, buf=0x5555559c1b30, nbytes=4096)
>>     at ../sysdeps/unix/sysv/linux/write.c:27
>> #1  0x00007ffff75fb56d in _IO_new_file_write (f=0x7ffff7739760 <_IO_2_1_stdout_>, 
>>     data=0x5555559c1b30, n=4096) at fileops.c:1203
>> #2  0x00007ffff75fa88f in new_do_write (fp=0x7ffff7739760 <_IO_2_1_stdout_>, 
>>     data=0x5555559c1b30 "pan></span></h2>\n<p>Emacs is primarily a <a
>> href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is
>> designed for manipulating pieces of text, although it is capable of formatting
>> and print"..., to_do=to_do@entry=4096) at fileops.c:457
>> #3  0x00007ffff75fc6f9 in _IO_new_do_write (fp=<optimized out>, data=<optimized out>, 
>>     to_do=4096) at fileops.c:433
>> #4  0x00007ffff75fa6d8 in _IO_new_file_sync (fp=0x7ffff7739760 <_IO_2_1_stdout_>)
>>     at fileops.c:813
>> #5 0x00007ffff75ef6ed in __GI__IO_fflush (fp=0x7ffff7739760 <_IO_2_1_stdout_>)
>> at iofflush.c:40
>> #6  0x00005555555918bb in write_data (out=0x7ffff7739760 <_IO_2_1_stdout_>, out2=0x0, 
>>     buf=0x5555559a9590 "pan></span></h2>\n<p>Emacs is primarily a <a
>> href=\"/wiki/Text_editor\" title=\"Text editor\">text editor</a> and is
>> designed for manipulating pieces of text, although it is capable of formatting
>> and print"..., bufsize=4096, skip=0x7fffffffd0e8,
>>     written=0x7fffffffd0e0) at retr.c:207
>> #7  0x000055555559204f in fd_read_body (downloaded_filename=0x5555555e18a0 "-", fd=4, 
>>     out=0x7ffff7739760 <_IO_2_1_stdout_>, toread=198224, startpos=0, qtyread=0x7fffffffda20, 
>>     qtywritten=0x7fffffffd9d0, elapsed=0x7fffffffda28, flags=1, out2=0x0) at retr.c:498
>
> Looks like the buffer of the pipe through which Emacs reads the stuff
> is full, and wget waits for some space there?

Would that imply that different threads/processes are (re)using the same
buffer/pipe?

FWIW, strace -p <pip of stuck emacs> gives:

pselect6(14, [6 7], [], NULL, {tv_sec=99975, tv_nsec=320947003}, {NULL, 8}

and strace -p <pip of stuck wget> gives:

write(1, ">]</span></span></h2>\n<p>Emacs i"..., 4096

The former reminded me of bug#24201: https://debbugs.gnu.org/24201

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 18:21:01 +0000
Resent-Message-ID: <handler.33018.B33018.15398004434119 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15398004434119
          (code B ref 33018); Wed, 17 Oct 2018 18:21:01 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 18:20:43 +0000
Received: from localhost ([127.0.0.1]:55606 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCqR0-00014M-MT
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:20:42 -0400
Received: from eggs.gnu.org ([208.118.235.92]:38169)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gCqQz-00014A-IP
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:20:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gCqQr-00039H-C4
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:20:36 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:42987)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gCqQq-00037n-NI; Wed, 17 Oct 2018 14:20:33 -0400
Received: from [176.228.60.248] (port=1403 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gCqQq-0003TQ-5I; Wed, 17 Oct 2018 14:20:32 -0400
Date: Wed, 17 Oct 2018 21:20:22 +0300
Message-Id: <83y3awqvtl.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <87o9bswirc.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN> <87o9bswirc.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
> Date: Wed, 17 Oct 2018 19:05:59 +0100
> 
> > Looks like the buffer of the pipe through which Emacs reads the stuff
> > is full, and wget waits for some space there?
> 
> Would that imply that different threads/processes are (re)using the same
> buffer/pipe?

Could be, but it's more likely that Emacs simply doesn't read the
output from wget.

> pselect6(14, [6 7], [], NULL, {tv_sec=99975, tv_nsec=320947003}, {NULL, 8}
> 
> and strace -p <pip of stuck wget> gives:
> 
> write(1, ">]</span></span></h2>\n<p>Emacs i"..., 4096
> 
> The former reminded me of bug#24201: https://debbugs.gnu.org/24201

I could be wrong, but it doesn't look similar to me.  In that bug, the
CPU was pegged, whereas you said that CPU is idle.

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 18:26:01 +0000
Resent-Message-ID: <handler.33018.B33018.15398007564665 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: contovob@HIDDEN
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.15398007564665
          (code B ref 33018); Wed, 17 Oct 2018 18:26:01 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 18:25:56 +0000
Received: from localhost ([127.0.0.1]:55610 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCqW4-0001DB-D0
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:25:56 -0400
Received: from eggs.gnu.org ([208.118.235.92]:39944)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gCqW2-0001Cu-Oz
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:25:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gCqVt-0001PI-5b
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 14:25:49 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:43140)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gCqVs-0001Og-Vs; Wed, 17 Oct 2018 14:25:45 -0400
Received: from [176.228.60.248] (port=1716 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gCqVs-0003w1-0w; Wed, 17 Oct 2018 14:25:44 -0400
Date: Wed, 17 Oct 2018 21:25:29 +0300
Message-Id: <83woqgqvl2.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <83y3awqvtl.fsf@HIDDEN> (message from Eli Zaretskii on Wed, 17
 Oct 2018 21:20:22 +0300)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN>
 <87o9bswirc.fsf@HIDDEN> <83y3awqvtl.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> Date: Wed, 17 Oct 2018 21:20:22 +0300
> From: Eli Zaretskii <eliz@HIDDEN>
> Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
> 
> > From: "Basil L. Contovounesios" <contovob@HIDDEN>
> > Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
> > Date: Wed, 17 Oct 2018 19:05:59 +0100
> > 
> > > Looks like the buffer of the pipe through which Emacs reads the stuff
> > > is full, and wget waits for some space there?
> > 
> > Would that imply that different threads/processes are (re)using the same
> > buffer/pipe?
> 
> Could be, but it's more likely that Emacs simply doesn't read the
> output from wget.

I think the relevant code should be instrumented to show which thread
waits for what process(es).

Btw, are you sure this is not a bug in your program?  Michael caused
your program to work twice by simple changes, AFAIU, no?

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 20:04:02 +0000
Resent-Message-ID: <handler.33018.B33018.153980659022728 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153980659022728
          (code B ref 33018); Wed, 17 Oct 2018 20:04:02 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 20:03:10 +0000
Received: from localhost ([127.0.0.1]:55697 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCs29-0005uW-U7
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 16:03:10 -0400
Received: from mail-ed1-f52.google.com ([209.85.208.52]:42515)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCs27-0005uH-Sp
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 16:03:08 -0400
Received: by mail-ed1-f52.google.com with SMTP id b7-v6so26055508edd.9
 for <33018 <at> debbugs.gnu.org>; Wed, 17 Oct 2018 13:03:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=1ILYLc6+Z7/cv1fOEdQHV//aZQTMPKxjLj2H3fhVWgg=;
 b=g+95zDl6fulcSZiFpNyT0w8OVBAPOzTW08K+uJ5iyuvqhRTsJ0kQuJ+kDiBoxD3T0E
 VoddNgQSgx8Tiu6nvY5kfhvS+ohl/rYfDaASQjKjWKKMVxJ4cD2UG4iKq1Fgd216m3U9
 qg8TPH7YgWG/AfeeY7oGt3l745ujYJFfIwgOlPSr40OzJApIm360aKChERwMrvGXFOYd
 tmEiQg6bhE+xKOxGRSLpF7uHGkPI/WltptMKDGgYUJzvePhNUmTxzsubXKhLb2WGW3Pj
 JmrUtOMmXzH42r8VVJp+n9kyrKqYBGGMRN2qzK6Q78PR+sSm/8MfD1LIHVZy7QI8muq2
 a+kg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=1ILYLc6+Z7/cv1fOEdQHV//aZQTMPKxjLj2H3fhVWgg=;
 b=TPAeXSC3ynu6/mtnswa/dnuw5SweRtdXhz3t4wbnk8x6K0SfWyc6AOh4ilq5XEzm1x
 20nYu7Yncvvw3tjgXmW58MJgcC+meLe1vbInWUD/Fxw6OEfxdbIePcW5uz5f/mvSL2y0
 SiG1LtoTU2nWJmFF+ui0SUN+Z9AmnobYluAoKANamSD7qvEmtYFrSyAghUjRQvbjO7rL
 NDqk2/PgTdCO7VMPyC4vg0d6IeSRTlqzuvqZ8MzfP7hD7EbusdHzXfzhIdhvppj5sltn
 lRQO9e+cOuZrbLttgTQr4W2Cr7pWhLmMHm47/AT61FAdxS9oEL15MxvIZKhEt80LIgMn
 1thw==
X-Gm-Message-State: ABuFfogxYmOYXnzppVEvmNWz5RGKlsnHXtt9Z+Deo4NpHJiti9bYhT20
 Fo+pzdnV845CzMvsQNm0vBfG2w==
X-Google-Smtp-Source: ACcGV630M40bM241a9yaHHzyL6kGvsUz4Xu1PQLqL2Tg0c8yWswBEJKgqniZLX6RD9zlRIO/b7ooyg==
X-Received: by 2002:a50:a0e2:: with SMTP id 89-v6mr1508380edo.95.1539806582200; 
 Wed, 17 Oct 2018 13:03:02 -0700 (PDT)
Received: from localhost (51-171-243-2-dynamic.agg2.clk.blp-srl.eircom.net.
 [51.171.243.2])
 by smtp.gmail.com with ESMTPSA id q3-v6sm3844747ejj.50.2018.10.17.13.03.00
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Wed, 17 Oct 2018 13:03:01 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN>
 <87o9bswirc.fsf@HIDDEN> <83y3awqvtl.fsf@HIDDEN>
Date: Wed, 17 Oct 2018 21:02:59 +0100
In-Reply-To: <83y3awqvtl.fsf@HIDDEN> (Eli Zaretskii's message of "Wed, 17 Oct
 2018 21:20:22 +0300")
Message-ID: <87k1mg49zg.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> From: "Basil L. Contovounesios" <contovob@HIDDEN>
>> Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
>> Date: Wed, 17 Oct 2018 19:05:59 +0100
>> 
>> pselect6(14, [6 7], [], NULL, {tv_sec=99975, tv_nsec=320947003}, {NULL, 8}
>> 
>> and strace -p <pip of stuck wget> gives:
>> 
>> write(1, ">]</span></span></h2>\n<p>Emacs i"..., 4096
>> 
>> The former reminded me of bug#24201: https://debbugs.gnu.org/24201
>
> I could be wrong, but it doesn't look similar to me.  In that bug, the
> CPU was pegged, whereas you said that CPU is idle.

Right, I was only reminded of that bug because of the common
accept-process-output/pselect hang in the context of a network-related
process.  I'm not suggesting the underlying cause is the same.

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: "Basil L. Contovounesios" <contovob@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Wed, 17 Oct 2018 20:47:02 +0000
Resent-Message-ID: <handler.33018.B33018.153980920927673 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: Eli Zaretskii <eliz@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.153980920927673
          (code B ref 33018); Wed, 17 Oct 2018 20:47:02 +0000
Received: (at 33018) by debbugs.gnu.org; 17 Oct 2018 20:46:49 +0000
Received: from localhost ([127.0.0.1]:55834 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gCsiP-0007CG-9s
	for submit <at> debbugs.gnu.org; Wed, 17 Oct 2018 16:46:49 -0400
Received: from mail-ed1-f41.google.com ([209.85.208.41]:39400)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <contovob@HIDDEN>) id 1gCsiN-0007C3-N5
 for 33018 <at> debbugs.gnu.org; Wed, 17 Oct 2018 16:46:48 -0400
Received: by mail-ed1-f41.google.com with SMTP id d15-v6so26190752edq.6
 for <33018 <at> debbugs.gnu.org>; Wed, 17 Oct 2018 13:46:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=tcd-ie.20150623.gappssmtp.com; s=20150623;
 h=from:to:cc:subject:references:date:in-reply-to:message-id
 :user-agent:mime-version;
 bh=0hILc1GM7YBstkA/IhmeqByE0F3D+HnFUecP9zJoSho=;
 b=xkRdFQc2bBrf83ODmqK9tIcNWPgtcDUFYytYzFZzzS0kd+nsC52wR8lTFzj35CNOkR
 lotcgWTlJ94779OEW7zEmjRw7k3p3tje28Hka1cNrPCACmIkGtuAUXIAzZVLGkChJZJI
 IbVGfrJ2j12njjA6KTRpsrL2bfYpW5J2MkjnGI4TW6RHdbo3UubEOH31VogewJd3DD3i
 aOWJwKPvEqMnB+YUOnvPf2FPxzQkNZLJq3kHRycNYAoANRLOU6IjjTCjFCTvHw1biHIy
 3k7sPXFabhKs1ShAF9zRFd7NwgZ98J0qWW8uJM6SJGQ2g7KD0YMU6MU4+bN7AfT40wS9
 Jlrw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to
 :message-id:user-agent:mime-version;
 bh=0hILc1GM7YBstkA/IhmeqByE0F3D+HnFUecP9zJoSho=;
 b=KUmGP4c1SLWXCqyypEqNjT6ik7PYQQGqXRKJIqx8YmlBQvTVCf0+eybLBK+cvfdrB5
 Qg1h/mK2zgCuBXdc6/iU8mFzZ03FFWzImWdLLKwB4/M/5x/US2ZnXxi2s1R7jvoZCqTZ
 GI+bschHU6W6SW0Gd+qRTGWZ1oe4YDYmA8FVJZ5JwsDwjFyIqujwc7Bu4P98G/KN5Mdv
 ox6NLv9fDTEkLXW6VrNfLqZPAohFUOFHLKRgdPMSZmnylVTZ7fjz1Mqw5lMtEp7HEFtB
 lBK3mG2JhGdMmxP7JvQAGPGPM/UG2NmcOiUBwq6qk4ZQDqfX5GzFRgXCOhoNJkNsZ398
 zxcg==
X-Gm-Message-State: ABuFfohG1JAkmLgYr0B5YA5GPgf1kUwHJDreeUIIB+/CIlh8TpvPsoRp
 ziPaPXJjlHmFS14L20qonPm85A==
X-Google-Smtp-Source: ACcGV63vcd9L06Wv3CajHnLQ5BA7xBUBiO0t2T677CvFr8ZO3PJdDrk+jPq7kNXi7Q6QT4BpDVP1YQ==
X-Received: by 2002:a17:906:88f:: with SMTP id
 n15-v6mr27897968eje.119.1539809201734; 
 Wed, 17 Oct 2018 13:46:41 -0700 (PDT)
Received: from localhost (51-171-243-2-dynamic.agg2.clk.blp-srl.eircom.net.
 [51.171.243.2])
 by smtp.gmail.com with ESMTPSA id z2-v6sm8007539edz.31.2018.10.17.13.46.40
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Wed, 17 Oct 2018 13:46:41 -0700 (PDT)
From: "Basil L. Contovounesios" <contovob@HIDDEN>
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN>
 <87o9bswirc.fsf@HIDDEN> <83y3awqvtl.fsf@HIDDEN>
 <83woqgqvl2.fsf@HIDDEN>
Date: Wed, 17 Oct 2018 21:46:39 +0100
In-Reply-To: <83woqgqvl2.fsf@HIDDEN> (Eli Zaretskii's message of "Wed, 17 Oct
 2018 21:25:29 +0300")
Message-ID: <87zhvc2te8.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: -0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Eli Zaretskii <eliz@HIDDEN> writes:

>> Date: Wed, 17 Oct 2018 21:20:22 +0300
>> From: Eli Zaretskii <eliz@HIDDEN>
>> Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
>> 
>> > From: "Basil L. Contovounesios" <contovob@HIDDEN>
>> > Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
>> > Date: Wed, 17 Oct 2018 19:05:59 +0100
>> > 
>> > > Looks like the buffer of the pipe through which Emacs reads the stuff
>> > > is full, and wget waits for some space there?
>> > 
>> > Would that imply that different threads/processes are (re)using the same
>> > buffer/pipe?
>> 
>> Could be, but it's more likely that Emacs simply doesn't read the
>> output from wget.
>
> I think the relevant code should be instrumented to show which thread
> waits for what process(es).

Each thread launches a single wget process which it then waits for
before dying, and current-thread is always eq to that thread around
calls to accept-process-output.  Or are you talking about some other
type of thread?

Either way, I'll report back when I've had a deeper look into what Emacs
is doing, unless someone beats me to it.

> Btw, are you sure this is not a bug in your program?

No, but the fact that it reliably works when Emacs is run with -batch,
and reliably hangs when run with -Q is at least somewhat intriguing.

> Michael caused your program to work twice by simple changes, AFAIU,
> no?

Michael avoided the hang by rewriting the create-join-...-create-join
sequence as create-create-...-join-join.  But if the latter is done
twice from within the same master thread, the hang still occurs.  As I
said:

> There's something about going through a complete create-join cycle more
> than once within a non-main-thread which is triggering this behaviour.

(Actually, I haven't checked whether the hang occurs when two
 create-join cycles are completed within main-thread; I was just
 specifically describing my sample program.)

In his second rewrite, Michael replaced wget with echo, which does not
suffer from any hangs.  As I said:

> Indeed, there are many subprocess-within-a-thread examples which don't
> suffer from a hang, e.g. by using a different URL.  I would like to get
> to the bottom of why network programs like wget/curl in particular
> eventually hang, though.

In other words, I don't (yet) see why my recipe shouldn't work, and I'm
curious to eventually get to the bottom of this.

I'm sorry I've been talking more than doing, but university and house
hunting will dominate my free time for the next few weeks.

Thanks,

-- 
Basil

Message sent to bug-gnu-emacs@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#33018: 26.1.50; thread starvation with async processes and accept-process-output
Resent-From: Eli Zaretskii <eliz@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Sat, 20 Oct 2018 08:36:01 +0000
Resent-Message-ID: <handler.33018.B33018.154002451010584 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 33018
X-GNU-PR-Package: emacs
X-GNU-PR-Keywords: 
To: "Basil L. Contovounesios" <contovob@HIDDEN>
Cc: michael.albinus@HIDDEN, 33018 <at> debbugs.gnu.org
Received: via spool by 33018-submit <at> debbugs.gnu.org id=B33018.154002451010584
          (code B ref 33018); Sat, 20 Oct 2018 08:36:01 +0000
Received: (at 33018) by debbugs.gnu.org; 20 Oct 2018 08:35:10 +0000
Received: from localhost ([127.0.0.1]:60249 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1gDmiz-0002kc-L3
	for submit <at> debbugs.gnu.org; Sat, 20 Oct 2018 04:35:09 -0400
Received: from eggs.gnu.org ([208.118.235.92]:51581)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eliz@HIDDEN>) id 1gDmiy-0002kK-2Q
 for 33018 <at> debbugs.gnu.org; Sat, 20 Oct 2018 04:35:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <eliz@HIDDEN>) id 1gDmip-0003KF-KI
 for 33018 <at> debbugs.gnu.org; Sat, 20 Oct 2018 04:35:02 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level: 
X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled
 version=3.3.2
Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36384)
 by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>)
 id 1gDmip-0003K6-Fe; Sat, 20 Oct 2018 04:34:59 -0400
Received: from [176.228.60.248] (port=2670 helo=home-c4e4a596f7)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <eliz@HIDDEN>)
 id 1gDmip-0002SL-2T; Sat, 20 Oct 2018 04:34:59 -0400
Date: Sat, 20 Oct 2018 11:34:40 +0300
Message-Id: <83efcluicf.fsf@HIDDEN>
From: Eli Zaretskii <eliz@HIDDEN>
In-reply-to: <87zhvc2te8.fsf@HIDDEN> (contovob@HIDDEN)
References: <87r2gwbmma.fsf@HIDDEN> <875zy7cqzd.fsf@HIDDEN>
 <878t30h823.fsf@HIDDEN> <87sh177i36.fsf@HIDDEN>
 <877eiipu8g.fsf@HIDDEN> <83k1mirl9n.fsf@HIDDEN>
 <874ldkjwzn.fsf@HIDDEN> <83zhvcqwxc.fsf@HIDDEN>
 <87o9bswirc.fsf@HIDDEN> <83y3awqvtl.fsf@HIDDEN>
 <83woqgqvl2.fsf@HIDDEN> <87zhvc2te8.fsf@HIDDEN>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2001:4830:134:3::e
X-Spam-Score: -5.0 (-----)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -6.0 (------)

> From: "Basil L. Contovounesios" <contovob@HIDDEN>
> Cc: <michael.albinus@HIDDEN>,  <33018 <at> debbugs.gnu.org>
> Date: Wed, 17 Oct 2018 21:46:39 +0100
> 
> > I think the relevant code should be instrumented to show which thread
> > waits for what process(es).
> 
> Each thread launches a single wget process which it then waits for
> before dying, and current-thread is always eq to that thread around
> calls to accept-process-output.  Or are you talking about some other
> type of thread?

I was talking about low-level details: we set up the file-descriptor
mask passed to pselect, to tell it which descriptors to wait on.  The
call to accept-process-output is supposed to arrange for the
descriptor where the corresponding process will write to be one of
those on which the corresponding pselect will wait.  I was thinking
that perhaps we become confused and don't ask pselect called by a
thread to wait on the process which was launched by that thread.

> Either way, I'll report back when I've had a deeper look into what Emacs
> is doing, unless someone beats me to it.

Thanks.

> I'm sorry I've been talking more than doing, but university and house
> hunting will dominate my free time for the next few weeks.

No need to be sorry, we all have our lives.

Last modified: Mon, 25 Nov 2019 12:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.