GNU bug report logs - #56674
[Shepherd] Use of ‘waitpid’, ‘system*’, etc. in service code can cause deadlocks

Message received at submit <at> debbugs.gnu.org:

Received: (at submit) by debbugs.gnu.org; 20 Jul 2022 21:39:14 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jul 20 17:39:14 2022
Received: from localhost ([127.0.0.1]:36196 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1oEHPW-0005H1-FD
	for submit <at> debbugs.gnu.org; Wed, 20 Jul 2022 17:39:14 -0400
Received: from lists.gnu.org ([209.51.188.17]:56904)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1oEHPT-0005Gs-NI
 for submit <at> debbugs.gnu.org; Wed, 20 Jul 2022 17:39:13 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34104)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1oEHPT-0001Ia-IN
 for bug-guix@HIDDEN; Wed, 20 Jul 2022 17:39:11 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:39164)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1oEHPT-0001TV-AK
 for bug-guix@HIDDEN; Wed, 20 Jul 2022 17:39:11 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to:
 references; bh=M8w5WeDxJS14gBGGsfc8CvBrW4CYqhv18DUTdfvx+EU=; b=HBRZBuMtbWN9u3
 zm3KRNpSPuVelXLB6WvH2dvU2YbVqCxFLNGQkiD8MfxsjY9q78jQeQQK5OB/8MRhI7UaSsfEjgy/w
 zDOEyY7mCcTyPZXHaGLEzO/3m+00nAz4K39gCA5KEHYkkYl18VW1r7GAyU6jvhBpikmlXtCPjr8vF
 CIds4JmF3KO6LCWdCEBf+23EPMNoerp9LFcaZisTscpqoTbhTVU4wsm/ZISSuO1Va7LWi2cc1ofKX
 0shWBs0cxQhIuqvl9bmnPZiqSoZ2pTgZ3Gy0/GyrtLu1SMRSoIoiwnUkn/3HctM2lxfZoGb0HDvxz
 udLc9tiWBxbuQjp9g9Dw==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:64292
 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1oEHPS-00062Y-F2
 for bug-guix@HIDDEN; Wed, 20 Jul 2022 17:39:11 -0400
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: bug-guix@HIDDEN
Subject: [Shepherd] Use of =?utf-8?B?4oCYd2FpdHBpZOKAmSwg4oCYc3lzdGVtKg==?=
 =?utf-8?B?4oCZLA==?= etc. in service code can cause deadlocks
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Duodi 2 Thermidor an 230 de la =?utf-8?Q?R=C3=A9volu?=
 =?utf-8?Q?tion=2C?= jour du Bouillon-blanc
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Wed, 20 Jul 2022 23:39:08 +0200
Message-ID: <8735evpipv.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi!

We=E2=80=99ve just had a bad experience with the nginx service on berlin, w=
here
=E2=80=98herd restart nginx=E2=80=99 would cause shepherd to get stuck fore=
ver in
=E2=80=98waitpid=E2=80=99 on the process that was supposed to start nginx.

The details are unclear, but one thing is clear is that using =E2=80=98wait=
pid=E2=80=99
(either directly or indirectly with =E2=80=98system*=E2=80=99, which is what
=E2=80=98nginx-service-type=E2=80=99 does) is not great:

  1. In the best case, shepherd (as of 0.9.1) is stuck while =E2=80=98syste=
m*=E2=80=99
     is in =E2=80=98waitpid=E2=80=99 waiting for child process completion (=
=E2=80=9Cstuck=E2=80=9D as
     in: doesn=E2=80=99t do anything, not even answering =E2=80=98herd=E2=
=80=99 requests or
     inetd connections.)

  2. I don=E2=80=99t think that can happen with =E2=80=98system*=E2=80=99 (=
because it=E2=80=99s in C),
     but generally speaking, there=E2=80=99s a possibility that shepherd=E2=
=80=99s event
     loop will handle child process termination before some other
     user-made =E2=80=98waitpid=E2=80=99 call does.

Anyway, that=E2=80=99s a bad situation.

So I can think of several ways to address it:

  1. Change the nginx service =E2=80=98stop=E2=80=99 method to just
     (make-kill-destructor), which should work just as well as invoking
     =E2=80=9Cnginx -s stop=E2=80=9D.

  2. Have Shepherd provide a replacement for =E2=80=98system*=E2=80=99.

Thoughts?

Ludo=E2=80=99.