GNU logs - #72166, boring messages

Message sent to bug-guix@HIDDEN:

Subject: bug#72166: Shepherd periodically goes unresponsive on one of my machines
Resent-From: "Jonathan Frederickson" <jonathan@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Thu, 18 Jul 2024 00:44:01 +0000
Resent-Message-ID: <handler.72166.B.17212634275733 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 72166 <at> debbugs.gnu.org
Feedback-ID: if4194509:Fastmail
User-Agent: Cyrus-JMAP/3.11.0-alpha0-568-g843fbadbe-fm-20240701.003-g843fbadb
MIME-Version: 1.0
Message-Id: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN>
Date: Wed, 17 Jul 2024 20:43:15 -0400
From: "Jonathan Frederickson" <jonathan@HIDDEN>
Content-Type: text/plain
Received-SPF: pass client-ip=103.168.172.153;
 envelope-from=jonathan@HIDDEN; helo=fhigh2-smtp.messagingengine.com
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.6 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.6 (--)

I've been running into an issue with Shepherd on one of my machines. Every so often (and I haven't figured out what conditions trigger it), my Shepherd instances (both home and PID 1) will go unresponsive. I thought I had tracked it down to a misbehaving home service that I had configured, but it's just happened again without that service running.

'herd status' hangs indefinitely:

jfred@terracard ~$ sudo herd status
Password: 
<never returns>

...on both instances:

jfred@terracard ~$ herd status
<never returns>

The PID 1 shepherd instance isn't reaping defunct processes:

jfred@terracard ~$ ps aux | grep -i lock
jfred      541  0.0  0.0   3700  2304 ?        S    18:30   0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
jfred     3111  0.0  0.0      0     0 ?        Z    18:53   0:00 [swaylock] <defunct>
jfred     3112  0.0  0.0      0     0 ?        Zs   18:53   0:00 [swaylock] <defunct>

Some further troubleshooting... strace indicates that it's waiting on a read() on its fd 9:

jfred@terracard ~ [env]$ sudo strace -fp 1
Password: 
strace: Process 1 attached with 5 threads
[pid   144] read(9,  <unfinished ...>
[pid   142] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   141] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   140] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY^

...which seems to be:

jfred@terracard ~ [env]$ sudo ls -l /proc/1/fd/9
lr-x------ 1 root root 64 Jul 17 20:39 /proc/1/fd/9 -> 'pipe:[4015]'
jfred@terracard ~ [env]$ sudo lsof -n | grep 4015
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
shepherd     1                      root    9r     FIFO               0,15       0t0       4015 pipe
shepherd     1                      root   11w     FIFO               0,15       0t0       4015 pipe
shepherd     1  140 GC-marker       root    9r     FIFO               0,15       0t0       4015 pipe
shepherd     1  140 GC-marker       root   11w     FIFO               0,15       0t0       4015 pipe
shepherd     1  141 GC-marker       root    9r     FIFO               0,15       0t0       4015 pipe
shepherd     1  141 GC-marker       root   11w     FIFO               0,15       0t0       4015 pipe
shepherd     1  142 GC-marker       root    9r     FIFO               0,15       0t0       4015 pipe
shepherd     1  142 GC-marker       root   11w     FIFO               0,15       0t0       4015 pipe
shepherd     1  144 shepherd        root    9r     FIFO               0,15       0t0       4015 pipe
shepherd     1  144 shepherd        root   11w     FIFO               0,15       0t0       4015 pipe

My system configuration for this machine can be found here, and I last ran a 'guix pull' on June 21: https://github.com/jfrederickson/dotfiles/blob/master/guix/guix/system/machines/terracard/config.scm

Has anyone else run into this?

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: "Jonathan Frederickson" <jonathan@HIDDEN>
Subject: bug#72166: Acknowledgement (Shepherd periodically goes
 unresponsive on one of my machines)
Message-ID: <handler.72166.B.17212634275733.ack <at> debbugs.gnu.org>
References: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN>
X-Gnu-PR-Message: ack 72166
X-Gnu-PR-Package: guix
Reply-To: 72166 <at> debbugs.gnu.org
Date: Thu, 18 Jul 2024 00:44:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 72166 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
72166: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D72166
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:

Subject: bug#72166: Shepherd periodically goes unresponsive on one of my machines
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 19 Jul 2024 15:37:02 +0000
Resent-Message-ID: <handler.72166.B72166.17214033726407 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: "Jonathan Frederickson" <jonathan@HIDDEN>
Cc: 72166 <at> debbugs.gnu.org
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN> (Jonathan
 Frederickson's message of "Wed, 17 Jul 2024 20:43:15 -0400")
References: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
Date: Fri, 19 Jul 2024 17:35:29 +0200
Message-ID: <878qxxtmwu.fsf@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Hi Jonathan,

"Jonathan Frederickson" <jonathan@HIDDEN> skribis:

> I've been running into an issue with Shepherd on one of my machines. Ever=
y so often (and I haven't figured out what conditions trigger it), my Sheph=
erd instances (both home and PID 1) will go unresponsive. I thought I had t=
racked it down to a misbehaving home service that I had configured, but it'=
s just happened again without that service running.
>
> 'herd status' hangs indefinitely:
>
> jfred@terracard ~$ sudo herd status
> Password:=20
> <never returns>
>
> ...on both instances:
>
> jfred@terracard ~$ herd status
> <never returns>

Ouch.  What version of shepherd is running?  (You can view it with
=E2=80=9Ccat /proc/1/cmdline | xargs -0=E2=80=9D.)

> The PID 1 shepherd instance isn't reaping defunct processes:
>
> jfred@terracard ~$ ps aux | grep -i lock
> jfred      541  0.0  0.0   3700  2304 ?        S    18:30   0:00 swayidle=
 -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if =
pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output=
 * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
> jfred     3111  0.0  0.0      0     0 ?        Z    18:53   0:00 [swayloc=
k] <defunct>
> jfred     3112  0.0  0.0      0     0 ?        Zs   18:53   0:00 [swayloc=
k] <defunct>
>
> Some further troubleshooting... strace indicates that it's waiting on a r=
ead() on its fd 9:

Interesting.  There were bugs in earlier 0.10.x version that could cause
this sort of thing; let=E2=80=99s see what version you have, first.

Ludo=E2=80=99.

Message sent to bug-guix@HIDDEN:

Subject: bug#72166: Shepherd periodically goes unresponsive on one of my machines
Resent-From: "Jonathan Frederickson" <jonathan@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Fri, 19 Jul 2024 16:27:01 +0000
Resent-Message-ID: <handler.72166.B72166.172140636611152 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 72166 <at> debbugs.gnu.org
Feedback-ID: if4194509:Fastmail
User-Agent: Cyrus-JMAP/3.11.0-alpha0-568-g843fbadbe-fm-20240701.003-g843fbadb
MIME-Version: 1.0
Message-Id: <7974c622-e7d8-48b3-9948-14e8d7654793@HIDDEN>
In-Reply-To: <878qxxtmwu.fsf@HIDDEN>
References: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN>
 <878qxxtmwu.fsf@HIDDEN>
Date: Fri, 19 Jul 2024 12:25:37 -0400
From: "Jonathan Frederickson" <jonathan@HIDDEN>
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On Fri, Jul 19, 2024, at 11:35 AM, Ludovic Court=C3=A8s wrote:
> Hi Jonathan,
>=20
> "Jonathan Frederickson" <jonathan@HIDDEN> skribis:
>=20
> > I've been running into an issue with Shepherd on one of my machines.=
 Every so often (and I haven't figured out what conditions trigger it), =
my Shepherd instances (both home and PID 1) will go unresponsive. I thou=
ght I had tracked it down to a misbehaving home service that I had confi=
gured, but it's just happened again without that service running.
> >
> > 'herd status' hangs indefinitely:
> >
> > jfred@terracard ~$ sudo herd status
> > Password:=20
> > <never returns>
> >
> > ...on both instances:
> >
> > jfred@terracard ~$ herd status
> > <never returns>
>=20
> Ouch.  What version of shepherd is running?  (You can view it with
> =E2=80=9Ccat /proc/1/cmdline | xargs -0=E2=80=9D.)
>=20
> > The PID 1 shepherd instance isn't reaping defunct processes:
> >
> > jfred@terracard ~$ ps aux | grep -i lock
> > jfred      541  0.0  0.0   3700  2304 ?        S    18:30   0:00 swa=
yidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeou=
t 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume sway=
msg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-ma=
nual.jpg
> > jfred     3111  0.0  0.0      0     0 ?        Z    18:53   0:00 [sw=
aylock] <defunct>
> > jfred     3112  0.0  0.0      0     0 ?        Zs   18:53   0:00 [sw=
aylock] <defunct>
> >
> > Some further troubleshooting... strace indicates that it's waiting o=
n a read() on its fd 9:
>=20
> Interesting.  There were bugs in earlier 0.10.x version that could cau=
se
> this sort of thing; let=E2=80=99s see what version you have, first.
>=20
> Ludo=E2=80=99.
>=20

Hi Ludo, thanks for the troubleshooting help. Looks like I'm running 0.1=
0.4:

jfred@terracard ~$ cat /proc/1/cmdline | xargs -0
/gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-a=
uto-compile /gnu/store/39li5qpiaj1lx89xgahlbgvfnjhpcpwg-shepherd-0.10.4/=
bin/shepherd --config /gnu/store/hfyri6ygfdjq4w3nkha2ypa2k98hhfxj-shephe=
rd.conf

I see now that 0.10.5 was released a few weeks ago, does that have a fix=
 that could be related?

Message sent to bug-guix@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: bug#72166: Shepherd periodically goes unresponsive on one of my machines
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Mon, 22 Jul 2024 07:15:01 +0000
Resent-Message-ID: <handler.72166.B72166.172163248223692 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 72166
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: "Jonathan Frederickson" <jonathan@HIDDEN>
Cc: 72166 <at> debbugs.gnu.org
Received: via spool by 72166-submit <at> debbugs.gnu.org id=B72166.172163248223692
          (code B ref 72166); Mon, 22 Jul 2024 07:15:01 +0000
Received: (at 72166) by debbugs.gnu.org; 22 Jul 2024 07:14:42 +0000
Received: from localhost ([127.0.0.1]:56741 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1sVnFq-0006A4-F6
	for submit <at> debbugs.gnu.org; Mon, 22 Jul 2024 03:14:42 -0400
Received: from hera.aquilenet.fr ([185.233.100.1]:43500)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1sVnFo-00069o-7Z
 for 72166 <at> debbugs.gnu.org; Mon, 22 Jul 2024 03:14:41 -0400
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id D09A4207;
 Mon, 22 Jul 2024 09:14:31 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at hera.aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id LtT8vylRAwr0; Mon, 22 Jul 2024 09:14:31 +0200 (CEST)
Received: from ribbon (unknown [193.50.110.239])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 59E803C;
 Mon, 22 Jul 2024 09:14:31 +0200 (CEST)
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <7974c622-e7d8-48b3-9948-14e8d7654793@HIDDEN> (Jonathan
 Frederickson's message of "Fri, 19 Jul 2024 12:25:37 -0400")
References: <df6e8894-fd84-446f-a67f-50cdcc9de5b3@HIDDEN>
 <878qxxtmwu.fsf@HIDDEN>
 <7974c622-e7d8-48b3-9948-14e8d7654793@HIDDEN>
Date: Mon, 22 Jul 2024 09:14:29 +0200
Message-ID: <87zfq9kiei.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi,

"Jonathan Frederickson" <jonathan@HIDDEN> skribis:

> Hi Ludo, thanks for the troubleshooting help. Looks like I'm running 0.10=
.4:
>
> jfred@terracard ~$ cat /proc/1/cmdline | xargs -0
> /gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-au=
to-compile /gnu/store/39li5qpiaj1lx89xgahlbgvfnjhpcpwg-shepherd-0.10.4/bin/=
shepherd --config /gnu/store/hfyri6ygfdjq4w3nkha2ypa2k98hhfxj-shepherd.conf
>
> I see now that 0.10.5 was released a few weeks ago, does that have a fix =
that could be related?

Yes, it could be related.  Per the =E2=80=98NEWS=E2=80=99 file of Shepherd:

  ** =E2=80=98herd unload root SERVICE=E2=80=99 no longer hands when there=
=E2=80=99s a replacement
     (<https://issues.guix.gnu.org/71478>)

  It used to be that, for a running service S that has a replacement regist=
ered,
  =E2=80=98herd unload root S=E2=80=99 would hang shepherd, making it total=
ly unresponsive=E2=80=94=E2=80=98herd
  status=E2=80=99, =E2=80=98halt=E2=80=99, etc. would hang forever, and ine=
td-style services would no
  longer start, etc.  This is now fixed.

Depending on previous =E2=80=98guix system reconfigure=E2=80=99 invocations=
 on these
machines, it=E2=80=99s possible that you ended up in this state.

Would be great if you could upgrade and see if the problem still occurs.

Thanks,
Ludo=E2=80=99.

Last modified: Mon, 22 Jul 2024 07:30:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.