GNU logs - #77283, boring messages

Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#77283: Shepherd can't parse logs.
Resent-From: MSavoritias <email@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Wed, 26 Mar 2025 13:43:01 +0000
Resent-Message-ID: <handler.77283.B.174299656118358 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: report 77283
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: 77283 <at> debbugs.gnu.org
X-Debbugs-Original-To: bug-guix@HIDDEN
Received: via spool by submit <at> debbugs.gnu.org id=B.174299656118358
          (code B ref -1); Wed, 26 Mar 2025 13:43:01 +0000
Received: (at submit) by debbugs.gnu.org; 26 Mar 2025 13:42:41 +0000
Received: from localhost ([127.0.0.1]:42129 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1txR1k-0004m2-PR
	for submit <at> debbugs.gnu.org; Wed, 26 Mar 2025 09:42:41 -0400
Received: from lists.gnu.org ([2001:470:142::17]:59538)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <email@HIDDEN>)
 id 1txR1i-0004lh-Dp
 for submit <at> debbugs.gnu.org; Wed, 26 Mar 2025 09:42:39 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <email@HIDDEN>)
 id 1txR1V-0001AK-AA
 for bug-guix@HIDDEN; Wed, 26 Mar 2025 09:42:25 -0400
Received: from mail.webarch.email ([81.95.52.48])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <email@HIDDEN>)
 id 1txR1S-0001BA-RL
 for bug-guix@HIDDEN; Wed, 26 Mar 2025 09:42:24 -0400
Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon)
 with ESMTPSA id E7B3F1A851BA
 for <bug-guix@HIDDEN>; Wed, 26 Mar 2025 13:42:08 +0000 (GMT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=msavoritias.me;
 s=20210930; t=1742996530;
 h=from:subject:date:message-id:to:mime-version:content-type:
 content-transfer-encoding:content-language;
 bh=F6J1A/+revhMwMgDfVdcfsWBSELTgZNVMx/ruuCivuo=;
 b=LbEcl/6Ikr32+fJQEM4bkqaBcQWg82TAbi5gb27e1/HeoftVeneUeV5bzmkjpyIQvaJBpG
 WGQisuTmGZHN3LYTB3lkS2MwDiFFXlyUvPWXD1celzwGgojcVKU8Ose4lMBw2e2fwhkQN3
 xNIfwOmGflGTQQPyFOlvpYoizl0G9FXyMO+DpfDfYcUhIZr4PgJpM7lIxuoJVHRiTAUGfp
 EZu37O2b1IsDHLCH/KWNkSZSaSbIOFL/oe/kh4oLT/3dubdRuQ0IPCofB3hIGKgvK/Lm4j
 vOyBSqubKhWl4XUGEj0NpqK5oF3Y2PkefA2GEZQIQd6lMdUaJODDXfAeY4gfwQ==
Message-ID: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
Date: Wed, 26 Mar 2025 15:42:06 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
From: MSavoritias <email@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Last-TLS-Session-Version: TLSv1.3
Received-SPF: pass client-ip=81.95.52.48; envelope-from=email@HIDDEN;
 helo=mail.webarch.email
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Shepherd has been crashing our system with the message:

https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11.png


This ends up with prosody and openssh crashing, nginx still working. the 
picture shows shepherd system-log crashing with:

uncaught exception, value out of range in fibers. It references an 
"unknown file" and a string with prosody.


I looked into git commits in the last month (bug didn't exist before end 
of January/beginning of February, it seems that the bug is either 
shepherd itself, it was updated recently and that is what i was told in 
the irc. Or it is the certbot timer. Certbot timer is the only service 
we have installed from the ones that changed.


Being certbot makes less sense since prosody is mentioned specifically 
tho so it seems a bug in how shepherd parses logs.


Our whole config is here -> https://codeberg.org/joinjabber/Infra


MSavoritias

Message sent:


Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: MSavoritias <email@HIDDEN>
Subject: bug#77283: Acknowledgement (Shepherd can't parse logs.)
Message-ID: <handler.77283.B.174299656118358.ack <at> debbugs.gnu.org>
References: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
X-Gnu-PR-Message: ack 77283
X-Gnu-PR-Package: guix
Reply-To: 77283 <at> debbugs.gnu.org
Date: Wed, 26 Mar 2025 13:43:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-guix@HIDDEN

If you wish to submit further information on this problem, please
send it to 77283 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
77283: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D77283
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-guix@HIDDEN:


X-Loop: help-debbugs@HIDDEN
Subject: bug#77283: Shepherd can't parse logs.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-guix@HIDDEN
Resent-Date: Sun, 30 Mar 2025 21:03:01 +0000
Resent-Message-ID: <handler.77283.B77283.174336854612863 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 77283
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: MSavoritias <email@HIDDEN>
Cc: 77283 <at> debbugs.gnu.org
Received: via spool by 77283-submit <at> debbugs.gnu.org id=B77283.174336854612863
          (code B ref 77283); Sun, 30 Mar 2025 21:03:01 +0000
Received: (at 77283) by debbugs.gnu.org; 30 Mar 2025 21:02:26 +0000
Received: from localhost ([127.0.0.1]:38249 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tyznW-0003LO-5V
	for submit <at> debbugs.gnu.org; Sun, 30 Mar 2025 17:02:26 -0400
Received: from hera.aquilenet.fr ([185.233.100.1]:60238)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1tyznT-0003Kt-Gq
 for 77283 <at> debbugs.gnu.org; Sun, 30 Mar 2025 17:02:24 -0400
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 27EC12B6;
 Sun, 30 Mar 2025 23:02:10 +0200 (CEST)
Authentication-Results: hera.aquilenet.fr;
	none
X-Virus-Scanned: Debian amavis at hera.aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavis, port 10024) with ESMTP
 id qNjXOr_OtY31; Sun, 30 Mar 2025 23:02:09 +0200 (CEST)
Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 7EA301F3;
 Sun, 30 Mar 2025 23:02:07 +0200 (CEST)
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
 (MSavoritias's message of "Wed, 26 Mar 2025 15:42:06 +0200")
References: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
Date: Sun, 30 Mar 2025 23:02:06 +0200
Message-ID: <87bjtidygh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Rspamd-Queue-Id: 27EC12B6
X-Spamd-Result: default: False [5.46 / 15.00]; SPAM_FLAG(5.00)[];
 BAYES_HAM(-3.00)[99.99%]; NEURAL_SPAM(3.00)[1.000];
 R_MIXED_CHARSET(0.56)[subject]; MIME_GOOD(-0.10)[text/plain];
 RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[];
 MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[];
 RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[];
 MID_RHS_MATCH_FROM(0.00)[]
X-Spam-Level: *****
X-Rspamd-Action: greylist
X-Spamd-Bar: +++++
X-Rspamd-Server: hera
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi MSavoritias,

MSavoritias <email@HIDDEN> skribis:

> Shepherd has been crashing our system with the message:
>
> https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/=
KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11=
.png
>
>
> This ends up with prosody and openssh crashing, nginx still
> working. the picture shows shepherd system-log crashing with:
>
> uncaught exception, value out of range in fibers. It references an
> "unknown file" and a string with prosody.

The code that throws that out-of-range exception on =E2=80=98substring=E2=
=80=99 is
(match:substring m 3) in =E2=80=98parse-system-log-message=E2=80=99.

Normally, =E2=80=98regexp-exec=E2=80=99 returns match structures with corre=
ct
boundaries, but in this case it didn=E2=80=99t.  I reported it here:

  https://issues.guix.gnu.org/77392

Several options come to mind:

  1. Run shepherd in a UTF-8 locale; a quick workaround is to run:

       sudo herd eval root '(setlocale LC_ALL "C.UTF-8")'

     but we could also do that in Guix System.

  2. Avoid (ice-9 regex) altogether in =E2=80=98system-log.scm=E2=80=99 and=
 use a pure
     Scheme implementation, SRFI-115-style.

Thanks for your bug report!

Ludo=E2=80=99.

Message received at control <at> debbugs.gnu.org:


Received: (at control) by debbugs.gnu.org; 31 Mar 2025 20:44:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Mar 31 16:44:28 2025
Received: from localhost ([127.0.0.1]:43125 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tzLzg-00027h-8z
	for submit <at> debbugs.gnu.org; Mon, 31 Mar 2025 16:44:28 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:45666)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1tzLzd-000272-U9
 for control <at> debbugs.gnu.org; Mon, 31 Mar 2025 16:44:26 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>) id 1tzLzU-0004dS-GM
 for control <at> debbugs.gnu.org; Mon, 31 Mar 2025 16:44:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to:
 references; bh=0AUilOlgCxVbSr4fSlsZAIGNWoQZnx2vvqCsIEXUiFA=; b=PTpthguslv+TnC
 vQ1COHOcr8MbJQ0FNrHeyBxd7cwg+00/DLKqq4VF21ywp+5X5OKdBA4sQdI8oTBuAM+G5Q6Ua79w6
 dMCMtaBWbhQJMLHob8NMP/yD8M9nNxzjeD44v8+Z+uWwqzoF0+Gq0q6YCIRx7VldIszN8WlVyhza7
 g3RdMjD3TnE7UwzgZUvr/vHvhLBUfCs3uvlMRcux3eraq9rHQxgib4Y0tmcFOyfH67JOEadW05VwA
 hFYfHNvAl3jGKCL35D4NS+JRdDpNvBKsQ0lSGg2Zsurm7U2SYl9Uri4yXjrEQRvs0caz+6IcpXaFK
 xYozGzIoMaE+CW5ibt7Q==;
Date: Mon, 31 Mar 2025 22:44:13 +0200
Message-Id: <874iz9j5gi.fsf@HIDDEN>
To: control <at> debbugs.gnu.org
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
Subject: control message for bug #77283
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

retitle 77283 [Shepherd] System log crashes when parsing message
quit
Last modified: Mon, 31 Mar 2025 21:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.