GNU bug report logs - #77283
[Shepherd] System log crashes when parsing message

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Reported by: MSavoritias <email@HIDDEN>; dated Wed, 26 Mar 2025 13:43:01 UTC; Maintainer for guix is bug-guix@HIDDEN.
Changed bug title to '[Shepherd] System log crashes when parsing message' from 'Shepherd can't parse logs.' Request was from Ludovic Courtès <ludo@HIDDEN> to control <at> debbugs.gnu.org. Full text available.

Message received at 77283 <at> debbugs.gnu.org:


Received: (at 77283) by debbugs.gnu.org; 30 Mar 2025 21:02:26 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Mar 30 17:02:26 2025
Received: from localhost ([127.0.0.1]:38249 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tyznW-0003LO-5V
	for submit <at> debbugs.gnu.org; Sun, 30 Mar 2025 17:02:26 -0400
Received: from hera.aquilenet.fr ([185.233.100.1]:60238)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1tyznT-0003Kt-Gq
 for 77283 <at> debbugs.gnu.org; Sun, 30 Mar 2025 17:02:24 -0400
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 27EC12B6;
 Sun, 30 Mar 2025 23:02:10 +0200 (CEST)
Authentication-Results: hera.aquilenet.fr;
	none
X-Virus-Scanned: Debian amavis at hera.aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavis, port 10024) with ESMTP
 id qNjXOr_OtY31; Sun, 30 Mar 2025 23:02:09 +0200 (CEST)
Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 7EA301F3;
 Sun, 30 Mar 2025 23:02:07 +0200 (CEST)
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: MSavoritias <email@HIDDEN>
Subject: Re: bug#77283: Shepherd can't parse logs.
In-Reply-To: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
 (MSavoritias's message of "Wed, 26 Mar 2025 15:42:06 +0200")
References: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
Date: Sun, 30 Mar 2025 23:02:06 +0200
Message-ID: <87bjtidygh.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Rspamd-Queue-Id: 27EC12B6
X-Spamd-Result: default: False [5.46 / 15.00]; SPAM_FLAG(5.00)[];
 BAYES_HAM(-3.00)[99.99%]; NEURAL_SPAM(3.00)[1.000];
 R_MIXED_CHARSET(0.56)[subject]; MIME_GOOD(-0.10)[text/plain];
 RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[];
 MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[];
 RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[];
 MID_RHS_MATCH_FROM(0.00)[]
X-Spam-Level: *****
X-Rspamd-Action: greylist
X-Spamd-Bar: +++++
X-Rspamd-Server: hera
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 77283
Cc: 77283 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Hi MSavoritias,

MSavoritias <email@HIDDEN> skribis:

> Shepherd has been crashing our system with the message:
>
> https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/=
KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11=
.png
>
>
> This ends up with prosody and openssh crashing, nginx still
> working. the picture shows shepherd system-log crashing with:
>
> uncaught exception, value out of range in fibers. It references an
> "unknown file" and a string with prosody.

The code that throws that out-of-range exception on =E2=80=98substring=E2=
=80=99 is
(match:substring m 3) in =E2=80=98parse-system-log-message=E2=80=99.

Normally, =E2=80=98regexp-exec=E2=80=99 returns match structures with corre=
ct
boundaries, but in this case it didn=E2=80=99t.  I reported it here:

  https://issues.guix.gnu.org/77392

Several options come to mind:

  1. Run shepherd in a UTF-8 locale; a quick workaround is to run:

       sudo herd eval root '(setlocale LC_ALL "C.UTF-8")'

     but we could also do that in Guix System.

  2. Avoid (ice-9 regex) altogether in =E2=80=98system-log.scm=E2=80=99 and=
 use a pure
     Scheme implementation, SRFI-115-style.

Thanks for your bug report!

Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#77283; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 26 Mar 2025 13:42:41 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Wed Mar 26 09:42:41 2025
Received: from localhost ([127.0.0.1]:42129 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1txR1k-0004m2-PR
	for submit <at> debbugs.gnu.org; Wed, 26 Mar 2025 09:42:41 -0400
Received: from lists.gnu.org ([2001:470:142::17]:59538)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <email@HIDDEN>)
 id 1txR1i-0004lh-Dp
 for submit <at> debbugs.gnu.org; Wed, 26 Mar 2025 09:42:39 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <email@HIDDEN>)
 id 1txR1V-0001AK-AA
 for bug-guix@HIDDEN; Wed, 26 Mar 2025 09:42:25 -0400
Received: from mail.webarch.email ([81.95.52.48])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <email@HIDDEN>)
 id 1txR1S-0001BA-RL
 for bug-guix@HIDDEN; Wed, 26 Mar 2025 09:42:24 -0400
Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon)
 with ESMTPSA id E7B3F1A851BA
 for <bug-guix@HIDDEN>; Wed, 26 Mar 2025 13:42:08 +0000 (GMT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=msavoritias.me;
 s=20210930; t=1742996530;
 h=from:subject:date:message-id:to:mime-version:content-type:
 content-transfer-encoding:content-language;
 bh=F6J1A/+revhMwMgDfVdcfsWBSELTgZNVMx/ruuCivuo=;
 b=LbEcl/6Ikr32+fJQEM4bkqaBcQWg82TAbi5gb27e1/HeoftVeneUeV5bzmkjpyIQvaJBpG
 WGQisuTmGZHN3LYTB3lkS2MwDiFFXlyUvPWXD1celzwGgojcVKU8Ose4lMBw2e2fwhkQN3
 xNIfwOmGflGTQQPyFOlvpYoizl0G9FXyMO+DpfDfYcUhIZr4PgJpM7lIxuoJVHRiTAUGfp
 EZu37O2b1IsDHLCH/KWNkSZSaSbIOFL/oe/kh4oLT/3dubdRuQ0IPCofB3hIGKgvK/Lm4j
 vOyBSqubKhWl4XUGEj0NpqK5oF3Y2PkefA2GEZQIQd6lMdUaJODDXfAeY4gfwQ==
Message-ID: <fca86846-a160-487a-9b53-cc64753557cb@HIDDEN>
Date: Wed, 26 Mar 2025 15:42:06 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: bug-guix@HIDDEN
From: MSavoritias <email@HIDDEN>
Subject: Shepherd can't parse logs.
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Last-TLS-Session-Version: TLSv1.3
Received-SPF: pass client-ip=81.95.52.48; envelope-from=email@HIDDEN;
 helo=mail.webarch.email
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

Shepherd has been crashing our system with the message:

https://upload.magicbroccoli.de/fb6e7843fef693538a5c7c156c0a98fc0aeb6bac/KmA7ovtC2Fcw0Nm0PSCtphwSyDLI1VUF3QrT9PtD/Kuvakaappaus_-_2025-03-26_13-27-11.png


This ends up with prosody and openssh crashing, nginx still working. the 
picture shows shepherd system-log crashing with:

uncaught exception, value out of range in fibers. It references an 
"unknown file" and a string with prosody.


I looked into git commits in the last month (bug didn't exist before end 
of January/beginning of February, it seems that the bug is either 
shepherd itself, it was updated recently and that is what i was told in 
the irc. Or it is the certbot timer. Certbot timer is the only service 
we have installed from the ones that changed.


Being certbot makes less sense since prosody is mentioned specifically 
tho so it seems a bug in how shepherd parses logs.


Our whole config is here -> https://codeberg.org/joinjabber/Infra


MSavoritias





Acknowledgement sent to MSavoritias <email@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#77283; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 31 Mar 2025 21:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.