GNU bug report logs - #78067
Conversion to unprivileged guix-daemon breaks ssh

Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.

Package: guix; Reported by: "Zack Weinberg" <zack@HIDDEN>; dated Fri, 25 Apr 2025 20:36:02 UTC; Maintainer for guix is bug-guix@HIDDEN.

Message received at 78067 <at> debbugs.gnu.org:


Received: (at 78067) by debbugs.gnu.org; 5 May 2025 15:35:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon May 05 11:35:17 2025
Received: from localhost ([127.0.0.1]:41789 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1uBxqf-0005s4-7C
	for submit <at> debbugs.gnu.org; Mon, 05 May 2025 11:35:17 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34304)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1uBxqX-0005m2-Uk
 for 78067 <at> debbugs.gnu.org; Mon, 05 May 2025 11:35:10 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1uBxqS-00023M-Ar; Mon, 05 May 2025 11:35:04 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=38WxeU53pI7KKehwt+9/WO6hCQfyoN+x6RrELgb4+S4=; b=sPE6eNShwgcImGb/0Obv
 qlql8wgVJmtcTRTgaMmFJqZAoBTcpWXM6iGUttMdwJ06ulcMqT0DvdSeZmF7dadbxkgbxreEGrvkb
 S7dlILRtS8YcDfaaYI75j7H7Qq4zehbtAQXy7G3y6eayROC5rsfyfbj8EVM/S5PhLJjkhv0bhSq+L
 iYcSv8EescZnGZ4D4Z4bTZHG5c3/PKJI87tD+0g7UZxjZ6mDxK+NCASN+OYulFZvJHemB1jJ2N0Xl
 XUGjFpZz1gjP/qpbCIJiTLx8iosI6hGbxVexP1Pyb88Nw10Bu3cka7RIuE+cXl4mTmAc6L6ihsCI1
 O7IIFzjmE5ReWA==;
From: =?utf-8?Q?Ludovic_Court=C3=A8s?= <ludo@HIDDEN>
To: "Zack Weinberg" <zack@HIDDEN>
Subject: Re: bug#78067: Conversion to unprivileged guix-daemon breaks ssh
In-Reply-To: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN> (Zack
 Weinberg's message of "Fri, 25 Apr 2025 16:34:45 -0400")
References: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN>
Date: Mon, 05 May 2025 14:58:28 +0200
Message-ID: <871pt35i4r.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 78067
Cc: 78067 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Zack,

"Zack Weinberg" <zack@HIDDEN> writes:

> I just switched my Guix System-based server over to unprivileged guix-
> daemon, after which I was unable to ssh into it.  From the client, the
> syndrome looks like this (shell variables indicate redactions):

[...]

> 2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running
> with value #<<process> id: 234 command:
> ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd"
> "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config"
> "-i")>.
> 2025-04-25 20:27:40 localhost sshd[234]: fatal:
> /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty
> must be owned by root and not group or world-writable.

D=E2=80=99oh.  The fix here is to tell OpenSSH to use /var/empty instead.  =
Do
you know how to do that via sshd_config?

> `chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0=
p1/var/empty`
> is sufficient to fix ssh, but I wonder if store contents in general shoul=
d maybe remain
> owned by root regardless of whether the daemon is running as an unprivile=
ged user.
> It seems likely to me that this will not be the only such problem.

You should never manually modify files in the store or change their
ownership.  In the case above, the daemon will now be unable to delete
this store item when you run =E2=80=98guix gc=E2=80=99.

Thanks for the bug report,
Ludo=E2=80=99.




Information forwarded to bug-guix@HIDDEN:
bug#78067; Package guix. Full text available.

Message received at submit <at> debbugs.gnu.org:


Received: (at submit) by debbugs.gnu.org; 25 Apr 2025 20:35:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Apr 25 16:35:22 2025
Received: from localhost ([127.0.0.1]:53758 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1u8PlZ-0006qU-Px
	for submit <at> debbugs.gnu.org; Fri, 25 Apr 2025 16:35:22 -0400
Received: from lists.gnu.org ([2001:470:142::17]:43772)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <zack@HIDDEN>) id 1u8PlW-0006q9-D9
 for submit <at> debbugs.gnu.org; Fri, 25 Apr 2025 16:35:19 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <zack@HIDDEN>) id 1u8PlP-0002qy-Sz
 for bug-guix@HIDDEN; Fri, 25 Apr 2025 16:35:11 -0400
Received: from fhigh-a8-smtp.messagingengine.com ([103.168.172.159])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <zack@HIDDEN>) id 1u8PlN-0007MI-3l
 for bug-guix@HIDDEN; Fri, 25 Apr 2025 16:35:11 -0400
Received: from phl-compute-06.internal (phl-compute-06.phl.internal
 [10.202.2.46])
 by mailfhigh.phl.internal (Postfix) with ESMTP id 380E011402F4
 for <bug-guix@HIDDEN>; Fri, 25 Apr 2025 16:35:06 -0400 (EDT)
Received: from phl-imap-05 ([10.202.2.95])
 by phl-compute-06.internal (MEProxy); Fri, 25 Apr 2025 16:35:06 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=owlfolio.org; h=
 cc:content-transfer-encoding:content-type:content-type:date:date
 :from:from:in-reply-to:message-id:mime-version:reply-to:subject
 :subject:to:to; s=fm2; t=1745613306; x=1745699706; bh=uwjMevqd+M
 pBuiOjjYjU7OLr7vHap+HSu4ITzdYIrLQ=; b=LrhFN1qK801/1wLyZKGTYwrVtc
 L5TPHbZBFnUSqCfhc/8XL+ojNuilmVOr+NsfP+9EWe8A+adVFJb4MqE6nhJOdSob
 9TLWDAltQrdYyFuhIyHAi6mT9XkBlRg+lx/KFl4EZJ+VvAe4enQvmv1wxSJNmtR2
 ioiRW2UrgoBxoSFA7V1fHIzXPGrGwHLe/m/wxaSwyTg0L5qtUdpcL6JrvbzV1aZY
 q5i1GnYYd+fiHZDSKr3pSKMDH4xZYY7Hj/cOVgMVebMfEzQqkHpHbFkYejEBQoAx
 n9erQ2eIUxPlmy7d6j4Rx1LmCxBCfC1wbtPVAf6AvRWIqca+pUQ9eqt97Mwg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :content-type:date:date:feedback-id:feedback-id:from:from
 :in-reply-to:message-id:mime-version:reply-to:subject:subject:to
 :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=
 1745613306; x=1745699706; bh=uwjMevqd+MpBuiOjjYjU7OLr7vHap+HSu4I
 TzdYIrLQ=; b=SK7z+WxYlMQa8n3gbWYME77w1HCpd4y4KbsGkHMYFMJsjn7Kry0
 CW4ROPKgi0p1JSnr3rwBlGlweXyZbnuU9S9DOohnNlboS+O+ifBcJwM+IiSknIB4
 hajxsClGPhGpleY4+NBtug1QIxSXZJAtBB/xiuZ8uppPCMTEqQa5z8cdrf/tm4J6
 QIm1WM1gNX/vx96mkSKdMm71+256rBaoeoAWHOh+AU9qLQUoOORyheSiK2Sikgvr
 X1dCAbw54ZILc/sjkTmZn/bGc7g9texnqZwS7/+mwchPAGY6hHYFKbrYHtJvvcYp
 c1AyjHbPZUNOcAk2w0Wli0/c8uWE8Cv6cvA==
X-ME-Sender: <xms:-fELaF-AK8699Fh3Tdj65cTQ-NTFKO3xbqEQZA2t-2WE4H8pZ-wA1A>
 <xme:-fELaJseZd-lRzu0o24-WbiVvX3s04HFGGWvAZls7cUnLm9akdt7Lx8Tx7gHxwqcv
 PN0h-VOHgUHAV7nxVo>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvheeffeduucetufdoteggodetrf
 dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv
 pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpefogg
 ffhffvkffutgfgsehtjeertdertddtnecuhfhrohhmpedfkggrtghkucghvghinhgsvghr
 ghdfuceoiigrtghksehofihlfhholhhiohdrohhrgheqnecuggftrfgrthhtvghrnhepfe
 duteeifeevuedtgeehvefgtedvjefhleejteduvddtvddvhedvgeekhfejffdtnecuvehl
 uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiigrtghksehofi
 hlfhholhhiohdrohhrghdpnhgspghrtghpthhtohepuddpmhhouggvpehsmhhtphhouhht
 pdhrtghpthhtohepsghughdqghhuihigsehgnhhurdhorhhg
X-ME-Proxy: <xmx:-fELaDCi0Y-77RKlWo50admy0VLIJWXcj347JEffwyS-MPqODL4qag>
 <xmx:-fELaJeaHP0tfcb_oGBo8oPZQwx0tRkiEL04V1MwGhF8u85pyx88tw>
 <xmx:-fELaKNAiSll9A074sye98IQ1Vo5N77VQq6YrnwR5VnVS2wUhIb1fQ>
 <xmx:-fELaLmTrXCkr_6DKagLZeJbZp-RMN0Ihdauk4arTNbzbo2JK80hww>
 <xmx:-vELaGC7hSY6p3hIcQmAwe--ysqG5di7CDVKhT4Q4vkgGm2H0uZNSbDy>
Feedback-ID: i876146a2:Fastmail
Received: by mailuser.phl.internal (Postfix, from userid 501)
 id CA3923020080; Fri, 25 Apr 2025 16:35:05 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
MIME-Version: 1.0
Date: Fri, 25 Apr 2025 16:34:45 -0400
From: "Zack Weinberg" <zack@HIDDEN>
To: bug-guix@HIDDEN
Message-Id: <8d70405b-7f96-43a2-90de-8b5adde8873d@HIDDEN>
Subject: Conversion to unprivileged guix-daemon breaks ssh
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Received-SPF: pass client-ip=103.168.172.159; envelope-from=zack@HIDDEN;
 helo=fhigh-a8-smtp.messagingengine.com
X-Spam_score_int: -27
X-Spam_score: -2.8
X-Spam_bar: --
X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001,
 RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 0.7 (/)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.3 (/)

I just switched my Guix System-based server over to unprivileged guix-
daemon, after which I was unable to ssh into it.  From the client, the
syndrome looks like this (shell variables indicate redactions):

$ ssh $my_server
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

or with -v:

$ ssh -v $my_server
OpenSSH_9.9p2, OpenSSL 3.3.3 11 Feb 2025
debug1: Reading configuration data $HOME/.ssh/config
debug1: $HOME/.ssh/config line 31: Applying options for tinka
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to $my_server [$ip_address] port 22.
debug1: Connection established.
debug1: identity file $HOME/.ssh/$private_key type 0
[more lines about identity files omitted...]
debug1: Local version string SSH-2.0-OpenSSH_9.9
kex_exchange_identification: read: Connection reset by peer
Connection reset by $ip_address port 22

Fortunately, I can get into the server using a serial console,
and the problem is quite clear from server-side logs:

2025-04-25 20:27:40 localhost shepherd[1]: Accepted connection on 0.0.0.0:22 from $client_ip:51626.
2025-04-25 20:27:40 localhost shepherd[1]: Starting service sshd-69...
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 started.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 running with value #<<process> id: 234 command: ("/gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/sbin/sshd" "-D" "-f" "/gnu/store/vwy5d5mj35rh147iwzkzxijld0gx06mb-sshd_config" "-i")>.
2025-04-25 20:27:40 localhost sshd[234]: fatal: /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty must be owned by root and not group or world-writable.
2025-04-25 20:27:40 localhost shepherd[1]: 0 connections still in use after sshd-69 termination.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 (PID 234) exited with 255.
2025-04-25 20:27:40 localhost shepherd[1]: Service sshd-69 has been disabled.
2025-04-25 20:27:40 localhost shepherd[1]: Transient service sshd-69 terminated, now unregistered.

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1
total 24
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 bin/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 etc/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 libexec/
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 sbin/
dr-xr-xr-x 4 guix-daemon guix-daemon 4096 Jan  1  1970 share/
dr-xr-xr-x 3 guix-daemon guix-daemon 4096 Jan  1  1970 var/

# ls -l /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var
total 4
dr-xr-xr-x 2 guix-daemon guix-daemon 4096 Jan  1  1970 empty/

`chown root:root /gnu/store/8kman284vvlzk2hgy1bv1xzys3rfdzlr-openssh-10.0p1/var/empty`
is sufficient to fix ssh, but I wonder if store contents in general should maybe remain
owned by root regardless of whether the daemon is running as an unprivileged user.
It seems likely to me that this will not be the only such problem.

zw




Acknowledgement sent to "Zack Weinberg" <zack@HIDDEN>:
New bug report received and forwarded. Copy sent to bug-guix@HIDDEN. Full text available.
Report forwarded to bug-guix@HIDDEN:
bug#78067; Package guix. Full text available.
Please note: This is a static page, with minimal formatting, updated once a day.
Click here to see this page with the latest information and nicer formatting.
Last modified: Mon, 5 May 2025 15:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.